Engineering case study

BCP Monitor: Independent Visibility into Oracle Disaster Recovery

Checking the continuity environment—and checking that its monitoring is still working.

01 · The problem

When monitoring needs monitoring

An Oracle database environment used a primary database and a physical standby for disaster recovery, alongside an existing enterprise monitoring platform. An operational incident exposed a gap: the standby’s condition and the health of the monitoring agents were not being independently checked.

BCP Monitor was developed to make those dependencies visible. Its purpose is to help operators understand whether the recovery environment is functioning, whether the monitoring agents are reporting, and whether backup storage needs attention.

02 · The approach

An independent layer of checks

A lightweight monitor runs on a separate host, using shell scripts and SQL probes. A scheduled cycle checks the environment every 15 minutes.

Server reachability
Checks network access and a successful monitoring login to the primary and standby servers.
Monitoring agent health
Checks whether each agent is running and whether its uploads are current.
Data Guard continuity
Examines transport errors, apply and transport lag, archive gaps, the recovery process, and progress over the preceding hour.
Backup storage and monitor health
Checks backup-volume capacity and a forecast based on 24-hour growth. A separate watchdog checks the monitor’s heartbeat.

Data Guard progress is evaluated over time, helping distinguish a standby that is catching up from one that is falling further behind.

03 · Engineering decisions

Useful signals, explicit uncertainty

  • Independent operation, restricted access. The monitor runs outside the database servers, with a dedicated service account and constrained access for its checks.
  • “Unknown” is a real status. A check that cannot obtain a measurement never reports the system as healthy.
  • Operating modes that end automatically. Resynchronisation mode ends when the standby catches up; maintenance mode expires so a temporary exception cannot be forgotten.
  • Email on change. Alerts mark a problem’s start, change and recovery. Critical reminders follow increasing intervals, and a daily summary lists open issues.
  • Readable messages. Each alert explains what happened, what it means, the relevant readings and the next action.

04 · Validation

Testing the failure paths

Phase 1 entered production in October 2026. A 12-scenario acceptance walkthrough used controlled failure conditions, including an unreachable server, a stopped monitoring agent and adjusted storage thresholds.

12acceptance scenarios defined
9confirmed with live alerts or logs

Live confirmation remains outstanding for three scenarios; simulation covered part of that remaining testing.

The walkthrough and retests exposed a stopped-agent condition being reported as “unknown” rather than critical. Correcting that distinction was essential to the monitor’s original purpose.

05 · Operational lessons

Refined through real operation

The monitor tracked periods of standby lag and distinguished delays in redo delivery from delays in applying received data. The readings narrowed the investigation to redo delivery; the underlying cause remained unresolved. It also recognised when the standby caught up and returned automatically to normal mode.

Operational feedback led to corrections for misleading storage forecasts, daylight-saving clock alerts and excessive reminders. A network outage exposed another dependency: the monitor could detect a problem while email delivery was unavailable.

Version 2.1 adds a mail timeout and an outbox that resends delayed alerts with a clear delay label. These changes have been deployed and verified in production. They support eventual delivery once the connection is restored.

06 · Scope and next steps

Visibility with clear boundaries

Alert delivery still depends on the site’s internet connection. An independent delivery channel remains under consideration. Backup-job verification, standby storage and application-tier checks are candidates for a future phase.

My contribution

I led requirements, engineering decisions, integration, deployment and validation, using AI coding assistants for implementation and design review.