Project snapshot · November 2026
01 · The problem
When monitoring needs monitoring
An Oracle database environment used a primary database and a physical standby for disaster recovery, alongside an existing enterprise monitoring platform. An operational incident exposed a gap: the standby’s condition and the health of the monitoring agents were not being independently checked.
BCP Monitor was developed to make those dependencies visible. Its purpose is to help operators understand whether the recovery environment is functioning, whether the monitoring agents are reporting, and whether backup storage needs attention.
02 · The approach
An independent layer of checks
A lightweight monitor runs on a separate host, using shell scripts and SQL probes. A scheduled cycle checks the environment every 15 minutes.
- Server reachability
- Checks network access and a successful monitoring login to the primary and standby servers.
- Monitoring agent health
- Checks whether each agent is running and whether its uploads are current.
- Data Guard continuity
- Examines transport errors, apply and transport lag, archive gaps, the recovery process, and progress over the preceding hour.
- Backup storage and monitor health
- Checks backup-volume capacity and a forecast based on 24-hour growth. A separate watchdog checks the monitor’s heartbeat.
Data Guard progress is evaluated over time, helping distinguish a standby that is catching up from one that is falling further behind.
03 · Engineering decisions
Useful signals, explicit uncertainty
- Independent operation, restricted access. The monitor runs outside the database servers, with a dedicated service account and constrained access for its checks.
- “Unknown” is a real status. A check that cannot obtain a measurement never reports the system as healthy.
- Operating modes that end automatically. Resynchronisation mode ends when the standby catches up; maintenance mode expires so a temporary exception cannot be forgotten.
- Email on change. Alerts mark a problem’s start, change and recovery. Critical reminders follow increasing intervals, and a daily summary lists open issues.
- Readable messages. Each alert explains what happened, what it means, the relevant readings and the next action.
04 · Validation
Testing the failure paths
Phase 1 entered production in October 2026. A 12-scenario acceptance walkthrough used controlled failure conditions, including an unreachable server, a stopped monitoring agent and adjusted storage thresholds.
Live confirmation remains outstanding for three scenarios; simulation covered part of that remaining testing.
The walkthrough and retests exposed a stopped-agent condition being reported as “unknown” rather than critical. Correcting that distinction was essential to the monitor’s original purpose.
05 · Operational lessons
Refined through real operation
The monitor tracked periods of standby lag and distinguished delays in redo delivery from delays in applying received data. The readings narrowed the investigation to redo delivery; the underlying cause remained unresolved. It also recognised when the standby caught up and returned automatically to normal mode.
Operational feedback led to corrections for misleading storage forecasts, daylight-saving clock alerts and excessive reminders. A network outage exposed another dependency: the monitor could detect a problem while email delivery was unavailable.
Version 2.1 adds a mail timeout and an outbox that resends delayed alerts with a clear delay label. These changes have been deployed and verified in production. They support eventual delivery once the connection is restored.
06 · Scope and next steps
Visibility with clear boundaries
Alert delivery still depends on the site’s internet connection. An independent delivery channel remains under consideration. Backup-job verification, standby storage and application-tier checks are candidates for a future phase.
My contribution
I led requirements, engineering decisions, integration, deployment and validation, using AI coding assistants for implementation and design review.
