Continuous Replication Validation

As a system drives its Recovery Point Objective toward zero, periodic backup verification stops being the relevant control. A nightly restore drill measures whether last night’s snapshot is recoverable; it says nothing about the two-second window of writes a synchronous standby is supposed to be holding right now. Once the recovery point is measured in seconds, the artifact under test is no longer a file on object storage — it is the live replication stream, and the question shifts from “can this backup restore?” to “is the replica, at this instant, a promotable copy of the primary within the loss budget?” This guide, part of Core DR Architecture & Validation Fundamentals, treats replication as a continuously validated control rather than a background daemon assumed to be healthy, and it sits directly downstream of your RTO/RPO mapping: the RPO budget derived there becomes the pass/fail threshold every lag sample is judged against.

The operational gap is that replication rarely fails loudly. A stalled apply process, a broken WAL or oplog shipping link, or a replica that has silently diverged byte-for-byte from its primary all present as a healthy-looking RUNNING status while the recoverable data point quietly ages past its budget. Continuous replication validation closes that gap by sampling position on both ends, distinguishing transport lag from apply lag, cross-checking for silent divergence with the same rigor a checksum validation pipeline applies to backup artifacts, and emitting a structured verdict that feeds validation telemetry and metrics and maps each failure onto the shared error categorization tiers. The two engine-specific validators built on top of this framework — one for PostgreSQL streaming replication, one for MySQL replica lag — turn the model below into runnable, exit-coded checks.

Architecture and Execution Workflow

Figure. One continuous-validation cycle: position sampling and lag decomposition feed an RPO comparison and a divergence check, whose combined verdict classifies the cycle and gates promotion readiness.

The loop above runs on a fixed cadence — typically every few seconds for tier-0 systems — rather than once per drill. Each cycle is stateless with respect to the last: it re-samples both endpoints, recomputes lag from first principles, and emits an independent verdict, so a transient stall self-heals in the next sample and a persistent one accumulates into a breach classification. The phases below decompose the cycle into the concerns a production validator must implement separately, because each one fails in a distinct way and carries a distinct remediation.

Synchronous and asynchronous topologies both fit this model, but they change what a breach means. Under synchronous replication the primary blocks its own commit until at least one standby acknowledges the write, so a healthy topology has a structurally bounded transport lag; a rising lag there is not merely a degraded recovery point but back-pressure that is actively slowing production, which raises the stakes of a stall. Under asynchronous streaming the primary never waits, so lag can grow without any production symptom at all — the failure is invisible from the application’s side and only this loop will surface it. The validator therefore has to treat the same lag number differently depending on the guarantee the topology claims to provide, and the RPO budget it enforces is what encodes that difference numerically.

Sampling Position on Both Endpoints

Replication lag can only be measured by comparing a coordinate on the primary against the corresponding coordinate on the replica, sampled as close to simultaneously as the topology allows. The coordinate is engine-specific — a log sequence number in PostgreSQL, a GTID set or binlog position in MySQL, an oplog timestamp in MongoDB — but the sampling discipline is universal: read the primary’s current write position, read the replica’s received and applied positions, and record the wall-clock instant of each read so clock skew and sampling jitter can be bounded. The sampler below is engine-agnostic: it accepts two callables that return an opaque position plus a timestamp, and it keeps sampling overhead off the hot path by never holding a transaction open.

#!/usr/bin/env python3
"""Engine-agnostic replication position sampler.

Returns a paired sample of primary and replica positions with the wall-clock
instant each was observed, so downstream lag math can bound sampling jitter.
"""
from __future__ import annotations

import time
from dataclasses import dataclass
from typing import Callable, Tuple


@dataclass(frozen=True)
class PositionSample:
    """One endpoint reading: an opaque position and when it was observed."""

    position: int
    observed_at: float


@dataclass(frozen=True)
class ReplicationSample:
    primary: PositionSample
    replica_received: PositionSample
    replica_applied: PositionSample

    @property
    def sampling_skew(self) -> float:
        """Spread between the earliest and latest endpoint read, in seconds."""
        stamps = (
            self.primary.observed_at,
            self.replica_received.observed_at,
            self.replica_applied.observed_at,
        )
        return max(stamps) - min(stamps)


def take_sample(
    read_primary: Callable[[], Tuple[int, float]],
    read_received: Callable[[], Tuple[int, float]],
    read_applied: Callable[[], Tuple[int, float]],
) -> ReplicationSample:
    def _wrap(reader: Callable[[], Tuple[int, float]]) -> PositionSample:
        pos, _ = reader()
        return PositionSample(position=pos, observed_at=time.monotonic())

    return ReplicationSample(
        primary=_wrap(read_primary),
        replica_received=_wrap(read_received),
        replica_applied=_wrap(read_applied),
    )

Decomposing Transport Lag from Apply Lag

A single “seconds behind” number conflates two independent failures. Transport lag is the gap between what the primary has written and what the replica has received over the network; it grows when the shipping link is saturated or broken. Apply lag is the further gap between what the replica has received and what it has actually replayed into its data files; it grows when the replica’s single replay thread cannot keep pace with the primary’s write concurrency, even though bytes are arriving fine. A replica can have near-zero transport lag and minutes of apply lag — the data is on the box but not yet queryable, so a promotion would still lose it. Keeping the two separate is what lets the classifier point at the right subsystem instead of paging on an undifferentiated number.

from dataclasses import dataclass


@dataclass(frozen=True)
class LagBreakdown:
    """Byte-domain lag split into its transport and apply components."""

    transport_bytes: int
    apply_bytes: int

    @property
    def total_bytes(self) -> int:
        return self.transport_bytes + self.apply_bytes


def decompose_lag(
    primary_pos: int, received_pos: int, applied_pos: int
) -> LagBreakdown:
    """Split end-to-end byte lag into transport (unsent) and apply (unreplayed)."""
    transport = max(0, primary_pos - received_pos)
    apply = max(0, received_pos - applied_pos)
    return LagBreakdown(transport_bytes=transport, apply_bytes=apply)

Comparing Lag Against the RPO Budget

Byte lag alone does not answer the recovery question; the budget is expressed in time, because RPO is a bound on how much recent state a promotion may lose. The bridge is the replica’s last applied transaction timestamp: the difference between the primary’s clock and the timestamp of the newest transaction the replica has replayed is the time-domain recovery-point lag, and that is what gets compared against the RPO. Byte lag remains valuable as a leading indicator and as the signal that distinguishes a stalled link from an idle one — zero write traffic produces zero byte lag but should not be read as a healthy replica if apply has actually stopped. The classifier therefore consumes both a time lag and the byte breakdown.

#!/usr/bin/env python3
"""Classify a replication sample against an RPO budget.

Exit codes (consumed by the promotion gate):
    0  replica is within the RPO budget          -> promotable
    1  RPO breach or stalled apply                -> not promotable
    2  usage / configuration error                -> abort the check
"""
import sys
from dataclasses import dataclass
from enum import Enum


class ReplState(str, Enum):
    HEALTHY = "healthy"
    DEGRADED = "degraded"
    BREACH = "breach"


@dataclass(frozen=True)
class RpoVerdict:
    state: ReplState
    time_lag_seconds: float
    reason: str


def classify(
    time_lag_seconds: float,
    apply_bytes: int,
    rpo_seconds: float,
    warn_fraction: float = 0.6,
) -> RpoVerdict:
    """Grade a sample: healthy < warn threshold, degraded up to RPO, breach beyond."""
    if rpo_seconds <= 0:
        raise ValueError("rpo_seconds must be positive")
    if time_lag_seconds > rpo_seconds:
        return RpoVerdict(ReplState.BREACH, time_lag_seconds,
                          f"time lag {time_lag_seconds:.1f}s exceeds RPO {rpo_seconds:.1f}s")
    if time_lag_seconds >= rpo_seconds * warn_fraction:
        return RpoVerdict(ReplState.DEGRADED, time_lag_seconds,
                          f"time lag {time_lag_seconds:.1f}s within {warn_fraction:.0%} of RPO")
    return RpoVerdict(ReplState.HEALTHY, time_lag_seconds, "within budget")


def main() -> int:
    if len(sys.argv) != 4:
        print("usage: classify.py <time_lag_s> <apply_bytes> <rpo_s>", file=sys.stderr)
        return 2
    try:
        time_lag = float(sys.argv[1])
        apply_bytes = int(sys.argv[2])
        rpo = float(sys.argv[3])
        verdict = classify(time_lag, apply_bytes, rpo)
    except ValueError as exc:
        print(f"config error: {exc}", file=sys.stderr)
        return 2
    print(f"{verdict.state.value.upper()} {verdict.reason}")
    return 0 if verdict.state is not ReplState.BREACH else 1


if __name__ == "__main__":
    sys.exit(main())

Detecting Silent Replica Divergence

Lag can read zero while the replica is quietly wrong. Logical replication conflicts, an errant write on a replica that was supposed to be read-only, or storage-level bit rot can leave the applied position perfectly caught up while the bytes on disk no longer match the primary. Lag monitoring cannot see this — both ends report the same position — so a periodic divergence check is a distinct control. The technique mirrors artifact checksumming: hash a bounded, deterministic slice of a table on both primary and replica at the same logical position and compare the digests. A drift means the replica is caught up but corrupt, which is a harder failure than lag because promotion would silently serve wrong data. Because a full-table hash is expensive, production checks sample rotating key ranges so the cost amortizes across cycles.

The correctness of this check hinges on comparing the two endpoints at the same logical point, not merely at the same wall-clock instant. A naive hash of the primary and a still-lagging replica will always differ simply because the replica has not yet applied the primary’s most recent writes — that is expected lag, not divergence. A robust implementation pins the comparison to a shared coordinate: it reads the replica’s applied position first, then hashes only the range of the primary that the replica has definitely already consumed, so any digest mismatch is genuine corruption rather than in-flight replication. This is why the divergence check is layered on top of the lag decomposition rather than run independently — it needs the applied coordinate the earlier phases already established.

Classifying and Gating Promotion Readiness

The terminal phase folds the RPO comparison and the divergence check into a single promotion verdict. A replica is promotable only when its time lag is inside the RPO budget, its apply process is advancing rather than stalled, and its most recent divergence check passed. Any one of those failing flips the gate closed. Crucially, the gate is a live property re-evaluated every cycle, not a one-time blessing: a replica that was promotable a second ago but has since stalled must be denied promotion the instant the next sample lands, which is exactly why the loop is continuous rather than triggered only at failover time.

Integration with DR Drill Orchestration

Continuous replication validation is the near-zero-RPO complement to periodic restore drills, and the two feed a shared decision surface. When a failover is requested, the orchestrator does not blindly promote the configured standby — it reads the most recent promotion verdict from this loop and refuses any replica currently classified as breached or diverged, falling through to the next candidate. That behavior is the runtime input to fallback chain configuration: the chain’s ordering is only safe if each candidate’s live promotability is known. Choosing whether a given system is governed by continuous replication validation at all, versus periodic restore verification, is itself a validation-model selection decision driven by the RPO tier.

Every verdict is also an event, not just a gate. Each cycle publishes its state transition — healthy to degraded, degraded to breach — so that trend analysis can catch a replica whose apply lag is creeping upward long before it crosses the budget. Those events flow into the same telemetry surface the restore drills write to, giving operators one place to see both “can we restore from backup?” and “is the live replica promotable?” without reconciling two monitoring systems.

Error Classification and Thresholds

Not every degraded sample is an incident. A single cycle at 65% of budget on a bursty write workload is expected noise; a sustained breach, or any divergence, is unconditional. Mapping replication verdicts onto severity tiers — with sustain windows rather than single-sample triggers — keeps the loop sensitive to genuine regression without paging on jitter.

A frozen apply position deserves special handling: if the applied coordinate has not advanced across several samples while the primary keeps writing, the replica is stalled regardless of what its reported “seconds behind” says, because that figure can read zero on an idle link. The classifier must treat a non-advancing apply position under active primary writes as a breach in its own right, not merely a large lag number.

Telemetry and Compliance Output

Each cycle emits structured telemetry so replication health is a time series, not a snapshot glimpsed during an incident. Metrics export through Prometheus-compatible endpoints and feed both SLO reporting and DR evidence.

The verdict stream is written to append-only, tamper-evident storage capturing the sampled positions, the computed lag breakdown, the RPO in force, and the promotion decision, so a post-incident review can prove whether the replica was promotable at the moment of failover. This aligns replication evidence with the demonstrable-measurement expectations of frameworks such as NIST SP 800-34 Rev. 1, which require that contingency capabilities be measured and recorded rather than asserted.

Operational Best Practices

• Measure time lag, gate on it, but alert on byte lag too. Time-domain lag answers the RPO question; the transport-versus-apply byte split tells you which subsystem to fix and catches a stalled apply that reports zero seconds behind.
• Treat a frozen apply position as a breach. Never trust a “seconds behind” figure in isolation — cross-check that the applied coordinate is actually advancing under active primary writes.
• Run divergence checks on a slower cadence than lag checks. Full-fidelity hashing is expensive; sample rotating key ranges so the cost amortizes while still catching silent corruption within a bounded window.
• Bound sampling skew. Record the wall-clock instant of every endpoint read and discard any sample whose skew is large enough to distort the lag math.
• Re-evaluate promotability every cycle. A promotion gate is a live property; a replica healthy a second ago may be stalled now, so the failover path must read the latest verdict, not a cached one.
• Derive the threshold from the RPO tier, not a hand-picked constant. The warn and breach lines should fall out of the mapped objective so they scale automatically as tiers are retuned.

Treating replication as a continuously validated control, rather than a daemon presumed healthy until failover proves otherwise, is what makes a near-zero RPO a measured capability instead of an aspiration. The engine-specific validators below turn this model into runnable checks for PostgreSQL and MySQL.

Frequently Asked Questions

Related

• Validating PostgreSQL Streaming Replication Lag — a runnable LSN-based lag validator for PostgreSQL streaming replicas.
• Monitoring MySQL Replica Lag for RPO — a heartbeat-and-GTID validator that gets past the unreliable Seconds_Behind_Source figure.
• RTO/RPO mapping frameworks — where the RPO budget these checks gate against is derived.
• Validation model selection — choosing continuous replication validation over periodic restore verification per tier.
• Validation telemetry and metrics — where each cycle’s verdict and lag series are recorded.

This guide is one component of the broader Core DR Architecture & Validation Fundamentals framework.