Exporting Validation Metrics to Prometheus

This page implements the push-collection path described in the parent guide on validation telemetry and metrics: a complete prometheus_client module that instruments a single ephemeral backup-validation drill and ships its results to a Prometheus Pushgateway. A drill job is too short-lived to scrape — it may run for under a minute and exit before the next scrape interval fires — so it must push its terminal metric values to a long-lived gateway that Prometheus collects on its own cadence. The verdict this module emits comes from a checksum validation pipeline, its restore-duration histogram is evaluated against the window your RTO and RPO mapping defines, and any non-zero verdict is routed through your error categorization frameworks rather than lost with the process.

Architecture and Execution Model

Figure. The drill builds an isolated registry, fills it during validation, keys the push idempotently, ships it to the gateway, and exits with a code the scheduler reads.

The module is a single process with a strictly linear lifecycle. Nothing here is a long-running server; it is a job that runs once, records what happened, hands those numbers to the gateway, and terminates with a meaningful status. The gateway then holds the last-pushed values until Prometheus scrapes them and until the job deletes its own group on teardown.

Prerequisites

• Python 3.8+ on the drill executor host.

• The Prometheus client library:

  pip install prometheus_client
  

• A reachable Pushgateway — the hostname and port of a running pushgateway instance (default port 9091), routable from wherever the drill executes.

• Network egress from the drill sandbox to the gateway; if the drill runs in an isolated VPC, the gateway endpoint must be explicitly allowed through the egress policy.

• A grouping-key convention agreed with the Prometheus operators: which labels form the group identity so that a re-run overwrites rather than accumulates.

Production Implementation

The module separates three concerns: constructing the registry and metrics, running the validation while observing them, and the push/teardown transport. Keeping the registry local to the job — not the library’s global default — is what makes the job safe to clone across regions without series collisions.

#!/usr/bin/env python3
"""Instrument an ephemeral backup-validation drill and push it to a Pushgateway.

Usage:
    python3 export_validation_metrics.py <gateway_host:port> <drill_id> <engine>

Exit codes (consumed by the DR drill orchestrator):
    0  drill validated cleanly and metrics were pushed
    1  drill produced a failing verdict (metrics still pushed for evidence)
    2  usage / configuration error, or the gateway push failed
"""
import sys
import time
from typing import Dict, Tuple

from prometheus_client import CollectorRegistry, Counter, Histogram, Gauge
from prometheus_client.exposition import (
    push_to_gateway, delete_from_gateway,
)

JOB_NAME = "backup_validation_drill"
RESTORE_BUCKETS = (30, 60, 120, 300, 600, 1200, 1800, 3600)


def build_registry() -> Tuple[CollectorRegistry, Dict[str, object]]:
    """Return an isolated registry and a dict of the metrics defined on it."""
    registry = CollectorRegistry()
    metrics = {
        "validated": Counter(
            "artifacts_validated_total",
            "Backup artifacts that completed a validation attempt.",
            ("engine", "verdict"), registry=registry,
        ),
        "corrupt": Counter(
            "corrupt_pages_found_total",
            "Corrupt storage-engine pages detected during validation.",
            ("engine",), registry=registry,
        ),
        "restore": Histogram(
            "restore_duration_seconds",
            "Wall-clock seconds to restore the artifact into the sandbox.",
            ("engine",), buckets=RESTORE_BUCKETS, registry=registry,
        ),
        "lag": Gauge(
            "replication_lag_seconds",
            "Replica lag against the RPO ceiling observed at drill start.",
            ("engine",), registry=registry,
        ),
    }
    return registry, metrics

The validation itself is stubbed here with a deterministic simulation so the module runs end-to-end without a live database, but the interface is the real one: it observes a restore duration into the histogram, sets the replication-lag gauge, increments the corruption counter when pages fail, and returns a verdict string plus a boolean success flag that becomes the process exit code.

def run_validation(metrics: Dict[str, object], engine: str) -> Tuple[str, bool]:
    """Simulate one artifact's restore+verify, observing metrics as it goes.

    Replace the body with a real restore + checksum gate; the observe/inc/set
    calls and the (verdict, ok) return contract stay identical.
    """
    # Observe replication lag read at drill start (RPO evidence).
    metrics["lag"].labels(engine=engine).set(4.2)

    # Time the restore region; the context manager buckets the elapsed seconds.
    with metrics["restore"].labels(engine=engine).time():
        time.sleep(0.02)  # stand-in for restore + integrity check

    corrupt_pages = 0            # a real gate derives this from the scan
    if corrupt_pages > 0:
        metrics["corrupt"].labels(engine=engine).inc(corrupt_pages)
        verdict, ok = "invalid", False
    else:
        verdict, ok = "valid", True

    metrics["validated"].labels(engine=engine, verdict=verdict).inc()
    return verdict, ok

The transport function is where the ephemeral-job discipline lives. The grouping_key is what makes the push idempotent: a re-run of the same drill_id and engine overwrites the prior group rather than piling up duplicate series. On teardown the job deletes its own group so the gateway does not accumulate dead series after the drill is gone.

def push_metrics(registry: CollectorRegistry, gateway: str,
                 drill_id: str, engine: str) -> None:
    """Push under idempotent grouping keys; raise on transport failure."""
    grouping_key = {"drill_id": drill_id, "engine": engine}
    push_to_gateway(gateway, job=JOB_NAME, grouping_key=grouping_key,
                    registry=registry)


def delete_metrics(gateway: str, drill_id: str, engine: str) -> None:
    """Remove this drill's group from the gateway on teardown."""
    grouping_key = {"drill_id": drill_id, "engine": engine}
    delete_from_gateway(gateway, job=JOB_NAME, grouping_key=grouping_key)


def main(argv: list) -> int:
    if len(argv) != 4:
        print("usage: export_validation_metrics.py "
              "<gateway_host:port> <drill_id> <engine>", file=sys.stderr)
        return 2

    gateway, drill_id, engine = argv[1], argv[2], argv[3]
    registry, metrics = build_registry()

    verdict, ok = run_validation(metrics, engine)

    try:
        push_metrics(registry, gateway, drill_id, engine)
    except (OSError, ValueError) as exc:
        print(f"push to gateway failed: {exc}", file=sys.stderr)
        return 2

    print(f"drill {drill_id} engine={engine} verdict={verdict} pushed")
    return 0 if ok else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Note the deliberate ordering: metrics are pushed before the exit code is chosen, so a failing drill still leaves durable evidence on the gateway. Teardown deletion is invoked by the orchestrator after Prometheus has scraped the group — not inline in main — because deleting immediately would race the scrape and lose the sample.

Step-by-Step Execution Walkthrough

1. Install the client library into the executor’s environment with pip install prometheus_client.
2. Confirm the gateway is reachable from where the drill runs — a quick curl http://<gateway_host:port>/metrics should return the gateway’s own exposition output.
3. Run the module with the gateway address, a unique drill id, and the engine label: python3 export_validation_metrics.py pushgateway.dr.internal:9091 drill-20260718-01 postgres.
4. Read the exit code. echo $? yields 0 for a clean drill, 1 for a failing verdict (metrics still pushed), or 2 for a usage or transport fault.
5. Verify the group landed by scraping the gateway and confirming the pushed series appear under the drill_id and engine grouping labels.
6. Delete the group on teardown once Prometheus has scraped it, by calling delete_metrics (or curl -X DELETE) so the gateway does not retain stale series.

Verification and Expected Output

A clean run prints one summary line and exits 0:

$ python3 export_validation_metrics.py pushgateway.dr.internal:9091 drill-20260718-01 postgres
drill drill-20260718-01 engine=postgres verdict=valid pushed
$ echo $?
0

Scraping the gateway afterward shows the pushed series carrying both the metric labels and the grouping labels the push injected:

artifacts_validated_total{drill_id="drill-20260718-01",engine="postgres",verdict="valid"} 1.0
replication_lag_seconds{drill_id="drill-20260718-01",engine="postgres"} 4.2
restore_duration_seconds_bucket{drill_id="drill-20260718-01",engine="postgres",le="60.0"} 1.0
restore_duration_seconds_count{drill_id="drill-20260718-01",engine="postgres"} 1.0

Success means three things held: run_validation returned a valid verdict, push_to_gateway did not raise, and the series are visible under the expected grouping key. A 1 exit with the same series present is a successful export of a failing drill — the evidence is intact and the orchestrator quarantines the artifact.

Failure Modes and Troubleshooting

Integration Notes

Because the module returns clean POSIX exit codes, any scheduler can gate on it. Under Airflow, wrap it in a BashOperator (or PythonOperator calling main) where a non-zero return fails the task and short-circuits the downstream promotion step; the DAG run history becomes part of the drill record. Under Celery, dispatch it from a task that raises on non-zero return so event-driven drills — kicked off when a fresh backup lands — get low-latency export and the broker logs every failure. Under cron or systemd, schedule the wrapper directly and attach an OnFailure handler that routes the quarantine alert.

Feed the classified verdict into your error categorization frameworks so a corruption verdict and a lag-breach verdict escalate as distinct events, and treat the exported series as the numeric input to the broader validation telemetry and metrics workflow that the dashboards and burn-rate alerts read from.

Frequently Asked Questions

Related

• Validation telemetry and metrics — the parent guide defining the taxonomy and label schema this module emits.
• Grafana dashboards for DR drill observability — rendering the pushed series into panels and alerts.
• Immutable audit trails — persisting the exported evidence into a tamper-evident store.
• Error categorization frameworks — turning a failing verdict into an actionable severity tier.

This module is one component of the broader validation telemetry and metrics workflow.

For authoritative API behavior, consult the Prometheus Python client library documentation and the Python sys module reference for exit-code semantics.