Mapping Backup Validation to NIST SP 800-34

This page implements one concrete task from the broader compliance evidence and control mapping guide: a self-contained Python tool that proves — or fails to prove — that your automated backup validation satisfies the NIST SP 800-34 Rev. 1 contingency-planning controls. It ingests a declarative control-to-component map covering CP-9 (Information System Backup), CP-10 (System Recovery and Reconstitution), and CP-4 (Contingency Plan Testing), joins it against a stream of drill outcome records produced by your checksum validation pipeline and restore drills, and emits a per-control coverage report. Because a “passing” restore is only meaningful when it lands inside the window your RTO and RPO mapping defines, the tool trusts the verdict already stamped on each record rather than re-deriving it, and it separates a failed drill from an errored one using the same distinctions your error categorization frameworks apply upstream. The tool exits non-zero the moment any required control has no passing evidence in the audit window, so it can gate a compliance pipeline directly.

Architecture and Execution Model

Figure. The tool loads the control map, ingests evidence, matches records to CP-family controls, computes per-control coverage, and returns a report plus a gating exit code.

The stages enforce one ordering rule that matters for audit defensibility: the control map is loaded and frozen before any evidence is read, so the set of controls under evaluation cannot be influenced by the evidence itself. This prevents the subtle failure where a coverage tool silently narrows its own scope to whatever happens to be present in the telemetry, reporting a clean pass because it never asked about the controls that produced no records. The map is the claim; the evidence either substantiates it or does not. Each run is pure with respect to its inputs — the same control map and the same evidence file at the same wall-clock instant always yield the same report and the same exit code, which is precisely the reproducibility an auditor exercises when re-running the tool months later.

Prerequisites

• Python 3.9+ on the executor host.

• PyYAML for the human-editable control map:

  pip install pyyaml
  

• A control map at nist_control_map.yaml naming each CP control, the component expected to prove it, the artifact type that counts, and its freshness limit.

• A drill evidence file — newline-delimited JSON or a JSON array — carrying one record per validation or restore run, each with a control_id, component, verdict, and ISO-8601 timestamp.

Production Implementation

The control map encodes exactly which NIST SP 800-34 controls the organization claims to satisfy through automated testing, and what evidence proves each one. CP-9 is proven by a passing checksum or backup-integrity report; CP-10 is proven by a successful restore drill that reconstituted the system; CP-4 is proven by the periodic execution of the drill itself. Keeping this map in YAML lets a compliance owner adjust freshness limits or add a control without a code change.

# nist_control_map.yaml
audit_window_days: 90
controls:
  - control_id: "CP-9"
    title: "Information System Backup"
    component: "checksum-validator"
    artifact_type: "checksum-report"
    max_age_days: 7
    required: true
  - control_id: "CP-10"
    title: "System Recovery and Reconstitution"
    component: "restore-drill-runner"
    artifact_type: "restore-drill"
    max_age_days: 30
    required: true
  - control_id: "CP-4"
    title: "Contingency Plan Testing"
    component: "drill-scheduler"
    artifact_type: "drill-execution"
    max_age_days: 90
    required: true

The three CP controls in the map are not interchangeable, and the distinction is the point. CP-9 asks whether backups exist and are intact — a checksum or backup-integrity report answers it. CP-10 asks whether the system can actually be brought back from those backups — only a completed restore drill answers it, because an intact backup that has never been restored is an untested assumption, not a proven capability. CP-4 asks whether the contingency plan is being exercised at all, which the periodic firing of the drill itself demonstrates. Collapsing these into a single “backups work” signal is the most common way automated evidence misleads an auditor, so the tool holds them apart by binding each control to a specific producing component.

The tool loads that map, ingests the evidence stream, and applies the matching logic. A control is covered only when a pass record exists for its component, dated inside the audit window and within the control’s own freshness limit. The evidence-matching function is intentionally strict about component identity so that a passing checksum run cannot be miscredited as proof of a restore drill.

#!/usr/bin/env python3
"""Evaluate NIST SP 800-34 Rev. 1 contingency-control coverage from drill evidence.

Usage:
    python3 nist_coverage.py <control_map.yaml> <evidence.jsonl>

Exit codes:
    0  every required control has fresh passing evidence in the audit window
    1  at least one required control is uncovered -> compliance gap
    2  usage or configuration error
"""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

import yaml


@dataclass(frozen=True)
class Control:
    control_id: str
    title: str
    component: str
    artifact_type: str
    max_age_days: int
    required: bool


def load_controls(path: str) -> tuple[list[Control], int]:
    with open(path, "r", encoding="utf-8") as fh:
        doc = yaml.safe_load(fh)
    window = int(doc["audit_window_days"])
    controls = [
        Control(
            control_id=c["control_id"],
            title=c["title"],
            component=c["component"],
            artifact_type=c["artifact_type"],
            max_age_days=int(c["max_age_days"]),
            required=bool(c.get("required", True)),
        )
        for c in doc["controls"]
    ]
    return controls, window


def load_evidence(path: str) -> list[dict]:
    text = open(path, "r", encoding="utf-8").read().strip()
    if not text:
        return []
    if text.lstrip().startswith("["):
        return json.loads(text)
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def _parse(ts: str) -> datetime:
    return datetime.fromisoformat(ts).astimezone(timezone.utc)

The coverage computation walks each control and searches the evidence for a qualifying record, then packages a per-control verdict. Returning the newest matching timestamp alongside the boolean lets the report show how fresh the proof is, not merely that it exists.

def evaluate(controls: list[Control], evidence: list[dict],
             window_days: int, now: datetime) -> list[dict]:
    """Produce a per-control coverage row for the report."""
    window_start = now - timedelta(days=window_days)
    rows: list[dict] = []
    for ctrl in controls:
        floor = max(window_start, now - timedelta(days=ctrl.max_age_days))
        matches = [
            _parse(rec["timestamp"])
            for rec in evidence
            if rec.get("control_id") == ctrl.control_id
            and rec.get("component") == ctrl.component
            and rec.get("verdict") == "pass"
            and _parse(rec["timestamp"]) >= floor
        ]
        newest = max(matches) if matches else None
        rows.append({
            "control_id": ctrl.control_id,
            "title": ctrl.title,
            "required": ctrl.required,
            "covered": bool(matches),
            "newest_evidence": newest.isoformat() if newest else None,
        })
    return rows


def render(rows: list[dict]) -> str:
    lines = ["NIST SP 800-34 Rev. 1 contingency-control coverage", "-" * 52]
    for row in rows:
        status = "COVERED" if row["covered"] else "GAP    "
        flag = "required" if row["required"] else "optional"
        proof = row["newest_evidence"] or "none"
        lines.append(f"{status} {row['control_id']:<6} [{flag}] newest={proof}")
    return "\n".join(lines)


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: nist_coverage.py <control_map.yaml> <evidence.jsonl>",
              file=sys.stderr)
        return 2
    try:
        controls, window = load_controls(sys.argv[1])
        evidence = load_evidence(sys.argv[2])
    except (OSError, KeyError, ValueError, yaml.YAMLError) as exc:
        print(f"config error: {exc}", file=sys.stderr)
        return 2
    rows = evaluate(controls, evidence, window, datetime.now(timezone.utc))
    print(render(rows))
    gaps = [r["control_id"] for r in rows if r["required"] and not r["covered"]]
    if gaps:
        print(f"UNCOVERED REQUIRED CONTROLS: {', '.join(gaps)}", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Step-by-Step Execution Walkthrough

1. Author the control map. Write nist_control_map.yaml with one entry per CP control you claim through automated testing; set each max_age_days to match how often that control is genuinely exercised.
2. Export the evidence stream. Have your telemetry layer emit one record per validation and restore run into evidence.jsonl, each carrying control_id, component, verdict, and an ISO-8601 UTC timestamp.
3. Run the tool. python3 nist_coverage.py nist_control_map.yaml evidence.jsonl. The per-control report prints to stdout; any uncovered required control prints to stderr.
4. Read the exit code. echo $? — 0 means every required CP control is covered and the audit window is clean, 1 names a compliance gap, 2 signals a malformed map or evidence file.
5. Gate on it. Wire the non-zero exit into the compliance pipeline exactly like a failed validation: block the auditor export and schedule an out-of-cycle drill for the named control.

Verification and Expected Output

A clean run over a fully covered window prints the coverage table and exits 0:

NIST SP 800-34 Rev. 1 contingency-control coverage
----------------------------------------------------
COVERED CP-9   [required] newest=2026-07-18T02:14:07+00:00
COVERED CP-10  [required] newest=2026-07-05T19:02:44+00:00
COVERED CP-4   [required] newest=2026-06-30T00:00:00+00:00

If the most recent CP-10 restore drill has aged past its 30-day freshness limit, that control flips to a gap and the process exits 1:

COVERED CP-9   [required] newest=2026-07-18T02:14:07+00:00
GAP     CP-10  [required] newest=none
COVERED CP-4   [required] newest=2026-06-30T00:00:00+00:00
UNCOVERED REQUIRED CONTROLS: CP-10

Success means all three conditions held: the YAML parsed, every evidence record carried a usable timestamp, and each required control had at least one fresh passing record. Treat a 1 exit as a hard finding, not a warning — an uncovered CP-10 means no recent drill actually proved the system can be reconstituted.

Failure Modes and Troubleshooting

Integration Notes

The tool returns clean POSIX exit codes, so any scheduler can gate on it. In Airflow, run it in a BashOperator downstream of the drill DAG; a non-zero exit fails the task and the DAG run history becomes part of the audit record. Under Celery, dispatch it from a task that raises on non-zero return so a fresh evidence file triggers immediate re-evaluation. A nightly cron invocation writes the report to the immutable audit trails store and alerts on any 1 exit. Feed the named gaps back into drill scheduling and orchestration so the next cycle prioritizes exactly the controls that fell out of coverage, and consult the NIST SP 800-34 Rev. 1 publication for the authoritative control definitions and the Python datetime documentation for the timezone-aware parsing this tool relies on.

Frequently Asked Questions

Related

• Compliance evidence and control mapping — the parent guide defining the evidence model this tool populates.
• Generating SOC 2 and ISO 22301 Evidence — the sibling projection for the Availability criteria and continuity clauses.
• Immutable audit trails — where each coverage report is retained as a chain-of-custody record.
• Validation telemetry and metrics — the upstream source of the evidence records this tool ingests.

This tool is one component of the broader compliance evidence and control mapping guide.