Smoke Test Routing Logic

Disaster recovery validation is only as trustworthy as its traffic control plane. Smoke test routing logic is the authoritative dispatcher that maps health checks, schema probes, and synthetic transactions onto freshly restored endpoints without ever touching production DNS, service-mesh state, or connection pools. Within the broader Restore Drill Orchestration & Environment Isolation framework, this routing layer is what converts a static backup artifact into a measurable recovery state: it resolves where the restored service actually lives, forces validation traffic to reach only that address, and proves the traffic never escaped the drill topology. The operational gap it closes is specific — a restore can complete successfully and still be untestable, because nothing yet knows how to address the new instance without a stray query landing on the primary cluster.

That gap sits directly downstream of two adjacent concerns. Routing consumes the endpoints created by sandbox provisioning automation, and it must align its synthetic workload with the exact coordinate produced by point-in-time recovery targeting so that a probe reads the state the drill claims to have recovered. It only produces a defensible number when the pass/fail threshold is expressed against a recovery envelope — a “routable” restore is one that answers inside the windows defined by your RTO and RPO mapping. Get the routing wrong and every downstream signal is contaminated: a smoke test that silently hit production reports a green check that certifies nothing.

Architecture and Execution Workflow

Figure. State-aware routing layer that resolves sandbox endpoints and dispatches synthetic validation traffic confined to the drill topology.

The routing control plane runs as an event-driven stage wedged between the orchestrator and the validation scripts. Drill orchestration follows a strict sequence — infrastructure materialization, endpoint registration, validation execution, teardown — and the routing layer intercepts the provisioning-completion event to construct a temporary traffic map that overrides default resolution. Rather than relying on static host files or manual environment-variable edits, the layer reads newly provisioned endpoint addresses directly from infrastructure state and serializes them into a routing manifest that the validation runner consumes. That manifest is the single source of truth for which synthetic request targets which restored service, so read-only smoke tests, write-path validations, and dependency checks all stay confined to the drill topology. Each phase below is idempotent: re-running it against the same provisioning event yields the same manifest and the same terminal reachability state, which makes the stage safe to retry under transient control-plane faults.

Endpoint Discovery and Registry Population

The layer initiates when triggered by a provisioning-completion webhook, a state-store change event, or a scheduled orchestrator poll. Discovery parses the authoritative record of what was just built — Terraform state outputs, cloud-provider metadata endpoints, or a Kubernetes service-discovery API — and maps each logical service identifier (orders-db, inventory-cache, payments-api) to its dynamically assigned hostname or address. Discovery is the phase most prone to false negatives: a partially applied Terraform run can expose an output whose backing resource is not yet Ready, so the resolver must reconcile the state record against a liveness signal before it registers an endpoint. Registration also captures the port, protocol, and the drill session tag, because those fields become the match conditions that keep confinement enforceable later.

Routing Manifest Construction and Precedence

Once the registry is populated, the layer serializes it into a routing manifest — a versioned mapping of logical identifier to resolved endpoint plus the environment rules that govern it. Construction is where precedence is resolved: a service may have a static override for a pinned dependency, a session-scoped sandbox address, and a fallback address, and the manifest must collapse those into one deterministic winner per identifier. Precedence is evaluated highest-specificity-first (explicit session override beats environment default beats global fallback) so that two concurrent drills never resolve the same logical name to each other’s sandbox. The manifest is written before any traffic moves, giving auditors a byte-stable artifact that records exactly where each probe was told to go.

Traffic Override Injection

With a manifest in hand, the layer applies the overrides that actually redirect traffic. For database-centric drills it rewrites connection strings so that psycopg2, pymysql, or sqlalchemy open sessions against the isolated restore replica rather than the primary. For service-to-service paths it injects a signed drill header (X-Drill-Context) or rewrites container-level /etc/hosts entries and sidecar-proxy routes so downstream calls resolve inside the sandbox. Injection is deliberately confined to the drill process tree and namespace — the override is process- or namespace-scoped, never a mutation of shared DNS zones or the production mesh, so a leaked override cannot outlive the session.

Synthetic Dispatch and Confinement Verification

Dispatch reads the manifest, builds a request context per target, and fans out health checks, schema probes, and synthetic transactions under a concurrency ceiling. Every request carries the session tag, and the layer performs the load-bearing step that distinguishes a real isolation control from wishful thinking: it verifies that the response came from the sandbox endpoint and that no probe egressed to a production CIDR. Confinement is asserted, not assumed — an unreachable sandbox endpoint is a routable failure that hands off to the fallback chain, whereas a probe observed leaving the drill boundary is a safety violation that aborts the session immediately.

Teardown and Route Reclamation

When validation completes, the layer reverses every override in dependency order: connection strings revert, injected hosts entries and proxy routes are removed, and the manifest is sealed into the audit trail. Reclamation is idempotent and defensive — a reconciliation sweep looks for any override still tagged to the finished session and force-purges it, so a crashed runner cannot leave a stale route pointing production callers at a torn-down sandbox.

Python Implementation Patterns

Python is the natural orchestration language for this stage: asyncio gives cheap concurrency for dozens of endpoint probes, and the stdlib alone is enough to resolve, dispatch, and gate without dragging heavyweight dependencies into the drill runner. The pattern that scales abstracts endpoint resolution behind a pluggable interface, so a team can back the resolver with Terraform state today and a Kubernetes API tomorrow without touching dispatch logic.

Pluggable Endpoint Resolvers

The resolver is an abstract base class with one responsibility: turn a logical service identifier into a concrete, session-scoped endpoint. Backends — Terraform JSON state, cloud metadata, or the Kubernetes service registry — implement the same method, so the manifest builder never learns where addresses come from. The engine reads a signed manifest, applies precedence, and returns a strict POSIX exit code so the DR drill orchestrator can gate on it directly:

#!/usr/bin/env python3
"""Smoke-test routing engine for DR drill validation.

Resolves logical service identifiers to session-scoped sandbox endpoints,
builds a deterministic routing manifest, and returns a POSIX exit code the
orchestrator can gate on.

Exit codes (consumed by the DR drill orchestrator):
    0  every logical service resolved to exactly one sandbox endpoint
    1  at least one service failed to resolve -> hand off to fallback chain
    2  usage / configuration error -> abort routing stage
"""
import json
import sys
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, List, Optional


@dataclass(frozen=True)
class Endpoint:
    """A resolved, confinement-tagged target for synthetic traffic."""

    service: str
    host: str
    port: int
    scheme: str
    session_tag: str

    @property
    def url(self) -> str:
        return f"{self.scheme}://{self.host}:{self.port}"


class EndpointResolver(ABC):
    """Turn a logical service id into a session-scoped sandbox endpoint."""

    @abstractmethod
    def resolve(self, service: str) -> Optional[Endpoint]:
        ...


class TerraformStateResolver(EndpointResolver):
    """Resolve endpoints from a `terraform show -json` state document.

    Only resources tagged with the active drill session are eligible, which is
    what prevents one drill from resolving another drill's sandbox address.
    """

    def __init__(self, state: Dict, session_tag: str) -> None:
        self.session_tag = session_tag
        self._index: Dict[str, Endpoint] = {}
        for res in state.get("values", {}).get("root_module", {}).get("resources", []):
            attrs = res.get("values", {})
            if attrs.get("tags", {}).get("drill_session") != session_tag:
                continue
            service = attrs.get("tags", {}).get("logical_service")
            host = attrs.get("private_dns") or attrs.get("address")
            if service and host:
                self._index[service] = Endpoint(
                    service=service,
                    host=host,
                    port=int(attrs.get("port", 5432)),
                    scheme=attrs.get("scheme", "postgres"),
                    session_tag=session_tag,
                )

    def resolve(self, service: str) -> Optional[Endpoint]:
        return self._index.get(service)


@dataclass
class RoutingManifest:
    """Deterministic mapping of logical service -> winning endpoint."""

    session_tag: str
    routes: Dict[str, Endpoint] = field(default_factory=dict)
    unresolved: List[str] = field(default_factory=list)

    def to_json(self) -> str:
        return json.dumps(
            {
                "session_tag": self.session_tag,
                "routes": {
                    svc: {"url": ep.url, "session_tag": ep.session_tag}
                    for svc, ep in self.routes.items()
                },
                "unresolved": self.unresolved,
            },
            indent=2,
            sort_keys=True,
        )


def build_manifest(
    services: List[str],
    resolvers: List[EndpointResolver],
    session_tag: str,
) -> RoutingManifest:
    """Apply resolvers highest-precedence-first; first hit wins per service."""
    manifest = RoutingManifest(session_tag=session_tag)
    for service in services:
        endpoint = next(
            (ep for r in resolvers if (ep := r.resolve(service)) is not None),
            None,
        )
        if endpoint is None:
            manifest.unresolved.append(service)
        else:
            manifest.routes[service] = endpoint
    return manifest


def main() -> int:
    if len(sys.argv) != 4:
        print(
            "usage: routing_engine.py <services.json> <tfstate.json> <session_tag>",
            file=sys.stderr,
        )
        return 2
    try:
        services = json.loads(Path(sys.argv[1]).read_text())
        state = json.loads(Path(sys.argv[2]).read_text())
    except (OSError, json.JSONDecodeError) as exc:
        print(f"config error: {exc}", file=sys.stderr)
        return 2

    session_tag = sys.argv[3]
    resolvers = [TerraformStateResolver(state, session_tag)]
    manifest = build_manifest(services, resolvers, session_tag)
    Path("routing_manifest.json").write_text(manifest.to_json())

    for service in manifest.unresolved:
        print(f"UNRESOLVED {service}", file=sys.stderr)
    for service, ep in manifest.routes.items():
        print(f"ROUTE {service} -> {ep.url}")

    return 0 if not manifest.unresolved else 1


if __name__ == "__main__":
    sys.exit(main())

The manifest the orchestrator ultimately gates on is a versioned mapping keyed by logical service, so two concurrent drills produce two distinct, self-describing artifacts:

{
  "session_tag": "drill-2026-07-05-0e1a",
  "routes": {
    "orders-db": { "url": "postgres://10.42.7.19:5432", "session_tag": "drill-2026-07-05-0e1a" },
    "payments-api": { "url": "https://10.42.7.31:8443", "session_tag": "drill-2026-07-05-0e1a" }
  },
  "unresolved": []
}

Bounded-Concurrency Dispatch and Confinement Assertion

Sequential probing wastes wall-clock time on a multi-service manifest, but unbounded fan-out triggers connection storms against the freshly restored instances and skews the very latency numbers the drill is meant to measure. The scaling pattern is a bounded fan-out: an asyncio.Semaphore caps in-flight probes while asyncio.to_thread pushes each blocking socket call off the event loop. Every probe injects the signed X-Drill-Context header and then asserts confinement — the resolved host must match the address that answered — so a misrouted response is caught rather than counted as a pass:

import asyncio
import json
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Dict, List, Tuple


@dataclass(frozen=True)
class ProbeResult:
    service: str
    reachable: bool
    confined: bool
    detail: str


async def probe_endpoint(
    service: str, url: str, session_tag: str, timeout: float = 5.0
) -> ProbeResult:
    """Health-check one sandbox endpoint and assert it is confined."""
    health_url = f"{url}/healthz"
    req = urllib.request.Request(
        health_url,
        headers={"X-Drill-Context": session_tag},
        method="GET",
    )

    def _call() -> Tuple[int, str]:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            served_by = resp.headers.get("X-Served-By", "")
            return resp.status, served_by

    try:
        status, served_by = await asyncio.to_thread(_call)
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        return ProbeResult(service, reachable=False, confined=True, detail=str(exc))

    reachable = status == 200
    # Confinement: the responder must echo the session tag it was routed under.
    confined = served_by == session_tag
    detail = f"status={status} served_by={served_by or 'unknown'}"
    return ProbeResult(service, reachable, confined, detail)


async def dispatch(
    manifest: Dict, max_parallel: int = 8
) -> List[ProbeResult]:
    """Fan out confined probes across the manifest under a concurrency ceiling."""
    sem = asyncio.Semaphore(max_parallel)
    session_tag = manifest["session_tag"]

    async def one(service: str, route: Dict) -> ProbeResult:
        async with sem:
            return await probe_endpoint(service, route["url"], session_tag)

    return await asyncio.gather(
        *(one(svc, route) for svc, route in manifest["routes"].items())
    )

A dispatch driver turns those results into the branch condition the pipeline acts on. A leak — any probe whose response is not confined — is treated as categorically more severe than an unreachable endpoint, and it exits with a distinct code so the orchestrator can abort rather than fall back:

def evaluate(results: List[ProbeResult]) -> int:
    """Map probe results to a POSIX exit code.

    0  every endpoint reachable and confined
    1  one or more unreachable (confined) -> hand off to fallback chain
    3  isolation leak detected -> abort the session immediately
    """
    leaked = [r for r in results if not r.confined]
    unreachable = [r for r in results if not r.reachable]
    for r in leaked:
        print(f"LEAK {r.service} {r.detail}")
    for r in unreachable:
        print(f"UNREACHABLE {r.service} {r.detail}")
    if leaked:
        return 3
    return 1 if unreachable else 0

Integration with DR Drill Orchestration

Routing achieves its value only when embedded in the orchestrated pipeline as a gate, not as a standalone script. The orchestrator invokes routing after provisioning succeeds and before validation begins, and it branches on the routing exit code exactly as it branches on the checksum validation pipeline that certified the artifact upstream. On exit 0 the drill proceeds to run its smoke suite against the confined endpoints; on exit 1 it routes to the fallback chain configuration, which can degrade the validation to an alternate source or an earlier coordinate rather than failing the whole drill; on exit 3 it aborts the session and flags an isolation incident. Because routing must dispatch the synthetic workload against the state the recovery targeted, it coordinates with point-in-time recovery targeting so that a probe reads the exact recovery window under test and never queries stale transaction logs or a misaligned coordinate.

Figure. The routing engine runs as an exit-code gate: the orchestrator invokes it after provisioning, it resolves endpoints and dispatches confined probes, then the exit code deterministically selects run-suite, fallback, or abort.

Modern topologies rarely stop at a single database. Steering synthetic traffic through interconnected service meshes, API gateways, and message brokers requires explicit dependency mapping so that a downstream call from the restored service does not route back to a production dependency. The distributed patterns — local DNS hijacking inside validation containers, sidecar-proxy configuration injection, and gRPC metadata routing — are documented in depth in Smoke Test Routing for Microservice DR Drills, which shows how to preserve referential integrity across polyglot persistence and event-driven paths.

Error Classification and Threshold Management

Not every routing anomaly is an incident. A resolver that returns nothing on the first poll because a resource is still initializing is materially different from a probe observed egressing to production. Mapping raw routing facts to severity tiers — rather than paging on every non-200 — is what keeps the stage sensitive to genuine isolation failures without generating alert fatigue, and it reuses the same taxonomy as the site’s error categorization frameworks.

The tier is assigned after dispatch, not during it: the probe layer emits raw LEAK / UNREACHABLE / UNRESOLVED facts, and the classification layer maps them to a tier using the session budget and historical baselines. Keeping policy out of the deterministic dispatch core means tolerance windows can be retuned without touching the code that moves traffic — and it guarantees that a leak, the one condition that can never be tolerated, always maps to the same terminal action regardless of how the softer thresholds drift.

Telemetry and Compliance Output

Every routing run emits structured telemetry to centralized observability, exported over Prometheus-compatible endpoints so that resolution latency, confinement outcomes, and fallback frequency are trended across drill generations rather than inspected one run at a time.

The manifest and per-probe results are written to write-once, append-only storage and cryptographically signed, so the record of which logical service was routed to which sandbox address, under which session tag, cannot be altered after the fact during a post-incident review. That structure aligns routing evidence with the data-integrity and access-control expectations of frameworks such as NIST SP 800-34 and ISO 22301, where an auditor must be able to prove that a validated recovery never touched production.

Operational Best Practices

1. Assert confinement, never assume it. Treat every probe as suspect until its response echoes the session tag it was routed under; a smoke test that cannot prove where it landed is not evidence of recovery.
2. Scope every override to the session lifetime. Inject connection-string, /etc/hosts, and proxy overrides at process or namespace scope only, and reconcile a purge sweep on teardown so a crashed runner cannot leave a stale route behind.
3. Resolve precedence deterministically. Collapse static, session, and fallback addresses into one winner per logical name highest-specificity-first, so concurrent drills can never resolve into each other’s sandbox.
4. Bound the fan-out. Cap in-flight probes with a semaphore sized to the restored instances’ capacity; unbounded dispatch both starves the host and inflates the latency numbers the drill reports.
5. Fence egress at the boundary. Enforce default-deny egress from the drill namespace or VPC to production subnets so that a routing bug fails closed instead of leaking, and reject any ingress lacking the drill authentication token.
6. Gate on explicit exit codes. Keep the leak / unreachable / clean distinction encoded as distinct POSIX exit codes so the orchestrator’s branch to run-suite, fallback, or abort is unambiguous and testable.

Treated as a first-class stage rather than a bag of /etc/hosts edits, smoke test routing turns theoretical recoverability into an empirically confined measurement. Dynamic endpoint resolution, deterministic precedence, asserted confinement, and Python-driven dispatch together guarantee that every backup artifact can be exercised under production-like conditions without a single synthetic request reaching the systems it was meant to protect.

Frequently Asked Questions

Related

• Sandbox Provisioning Automation — the ephemeral environments whose endpoints this routing layer resolves and confines.
• Point-in-Time Recovery Targeting — the exact coordinate the synthetic workload must be dispatched against.
• Fallback Chain Configuration — the deterministic degradation path a routing exit code 1 hands off to.
• Smoke Test Routing for Microservice DR Drills — header-based steering across service meshes and polyglot persistence.
• Error Categorization Frameworks — the shared severity taxonomy that maps raw routing facts to actions.

This topic is one component of the broader Restore Drill Orchestration & Environment Isolation framework.