Designing pipeline health around the consumer

This is a reference architecture, not a client engagement. It demonstrates how I would reason about observability for a batch data product whose consumers care about trustworthy publication times.

The design question

A scheduler knows whether tasks ran. A consumer needs to know whether a dataset is safe to use.

Those views overlap, but they are not identical. A successful job can publish incomplete data; a failed optional enrichment can leave the primary dataset usable. The health model therefore begins at the publication boundary.

Release-oriented architecture

Source manifests ─┐
Contract checks ──┼──→ Release evaluator ──→ Dataset status API
Pipeline events ──┤            │                      │
Reconciliation ───┘            └──→ Alert routing     └──→ Catalog / BI

Each dataset release receives an explicit state:

• pending — expected inputs are not complete;
• validating — structural and semantic checks are running;
• published — the consumer contract has been satisfied;
• degraded — usable with a documented limitation;
• quarantined — withheld because the contract failed;
• superseded — replaced by a corrected release.

A latency view

The chart below is an illustrative dataset rendered as an MDX component. It shows how one slow transformation stage can dominate end-to-end freshness even when every stage succeeds.

Pipeline latency by stage

Stage	Minutes
Ingest	7
Validate	4
Transform	13
Publish	3

The next question is not simply “how do we make Spark faster?” It is “which part of the freshness promise is expensive, and what trade-off would a consumer accept?”

Failure should lead to a repair action

Every alert should carry enough context to choose a response:

State	Default action
Missing input	Hold publication and notify the source owner
Contract violation	Quarantine affected records or the release
Reconciliation drift	Compare source and target partitions
Late but valid	Publish with freshness status visible
Corrected backfill	Supersede the earlier release with lineage

This keeps observability connected to operation. A red dashboard without an ownership and repair model is only decoration.