Data Lineage Break Investigation

Investigate a suspected break in data lineage from authoritative source through transformations to a consequential dataset, metric, report, model, export, or decision. Identify the first evidenced divergence and the smallest safe recovery boundary.

Provide:
- Affected data product, symptom, discovery time, users, decisions, and known exposure: [Affected data product and incident]
- Intended sources, contracts, schemas, grain, keys, transformations, quality rules, lineage graph, and expected outputs: [Expected lineage and contracts]
- Relevant repository files, queries, models, jobs, orchestration, configuration, versions, deployments, and migrations: [Code configuration and deployment evidence]
- Authoritative source samples, intermediate outputs, affected outputs, reconciliation records, and timestamps: [Source transformation and output samples]
- Run history, scheduler/orchestrator events, retries, failures, row counts, checks, alerts, and backfills: [Operational logs and timing evidence]
- Data retention, reprocessing limits, downstream dependencies, data owner, engineering owner, metric/report owner, and decision owner: [Recovery constraints and accountable owners]

Inspect before proposing changes. Do not claim a query, job, repository, or warehouse was executed unless results are supplied. Do not infer completeness from a successful job status. Separate code/config facts, data observations, operational observations, and causal hypotheses. Preserve evidence before recommending destructive reprocessing.

Investigation:

1. Define expected lineage and invariants.
   State source of truth, grain, keys, time semantics, filters, joins, aggregation, late-arrival behavior, deduplication, schema, and downstream contract. Identify undocumented assumptions.

2. Reconstruct the incident timeline.
   Align source changes, deployments, schema events, job runs, retries, backfills, alerts, and first bad outputs. Note clock, retention, and logging gaps.

3. Locate the first divergence.
   Compare authoritative source to each available stage using stable keys and bounded samples or counts. Classify divergence as missing, duplicated, stale, misjoined, filtered, remapped, truncated, shifted in time, misaggregated, unauthorized, or not assessable.

4. Develop and test hypotheses.
   Rank code, schema, configuration, dependency, orchestration, source-quality, permission, and backfill hypotheses. For each, cite supporting and contradicting evidence and define the smallest read-only reproduction or reconciliation needed.

5. Bound impact.
   Identify affected partitions, entities, periods, metrics, dashboards, models, exports, customers, and decisions. Separate confirmed impact from potential dependency reach. Identify outputs that must be labelled, paused, recalled, or revalidated.

6. Design the minimal repair.
   Specify exact code/config/data correction, preserved architecture, migration or backfill scope, idempotency, validation, review, rollback, and changed files if repository evidence is supplied. Avoid unrelated refactoring.

7. Plan reprocessing and reconciliation.
   Define source snapshot, partitions, ordering, duplicate prevention, side effects, cost/capacity, checksums or totals, downstream refresh, and stop conditions. Do not authorize deletion or production reprocessing.

8. Define prevention.
   Add targeted contract, lineage, freshness, volume, reconciliation, canary, and alert controls tied to the failure—not a generic monitoring list.

For each proposed repair, define acceptance evidence that names the expected observation at the authoritative source and every downstream checkpoint, the actual observation when tested, and reconciliation of record counts, keys, timestamps, and affected metrics. Keep an untested repair marked Proposed.

Required deliverable:

# Data Lineage Break Investigation

## Expected Lineage and Invariants
| Stage | Source/input | Grain/key/time | Transformation | Contract/check | Owner |
|---|---|---|---|---|---|

## Incident Timeline
| Time/window | Change/run/event | Expected | Observed | Evidence | Relevance |
|---|---|---|---|---|---|

## First Divergence and Hypotheses
| Hypothesis | Supporting evidence | Contradicting evidence | Confidence | Read-only test needed |
|---|---|---|---|---|

## Impact Register
| Partition/output/decision | Confirmed/potential | Exposure window | Consequence | Containment | Owner |
|---|---|---|---|---|---|

## Minimal Repair and Reprocessing Plan
| Step | Change/scope | Idempotency/rollback | Validation | Authorization | Stop condition |
|---|---|---|---|---|---|

## Prevention Controls
| Control | Failure detected | Evidence retained | Threshold | Owner |
|---|---|---|---|---|

Completion requires a trace-supported first divergence, bounded downstream impact, a minimal reversible repair, and reconciliation evidence before affected outputs are restored for decision use.