# Rebuild Confidence in an Analytics Decision

Workflow ID: AMO-W-000024
Workflow URL: https://amo.ng/workflows/rebuild-confidence-analytics-decision

## Outcome

A decision-integrity package containing a repaired lineage boundary, affected-output and reprocessing scope, forecast assumption and drift findings, sensitivity results, residual uncertainty, and a Use, Use with limits, Recompute, or Hold decision.

## Before you begin

- Exact analytical conclusion, forecast or decision, decision owner, deadline, and consequence of error
- Source systems, transformations, schemas, lineage metadata, code or queries, data-quality checks, and affected outputs
- Forecast assumptions, model versions, training and backtest periods, scenarios, external drivers, and observed performance
- Analysis data, cohort and metric definitions, missing-data treatment, model choices, alternative specifications, and decision thresholds
- Incident or change timeline, prior decisions, reprocessing options, constraints, and accountable data, model, analytics, and business owners

## Step 1 — Reconstruct the data-lineage break

**Prompt**

Data Lineage Break Investigation

**Instructions**

Trace the data product from authoritative sources through transformations and checks to affected outputs. Identify the first supported divergence, blast radius, smallest safe repair, and reprocessing boundary.

**Input for this step**

Provide the decision and affected outputs, source and target data, lineage metadata, schemas, queries or code, orchestration logs, checks, changes, incident timeline, and known-good comparisons.

**Carry forward**

Carry the lineage graph, first divergence, affected outputs and decisions, evidence gaps, repair and reprocessing options, and validation requirements into forecast review.

**Review note**

Data and service owners approve repair and reprocessing scope; the decision owner confirms which downstream decisions require suspension or reassessment.

**Prompt ID**

AMO-P-000318

**Prompt URL**

https://amo.ng/prompts/data-lineage-break-investigation

**Prompt content**

Investigate a suspected break in data lineage from authoritative source through transformations to a consequential dataset, metric, report, model, export, or decision. Identify the first evidenced divergence and the smallest safe recovery boundary.

Provide:
- Affected data product, symptom, discovery time, users, decisions, and known exposure: [Affected data product and incident]
- Intended sources, contracts, schemas, grain, keys, transformations, quality rules, lineage graph, and expected outputs: [Expected lineage and contracts]
- Relevant repository files, queries, models, jobs, orchestration, configuration, versions, deployments, and migrations: [Code configuration and deployment evidence]
- Authoritative source samples, intermediate outputs, affected outputs, reconciliation records, and timestamps: [Source transformation and output samples]
- Run history, scheduler/orchestrator events, retries, failures, row counts, checks, alerts, and backfills: [Operational logs and timing evidence]
- Data retention, reprocessing limits, downstream dependencies, data owner, engineering owner, metric/report owner, and decision owner: [Recovery constraints and accountable owners]

Inspect before proposing changes. Do not claim a query, job, repository, or warehouse was executed unless results are supplied. Do not infer completeness from a successful job status. Separate code/config facts, data observations, operational observations, and causal hypotheses. Preserve evidence before recommending destructive reprocessing.

Investigation:

1. Define expected lineage and invariants.
   State source of truth, grain, keys, time semantics, filters, joins, aggregation, late-arrival behavior, deduplication, schema, and downstream contract. Identify undocumented assumptions.

2. Reconstruct the incident timeline.
   Align source changes, deployments, schema events, job runs, retries, backfills, alerts, and first bad outputs. Note clock, retention, and logging gaps.

3. Locate the first divergence.
   Compare authoritative source to each available stage using stable keys and bounded samples or counts. Classify divergence as missing, duplicated, stale, misjoined, filtered, remapped, truncated, shifted in time, misaggregated, unauthorized, or not assessable.

4. Develop and test hypotheses.
   Rank code, schema, configuration, dependency, orchestration, source-quality, permission, and backfill hypotheses. For each, cite supporting and contradicting evidence and define the smallest read-only reproduction or reconciliation needed.

5. Bound impact.
   Identify affected partitions, entities, periods, metrics, dashboards, models, exports, customers, and decisions. Separate confirmed impact from potential dependency reach. Identify outputs that must be labelled, paused, recalled, or revalidated.

6. Design the minimal repair.
   Specify exact code/config/data correction, preserved architecture, migration or backfill scope, idempotency, validation, review, rollback, and changed files if repository evidence is supplied. Avoid unrelated refactoring.

7. Plan reprocessing and reconciliation.
   Define source snapshot, partitions, ordering, duplicate prevention, side effects, cost/capacity, checksums or totals, downstream refresh, and stop conditions. Do not authorize deletion or production reprocessing.

8. Define prevention.
   Add targeted contract, lineage, freshness, volume, reconciliation, canary, and alert controls tied to the failure—not a generic monitoring list.

For each proposed repair, define acceptance evidence that names the expected observation at the authoritative source and every downstream checkpoint, the actual observation when tested, and reconciliation of record counts, keys, timestamps, and affected metrics. Keep an untested repair marked Proposed.

Required deliverable:

# Data Lineage Break Investigation

## Expected Lineage and Invariants
| Stage | Source/input | Grain/key/time | Transformation | Contract/check | Owner |
|---|---|---|---|---|---|

## Incident Timeline
| Time/window | Change/run/event | Expected | Observed | Evidence | Relevance |
|---|---|---|---|---|---|

## First Divergence and Hypotheses
| Hypothesis | Supporting evidence | Contradicting evidence | Confidence | Read-only test needed |
|---|---|---|---|---|

## Impact Register
| Partition/output/decision | Confirmed/potential | Exposure window | Consequence | Containment | Owner |
|---|---|---|---|---|---|

## Minimal Repair and Reprocessing Plan
| Step | Change/scope | Idempotency/rollback | Validation | Authorization | Stop condition |
|---|---|---|---|---|---|

## Prevention Controls
| Control | Failure detected | Evidence retained | Threshold | Owner |
|---|---|---|---|---|

Completion requires a trace-supported first divergence, bounded downstream impact, a minimal reversible repair, and reconciliation evidence before affected outputs are restored for decision use.


## Step 2 — Challenge forecast assumptions and model drift

**Prompt**

Forecast Assumption and Model Drift Challenge

**Instructions**

Test whether assumptions, structural relationships, input distributions, backtest evidence, and scenario boundaries remain credible after the lineage finding or other material changes.

**Input for this step**

Provide repaired or bounded data, forecast purpose, assumptions, model and feature versions, training and backtest evidence, historical errors, external drivers, scenarios, and decision thresholds.

**Carry forward**

Carry the assumption register, drift findings, backtest limitations, scenario bounds, unstable relationships, and required recalibration into conclusion sensitivity.

**Review note**

The model owner, domain reviewer, and decision owner approve assumption interpretation and any continued use of a degraded or recalibrated forecast.

**Prompt ID**

AMO-P-000317

**Prompt URL**

https://amo.ng/prompts/forecast-assumption-model-drift-challenge

**Prompt content**

Challenge whether a forecast is fit for a specific operating, financial, capacity, or investment decision. Test its assumptions, data, backtest behavior, structural stability, and sensitivity without presenting an unexecuted model run as fact.

Provide:
- Decision, forecast target, horizon, granularity, update cadence, required accuracy, and consequence of error: [Forecast decision and horizon]
- Model/formula, feature or driver logic, transformations, assumptions, overrides, constraints, and version: [Model logic and assumption register]
- Historical observations, training/calibration window, holdouts, backtests, errors, residuals, revisions, and benchmark forecasts: [Historical data and backtest evidence]
- Recent driver changes, regime shifts, policy/market/process changes, anomalies, missingness, and leading indicators: [Current driver and structural-change evidence]
- Base, upside, downside, stress cases, decision thresholds, risk limits, and alternative actions: [Scenarios thresholds and constraints]
- Forecast owner, data owner, finance or planning reviewer, decision owner, and analysis limitations: [Accountable owners and review limits]

Do not invent forecast values, confidence intervals, driver effects, or backtest statistics. Distinguish model output, planner override, observed fact, assumption, and scenario. Do not infer causal drivers from correlation alone. When raw data or executable logic is absent, assess the evidence and specify required tests rather than claiming to have rerun the forecast.

Challenge:

1. Define the decision contract.
   State which decision the forecast informs, the timing, relevant error direction, tolerance, and alternative if evidence is insufficient.

2. Build an assumption register.
   Record every decision-critical assumption, source, rationale, range, dependency, owner, last validation, and failure consequence. Include missing-data treatment, overrides, external drivers, capacity limits, and behavioral responses.

3. Review data and model lineage.
   Trace target and drivers through sources, transformations, update timing, revisions, exclusions, and version changes. Flag leakage, survivorship, regime mixing, inconsistent definitions, and unavailable lineage.

4. Assess backtest evidence.
   Compare forecast errors across time, horizon, segment, regime, and relevant benchmark using supplied results. Check bias, variance, tail misses, turning points, coverage of intervals, and revision behavior. Avoid extrapolating from one period.

5. Identify structural drift.
   Map changes that weaken learned relationships or operational assumptions. Separate observed drift signals from hypotheses and specify evidence that would confirm them.

6. Run or design sensitivity analysis.
   Where supplied calculations permit, show how decision outcomes change across plausible assumptions. Otherwise provide exact scenario runs needed. Identify tipping points and assumptions that do not affect the decision.

7. Compare decision robustness.
   Determine whether Base, Upside, Downside, Stress, delayed-decision, staged-commitment, or no-action cases cross material thresholds. Keep probability claims evidence-bound.

8. Make a fitness decision.
   Choose Fit for stated decision, Fit with range/conditions, Use only as scenario input, Recalibrate/rebuild, or Not decision-fit. Assign owner actions and refresh triggers.

Use ChatGPT to analyze supplied forecast artifacts and results; do not claim that backtests, refits, or sensitivity runs were executed unless their outputs are provided. Protect sensitive financial or operational data and stop where access or methodology is unauthorized. The model owner validates specification evidence, the finance or decision owner accepts decision risk, and the release owner authorizes operational use. Require acceptance evidence for each challenge: expected observation, actual observation, sensitivity boundary, reconciliation, and rollback or reforecast trigger.

Required deliverable:

# Forecast Assumption and Model Drift Challenge

## Decision Contract
- Forecast and horizon:
- Decision/threshold:
- Cost of over-forecast and under-forecast:
- Evidence limitations:

## Assumption Register
| Assumption | Value/range | Evidence/source | Dependency | Last validated | Sensitivity | Owner |
|---|---|---|---|---|---|---|

## Backtest and Drift Findings
| Period/slice | Error evidence | Bias/tail issue | Structural change | Confidence | Decision impact |
|---|---|---|---|---|---|

## Scenario and Tipping-Point Table
| Scenario/assumption change | Forecast effect supplied or test needed | Decision threshold crossed? | Evidence | Action |
|---|---|---|---|---|

## Forecast Fitness Decision
- Decision:
- Valid use and range:
- Prohibited overclaim:
- Required recalibration/data:
- Monitoring and refresh trigger:
- Accountable decision owner:

Completion requires a traceable assumption register, honest use of backtest evidence, explicit structural-drift risks, and a decision that remains within the tested horizon and data regime.


## Step 3 — Test analytical conclusion sensitivity

**Prompt**

Analytical Conclusion Sensitivity Review

**Instructions**

Determine whether the consequential conclusion survives plausible changes to data, cohort, definitions, missing-information treatment, assumptions, and model choices. Separate robust direction from fragile magnitude or threshold crossing.

**Input for this step**

Provide the conclusion and decision threshold, analysis data and code or outputs, metric and cohort definitions, missing-data treatment, assumptions, model choices, prior findings, and authorized sensitivity scope.

**Carry forward**

Produce the final lineage, forecast, and sensitivity package with robust and fragile claims, decision impact, repair or recomputation needs, residual uncertainty, and Use, Use with limits, Recompute, or Hold disposition.

**Review note**

The analytics owner and domain reviewer approve analytical interpretation; the accountable decision owner decides whether evidence is sufficient for use.

**Prompt ID**

AMO-P-000319

**Prompt URL**

https://amo.ng/prompts/analytical-conclusion-sensitivity-review

**Prompt content**

Review how robust a consequential analytical conclusion is to plausible changes in data, definitions, cohort, time window, missingness, assumptions, model specification, and decision threshold.

Provide:
- Exact conclusion, decision it supports, affected population, time horizon, and consequence of error: [Decision and analytical conclusion]
- Source data or extracts, calculations, queries, outputs, uncertainty, validation, and provenance evidence: [Data calculations and evidence]
- Metric definitions, inclusion/exclusion, transformations, causal or forecasting assumptions, model choices, overrides, and missing-data treatment: [Definitions assumptions and model choices]
- Plausible alternative cohorts, periods, definitions, models, thresholds, corrections, and scenarios already tested or proposed: [Alternative specifications and scenarios]
- Materiality threshold, risk tolerance, data owner, analyst, domain reviewer, and decision owner: [Risk thresholds and accountable owners]

Do not invent rerun results. If executable data or results are not supplied, specify exact sensitivity analyses and how to interpret them. Distinguish observed evidence from inference. Separate arithmetic verification, specification sensitivity, sampling uncertainty, causal uncertainty, external validity, and data-quality exposure. Do not call a conclusion robust merely because its sign is unchanged if decision magnitude or affected population changes materially.

Review:

1. State the decision claim.
   Rewrite the conclusion as a bounded claim with population, period, measure, comparison, magnitude, uncertainty, and decision threshold. Identify language broader than evidence.

2. Reconstruct the analytical path.
   Trace source, filters, joins, transformations, definitions, calculation/model, aggregation, uncertainty, and final claim. Mark steps not reproducible from supplied evidence.

3. Identify decision-critical choices.
   List choices that could change sign, magnitude, uncertainty, ranking, threshold crossing, or affected group. Distinguish defensible alternatives from arbitrary perturbations.

4. Verify supplied sensitivity evidence.
   Compare base and alternative specifications across outcome, uncertainty, sample/population, fit or diagnostics, and decision implication. Do not select only alternatives favorable to the original conclusion.

5. Design missing sensitivity tests.
   Prioritize data corrections, cohort/time-window variants, metric definitions, missing-data bounds, outlier influence, placebo/negative controls, alternative models, priors, thresholds, and external benchmarks appropriate to the analysis.

6. Map the robustness envelope.
   Identify ranges in which the decision is Stable, Stable with narrower claim, Threshold-sensitive, Direction-sensitive, Population-sensitive, or Not assessable. State tipping points.

7. Determine action.
   Choose Use for decision, Use with stated limits, Stage or hedge decision, Collect evidence, Reanalyze, or Do not use. Assign owner verification and avoid implying approval.

Define acceptance evidence for every sensitivity conclusion: the expected observation under each perturbation, actual observation when a run is supplied, decision-boundary crossing, and reconciliation to the original claim. Unexecuted analyses remain proposed and cannot establish robustness.

Required deliverable:

# Analytical Conclusion Sensitivity Review

## Bounded Decision Claim
- Original conclusion:
- Evidence-supported wording:
- Population/period:
- Materiality threshold:
- Consequence of error:

## Analytical Path and Reproducibility
| Step | Input/choice | Evidence | Reproducible? | Decision exposure |
|---|---|---|---|---|

## Decision-Critical Choice Register
| Choice/assumption | Base | Plausible alternative/range | Rationale | Expected decision effect |
|---|---|---|---|---|

## Sensitivity Evidence
| Specification/scenario | Result supplied or test needed | Uncertainty | Threshold/sign effect | Interpretation |
|---|---|---|---|---|

## Robustness Envelope
| Condition/range | Claim status | Decision status | Population affected | Confidence |
|---|---|---|---|---|

## Decision Recommendation
- Disposition:
- Claim allowed:
- Claim prohibited:
- Tests/evidence required:
- Owner verification:
- Revisit trigger:

Completion requires a reproducible analytical path or explicit gaps, a defensible set of alternatives, and a decision recommendation that changes when material tipping points are crossed.


## Completion criteria

The workflow is complete when:

- The first supported lineage break, affected data products, decision exposure, repair boundary, and reprocessing plan are explicit.
- Forecast assumptions and structural stability have evidence-linked supported, contradicted, or unresolved dispositions.
- Material conclusions have sensitivity results across plausible data, cohort, definition, missing-data, assumption, and model choices.
- Residual uncertainty and unrun checks remain explicit.
- The decision owner has a Use, Use with limits, Recompute, or Hold recommendation with observable repair, validation, and re-review conditions.