Reusable AI capability
Reconstruct Broken Data Lineage
Apply a repeatable incident method to locate the first data-lineage divergence, bound affected outputs and decisions, and gate repair and reprocessing.
This Skill packages a reusable way to use the linked Prompt or Workflow; Amo.ng does not run it for you.
# Reconstruct Broken Data Lineage Skill ID: AMO-S-000025 Skill URL: https://amo.ng/skills/reconstruct-broken-data-lineage Purpose: Give data, analytics, service, and decision owners a reusable capability for investigating lineage failures across pipelines, metrics, models, dashboards, and downstream decisions. Required inputs: - Exact affected data product, output, metric, model, report, or decision - Authoritative sources, schemas, transformations, queries or code, lineage metadata, orchestration and quality checks - Known-good and affected samples, change and incident timeline, runtime logs, and owner records - Repair, rollback, reprocessing, retention, audit, and decision-suspension constraints How to use: When to use: - A downstream value or decision no longer reconciles to its authoritative source. - A schema, transformation, mapping, orchestration, or quality-control change may have broken lineage. When not to use: - General data catalog design with no observed divergence. - Repairing or reprocessing production data without owner authorization. Reusable method: 1. Freeze the affected outputs, decision window, expected lineage, authoritative source, and known-good comparison. 2. Trace source, extraction, transformation, mapping, storage, aggregation, semantic, and consumption edges with versions and evidence. 3. Compare values and contracts at each boundary to locate the first supported divergence. 4. Bound affected periods, entities, outputs, models, reports, and decisions; distinguish confirmed from possible exposure. 5. Compare smallest repair, rollback, targeted reprocessing, full reprocessing, and continue-investigation paths. 6. Define preservation, validation, reconciliation, downstream notification, and decision-reassessment gates. Expected output: A lineage graph, evidence ledger, first-divergence finding, impact and decision register, repair and reprocessing options, validation plan, unresolved gaps, and owner decisions. Boundaries: Do not invent data, lineage, code, queries, runs, or repair results. Data and service owners authorize changes and reprocessing; analytics and model owners validate downstream artifacts; the accountable decision owner decides reassessment. Source: AMO-P-000318. Applicable Workflow: Rebuild Confidence in an Analytics Decision. Powered by Prompt: Data Lineage Break Investigation Source ID: AMO-P-000318 https://amo.ng/prompts/data-lineage-break-investigation Completion criteria: Complete when the first supported divergence and all affected downstream edges are evidence-linked; impact is bounded; a repair and reprocessing disposition has owner, preservation, rollback, and validation gates; and affected decisions are marked usable, limited, suspended, or pending reassessment. Use this Amo.ng Skill with your preferred AI tool. Supply the required inputs and follow the usage instructions. # Reconstruct Broken Data Lineage Skill ID: AMO-S-000025 Skill URL: https://amo.ng/skills/reconstruct-broken-data-lineage Purpose: Give data, analytics, service, and decision owners a reusable capability for investigating lineage failures across pipelines, metrics, models, dashboards, and downstream decisions. Required inputs: - Exact affected data product, output, metric, model, report, or decision - Authoritative sources, schemas, transformations, queries or code, lineage metadata, orchestration and quality checks - Known-good and affected samples, change and incident timeline, runtime logs, and owner records - Repair, rollback, reprocessing, retention, audit, and decision-suspension constraints How to use: When to use: - A downstream value or decision no longer reconciles to its authoritative source. - A schema, transformation, mapping, orchestration, or quality-control change may have broken lineage. When not to use: - General data catalog design with no observed divergence. - Repairing or reprocessing production data without owner authorization. Reusable method: 1. Freeze the affected outputs, decision window, expected lineage, authoritative source, and known-good comparison. 2. Trace source, extraction, transformation, mapping, storage, aggregation, semantic, and consumption edges with versions and evidence. 3. Compare values and contracts at each boundary to locate the first supported divergence. 4. Bound affected periods, entities, outputs, models, reports, and decisions; distinguish confirmed from possible exposure. 5. Compare smallest repair, rollback, targeted reprocessing, full reprocessing, and continue-investigation paths. 6. Define preservation, validation, reconciliation, downstream notification, and decision-reassessment gates. Expected output: A lineage graph, evidence ledger, first-divergence finding, impact and decision register, repair and reprocessing options, validation plan, unresolved gaps, and owner decisions. Boundaries: Do not invent data, lineage, code, queries, runs, or repair results. Data and service owners authorize changes and reprocessing; analytics and model owners validate downstream artifacts; the accountable decision owner decides reassessment. Source: AMO-P-000318. Applicable Workflow: Rebuild Confidence in an Analytics Decision. Powered by Prompt: Data Lineage Break Investigation Source ID: AMO-P-000318 https://amo.ng/prompts/data-lineage-break-investigation Completion criteria: Complete when the first supported divergence and all affected downstream edges are evidence-linked; impact is bounded; a repair and reprocessing disposition has owner, preservation, rollback, and validation gates; and affected decisions are marked usable, limited, suspended, or pending reassessment.Copy skill copies the Skill details. Use with AI adds a short instruction for your preferred AI tool; neither action runs the Skill.
Purpose
Give data, analytics, service, and decision owners a reusable capability for investigating lineage failures across pipelines, metrics, models, dashboards, and downstream decisions.
Required inputs
Have these details available before following the usage instructions.
- Exact affected data product, output, metric, model, report, or decision
- Authoritative sources, schemas, transformations, queries or code, lineage metadata, orchestration and quality checks
- Known-good and affected samples, change and incident timeline, runtime logs, and owner records
- Repair, rollback, reprocessing, retention, audit, and decision-suspension constraints
How to use this Skill
When to use:
- A downstream value or decision no longer reconciles to its authoritative source.
- A schema, transformation, mapping, orchestration, or quality-control change may have broken lineage.
When not to use:
- General data catalog design with no observed divergence.
- Repairing or reprocessing production data without owner authorization.
Reusable method:
1. Freeze the affected outputs, decision window, expected lineage, authoritative source, and known-good comparison.
2. Trace source, extraction, transformation, mapping, storage, aggregation, semantic, and consumption edges with versions and evidence.
3. Compare values and contracts at each boundary to locate the first supported divergence.
4. Bound affected periods, entities, outputs, models, reports, and decisions; distinguish confirmed from possible exposure.
5. Compare smallest repair, rollback, targeted reprocessing, full reprocessing, and continue-investigation paths.
6. Define preservation, validation, reconciliation, downstream notification, and decision-reassessment gates.
Expected output:
A lineage graph, evidence ledger, first-divergence finding, impact and decision register, repair and reprocessing options, validation plan, unresolved gaps, and owner decisions.
Boundaries:
Do not invent data, lineage, code, queries, runs, or repair results. Data and service owners authorize changes and reprocessing; analytics and model owners validate downstream artifacts; the accountable decision owner decides reassessment. Source: AMO-P-000318. Applicable Workflow: Rebuild Confidence in an Analytics Decision.
Powered by an Amo.ng Prompt
Data Lineage Break Investigation
Open the linked prompt to use the instructions that power this Skill.
Open prompt
Investigate a suspected break in data lineage from authoritative source through transformations to a consequential dataset, metric, report, model, export, or decision. Identify the first evidenced divergence and the smallest safe recovery boundary.
Provide:
- Affected data product, symptom, discovery time, users, decisions, and known exposure: [Affected data product and incident]
- Intended sources, contracts, schemas, grain, keys, transformations, quality rules, lineage graph, and expected outputs: [Expected lineage and contracts]
- Relevant repository files, queries, models, jobs, orchestration, configuration, versions, deployments, and migrations: [Code configuration and deployment evidence]
- Authoritative source samples, intermediate outputs, affected outputs, reconciliation records, and timestamps: [Source transformation and output samples]
- Run history, scheduler/orchestrator events, retries, failures, row counts, checks, alerts, and backfills: [Operational logs and timing evidence]
- Data retention, reprocessing limits, downstream dependencies, data owner, engineering owner, metric/report owner, and decision owner: [Recovery constraints and accountable owners]
Inspect before proposing changes. Do not claim a query, job, repository, or warehouse was executed unless results are supplied. Do not infer completeness from a successful job status. Separate code/config facts, data observations, operational observations, and causal hypotheses. Preserve evidence before recommending destructive reprocessing.
Investigation:
1. Define expected lineage and invariants.
State source of truth, grain, keys, time semantics, filters, joins, aggregation, late-arrival behavior, deduplication, schema, and downstream contract. Identify undocumented assumptions.
2. Reconstruct the incident timeline.
Align source changes, deployments, schema events, job runs, retries, backfills, alerts, and first bad outputs. Note clock, retention, and logging gaps.
3. Locate the first divergence.
Compare authoritative source to each available stage using stable keys and bounded samples or counts. Classify divergence as missing, duplicated, stale, misjoined, filtered, remapped, truncated, shifted in time, misaggregated, unauthorized, or not assessable.
4. Develop and test hypotheses.
Rank code, schema, configuration, dependency, orchestration, source-quality, permission, and backfill hypotheses. For each, cite supporting and contradicting evidence and define the smallest read-only reproduction or reconciliation needed.
5. Bound impact.
Identify affected partitions, entities, periods, metrics, dashboards, models, exports, customers, and decisions. Separate confirmed impact from potential dependency reach. Identify outputs that must be labelled, paused, recalled, or revalidated.
6. Design the minimal repair.
Specify exact code/config/data correction, preserved architecture, migration or backfill scope, idempotency, validation, review, rollback, and changed files if repository evidence is supplied. Avoid unrelated refactoring.
7. Plan reprocessing and reconciliation.
Define source snapshot, partitions, ordering, duplicate prevention, side effects, cost/capacity, checksums or totals, downstream refresh, and stop conditions. Do not authorize deletion or production reprocessing.
8. Define prevention.
Add targeted contract, lineage, freshness, volume, reconciliation, canary, and alert controls tied to the failure—not a generic monitoring list.
For each proposed repair, define acceptance evidence that names the expected observation at the authoritative source and every downstream checkpoint, the actual observation when tested, and reconciliation of record counts, keys, timestamps, and affected metrics. Keep an untested repair marked Proposed.
Required deliverable:
# Data Lineage Break Investigation
## Expected Lineage and Invariants
| Stage | Source/input | Grain/key/time | Transformation | Contract/check | Owner |
|---|---|---|---|---|---|
## Incident Timeline
| Time/window | Change/run/event | Expected | Observed | Evidence | Relevance |
|---|---|---|---|---|---|
## First Divergence and Hypotheses
| Hypothesis | Supporting evidence | Contradicting evidence | Confidence | Read-only test needed |
|---|---|---|---|---|
## Impact Register
| Partition/output/decision | Confirmed/potential | Exposure window | Consequence | Containment | Owner |
|---|---|---|---|---|---|
## Minimal Repair and Reprocessing Plan
| Step | Change/scope | Idempotency/rollback | Validation | Authorization | Stop condition |
|---|---|---|---|---|---|
## Prevention Controls
| Control | Failure detected | Evidence retained | Threshold | Owner |
|---|---|---|---|---|
Completion requires a trace-supported first divergence, bounded downstream impact, a minimal reversible repair, and reconciliation evidence before affected outputs are restored for decision use.
Completion criteria
Complete when the first supported divergence and all affected downstream edges are evidence-linked; impact is bounded; a repair and reprocessing disposition has owner, preservation, rollback, and validation gates; and affected decisions are marked usable, limited, suspended, or pending reassessment.
Explore related Workflows
Browse WorkflowsRebuild Confidence in an Analytics Decision
Reconstruct broken data lineage, challenge forecast assumptions and drift, and test conclusion sensitivity before relying on a consequential analytical decision.
Related Prompts
Browse PromptsAnalytical Conclusion Sensitivity Review
Test whether a consequential analytical conclusion survives plausible changes to data, cohort, definitions, assumptions, model choices, and missing-information treatment.
Review how robust a consequential analytical conclusion is to plausible changes in data, definitions, cohort, time window, missingness, assumptions, model specification, and decision threshold. Provide: - Exact conclusion, decision it supports, affected population, time horizon, and consequence of error: [Decision and analytical conclusion] - Source data or extracts, calculations, queries, outputs, uncertainty, validation, and provenance evidence: [Data calculations and evidence] - Metric definitions, inclusion/exclusion, transformations, causal or forecasting assumptions, model choices, overrides, and missing-data treatment: [Definitions assumptions and model choices] - Plausible alternative cohorts, periods, definitions, models, thresholds, corrections, and scenarios already tested or proposed: [Alternative specifications and scenarios] - Materiality threshold, risk tolerance, data owner, analyst, domain reviewer, and decision owner: [Risk thresholds and accountable owners] Do not invent rerun results. If executable data or results are not supplied, specify exact sensitivity analyses and how to interpret them. Distinguish observed evidence from inference. Separate arithmetic verification, specification sensitivity, sampling uncertainty, causal uncertainty, external validity, and data-quality exposure. Do not call a conclusion robust merely because its sign is unchanged if decision magnitude or affected population changes materially. Review: 1. State the decision claim. Rewrite the conclusion as a bounded claim with population, period, measure, comparison, magnitude, uncertainty, and decision threshold. Identify language broader than evidence. 2. Reconstruct the analytical path. Trace source, filters, joins, transformations, definitions, calculation/model, aggregation, uncertainty, and final claim. Mark steps not reproducible from supplied evidence. 3. Identify decision-critical choices. List choices that could change sign, magnitude, uncertainty, ranking, threshold crossing, or affected group. Distinguish defensible alternatives from arbitrary perturbations. 4. Verify supplied sensitivity evidence. Compare base and alternative specifications across outcome, uncertainty, sample/population, fit or diagnostics, and decision implication. Do not select only alternatives favorable to the original conclusion. 5. Design missing sensitivity tests. Prioritize data corrections, cohort/time-window variants, metric definitions, missing-data bounds, outlier influence, placebo/negative controls, alternative models, priors, thresholds, and external benchmarks appropriate to the analysis. 6. Map the robustness envelope. Identify ranges in which the decision is Stable, Stable with narrower claim, Threshold-sensitive, Direction-sensitive, Population-sensitive, or Not assessable. State tipping points. 7. Determine action. Choose Use for decision, Use with stated limits, Stage or hedge decision, Collect evidence, Reanalyze, or Do not use. Assign owner verification and avoid implying approval. Define acceptance evidence for every sensitivity conclusion: the expected observation under each perturbation, actual observation when a run is supplied, decision-boundary crossing, and reconciliation to the original claim. Unexecuted analyses remain proposed and cannot establish robustness. Required deliverable: # Analytical Conclusion Sensitivity Review ## Bounded Decision Claim - Original conclusion: - Evidence-supported wording: - Population/period: - Materiality threshold: - Consequence of error: ## Analytical Path and Reproducibility | Step | Input/choice | Evidence | Reproducible? | Decision exposure | |---|---|---|---|---| ## Decision-Critical Choice Register | Choice/assumption | Base | Plausible alternative/range | Rationale | Expected decision effect | |---|---|---|---|---| ## Sensitivity Evidence | Specification/scenario | Result supplied or test needed | Uncertainty | Threshold/sign effect | Interpretation | |---|---|---|---|---| ## Robustness Envelope | Condition/range | Claim status | Decision status | Population affected | Confidence | |---|---|---|---|---| ## Decision Recommendation - Disposition: - Claim allowed: - Claim prohibited: - Tests/evidence required: - Owner verification: - Revisit trigger: Completion requires a reproducible analytical path or explicit gaps, a defensible set of alternatives, and a decision recommendation that changes when material tipping points are crossed.
View prompt
4 views
Forecast Assumption and Model Drift Challenge
Challenge forecast assumptions, structural stability, backtest evidence, scenario sensitivity, and decision thresholds before relying on projected outcomes.
Challenge whether a forecast is fit for a specific operating, financial, capacity, or investment decision. Test its assumptions, data, backtest behavior, structural stability, and sensitivity without presenting an unexecuted model run as fact. Provide: - Decision, forecast target, horizon, granularity, update cadence, required accuracy, and consequence of error: [Forecast decision and horizon] - Model/formula, feature or driver logic, transformations, assumptions, overrides, constraints, and version: [Model logic and assumption register] - Historical observations, training/calibration window, holdouts, backtests, errors, residuals, revisions, and benchmark forecasts: [Historical data and backtest evidence] - Recent driver changes, regime shifts, policy/market/process changes, anomalies, missingness, and leading indicators: [Current driver and structural-change evidence] - Base, upside, downside, stress cases, decision thresholds, risk limits, and alternative actions: [Scenarios thresholds and constraints] - Forecast owner, data owner, finance or planning reviewer, decision owner, and analysis limitations: [Accountable owners and review limits] Do not invent forecast values, confidence intervals, driver effects, or backtest statistics. Distinguish model output, planner override, observed fact, assumption, and scenario. Do not infer causal drivers from correlation alone. When raw data or executable logic is absent, assess the evidence and specify required tests rather than claiming to have rerun the forecast. Challenge: 1. Define the decision contract. State which decision the forecast informs, the timing, relevant error direction, tolerance, and alternative if evidence is insufficient. 2. Build an assumption register. Record every decision-critical assumption, source, rationale, range, dependency, owner, last validation, and failure consequence. Include missing-data treatment, overrides, external drivers, capacity limits, and behavioral responses. 3. Review data and model lineage. Trace target and drivers through sources, transformations, update timing, revisions, exclusions, and version changes. Flag leakage, survivorship, regime mixing, inconsistent definitions, and unavailable lineage. 4. Assess backtest evidence. Compare forecast errors across time, horizon, segment, regime, and relevant benchmark using supplied results. Check bias, variance, tail misses, turning points, coverage of intervals, and revision behavior. Avoid extrapolating from one period. 5. Identify structural drift. Map changes that weaken learned relationships or operational assumptions. Separate observed drift signals from hypotheses and specify evidence that would confirm them. 6. Run or design sensitivity analysis. Where supplied calculations permit, show how decision outcomes change across plausible assumptions. Otherwise provide exact scenario runs needed. Identify tipping points and assumptions that do not affect the decision. 7. Compare decision robustness. Determine whether Base, Upside, Downside, Stress, delayed-decision, staged-commitment, or no-action cases cross material thresholds. Keep probability claims evidence-bound. 8. Make a fitness decision. Choose Fit for stated decision, Fit with range/conditions, Use only as scenario input, Recalibrate/rebuild, or Not decision-fit. Assign owner actions and refresh triggers. Use ChatGPT to analyze supplied forecast artifacts and results; do not claim that backtests, refits, or sensitivity runs were executed unless their outputs are provided. Protect sensitive financial or operational data and stop where access or methodology is unauthorized. The model owner validates specification evidence, the finance or decision owner accepts decision risk, and the release owner authorizes operational use. Require acceptance evidence for each challenge: expected observation, actual observation, sensitivity boundary, reconciliation, and rollback or reforecast trigger. Required deliverable: # Forecast Assumption and Model Drift Challenge ## Decision Contract - Forecast and horizon: - Decision/threshold: - Cost of over-forecast and under-forecast: - Evidence limitations: ## Assumption Register | Assumption | Value/range | Evidence/source | Dependency | Last validated | Sensitivity | Owner | |---|---|---|---|---|---|---| ## Backtest and Drift Findings | Period/slice | Error evidence | Bias/tail issue | Structural change | Confidence | Decision impact | |---|---|---|---|---|---| ## Scenario and Tipping-Point Table | Scenario/assumption change | Forecast effect supplied or test needed | Decision threshold crossed? | Evidence | Action | |---|---|---|---|---| ## Forecast Fitness Decision - Decision: - Valid use and range: - Prohibited overclaim: - Required recalibration/data: - Monitoring and refresh trigger: - Accountable decision owner: Completion requires a traceable assumption register, honest use of backtest evidence, explicit structural-drift risks, and a decision that remains within the tested horizon and data regime.
View prompt
3 views
Knowledge Entitlement Drift Review
Reconcile authoritative access policy with effective permissions across source, ingestion, index, cache, retrieval, citation, and response layers.
Review whether effective access to enterprise knowledge has drifted from authoritative entitlement policy anywhere between the source system and the final retrieval response. Provide: - Approved subject, group, tenant, region, purpose, document, field, and time-bound access rules: [Authoritative entitlement policy] - Source repositories, ingestion, transformations, embeddings, indexes, namespaces, caches, retrievers, rerankers, citations, response filters, and identity propagation: [Knowledge architecture and identity flow] - Current ACLs, groups, policies, index filters, namespace rules, token claims, cache keys, and configuration exports: [Effective permission and configuration evidence] - Sanitized query/access traces, denials, sampled results, recertification records, deployments, migrations, and permission changes: [Retrieval access and change logs] - Data classification, privacy, residency, incident, availability, knowledge-owner, identity-owner, security-reviewer, and service-owner constraints: [Risk constraints and accountable owners] Do not claim a leakage event from configuration drift alone, and do not treat absence of logged leakage as proof of correct enforcement. Do not claim live access tests unless supplied. Distinguish observed evidence from inference, as well as authoritative policy, configured control, effective access, observed use, and confirmed exposure. Preserve uncertainty where identities or document lineage cannot be joined. Review: 1. Define authoritative entitlement contracts. Translate policy into subject-resource-action-context rules, including inheritance, deny precedence, purpose, tenant, region, embargo, expiry, field-level, and break-glass conditions. 2. Trace identity and entitlement propagation. Map how the caller identity and claims reach source filtering, ingestion metadata, index namespaces, retrieval filters, caches, citations, and response controls. Flag stages where identity is dropped, transformed, defaulted, or cached. 3. Reconcile effective controls. Compare authoritative rules with source ACLs, indexed metadata, filter logic, namespace membership, cache partitioning, service credentials, and post-retrieval controls. Account for stale groups, deleted users, broad service accounts, and shared indexes. 4. Build a drift register. Classify differences as Stale grant, Missing grant, Metadata loss, Filter mismatch, Namespace error, Cache-key weakness, Inherited expansion, Migration residue, Exception, or Not assessable. Record potential exposure and confirmed use separately. 5. Assess change and time behavior. Measure supplied lag between source permission changes and downstream propagation. Review revocation, role change, tenant move, embargo, and expiry handling. Do not invent propagation time. 6. Define the smallest safe correction. Recommend targeted re-indexing, metadata repair, filter correction, cache invalidation, token/session refresh, service-account restriction, or temporary serving restriction. Preserve availability and forensic evidence. 7. Design recertification proof. Specify positive and negative access cases, identity/tenant/region slices, document lineage checks, revocation timing, cache isolation, monitoring, and owner approval. Required deliverable: # Knowledge Entitlement Drift Review ## Authoritative Entitlement Matrix | Subject/context | Resource/class | Allowed/denied action | Condition/expiry | Policy evidence | Owner | |---|---|---|---|---|---| ## Propagation Map | Stage | Identity/entitlement input | Transformation | Enforcement | Evidence | Gap | |---|---|---|---|---|---| ## Drift Register | Drift | Classification | Policy state | Effective state | Potential exposure | Observed use | Confidence | Owner | |---|---|---|---|---|---|---|---| ## Correction and Restriction Plan | Priority | Smallest safe action | Affected layer | Availability effect | Authorization | Verification | |---|---|---|---|---|---| ## Recertification Gate | Test slice | Positive expectation | Negative expectation | Evidence source | Owner | Status | |---|---|---|---|---|---| ## Decision - Entitlement state: Aligned / Aligned with exceptions / Drift present / Not assessable - Serving restrictions: - Required corrections: - Unresolved exposure: - Revalidation trigger: Completion requires policy-to-response traceability for material knowledge classes, explicit separation of drift from confirmed exposure, and negative access evidence before removed permissions are considered effective.
View prompt
3 views
AI-Generated SQL Result Verification and Reconciliation
Validate AI-generated SQL and its reported results before they are used for a consequential decision.
Verify the AI-generated SQL and its claimed result before anyone relies on it for the stated decision. Treat this as a defensible review artifact, not a general SQL improvement exercise. Context to provide: - Repository scope and task: [Repository scope and task] - Relevant files and instructions: [Relevant files and instructions] - Observed evidence: [Observed evidence] - Constraints and authorized changes: [Constraints and authorized changes] - Environment details without secrets: [Environment details without secrets] - Verification commands and acceptance criteria: [Verification commands and acceptance criteria] Working rules: 1. Inspect the relevant files, saved queries, schemas, models, tests, lineage notes, and query artifacts first where they are available. 2. Do not claim that a table, file, source, command, test, query execution, permission boundary, or approval was inspected unless there is direct evidence in the provided materials or accessible workspace. 3. Identify the likely root cause of any discrepancy before editing SQL or proposing a corrected query. 4. Apply the smallest safe change needed to correct the query. Preserve existing intended behavior unless the evidence shows it is wrong. 5. Avoid broad rewrites, style-only edits, or migration to a different modeling pattern unless required to remove a proven defect. 6. Distinguish observations from inference. Mark unsupported assumptions explicitly. 7. Respect access boundaries. Do not suggest bypassing row-level security, protected schemas, production safeguards, or data owner controls. 8. Run syntax checks, query compilation, dry runs, unit tests, dbt tests, warehouse explain plans, or limited validation queries where available and appropriate. If execution is unavailable, state exactly what could not be run and what evidence substitutes for it. 9. Do not present a corrected result as final unless it is reproducible from inspected SQL and accessible evidence. Missing-input gate: - Treat the exact SQL, intended metric or decision and grain, relevant schema or lineage, and comparison source or acceptance threshold as blocking when their absence or conflict prevents semantic review or reconciliation. Request all blocking items in one consolidated clarification and stop the affected conclusion until they are supplied. - Continue with non-blocking gaps only as explicit Unknowns with their decision impact. Review procedure: A. Build the query intent contract - Restate the decision or metric question in operational terms. - Define the expected grain of the result. - Identify required dimensions, filters, joins, exclusions, deduplication rules, aggregation logic, time window, timezone, and freshness expectations. - Identify the authoritative definition owner when evident, such as data owner, analytics owner, finance owner, product owner, or compliance owner. - List evidence supplied versus evidence missing. B. Review SQL semantics before result claims - Check whether selected columns, grouping, joins, filters, CTEs, window functions, null handling, distinct logic, and aggregation match the query intent contract. - Check for join multiplication, accidental inner joins, incomplete predicates, slowly changing dimension issues, many-to-many joins, late-arriving data, timezone drift, partition filters, and snapshot-versus-current-state confusion. - Check whether access constraints or row-level filters could change the observed result. - Identify whether the SQL answers the stated question, a narrower question, a broader question, or a different question. C. Review execution and reproducibility evidence - Determine whether the claimed result can be traced to a specific SQL text, execution environment, parameters, source versions, run time, and data freshness state. - If query execution is available, run the safest reproducible check permitted by the environment and record the command/query used, result shape, row counts, and relevant totals. - If execution is not available, perform static validation and specify the minimum evidence needed from the data owner or analytics owner to complete reconciliation. D. Reconcile against authoritative totals - Compare the claimed result to authoritative totals, source-of-truth reports, known control totals, prior certified extracts, ledger totals, or approved metric definitions where supplied. - Reconcile at the most useful level: total, time bucket, segment, source system, account, customer, product, or other relevant dimension. - Explain every material variance using evidence where possible. Separate confirmed causes from plausible causes. E. Correct only what is justified - If the SQL is wrong and enough evidence exists, provide a corrected query or minimal patch. - If there is not enough evidence to correct the query safely, provide a bounded correction proposal and list the exact evidence required before use. - Preserve naming, output shape, filters, permissions, and downstream expectations unless the discrepancy requires a specific change. Deliverable format: 1. Query intent contract - Decision or metric question: - Intended grain: - Required population and exclusions: - Required time logic and timezone: - Required joins and source precedence: - Authoritative definitions or totals used: - Accountable owner to verify final use: - Missing context that limits certainty: 2. Semantic and execution review Create a table with columns: - Review area - Evidence inspected - Observation - Risk to claimed result - Status: Pass / Fail / Unverified - Notes or required follow-up Include at minimum: grain, joins, filters, aggregation, deduplication, time logic, null handling, access boundaries, source freshness, result reproducibility, and output shape. 3. Reconciliation table Create a table with columns: - Reconciliation item - AI-claimed value - Verified or control value - Difference - Materiality assessment - Evidence source - Explanation - Status: Reconciled / Variance explained / Unresolved 4. Unsupported-result register Create a table with columns: - Unsupported claim or result component - Why it is unsupported - Evidence needed - Accountable owner or source to confirm - Decision impact 5. Corrected-query and acceptance record - Root cause summary before any SQL change: - Minimal corrected SQL or patch, if justified: - Behavior intentionally preserved: - Behavior intentionally changed: - Files changed, if any: - Syntax checks, tests, dry runs, or executions performed: - Verification results: - Remaining uncertainty: - Acceptance decision: Accept / Accept with caveats / Reject / Cannot determine - Acceptance criteria met or unmet: - Owner verification required before decision use: Completion check: - The claimed result is either reconciled, corrected and reproducible, or explicitly rejected as unsupported. - All material assumptions are labeled. - Any changed files are summarized. - Verification results and unavailable checks are stated plainly. - The final acceptance decision is tied to the acceptance threshold or materiality standard.Retrieval Chunking and Metadata Experiment Design
Design a controlled retrieval experiment that compares chunking and metadata choices without turning into a full RAG redesign.
Design a controlled retrieval experiment to isolate how chunking choices and metadata choices affect retrieval and answer quality. Keep the scope limited to corpus configuration decisions: do not redesign the full RAG architecture, diagnose the whole pipeline, change model selection, or introduce unrelated retrieval changes unless they are explicitly controlled constants. Context to use: - Corpus description and sample documents: [Corpus description and sample documents] - Current retrieval setup: [Current retrieval setup] - Candidate chunking options: [Candidate chunking options] - Candidate metadata fields: [Candidate metadata fields] - Query logs or benchmark questions: [Query logs or benchmark questions] - Answer evaluation standard: [Answer evaluation standard] - Decision owners and constraints: [Decision owners and constraints] Evidence discipline: - Separate observations supported by the provided inputs from inferences and assumptions. - Do not claim that experiments, tests, retrieval runs, indexing jobs, or evaluations were completed unless results are provided. - If required information is missing, state the missing evidence and design the smallest reasonable way to obtain it. - Preserve uncertainty where the available evidence does not justify a firm conclusion. - Before defining experiment arms, treat the baseline retrieval configuration, candidate changes, representative corpus and query evidence, and answer-evaluation standard as blocking when absent or materially conflicting. Request all blocking items in one consolidated clarification and stop the affected design decision until they are supplied. Record other gaps as Unknown with their effect on sizing, confounding, metrics, and acceptance criteria. - Treat the retrieval owner as accountable for experiment execution, the data owner as accountable for corpus and metadata validity, and the product owner or domain owner as accountable for acceptance criteria tied to user impact. Produce the following deliverable: 1. Experiment objective and decision boundary - State the configuration decision this experiment will support. - Define what is in scope: chunking strategy, overlap, boundary rules, parent-child or hierarchical chunking if relevant, metadata extraction, metadata normalization, and metadata use in filtering or ranking. - Define what is out of scope and should be held constant: embedding model, reranker, generation prompt, answer model, UI behavior, permissions, and production rollout unless the inputs require otherwise. - Identify the baseline configuration and the candidate configurations to compare. 2. Controlled variables and constants Create a table with: - Variable under test - Candidate levels - Why it matters - Expected retrieval effect - Required implementation evidence - What must remain constant Include at minimum: - Chunk size or token range - Chunk overlap - Chunk boundary rule - Structural handling of headings, tables, lists, or sections - Metadata fields to attach - Metadata normalization rules - Metadata use pattern: display only, filter, boost, grouping, or attribution 3. Factorial experiment matrix Build a practical factorial or fractional-factorial matrix that isolates chunking and metadata effects without creating unnecessary runs. For each experiment arm include: - Arm ID - Chunking configuration - Metadata configuration - Retrieval settings held constant - Indexing requirements - Expected comparison value - Risk of confounding - Minimum evidence needed before execution If the full factorial design is too large, propose a staged design: - Screening stage to remove weak options - Focused comparison stage for finalists - Confirmation stage against the baseline 4. Dataset and query slices Define the evaluation dataset and query slices needed to detect meaningful differences. Include: - Document sample selection rules - Minimum corpus coverage requirements - Query source: logs, expert-written questions, synthetic-but-reviewed questions, or known support cases - Query slice taxonomy - Required number of queries per slice if enough information is available; otherwise give a sizing rule Use query slices such as: - Exact lookup questions - Multi-section synthesis questions - Long-tail entity questions - Recent or version-specific questions - Table or list extraction questions - Procedure or policy questions - Ambiguous terminology questions - Metadata-dependent questions - Queries where the answer should not be in the corpus For each slice, specify the retrieval failure mode it is intended to expose. 5. Relevance judgments and answer evaluation setup Define how ground truth or reference judgments should be created. Include: - Who should judge relevance and answer correctness - What evidence they need - How to label relevant, partially relevant, irrelevant, stale, duplicate, and misleading chunks - How to handle multiple valid source passages - How to record uncertainty and disagreements - How to prevent evaluators from seeing the tested configuration when practical 6. Retrieval metrics Specify retrieval metrics that directly measure chunking and metadata effects. Include: - Recall@k - Precision@k or context precision - MRR or nDCG where graded relevance exists - Source coverage for multi-source answers - Duplicate or near-duplicate rate in top-k - Metadata filter precision and filter fallout - Freshness or version correctness when metadata includes time or version fields - No-answer retrieval behavior for queries outside the corpus For each metric, define: - Formula or scoring method in plain language - Required inputs - Which query slices it applies to - What kind of configuration failure it reveals 7. Answer-level metrics Define answer metrics only as downstream checks of retrieval configuration, not as a full generation evaluation. Include: - Answer correctness against the evaluation standard - Citation/source support - Missing critical fact rate - Unsupported claim rate - Wrong-version or wrong-jurisdiction rate if relevant - Refusal or no-answer correctness where the corpus lacks the answer Explain how to attribute answer failures to retrieval versus generation when the top-k evidence is sufficient but the answer is wrong. 8. Error attribution rules Create a concrete error taxonomy with decision rules. Include at minimum: - Chunk boundary split error - Chunk too broad or noisy - Chunk too narrow or missing context - Overlap redundancy error - Metadata absent - Metadata incorrect - Metadata too coarse - Metadata filter excludes relevant evidence - Metadata boost overpromotes irrelevant evidence - Duplicate chunk crowding - Stale or wrong-version retrieval - Relevant evidence not indexed - Relevant evidence indexed but not retrieved - Answer failure despite sufficient retrieved evidence For each error type, define: - Observable evidence - How to distinguish it from similar errors - Likely configuration implication - Whether it should count against chunking, metadata, indexing, retrieval settings, or answer generation 9. Analysis plan by query slice Define how results should be compared across arms and slices. Include: - Primary metric and secondary metrics - Minimum practical improvement threshold - Regression checks where a configuration improves one slice but harms another - Treatment of ties - Treatment of small sample sizes - Required confidence or stability checks if repeated runs are possible - How to summarize tradeoffs for decision owners without hiding slice-level failures 10. Configuration acceptance decision Create a decision framework the retrieval owner, data owner, and product owner can use after the experiment is run. Include: - Acceptance criteria for adopting a new chunking and metadata configuration - Rejection criteria - Conditional acceptance criteria requiring remediation - Required evidence package before approval - Rollback or re-indexing considerations if adopted - Open questions that must be resolved before production use 11. Completion checklist End with a checklist that confirms whether the experiment design is ready to execute. Include checks for: - Baseline defined - Candidate arms defined - Constants identified - Query slices complete - Relevance judgment process defined - Metrics mapped to slices - Error attribution rules defined - Acceptance criteria agreed by named accountable owners - Missing evidence listed - No unsupported claims of completed executionRAG Access-Control Leakage Investigation
Investigate whether a RAG system crossed identity, tenant, purpose, region, or document authorization boundaries during retrieval, embedding, caching, citation, or response exposure.
Investigate whether the RAG system described below retrieved, embedded, cached, cited, or exposed content outside the requesting identity, tenant, purpose, region, or document authorization boundary. Do not perform a general RAG quality review, connector security review, model safety review, or relevance evaluation unless it directly affects authorization leakage evidence. Use only the evidence provided. Do not claim that a log, policy, source, command, system, test, approval, or remediation was inspected or completed unless it is present in the supplied materials. Context and inputs: - RAG system and deployment context: [RAG system and deployment context] - Identity, tenant, purpose, region, and document authorization model: [Identity tenant purpose region and document authorization model] - Evidence bundle, including any retrieval logs, citation traces, document ACLs, embedding ingestion records, cache records, query logs, response samples, tickets, diagrams, or policy excerpts: [Evidence bundle] - Suspected leakage scenarios or query samples: [Suspected leakage scenarios or query samples] - Time window and affected environments: [Time window and affected environments] - Accountable owners and decision deadline: [Accountable owners and decision deadline] Investigation rules: 1. Separate observations from inference. Label each material statement as Evidence, Inference, Gap, or Decision. 2. Preserve uncertainty. If evidence is missing, conflicting, sampled, redacted, or outside the time window, say so plainly. Missing-input gate: - Treat the requesting identity, applicable authorization model, affected environment and time window, and sufficient retrieval or exposure traces as blocking for any confirmed leakage conclusion. Request all blocking items in one consolidated clarification and stop the affected conclusion until they are supplied. - Continue with non-blocking gaps only when they are marked Unknown and their effect on path confidence, affected scope, containment, and regression coverage is explicit. 3. Trace authorization from requesting identity to document exposure. Include retrieval, ranking, embedding, cache, citation, prompt assembly, generated response, export, telemetry, and feedback paths where evidence exists. 4. Treat access-control boundaries independently: identity, tenant, purpose, region, document-level authorization, group membership, delegated access, service account access, and temporal entitlement changes. 5. Distinguish content that was indexed, embedded, retrieved, ranked, cached, cited, included in context, generated in the answer, logged, or externally exposed. 6. Do not assume that document-level authorization at source ingestion remains valid at query time. Look for entitlement drift, stale indexes, overbroad service accounts, cache reuse, citation leakage, and post-filter bypass. 7. Do not recommend broad rewrites. Recommend the smallest containment and verification actions that fit the observed risk. 8. Assign owner accountability by role where needed: security reviewer, data owner, product owner, release owner, search/retrieval owner, platform owner, or legal/privacy owner. Deliverable: 1. Investigation scope - State the system, environments, time window, suspected boundary, and query/document populations covered. - State what is explicitly out of scope. - List the supplied evidence types and any major missing evidence. 2. Evidence register Create a table with columns: - Evidence ID - Evidence type - Source or artifact name - Time range - What it shows - Authorization boundary relevance - Limitations or uncertainty 3. Identity-to-document authorization map Create a map or table showing: - Requesting identity or role - Tenant or account - Purpose or workflow - Region or data residency boundary - Entitlement source - Allowed document classes or IDs - Disallowed document classes or IDs - Query-time enforcement point - Retrieval/index/cache/citation enforcement point - Evidence supporting the mapping - Gaps requiring owner confirmation 4. Leakage-path reconstruction For each suspected or observed leakage path, reconstruct the sequence from request to exposure: - Query or trigger - Requesting identity and authorization state at the time - Retrieval/index candidate set - Filter or authorization check expected - Filter or authorization check observed - Document, chunk, citation, cache entry, embedding, or response involved - Exposure mode: retrieved, ranked, cached, cited, placed in prompt context, generated in response, logged, exported, or visible in UI/API - Boundary crossed: identity, tenant, purpose, region, document authorization, or timing/entitlement drift - Evidence supporting the path - Alternative explanations - Confidence level: confirmed, probable, possible, or not supported 5. Affected corpus and query scope Estimate the likely scope without overstating certainty: - Affected tenants, identities, groups, regions, document classes, indexes, embeddings, caches, and environments - Query patterns likely to trigger the issue - Whether exposure appears isolated, systemic, configuration-specific, connector-specific, cache-specific, or time-bound - Known false-positive or false-negative risks in the evidence - Additional evidence needed to narrow scope 6. Containment decision Provide a decision-ready containment assessment for the accountable owners: - Immediate risk level and rationale - Recommended containment action: no action, monitor, disable affected queries, invalidate cache, remove affected index, restrict service account, re-ingest with corrected ACLs, disable citation/source display, tenant isolate, region isolate, or pause release - Why this is the smallest safe containment action supported by the evidence - Operational impact - Data owner, security reviewer, product owner, and release owner decisions required - Conditions for lifting containment 7. Regression test matrix Create a test matrix that can be used by engineering and security reviewers. Include: - Test ID - Boundary tested - Test identity or role - Allowed corpus - Forbidden corpus - Query pattern - Expected retrieval behavior - Expected citation behavior - Expected cache behavior - Expected response behavior - Required logs or traces - Pass/fail criteria - Owner responsible for verification Include tests for at least: - Same-tenant allowed document retrieval - Same-tenant forbidden document exclusion - Cross-tenant exclusion - Region boundary exclusion - Purpose-boundary exclusion where applicable - Stale group membership or entitlement removal - Cache reuse across identities or tenants - Citation/source leakage without answer leakage - Embedding/index rebuild after ACL correction - Service account or delegated access path 8. Open questions and evidence requests List only the missing items that materially affect the containment decision, scope estimate, or regression coverage. Assign each request to a specific owner role. 9. Completion checks End with a concise checklist confirming whether the current evidence is sufficient to: - Determine if leakage occurred - Identify the likely leakage path - Estimate affected corpus and query scope - Make a containment decision - Define regression tests - Proceed to remediation planning If any check cannot be completed from the supplied evidence, mark it as incomplete and explain the exact missing evidence.Was this useful?