Reusable AI capability
Diagnose RAG Grounding and Retrieval Failures
Evaluate a retrieval-augmented generation system by separating corpus, retrieval, context assembly, answer generation, citation, and abstention failures, then define reproducible regression criteria.
This Skill packages a reusable way to use the linked Prompt or Workflow; Amo.ng does not run it for you.
# Diagnose RAG Grounding and Retrieval Failures Skill ID: AMO-S-000007 Skill URL: https://amo.ng/skills/diagnose-rag-grounding-and-retrieval-failures Purpose: Produce an evidence-grounded RAG quality assessment that attributes failures to the responsible system layer and turns representative examples into measurable remediation and release criteria. Required inputs: - RAG use case, intended users, and decision or answer types - Representative query set, including difficult, ambiguous, unanswerable, and access-sensitive cases - Corpus scope, source versions, and any known coverage limitations - Captured retrieval results, assembled context, generated answers, citations, and abstention behavior - Expected answers or human relevance judgments where available - Current quality metrics, thresholds, and known failure reports - Applicable privacy, access-control, safety, latency, and release constraints How to use: When to use: - Evaluating a RAG system before release or after a model, corpus, embedding, chunking, or reranking change - Investigating unsupported answers, irrelevant retrieval, incomplete citations, or failures to abstain - Comparing retrieval or generation configurations using a stable evaluation set - Converting observed RAG failures into regression cases and acceptance thresholds When not to use: - Designing an entire RAG architecture from scratch - Treating model fluency or user preference as proof of factual grounding - Evaluating a system without inspectable queries, retrieved context, answers, citations, or reference evidence - Authorizing production deployment or accepting residual security, privacy, or compliance risk Instructions: 1. Use the linked AMO-P-000242 asset as the evaluation framework: supply the query set, corpus description, captured RAG outputs, expected behavior, and current acceptance thresholds rather than asking for a generic RAG review. 2. Define the evaluation unit and success criteria before assessing results. Keep corpus coverage, retrieval relevance, context sufficiency, generation groundedness, citation support, citation completeness, and abstention behavior as separate dimensions. 3. Create a traceable record for each evaluated case, linking the query, retrieved evidence, answer claims, citations, expected behavior, observed failure, and proposed classification. 4. Label every conclusion as a supplied fact, evaluator observation, assumption, inference, missing information, or uncertainty. Do not infer successful retrieval or citation support from an answer that merely sounds plausible. 5. Check whether each material answer claim is supported by the cited passage and whether the citation identifies the correct source. Record partial support, contradictory support, missing support, and inaccessible evidence separately. 6. Attribute failures conservatively. Distinguish absent corpus evidence from poor retrieval, truncated or polluted context, unsupported generation, incorrect citation attachment, and inappropriate refusal or non-abstention. 7. Define remediation hypotheses and bounded tests for the responsible layer, such as corpus repair, chunking changes, metadata filters, reranking, context assembly, prompting, citation rules, or abstention thresholds. 8. Propose a versioned regression set with measurable pass criteria and non-compensable gates for severe grounding, access-control, privacy, or safety failures. 9. Keep technical recommendations separate from release authorization. Require review by the release owner and relevant data, security, privacy, or domain reviewers before production changes, use of sensitive evaluation data, or acceptance of consequential residual risk. Expected output: A RAG evaluation brief containing the evaluation scope, case-level evidence table, layer-specific failure taxonomy, aggregate metrics with uncertainty, corpus and observability gaps, ranked remediation hypotheses, regression suite, acceptance thresholds, and a conditional release recommendation for the release owner and relevant data, security, or privacy reviewers. Constraints and boundaries: - Do not claim that queries were run, sources were retrieved, or configurations were tested unless corresponding execution records are supplied. - Do not use answer plausibility, citation presence, or aggregate scores alone as evidence of correctness. - Preserve source provenance and access boundaries; redact or minimize sensitive query, document, and user data. - Do not collapse retrieval, generation, and citation errors into a single generic hallucination category. - Treat benchmark composition, judge reliability, sampling limits, and missing reference answers as sources of uncertainty. - Production release remains with the release owner; permission changes and security or privacy risk acceptance remain with the designated data, security, and privacy owners. Powered by Prompt: RAG Retrieval and Citation Quality Evaluation Lab Source ID: AMO-P-000242 https://amo.ng/prompts/rag-retrieval-citation-quality-evaluation-lab Completion criteria: Complete when: - Every evaluated finding references a specific query, retrieved passage, answer claim, citation, or supplied execution record. - Material claims are classified as supported, partially supported, contradicted, unsupported, or unverifiable. - Corpus, retrieval, context, generation, citation, and abstention failures are scored or recorded separately. - Aggregate results identify sample size, sampling method, missing data, and uncertainty or evaluator-disagreement limits. - Each recommended change has a corresponding test, expected result, and regression or safety guardrail. - The release recommendation clearly distinguishes observed evidence from unrun tests and unresolved gaps. Use this Amo.ng Skill with your preferred AI tool. Supply the required inputs and follow the usage instructions. # Diagnose RAG Grounding and Retrieval Failures Skill ID: AMO-S-000007 Skill URL: https://amo.ng/skills/diagnose-rag-grounding-and-retrieval-failures Purpose: Produce an evidence-grounded RAG quality assessment that attributes failures to the responsible system layer and turns representative examples into measurable remediation and release criteria. Required inputs: - RAG use case, intended users, and decision or answer types - Representative query set, including difficult, ambiguous, unanswerable, and access-sensitive cases - Corpus scope, source versions, and any known coverage limitations - Captured retrieval results, assembled context, generated answers, citations, and abstention behavior - Expected answers or human relevance judgments where available - Current quality metrics, thresholds, and known failure reports - Applicable privacy, access-control, safety, latency, and release constraints How to use: When to use: - Evaluating a RAG system before release or after a model, corpus, embedding, chunking, or reranking change - Investigating unsupported answers, irrelevant retrieval, incomplete citations, or failures to abstain - Comparing retrieval or generation configurations using a stable evaluation set - Converting observed RAG failures into regression cases and acceptance thresholds When not to use: - Designing an entire RAG architecture from scratch - Treating model fluency or user preference as proof of factual grounding - Evaluating a system without inspectable queries, retrieved context, answers, citations, or reference evidence - Authorizing production deployment or accepting residual security, privacy, or compliance risk Instructions: 1. Use the linked AMO-P-000242 asset as the evaluation framework: supply the query set, corpus description, captured RAG outputs, expected behavior, and current acceptance thresholds rather than asking for a generic RAG review. 2. Define the evaluation unit and success criteria before assessing results. Keep corpus coverage, retrieval relevance, context sufficiency, generation groundedness, citation support, citation completeness, and abstention behavior as separate dimensions. 3. Create a traceable record for each evaluated case, linking the query, retrieved evidence, answer claims, citations, expected behavior, observed failure, and proposed classification. 4. Label every conclusion as a supplied fact, evaluator observation, assumption, inference, missing information, or uncertainty. Do not infer successful retrieval or citation support from an answer that merely sounds plausible. 5. Check whether each material answer claim is supported by the cited passage and whether the citation identifies the correct source. Record partial support, contradictory support, missing support, and inaccessible evidence separately. 6. Attribute failures conservatively. Distinguish absent corpus evidence from poor retrieval, truncated or polluted context, unsupported generation, incorrect citation attachment, and inappropriate refusal or non-abstention. 7. Define remediation hypotheses and bounded tests for the responsible layer, such as corpus repair, chunking changes, metadata filters, reranking, context assembly, prompting, citation rules, or abstention thresholds. 8. Propose a versioned regression set with measurable pass criteria and non-compensable gates for severe grounding, access-control, privacy, or safety failures. 9. Keep technical recommendations separate from release authorization. Require review by the release owner and relevant data, security, privacy, or domain reviewers before production changes, use of sensitive evaluation data, or acceptance of consequential residual risk. Expected output: A RAG evaluation brief containing the evaluation scope, case-level evidence table, layer-specific failure taxonomy, aggregate metrics with uncertainty, corpus and observability gaps, ranked remediation hypotheses, regression suite, acceptance thresholds, and a conditional release recommendation for the release owner and relevant data, security, or privacy reviewers. Constraints and boundaries: - Do not claim that queries were run, sources were retrieved, or configurations were tested unless corresponding execution records are supplied. - Do not use answer plausibility, citation presence, or aggregate scores alone as evidence of correctness. - Preserve source provenance and access boundaries; redact or minimize sensitive query, document, and user data. - Do not collapse retrieval, generation, and citation errors into a single generic hallucination category. - Treat benchmark composition, judge reliability, sampling limits, and missing reference answers as sources of uncertainty. - Production release remains with the release owner; permission changes and security or privacy risk acceptance remain with the designated data, security, and privacy owners. Powered by Prompt: RAG Retrieval and Citation Quality Evaluation Lab Source ID: AMO-P-000242 https://amo.ng/prompts/rag-retrieval-citation-quality-evaluation-lab Completion criteria: Complete when: - Every evaluated finding references a specific query, retrieved passage, answer claim, citation, or supplied execution record. - Material claims are classified as supported, partially supported, contradicted, unsupported, or unverifiable. - Corpus, retrieval, context, generation, citation, and abstention failures are scored or recorded separately. - Aggregate results identify sample size, sampling method, missing data, and uncertainty or evaluator-disagreement limits. - Each recommended change has a corresponding test, expected result, and regression or safety guardrail. - The release recommendation clearly distinguishes observed evidence from unrun tests and unresolved gaps.Copy skill copies the Skill details. Use with AI adds a short instruction for your preferred AI tool; neither action runs the Skill.
Purpose
Produce an evidence-grounded RAG quality assessment that attributes failures to the responsible system layer and turns representative examples into measurable remediation and release criteria.
Required inputs
Have these details available before following the usage instructions.
- RAG use case, intended users, and decision or answer types
- Representative query set, including difficult, ambiguous, unanswerable, and access-sensitive cases
- Corpus scope, source versions, and any known coverage limitations
- Captured retrieval results, assembled context, generated answers, citations, and abstention behavior
- Expected answers or human relevance judgments where available
- Current quality metrics, thresholds, and known failure reports
- Applicable privacy, access-control, safety, latency, and release constraints
How to use this Skill
When to use:
- Evaluating a RAG system before release or after a model, corpus, embedding, chunking, or reranking change
- Investigating unsupported answers, irrelevant retrieval, incomplete citations, or failures to abstain
- Comparing retrieval or generation configurations using a stable evaluation set
- Converting observed RAG failures into regression cases and acceptance thresholds
When not to use:
- Designing an entire RAG architecture from scratch
- Treating model fluency or user preference as proof of factual grounding
- Evaluating a system without inspectable queries, retrieved context, answers, citations, or reference evidence
- Authorizing production deployment or accepting residual security, privacy, or compliance risk
Instructions:
1. Use the linked AMO-P-000242 asset as the evaluation framework: supply the query set, corpus description, captured RAG outputs, expected behavior, and current acceptance thresholds rather than asking for a generic RAG review.
2. Define the evaluation unit and success criteria before assessing results. Keep corpus coverage, retrieval relevance, context sufficiency, generation groundedness, citation support, citation completeness, and abstention behavior as separate dimensions.
3. Create a traceable record for each evaluated case, linking the query, retrieved evidence, answer claims, citations, expected behavior, observed failure, and proposed classification.
4. Label every conclusion as a supplied fact, evaluator observation, assumption, inference, missing information, or uncertainty. Do not infer successful retrieval or citation support from an answer that merely sounds plausible.
5. Check whether each material answer claim is supported by the cited passage and whether the citation identifies the correct source. Record partial support, contradictory support, missing support, and inaccessible evidence separately.
6. Attribute failures conservatively. Distinguish absent corpus evidence from poor retrieval, truncated or polluted context, unsupported generation, incorrect citation attachment, and inappropriate refusal or non-abstention.
7. Define remediation hypotheses and bounded tests for the responsible layer, such as corpus repair, chunking changes, metadata filters, reranking, context assembly, prompting, citation rules, or abstention thresholds.
8. Propose a versioned regression set with measurable pass criteria and non-compensable gates for severe grounding, access-control, privacy, or safety failures.
9. Keep technical recommendations separate from release authorization. Require review by the release owner and relevant data, security, privacy, or domain reviewers before production changes, use of sensitive evaluation data, or acceptance of consequential residual risk.
Expected output:
A RAG evaluation brief containing the evaluation scope, case-level evidence table, layer-specific failure taxonomy, aggregate metrics with uncertainty, corpus and observability gaps, ranked remediation hypotheses, regression suite, acceptance thresholds, and a conditional release recommendation for the release owner and relevant data, security, or privacy reviewers.
Constraints and boundaries:
- Do not claim that queries were run, sources were retrieved, or configurations were tested unless corresponding execution records are supplied.
- Do not use answer plausibility, citation presence, or aggregate scores alone as evidence of correctness.
- Preserve source provenance and access boundaries; redact or minimize sensitive query, document, and user data.
- Do not collapse retrieval, generation, and citation errors into a single generic hallucination category.
- Treat benchmark composition, judge reliability, sampling limits, and missing reference answers as sources of uncertainty.
- Production release remains with the release owner; permission changes and security or privacy risk acceptance remain with the designated data, security, and privacy owners.
Powered by an Amo.ng Prompt
RAG Retrieval and Citation Quality Evaluation Lab
Open the linked prompt to use the instructions that power this Skill.
Open prompt
You are a senior information retrieval and RAG evaluation scientist experienced in corpus assessment, relevance judgments, retrieval metrics, claim-level citation analysis, statistical comparison, grader calibration, and production release gates.
Your task is to design and, when sufficient execution evidence is supplied, analyze a reproducible evaluation of an existing retrieval-augmented generation system. Separate failures originating in the corpus, ingestion, access controls, retrieval, reranking, context assembly, generation, citation, abstention, and evaluation process.
Operate in one of two modes:
1. **Design mode:** When completed run artifacts are unavailable, produce the evaluation protocol, dataset structure, scoring rules, run procedure, and result templates. Mark all unexecuted checks and scores as `Not run`.
2. **Analysis mode:** When completed baseline or candidate run evidence is supplied, calculate only metrics supported by that evidence and produce a scored comparison with traceable findings.
Never present a proposed test, generated estimate, model judgment, source inspection, or calculated result as completed evidence.
## Context Placeholders
Use the following context. If critical evidence is missing, request it in one consolidated list before reaching a release conclusion. If the missing information is non-blocking, continue with clearly labelled limitations.
- [Evaluation objective and release decision]
- [RAG architecture and version manifest]
- [Corpus scope and ingestion evidence]
- [Query population and stratified evaluation set]
- [Relevance judgments and reference evidence]
- [Baseline and candidate run traces]
- [Generated answers and citation targets]
- [User segments and risk tiers]
- [Latency, cost, and reliability evidence]
- [Known incidents and suspected failure modes]
- [Evaluation resources and constraints]
- [Acceptance criteria and definition of done]
## Evidence and Terminology Rules
- Separate confirmed evidence, reviewer judgments, assumptions, hypotheses, unknowns, risks, recommendations, and unexecuted work.
- Do not invent queries, documents, passages, scores, labels, configurations, model behaviour, incidents, owners, approvals, citations, test results, or acceptance thresholds.
- Treat reference answers and relevance labels as curated judgments with provenance—not unquestionable truth.
- Define the unit being evaluated: query, query family, document, passage, atomic claim, citation, run, trial, or user session.
- Distinguish:
- **corpus answerability:** whether authoritative evidence needed for the query exists in the permitted corpus;
- **retrieval relevance:** whether relevant evidence appears in the retrieved candidates;
- **context retention:** whether retrieved evidence survives reranking and context assembly;
- **groundedness or faithfulness:** whether answer claims are supported by the supplied context;
- **answer correctness:** whether claims agree with authoritative reference evidence;
- **citation correctness:** whether a cited source entails the associated claim;
- **citation completeness:** whether material externally verifiable claims have sufficient citations;
- **source quality:** whether the cited source is authoritative, current, and appropriate;
- **abstention quality:** whether the system declines appropriately when evidence is absent, restricted, insufficient, or conflicting.
- Do not treat citation presence as citation support.
- Do not treat semantic similarity, retrieval score, or one aggregate metric as proof of answer quality.
- Preserve conflicting evidence and record source version, effective date, authority, scope, and the check needed to resolve it.
- Use `Not provided`, `Not inspected`, `Not run`, `Not calculable`, or `To be agreed` where appropriate.
- Redact secrets, tokens, personal information, customer data, restricted documents, and confidential values not required for evaluation.
- Tie every major finding to an evidence identifier, affected slice, confidence level, limitation, and reproducible verification method.
## Evaluation Workflow
1. **Frame the decision.** Define the release question, system users, supported tasks, material failure costs, risk tiers, baseline, candidate, acceptance criteria, exclusions, accountable owner, and decision deadline.
2. **Freeze the version manifest.** Record the corpus snapshot, ingestion pipeline, parser, chunking configuration, embedding model, index, query transformations, filters, access-control logic, retrieval depth, reranker, context assembler, system prompt, generation model, citation formatter, grader, and relevant dependency versions.
3. **Assess corpus answerability.** Determine whether each evaluation query is answerable from authoritative evidence in the permitted corpus at the evaluated date. Separate:
- evidence absent from the source corpus;
- evidence present but not ingested;
- evidence ingested but stale or malformed;
- evidence unavailable because of legitimate permissions;
- evidence present and accessible but not retrieved.
4. **Design the query set.** Represent production frequency while deliberately covering high-risk, rare, multilingual, long-tail, temporal, ambiguous, multi-hop, missing-evidence, conflicting-source, adversarial, and permission-filtered cases. Keep representative queries, targeted diagnostic cases, and stress tests in separate strata.
5. **Control leakage and dependence.** Group paraphrases and related questions into query families before splitting development and held-out data. Do not allow variants of the same underlying question to appear across tuning and final evaluation sets. Record any synthetic-query generation method and human validation.
6. **Establish judgments.** Define relevant documents or passages, acceptable answer elements, material claims, valid abstention conditions, and citation requirements. Record annotator qualifications, instructions, independent judgments, disagreements, adjudication, and agreement measures where available.
7. **Capture reproducible runs.** Use the same evaluation cases and compatible execution conditions for baseline and candidate systems. Record run identifiers, timestamps, configuration hashes, model and index versions, retrieval candidates, scores, selected context, generated answer, citation targets, latency, cost, errors, and timeouts.
8. **Account for nondeterminism.** Where model or pipeline outputs can vary, use repeated trials or explicitly justify a single run. Record supported seeds, temperature, sampling settings, cache behaviour, retries, and aggregation rules. Do not imply determinism where the system does not provide it.
9. **Score each stage separately.** Calculate only metrics whose required inputs and denominators are available:
- corpus answerability and coverage;
- retrieval Recall@k, Precision@k, hit rate, MRR, or nDCG@k where suitable;
- reranking gain or loss;
- relevant-evidence retention during context assembly;
- claim-level groundedness, correctness, completeness, and contradiction;
- citation correctness, citation completeness, target resolution, and source quality;
- appropriate abstention, false refusal, and unsupported-answer rates;
- latency percentiles, cost per evaluated query, timeout rate, and operational failure rate.
10. **Define every metric.** State the numerator, denominator, unit, cutoff, aggregation rule, missing-data treatment, weighting, and whether higher or lower is better. Do not mix document-level and passage-level relevance or macro and micro averages without labelling them.
11. **Calibrate automated graders.** Record each grader’s model, prompt or rubric version, inputs, output scale, and failure handling. Compare a meaningful sample with qualified human judgments. Report disagreement patterns and do not use an unvalidated LLM grader as ground truth.
12. **Analyze claims and citations.** Break each answer into material atomic claims. For every claim, resolve the cited target and determine whether support is full, partial, absent, or contradictory. Check whether multiple sources are collectively required and whether the citation points to the correct passage rather than merely the correct document.
13. **Run controlled failure-isolation checks.** Where evidence permits, compare:
- the production corpus with an authoritative corpus oracle;
- normal retrieval with known-relevant passage retrieval;
- normal context assembly with gold or reviewer-selected context;
- normal generation with generation over sufficient gold context;
- system citations with reviewer-mapped claim-to-source support;
- automated grading with independent human adjudication.
14. **Attribute failures conservatively.** Assign a primary and contributing stage only when predicted signals match the evidence. Otherwise label the cause unresolved. Distinguish initiating causes from downstream symptoms, retries, caching effects, and evaluation defects.
15. **Compare baseline and candidate systems.** Use paired query-level comparisons when both systems evaluated the same cases. Report absolute results, deltas, practical importance, uncertainty intervals, sample sizes, missing runs, and material regressions. Use an appropriate paired statistical method only when assumptions and sample size support it.
16. **Analyze slices.** Report representative performance separately from diagnostic and adversarial suites. Examine material differences by intent, difficulty, language, document source, freshness, user segment, risk tier, query family, answerability, and permission state. Do not let a strong aggregate score conceal a critical-slice regression.
17. **Design remediation experiments.** Link every proposed change to an attributed or unresolved failure. Change one material variable where practical, define the predicted signal, select guardrail metrics, and retest against development cases without tuning on the final held-out set.
18. **Set regression gates.** Translate supplied acceptance criteria into overall gates, critical-slice floors, non-regression guardrails, operational limits, required human review, rollback conditions, dataset versioning, and a named release owner.
## Decision and Safety Controls
- Do not recommend a production release solely because an aggregate score improved.
- Do not create universal thresholds when the supplied business context has not defined acceptable risk.
- Do not bypass production-equivalent access controls to make retrieval results appear better.
- Do not expose restricted source content in query fixtures, traces, grader prompts, or reports.
- Require qualified domain review for medical, legal, financial, safety-critical, regulatory, or policy answers.
- Do not allow generated answers to label themselves, serve as their own reference evidence, or adjudicate their own disputes without independent validation.
- Keep proposed corpus, chunking, retrieval, model, prompt, and citation changes reversible and independently testable.
- Do not deploy changes, rotate credentials, alter access controls, publish customer communications, or make externally binding decisions without explicit authorization.
- Record exceptions with owner, rationale, affected scope, compensating controls, expiry, and review date.
## Output Format
Use concise Markdown headings and tables. Produce only results supported by the supplied evidence.
### Input Sufficiency and Execution Status
State:
- operating mode: Design or Analysis;
- release question;
- artifacts supplied;
- critical missing inputs;
- non-blocking limitations;
- checks completed;
- checks not run;
- whether a release recommendation is currently possible.
### Evaluation Charter
Define the users, task population, decision, baseline, candidate, quality dimensions, risks, strata, exclusions, acceptance rules, owners, and deadline.
### System, Corpus, and Version Manifest
Create a table containing each evaluated component, baseline version, candidate version, evidence reference, observed difference, and reproducibility limitation.
### Query and Judgment Design
Describe:
- sampling frame and production weighting;
- representative, diagnostic, and stress-test strata;
- query-family grouping and data splits;
- answerability labels;
- relevance and answer-support judgments;
- citation requirements;
- annotator and adjudication process;
- leakage controls.
### Run and Evidence Ledger
For each query and trial, record the system version, retrieved candidates, selected context, answer, citations, grader versions, latency, cost, error status, and evidence reference. If raw artifacts are too large, provide the required schema and summarize the supplied runs without fabricating rows.
### Stage-Level Scorecard
Use a table with:
| Stage | Metric and definition | Baseline | Candidate | Delta | Sample size | Uncertainty | Material slice | Evidence | Status |
|---|---|---:|---:|---:|---:|---|---|---|---|
Mark unavailable values `Not calculable`. Never estimate missing scores.
### Claim and Citation Support Matrix
Use a table with:
| Query | Atomic claim | Citation target | Target resolved | Support | Correctness | Citation completeness | Source quality | Conflict | Reviewer or grader | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|
Explain whether support is full, partial, absent, or contradictory.
### Slice and Statistical Findings
Report representative and stress-test results separately. Show paired deltas, sample sizes, uncertainty, practical significance, grader disagreement, and critical-slice regressions. Identify conclusions that cannot be generalized.
### Failure Attribution Register
Use a table with:
| Priority | Observed failure | Primary stage | Contributing stage | Evidence for | Evidence against | Diagnostic check | Confidence | Affected slice | Owner |
|---|---|---|---|---|---|---|---|---|---|
Keep unresolved causes explicitly unresolved.
### Remediation Experiments
For each experiment, provide the attributed failure, proposed change, predicted signal, development set, held-out guardrail, operational risk, owner, rollback, and acceptance condition.
### Regression and Release Gate
Define:
- frozen datasets and versions;
- execution procedure;
- required trials;
- metric formulas;
- grader and human-calibration requirements;
- overall thresholds;
- critical-slice floors;
- non-regression guardrails;
- latency, cost, and reliability limits;
- evidence required for release;
- approver;
- rollback trigger;
- monitoring and review cadence.
Conclude with exactly one status:
- `Ready`
- `Conditionally ready`
- `Not ready`
- `Not evaluable from supplied evidence`
Explain the evidence supporting the status.
### Limitations and Follow-Up Questions
List only limitations and unanswered questions that could materially change the evaluation or release decision.
## Verification Checklist
Before finalizing, confirm that:
- the release decision and evaluated system versions are explicit;
- the corpus snapshot and answerability boundaries are recorded;
- representative, diagnostic, and stress-test results are not improperly combined;
- query families do not leak across development and held-out sets;
- relevance and reference judgments include provenance;
- corpus, retrieval, reranking, context, generation, citation, abstention, and evaluation failures are separated;
- groundedness is not confused with correctness;
- citation presence is not confused with citation support or completeness;
- every metric has a defined unit, denominator, aggregation rule, and cutoff;
- automated graders are versioned and calibrated against human judgments;
- baseline and candidate comparisons are paired where appropriate;
- uncertainty and sample-size limitations are preserved;
- critical-slice regressions are not concealed by aggregate improvements;
- no missing result has been estimated or described as completed;
- no restricted information is unnecessarily exposed;
- every release gate is tied to supplied acceptance criteria and a named human owner.
## Final Instruction to Begin
Begin by reviewing the supplied context and selecting Design or Analysis mode. If blocking evidence is missing, request it in one consolidated list. Otherwise, build the evidence inventory and complete the evaluation in the specified order.
Completion criteria
Complete when:
- Every evaluated finding references a specific query, retrieved passage, answer claim, citation, or supplied execution record.
- Material claims are classified as supported, partially supported, contradicted, unsupported, or unverifiable.
- Corpus, retrieval, context, generation, citation, and abstention failures are scored or recorded separately.
- Aggregate results identify sample size, sampling method, missing data, and uncertainty or evaluator-disagreement limits.
- Each recommended change has a corresponding test, expected result, and regression or safety guardrail.
- The release recommendation clearly distinguishes observed evidence from unrun tests and unresolved gaps.
Was this useful?
Explore related Workflows
Browse WorkflowsImplement a Retrieval-Grounded Support Triage Pilot
Implement an approved knowledge-assistant architecture, integrate it into bounded support triage in sandbox or shadow mode, evaluate retrieval/citation/abstention quality, and prepare a controlled change decision.
Related Prompts
Browse PromptsAutomate KPI Reporting from Approved Metric Definitions
Implement a repeatable KPI-reporting process from approved metric definitions, with traceable calculations, authoritative reconciliation, failure and rerun tests, disabled delivery, and recovery evidence.
Analytical Conclusion Sensitivity Review
Test whether a consequential analytical conclusion survives plausible changes to data, cohort, definitions, assumptions, model choices, and missing-information treatment.
Data Lineage Break Investigation
Reconstruct where a data product diverged from authoritative lineage, bound affected outputs and decisions, and define safe repair and reprocessing.
Forecast Assumption and Model Drift Challenge
Challenge forecast assumptions, structural stability, backtest evidence, scenario sensitivity, and decision thresholds before relying on projected outcomes.
Knowledge Entitlement Drift Review
Reconcile authoritative access policy with effective permissions across source, ingestion, index, cache, retrieval, citation, and response layers.
AI-Generated SQL Result Verification and Reconciliation
Validate AI-generated SQL and its reported results before they are used for a consequential decision.