AI Trace Review and Failure Taxonomy

Public field comparison

AI Trace Review and Failure Taxonomy

AI Trace Review and Failure Taxonomy

Reconstruct AI and agent traces, classify primary and contributing failures, quantify patterns with sampling limits, and convert evidence into measurable prevention work.

Use ChatGPT to reconstruct evidence-supported AI and agent traces, govern failure classifications, analyze recurring patterns within sampling limits, identify observability gaps, and propose sanitized regression cases and measurable prevention work.

Turning AI traces and incidents into a governed failure taxonomy, evidence-backed root-cause patterns, observability improvements, and measurable prevention loops.

Turn AI and agent trace evidence into a governed failure taxonomy, defensible pattern analysis, observability improvements, and measurable prevention work.

AI Trace Reconstruction
Agent Failure Classification
Recurring Failure Pattern Analysis
AI Observability Gap Review
Trace-to-Evaluation Case Mining

AI Trace Reconstruction
Agent Failure Classification
Recurring Failure Pattern Analysis
AI Observability Gap Review
Trace-to-Evaluation Case Mining

Review objective, decisions, period, and definition of done
AI system architecture, environments, and component versions
Analysis unit, eligible population, and sampling design
Sanitized traces, spans, logs, metrics, and event schemas
User intents, expected outcomes, observed outcomes, and impact evidence
Prompt, context, memory, retrieval, and data-source artifacts
Model requests, observable responses, validations, and post-processing
Tool definitions, calls, approvals, results, side effects, and reconciliation evidence
Safeguard, policy, moderation, fallback, escalation, and human-review events
Latency, token, error, reliability, and cost definitions
Incident, support, evaluation, experiment, release, and change records
Privacy, retention, tenant-separation, and reviewer-access constraints
Existing failure taxonomy, owners, reporting needs, and review cadence
Allowed analyses, prohibited actions, and acceptance criteria

Review scope and decisions
System and telemetry map
Population and sampling design
Sanitized trace evidence
Outcome and incident evidence
Taxonomy and measurement definitions
Governance and privacy constraints
Acceptance criteria and authorized analyses

Provide ChatGPT with the review objective, analysis period, AI-system architecture, component versions, telemetry schema, population and sampling method, sanitized traces, user outcomes, retrieval and tool evidence, safeguard events, latency and cost definitions, existing taxonomy, and privacy constraints.

Start with a representative pilot sample. Ask ChatGPT to reconstruct the traces and propose the taxonomy before classifying the full authorized sample. Have two or more qualified reviewers calibrate ambiguous or high-impact labels where practical, then revise the inclusion and exclusion rules before wider use.

Do not upload secrets, raw credentials, unnecessary customer content, confidential prompts, or unrestricted tool payloads. Do not ask ChatGPT to recover hidden chain-of-thought. Verify all calculated rates, classifications, causal claims, remediation priorities, and taxonomy changes with accountable human owners.

Open ChatGPT and replace every bracketed variable in the prompt with the corresponding review scope, system map, sampling information, sanitized telemetry, outcome evidence, definitions, governance constraints, and acceptance criteria. Provide the access-approved trace exports, schema documentation, version records, incident or support evidence, reviewer labels, and authoritative outcome or reconciliation records relevant to the task. Remove secrets and unnecessary customer content before submission. Run the prompt first on a representative pilot sample; have qualified human reviewers examine ambiguous or high-impact classifications and independently verify calculations, side effects, priorities, and acceptance evidence before using the outputs for reporting or change decisions. ChatGPT is being asked to analyze supplied evidence, not to access telemetry systems, execute remediations, approve changes, or verify unavailable external state.

An AI operations team supplies 200 access-approved sampled traces from a customer-service agent, including model and prompt versions, retrieval metadata, tool-call states, safeguard events, user outcomes, support incidents, latency, token and cost data, sampling rules, and an existing informal error list. The team needs a governed taxonomy, recurring-failure analysis, observability backlog, and sanitized regression-evaluation cases.

An AI operations team has 200 access-approved sampled traces from a customer-service agent, plus model and prompt versions, retrieval metadata, tool authorization and invocation states, safeguard events, user outcomes, support incidents, latency and cost definitions, sampling rules, and an informal error list. The team runs the prompt in ChatGPT on a representative pilot to obtain proposed trace reconstructions and taxonomy rules, then asks qualified reviewers to calibrate disputed labels. After supplying the adjudicated evidence, the team uses the prompt to prepare a classified-case register, denominator-limited pattern analysis, observability-gap register, proposed remediation backlog, and sanitized regression-case candidates. No production change, external side effect, reviewer agreement, or successful evaluation is treated as completed without separate evidence.

Expert

Expert

ChatGPT

ChatGPT

analysis

analysis

ai-observability
distributed-tracing
failure-taxonomy
agent-evaluation
tool-calling
root-cause-analysis
telemetry
llmops
incident-analysis
chatgpt

ai-observability
distributed-tracing
failure-taxonomy
agent-evaluation
tool-calling
telemetry
incident-analysis
llmops
chatgpt

AI Trace Review and Failure Taxonomy Prompt

AI Trace Review and Failure Taxonomy Prompt

Reconstruct AI and agent traces, classify failure mechanisms, account for sampling gaps, and convert recurring patterns into measurable prevention work.

Reconstruct AI traces, govern failure labels, account for sampling gaps, and propose evidence-backed observability and evaluation improvements.

Prompt-body line comparison

Removed Added Unchanged context

−You are a senior AI observability and quality analyst experienced in distributed tracing, agent workflows, retrieval systems, tool execution, failure classification, evaluation design, privacy-preserving telemetry, and remediation analytics.
−
−Reconstruct the supplied AI or agent traces, identify what the observable evidence supports, classify failures using a stable and useful taxonomy, quantify recurring patterns within the limits of the sampling design, and convert the findings into prioritized prevention work.
−
−Produce a review boundary, telemetry inventory, trace reconstruction set, governed failure taxonomy, classified case register, pattern analysis, observability-gap register, remediation backlog, evaluation feedback loop, and taxonomy-governance plan.
−
−A trace records observable system activity. It does not automatically reveal hidden internal reasoning, prove causation, capture every operation, confirm a downstream side effect, or represent the complete production population.
−
−Do not present an inspection, query, reconstruction, classification, calculation, test, remediation, approval, or outcome as completed unless its actual evidence is supplied.
−
−## Context to Provide
−
−Replace every bracketed placeholder. If a blocking input is missing, ask for it in one consolidated list before reaching conclusions. Continue with clearly labelled assumptions only when missing information is non-blocking.
−
−- [Review objective, decisions, period, and definition of done]
−- [AI system architecture, environments, and component versions]
−- [Analysis unit, eligible population, and sampling design]
−- [Sanitized traces, spans, logs, metrics, and event schemas]
−- [User intents, expected outcomes, observed outcomes, and impact evidence]
−- [Prompt, context, memory, retrieval, and data-source artifacts]
−- [Model requests, observable responses, validations, and post-processing]
−- [Tool definitions, calls, approvals, results, side effects, and reconciliation evidence]
−- [Safeguard, policy, moderation, fallback, escalation, and human-review events]
−- [Latency, token, error, reliability, and cost definitions]
−- [Incident, support, evaluation, experiment, release, and change records]
−- [Privacy, retention, tenant-separation, and reviewer-access constraints]
−- [Existing failure taxonomy, owners, reporting needs, and review cadence]
−- [Allowed analyses, prohibited actions, and acceptance criteria]
−
−## Evidence and Privacy Rules
−
−- Separate confirmed observations, derived calculations, assumptions, hypotheses, unknowns, risks, recommendations, and verified results.
−- Do not request, expose, reconstruct, or infer hidden chain-of-thought or private model reasoning.
−- Use only observable artifacts such as messages, retrieved references, tool selections, validation results, policy signals, structured outputs, timestamps, status codes, and supplied outcome evidence.
−- Do not invent traces, spans, timestamps, versions, prompts, retrieval results, tool activity, side effects, costs, incidents, users, owners, approvals, or test results.
−- Record each material source with its identifier, schema or convention version, environment, component version, time window, sampling status, and known limitations.
−- Treat a missing span or event as missing evidence, not proof that the operation did not occur.
−- Treat a successful tool response as evidence of a response, not automatic proof that the intended external side effect occurred correctly.
−- Preserve conflicting timestamps, statuses, outcomes, and versions until a discriminating check is available.
−- Account for asynchronous operations, parallel branches, queues, retries, fallbacks, trace links, clock skew, and out-of-order arrival before constructing chronology.
−- Do not assume telemetry field names or generative-AI semantic conventions are stable. Record the emitted schema and convention version and map fields explicitly.
−- Minimize prompt, completion, retrieval, and tool content. Prefer identifiers, hashes, classifications, derived attributes, or redacted excerpts when full content is unnecessary.
−- Do not expose credentials, tokens, cookies, personal data, customer records, confidential prompts, private endpoints, or sensitive tool payloads.
−- Enforce supplied tenant, retention, regional, access, and purpose limitations.
−- Use `Not provided`, `Not captured`, `Not sampled`, `Not linked`, `Not verified`, `Not assessable`, or `Owner decision required` when evidence is unavailable.
−- Tie every recommendation to a finding, affected cases, accountable owner, verification method, observable acceptance condition, and rollback or retirement path.
−
−## Review Boundary and Denominator
−
−Define the unit being analyzed before calculating any rate. Examples include:
−
−- user request;
−- conversation turn;
−- task run;
−- trace;
−- span;
−- model call;
−- retrieval operation;
−- tool call;
−- workflow;
−- user session;
−- incident.
−
−Record:
−
−- eligible population;
−- included and excluded cases;
−- review period;
−- sampling method;
−- sample size;
−- sampling probability where known;
−- dropped, unsampled, corrupt, incomplete, or inaccessible traces;
−- stratification or weighting;
−- known coverage differences by environment, version, customer segment, failure type, or severity;
−- duplicate and retry treatment;
−- denominator used for every reported rate.
−
−If the eligible population or sampling process is unknown, report observed counts and proportions within the reviewed set. Do not present them as production failure rates.
−
−Do not let frequency alone determine priority. Consider severity, affected users, reversibility, detectability, recurrence, exposure duration, and confidence.
−
−## System and Telemetry Map
−
−Map the relevant observable stages:
−
−1. Request ingress and authentication.
−2. Intent recognition and routing.
−3. Prompt-template selection and variable assembly.
−4. Memory and conversation-context assembly.
−5. Retrieval query construction, filtering, ranking, and document selection.
−6. Model request and observable response.
−7. Structured-output parsing, validation, post-processing, and formatting.
−8. Safeguard, moderation, policy, and approval decisions.
−9. Tool discovery, selection, authorization, invocation, response, and reconciliation.
−10. Human review, fallback, escalation, or handoff.
−11. User-visible response.
−12. Downstream action, record, message, transaction, or state change.
−13. User, business, safety, operational, or support outcome.
−
−For each stage, record:
−
−- component and owner;
−- environment and version;
−- trace, span, event, metric, and log sources;
−- identifiers and correlation method;
−- captured attributes;
−- content-capture policy;
−- sampling and retention;
−- known blind spots;
−- authoritative outcome source.
−
−Do not force every system into this sequence. Preserve actual branching, concurrency, loops, retries, and asynchronous work.
−
−## Trace Reconstruction Model
−
−For every reviewed case, reconstruct the smallest evidence-supported chronology.
−
−Record:
−
−- case and trace identifier;
−- task or user intent;
−- expected outcome;
−- environment and component versions;
−- start and end time;
−- parent-child or linked-span relationships;
−- ordered stages, spans, and events;
−- retry and attempt numbers;
−- parallel or asynchronous branches;
−- input and output references;
−- status, error, timeout, cancellation, or fallback;
−- safeguard and approval decisions;
−- user-visible output;
−- downstream side-effect evidence;
−- user or business outcome;
−- missing evidence and trace-completeness assessment.
−
−Distinguish:
−
−- recorded fact;
−- derived timing or calculation;
−- inferred transition;
−- unverified claim;
−- missing evidence;
−- contradictory evidence.
−
−For tool activity, distinguish these states:
−
−1. Tool was available.
−2. Tool was selected or proposed.
−3. Call arguments were produced.
−4. Authorization or approval was requested.
−5. Authorization was granted or denied.
−6. Invocation was attempted.
−7. A response or error was returned.
−8. The intended side effect was independently confirmed.
−9. Retry or compensation occurred.
−10. Final state was reconciled.
−
−Do not collapse these states into a single “tool succeeded” label.
−
−## Failure Taxonomy Architecture
−
−Build a hierarchical, versioned taxonomy that separates different analytical dimensions.
−
−### Outcome Status
−
−Classify the final task outcome as:
−
−- `Successful`
−- `Partially successful`
−- `Failed safely`
−- `Failed unsafely`
−- `Abandoned or timed out`
−- `Escalated`
−- `Outcome unknown`
−- `Not assessable`
−
−Define outcome status using supplied user, business, safety, and system expectations.
−
−### Failure Stage
−
−Identify where the first supported failure became observable:
−
−- input or intent;
−- routing or orchestration;
−- context or memory;
−- retrieval;
−- model generation;
−- structured-output validation;
−- safeguard or policy;
−- tool authorization;
−- tool execution;
−- downstream side effect;
−- human review or escalation;
−- presentation or delivery;
−- infrastructure or provider;
−- observability;
−- user or business outcome;
−- unresolved.
−
−### Failure Mechanism
−
−Define the causal mechanism supported by evidence, such as:
−
−- ambiguous or unsupported request;
−- incorrect routing;
−- missing, stale, excessive, conflicting, or truncated context;
−- retrieval miss, contamination, authorization error, ranking defect, or stale source;
−- unsupported model claim, instruction failure, refusal error, format defect, or non-deterministic behaviour;
−- schema, parsing, validation, or post-processing failure;
−- policy false positive, false negative, inconsistent enforcement, or missing review;
−- incorrect tool selection, invalid arguments, insufficient authorization, timeout, retry defect, or non-idempotent execution;
−- downstream mapping, persistence, delivery, or reconciliation failure;
−- unavailable reviewer, incomplete handoff, or incorrect escalation;
−- deployment, capacity, queue, network, provider, cache, or configuration failure;
−- incomplete, corrupt, unlinked, oversampled, or undersampled telemetry;
−- unresolved competing mechanisms.
−
−Use organization-specific subcategories where supplied evidence supports them.
−
−### Symptom
−
−Record what the user or system experienced, such as:
−
−- incorrect answer;
−- unsupported claim;
−- missing answer;
−- unnecessary refusal;
−- unsafe output;
−- wrong tool action;
−- duplicate action;
−- incomplete action;
−- delayed response;
−- excessive cost;
−- broken format;
−- missing citation;
−- failed escalation;
−- inconsistent result;
−- unavailable service.
−
−Do not use the symptom as the root cause.
−
−### Contributing Factors
−
−Allow multiple contributing factors, including:
−
−- prompt design;
−- data quality;
−- missing version metadata;
−- context truncation;
−- stale retrieval index;
−- permission design;
−- ambiguous tool contract;
−- timeout budget;
−- retry policy;
−- concurrency;
−- experiment assignment;
−- insufficient validation;
−- observability gap;
−- human-process gap;
−- vendor dependency.
−
−### Impact and Severity
−
−Assess severity using supplied criteria covering:
−
−- user harm;
−- safety;
−- privacy or security;
−- financial or operational consequence;
−- affected scale;
−- exposure duration;
−- reversibility;
−- detectability;
−- recurrence;
−- regulatory or contractual relevance requiring qualified review.
−
−Do not invent legal or regulatory conclusions. Assign them to the appropriate human owner.
−
−### Recoverability
−
−Classify recovery as:
−
−- automatic retry;
−- automatic fallback;
−- user retry;
−- human correction;
−- compensation or reconciliation;
−- rollback;
−- data repair;
−- irreversible;
−- unknown.
−
−### Confidence
−
−Use:
−
−- `High`: direct, consistent evidence demonstrates the mechanism.
−- `Moderate`: evidence supports the mechanism, but a material gap remains.
−- `Low`: several mechanisms remain plausible.
−- `Unresolved`: available evidence cannot discriminate between competing mechanisms.
−
−Do not force a primary mechanism when confidence is insufficient.
−
−## Multi-Label and Causal Rules
−
−For every failed or degraded case, assign:
−
−- one primary failure mechanism when supported;
−- zero or more contributing factors;
−- zero or more downstream symptoms;
−- impact and severity;
−- detection source;
−- recoverability;
−- primary accountable system or process owner;
−- classification confidence.
−
−Distinguish:
−
−1. initiating failure;
−2. propagation or amplification;
−3. visible symptom;
−4. detection event;
−5. recovery attempt;
−6. final outcome.
−
−Do not assign several competing mechanisms as simultaneous primary causes. Keep them as unresolved hypotheses until evidence discriminates between them.
−
−A taxonomy label should have:
−
−- stable code;
−- name;
−- parent category;
−- definition;
−- inclusion criteria;
−- exclusion criteria;
−- positive example;
−- near-miss or counterexample;
−- default owner;
−- related evaluation;
−- introduction version;
−- retirement or replacement mapping.
−
−## Failure Modes to Test
−
−Treat these as hypotheses until supported:
−
−- incomplete or sampled telemetry creates a false single-cause narrative;
−- trace identifiers fail to propagate across services, queues, tools, or vendors;
−- clock skew or out-of-order arrival creates an incorrect chronology;
−- retries, fallbacks, or recovery attempts hide the initiating failure;
−- one user symptom receives multiple overlapping primary labels;
−- stage, mechanism, symptom, severity, and owner are conflated;
−- tool-response success is mistaken for confirmed external-state success;
−- retrieval, model, policy, tool, and presentation failures are grouped together;
−- taxonomy categories are too broad to guide remediation or too narrow to compare over time;
−- redaction removes necessary diagnostic attributes;
−- logging captures excessive sensitive content;
−- model, prompt, index, policy, tool, experiment, or deployment versions are missing;
−- observed frequency is biased by sampling, detectability, support reporting, or trace loss;
−- common low-impact errors overshadow rare severe failures;
−- resolved incidents are absent from the reviewed trace sample;
−- cases without telemetry are silently excluded from the denominator;
−- a correlation with a version or segment is described as causal without a controlled comparison.
−
−For every material hypothesis, state:
−
−- predicted signal;
−- evidence for;
−- evidence against;
−- missing evidence;
−- affected cases;
−- confidence;
−- cheapest safe discriminating check;
−- result that would confirm or reject it.
−
−## Pattern and Measurement Rules
−
−Analyze only dimensions supported by the supplied data:
−
−- failure count and rate;
−- severity-weighted occurrence;
−- affected user or task segment;
−- model, prompt, retrieval, policy, tool, and deployment version;
−- environment and experiment assignment;
−- first occurrence and recurrence;
−- detection source and detection delay;
−- recovery path and recovery success;
−- end-to-end and stage latency;
−- time to first response or first chunk;
−- queue, retrieval, model, tool, validation, and human-review latency;
−- input, output, cached, and other supplied token categories;
−- tool and infrastructure usage;
−- cost per request, successful outcome, or failure;
−- user feedback, support contact, correction, abandonment, or repeat request;
−- trace and logging coverage.
−
−For every metric:
−
−- define the numerator and denominator;
−- state the unit;
−- identify the authoritative source;
−- state whether it is observed, derived, estimated, or unavailable;
−- identify sampling and missing-data limitations;
−- record the relevant pricing or rate-card version for monetary cost;
−- avoid percentage or percentile claims when the sample cannot support them.
−
−Use medians and tail percentiles only when the sample size and measurement method support them. Do not hide severe outliers behind averages.
−
−Treat correlations as hypotheses. Use controlled comparisons, matched cases, change timelines, or targeted evaluations before claiming causation.
−
−## Remediation and Evaluation Feedback Loop
−
−Connect every prioritized pattern to the narrowest appropriate intervention:
−
−- prompt or instruction change;
−- context or memory correction;
−- retrieval indexing, ranking, filtering, authorization, or freshness improvement;
−- model, routing, or fallback change;
−- structured-output or validation improvement;
−- safeguard, escalation, or human-review change;
−- tool schema, authorization, idempotency, timeout, retry, or reconciliation improvement;
−- product or user-experience change;
−- infrastructure, capacity, cache, queue, or provider improvement;
−- tracing, logging, metrics, sampling, retention, or alerting improvement;
−- policy, ownership, training, or operational-process improvement.
−
−For every remediation, define:
−
−- supporting cases and pattern;
−- responsible owner;
−- proposed change;
−- expected mechanism;
−- affected users and systems;
−- regression and unintended-effect risks;
−- evaluation cases;
−- telemetry needed;
−- success and failure measures;
−- approval gate;
−- rollback or retirement path;
−- review date.
−
−Convert suitable severe, recurring, or previously undetected failures into sanitized evaluation cases. Preserve the original failure mechanism without retaining unnecessary sensitive content.
−
−Keep historical failure cases separate from held-out evaluation data where contamination would invalidate measurement.
−
−## Taxonomy Governance
−
−Define:
−
−- taxonomy owner and reviewers;
−- version identifier and effective date;
−- stable codes and naming rules;
−- hierarchy and allowed multi-label relationships;
−- inclusion and exclusion criteria;
−- reviewer guidance;
−- adjudication process;
−- calibration sample;
−- disagreement tracking;
−- additions, merges, splits, deprecations, and retirements;
−- mappings between old and new versions;
−- backfill policy;
−- reporting comparability notes;
−- privacy and access review;
−- review cadence;
−- drift and usefulness measures.
−
−Do not silently rewrite historical classifications. Preserve the original taxonomy version and add mapped classifications where authorized.
−
−## Workflow
−
−1. Confirm the review objective, decisions, analysis unit, population, period, sampling design, privacy boundary, owners, and definition of done.
−2. Inventory architecture, telemetry schemas, semantic-convention versions, environments, component versions, identifiers, sampling, retention, and known blind spots.
−3. Validate trace integrity, correlation, parent-child relationships, links, event order, retries, branches, clock assumptions, and missing spans.
−4. Reconstruct each selected case chronologically without inferring hidden reasoning or treating missing telemetry as absence.
−5. Determine task outcome, first observable divergence, initiating failure, propagation, symptom, detection, recovery, and final outcome.
−6. Draft the taxonomy with stable codes, definitions, inclusion and exclusion rules, examples, severity, recoverability, ownership, and confidence.
−7. Calibrate the taxonomy on a representative pilot sample. Record reviewer disagreements and revise ambiguous categories.
−8. Classify the authorized sample using primary, contributing, and downstream labels.
−9. Analyze patterns using explicit denominators, sampling limitations, version dimensions, severity, latency, cost, detection, and recovery evidence.
−10. Connect findings to incidents, evaluations, support records, releases, experiments, and existing remediation work.
−11. Prioritize product, prompt, retrieval, model, tool, safeguard, infrastructure, observability, and operational changes.
−12. Create sanitized evaluation cases and measurable follow-up for prioritized failure mechanisms.
−13. Version the taxonomy, preserve mappings, define governance, and schedule calibration and drift review.
−14. End with the smallest safe next action that materially reduces uncertainty or recurring risk.
−
−## Decision and Safety Controls
−
−- Do not request or infer hidden chain-of-thought.
−- Do not collect raw prompts, outputs, retrieval passages, tool payloads, identities, or customer records unless explicitly approved and necessary.
−- Do not treat missing traces or spans as successful execution.
−- Do not treat a returned tool result as confirmed downstream state without reconciliation evidence.
−- Do not force a single root cause when evidence supports unresolved competing mechanisms.
−- Do not rank individual employees or reviewers from trace activity.
−- Do not compare failure rates across versions, segments, or periods without compatible denominators and telemetry coverage.
−- Do not weaken safeguards, validation, escalation, or logging merely to improve apparent success, latency, or cost.
−- Require human review for severe safety, privacy, security, financial, legal, regulatory, or other high-impact cases.
−- Require approval before changing production telemetry, retention, sampling, access, routing, models, prompts, tools, policies, or customer-facing behaviour.
−- Keep taxonomy recommendations separate from authorization to implement changes.
−- Do not mutate production systems or external services.
−
−## Output Contract
−
−Use concise markdown. Use tables for chronology, comparison, classification, ownership, and measurement evidence.
−
−### 1. Review and Data Boundary
−
−State:
−
−- objective and decisions;
−- analysis unit and period;
−- population and sample;
−- environments and versions;
−- supplied artifacts;
−- privacy and retention boundary;
−- allowed analyses;
−- missing inputs;
−- assumptions;
−- definition of done.
−
−### 2. Telemetry and Coverage Inventory
−
−Provide:
−
−| Stage or component | Telemetry source | Identifiers and links | Schema or convention version | Sampling and retention | Sensitive-content handling | Known gap | Owner |
−|---|---|---|---|---|---|---|---|
−
−### 3. Sample and Denominator Ledger
−
−Provide:
−
−| Population or stratum | Eligible cases | Reviewed cases | Sampling method | Missing or excluded cases | Weighting | Permitted inference | Limitation |
−|---|---:|---:|---|---|---|---|---|
−
−Do not populate unavailable counts with estimates unless an approved estimation method is supplied.
−
−### 4. Trace Reconstruction Register
−
−Provide:
−
−| Case | Ordered stage or event | Time and duration | Parent, link, branch, or retry | Observable evidence | Derived or inferred detail | Status or outcome | Missing evidence |
−|---|---|---|---|---|---|---|---|
−
−### 5. Failure Taxonomy Specification
−
−Provide:
−
−| Code | Parent | Stage | Mechanism | Definition | Inclusion criteria | Exclusions | Example | Default owner | Version |
−|---|---|---|---|---|---|---|---|---|---|
−
−### 6. Classified Case Register
−
−Provide:
−
−| Case | Outcome status | Primary mechanism | Contributing factors | Symptoms | Severity | Recoverability | Detection | Confidence | Reviewer |
−|---|---|---|---|---|---|---|---|---|---|
−
−Mark unresolved primary mechanisms explicitly.
−
−### 7. Pattern and Impact Analysis
−
−For every material pattern, report:
−
−- observed count;
−- valid denominator;
−- rate where supported;
−- severity and impact;
−- affected segments and versions;
−- latency and cost evidence;
−- detection and recovery evidence;
−- sampling and observability limitations;
−- confidence;
−- alternative explanation.
−
−### 8. Observability Gap Register
−
−Provide:
−
−| Priority | Blind spot | Affected stages or cases | Diagnostic consequence | Privacy risk | Minimum necessary telemetry | Owner | Acceptance condition |
−|---:|---|---|---|---|---|---|---|
−
−### 9. Remediation and Evaluation Backlog
−
−Provide:
−
−| Priority | Pattern or failure | Proposed intervention | Owner | Evaluation case | Required telemetry | Success measure | Risk or trade-off | Approval gate |
−|---:|---|---|---|---|---|---|---|---|
−
−### 10. Governance and Smallest Safe Next Action
−
−State:
−
−- taxonomy version;
−- owner and reviewers;
−- calibration outcome;
−- unresolved disagreements;
−- mapping and backfill policy;
−- privacy and access controls;
−- reporting cadence;
−- drift triggers;
−- retest plan;
−- smallest safe next action;
−- owner, evidence expected, and completion condition.
−
−## Verification Checklist
−
−Before finalizing, confirm that:
−
−- the analysis unit, population, sample, and denominator are explicit;
−- sampling, missingness, and trace loss are reflected in every rate;
−- telemetry schemas and semantic-convention versions are recorded;
−- trace chronology preserves links, branches, retries, fallbacks, and recovery attempts;
−- clock assumptions and out-of-order evidence were considered;
−- observable evidence is distinguished from inferred internal state;
−- hidden chain-of-thought was neither requested nor inferred;
−- missing spans were not treated as evidence of successful execution;
−- tool availability, selection, authorization, invocation, response, side effect, and reconciliation were not conflated;
−- failure stage, mechanism, symptom, impact, owner, and recoverability remain distinct;
−- unresolved competing mechanisms were not converted into a confirmed root cause;
−- taxonomy labels have stable codes and clear inclusion and exclusion rules;
−- version comparisons use compatible telemetry coverage and denominators;
−- common failures do not hide rare severe outcomes;
−- sensitive telemetry is minimized, redacted, access-controlled, and retention-bounded;
−- historical cases remain traceable to the taxonomy version used;
−- remediations produce evaluation cases and measurable follow-up;
−- no unperformed analysis, unapproved change, or unresolved conflict is described as complete;
−- every major conclusion is supported by supplied evidence or explicitly labelled as an assumption.
−
−Begin by checking the supplied context for blocking gaps. If none remain, define the review boundary, analysis unit, population, sampling design, and telemetry inventory before reconstructing or classifying any trace.
+Analyze the supplied AI-system evidence to reconstruct observable traces, classify supported failure mechanisms, measure recurring patterns within the sampling design, identify observability gaps, and propose prevention and regression-evaluation work.
+
+Inputs
+- Review scope and decisions: [Review scope and decisions]
+- System and telemetry map: [System and telemetry map]
+- Population and sampling design: [Population and sampling design]
+- Sanitized trace evidence: [Sanitized trace evidence]
+- Outcome and incident evidence: [Outcome and incident evidence]
+- Taxonomy and measurement definitions: [Taxonomy and measurement definitions]
+- Governance and privacy constraints: [Governance and privacy constraints]
+- Acceptance criteria and authorized analyses: [Acceptance criteria and authorized analyses]
+
+Input handling
+1. First check whether the review objective, analysis unit, review period, eligible population, sample construction, trace identifiers, environment and component versions, outcome criteria, privacy boundary, and authorized analyses are sufficiently defined.
+2. If a missing input blocks reconstruction, classification, denominator selection, or safe handling, ask for all blocking items in one consolidated list and stop before drawing conclusions.
+3. If a gap is non-blocking, continue only after labeling the relevant statement as an assumption, hypothesis, unknown, unavailable measurement, or owner decision.
+4. Preserve conflicting timestamps, versions, statuses, labels, and outcomes. Do not resolve a conflict by preference; identify the evidence needed to discriminate between alternatives.
+5. Use only sanitized, access-approved material supplied in the conversation. Do not request secrets, credentials, unrestricted customer content, confidential prompts, private endpoints, or unnecessary tool payloads.
+
+ChatGPT operating boundary
+ChatGPT may organize, compare, classify, and calculate from the supplied material. It must not claim to have queried telemetry stores, opened unavailable files, inspected production, contacted providers, executed tools, confirmed external state, changed sampling, altered retention, deployed a fix, or run an evaluation unless direct evidence of that action is supplied. Do not request or infer hidden chain-of-thought.
+
+Use these work-status terms consistently:
+- Requested: an action or artifact was sought, but completion evidence was not supplied.
+- Proposed: a recommendation, taxonomy change, check, or evaluation that has not been authorized or performed.
+- Executed: contemporaneous evidence shows that an operation was attempted; this does not by itself prove its intended effect.
+- Unavailable: the required source or field was not supplied or accessible for this review.
+- Unverified: a claim, result, or side effect lacks independent confirmation.
+- Verified: the supplied authoritative source and acceptance evidence support the claim.
+
+Evidence rules
+- Separate supplied facts, direct observations, derived calculations, assumptions, hypotheses, unknowns, unsupported claims, conflicts, recommendations, and verified results.
+- Cite each material source by its supplied identifier, environment, time window, component version, schema or semantic-convention version, and sampling status where available.
+- Treat a missing span as missing evidence, not proof that an operation did not happen.
+- Account for parent-child links, trace links, retries, fallbacks, parallel branches, queues, asynchronous work, clock skew, and out-of-order arrival before asserting chronology.
+- Distinguish tool availability, selection, argument construction, approval request, approval result, invocation attempt, returned response or error, independently confirmed side effect, compensation, and final reconciliation. Never collapse these into a single tool-success claim.
+- Minimize sensitive content. Prefer identifiers, hashes, classifications, derived attributes, and redacted excerpts when full content is unnecessary.
+- Apply supplied tenant, purpose, region, access, retention, and reviewer restrictions. Refer high-impact privacy, security, safety, financial, legal, regulatory, or contractual judgments to accountable human owners.
+
+Focused workflow
+1. Define the review boundary: objective, decisions, analysis unit, period, population, sample, exclusions, privacy limits, environments, versions, and definition of done.
+2. Inventory telemetry by system stage, including identifiers, schemas, sampling, retention, content-capture policy, owners, authoritative outcome sources, and blind spots.
+3. Validate trace integrity and reconstruct the smallest evidence-supported chronology for each authorized case. Mark recorded events, derived timings, inferred transitions, contradictions, and missing evidence separately.
+4. Determine the expected outcome, observed outcome, first supported divergence, initiating failure, propagation, visible symptom, detection event, recovery attempt, and final outcome.
+5. Draft or refine a hierarchical, versioned taxonomy. Keep outcome status, failure stage, mechanism, symptom, contributing factors, severity, recoverability, owner, and confidence as separate dimensions.
+6. Calibrate the taxonomy on a representative pilot when reviewer labels are supplied. Record disagreements and ambiguous rules; do not invent reviewer consensus.
+7. Classify the authorized sample. Assign one primary mechanism only when evidence discriminates it from alternatives; otherwise mark it unresolved and retain competing hypotheses.
+8. Analyze recurring patterns using explicit numerators and denominators. Reflect sampling, trace loss, missingness, version coverage, detectability, duplicates, retries, and weighting in every rate or comparison.
+9. Prioritize patterns using severity, affected scale, exposure, reversibility, detectability, recurrence, and confidence rather than frequency alone.
+10. Convert supported findings into proposed observability changes, remediations, and sanitized evaluation cases. Keep historical evidence separate from held-out evaluation data when contamination would invalidate measurement.
+11. Finish with the smallest safe next action that reduces material uncertainty or recurring risk without changing production.
+
+Classification model
+Use these outcome states where applicable: Successful, Partially successful, Failed safely, Failed unsafely, Abandoned or timed out, Escalated, Outcome unknown, and Not assessable.
+
+Locate the first observable failure stage, such as input or intent, routing, context or memory, retrieval, model generation, validation, safeguard, tool authorization, tool execution, downstream side effect, human escalation, delivery, infrastructure, observability, or user outcome.
+
+Define a mechanism narrowly enough to guide prevention. Examples may include retrieval miss, stale context, unsupported model claim, format validation failure, policy false positive, invalid tool arguments, authorization failure, timeout, retry defect, non-idempotent execution, reconciliation failure, queue failure, configuration error, or incomplete telemetry. Treat these as candidate mechanisms, not findings, until supported.
+
+Use confidence levels:
+- High: direct and consistent evidence demonstrates the mechanism.
+- Moderate: evidence supports the mechanism, but a material gap remains.
+- Low: multiple mechanisms remain plausible.
+- Unresolved: available evidence cannot discriminate between competing mechanisms.
+
+Every taxonomy label must include a stable code, parent, name, definition, inclusion criteria, exclusions, positive example, near-miss, default owner, related evaluation, introduction version, and retirement or replacement mapping.
+
+Measurement rules
+For each metric, state the numerator, denominator, unit, authoritative source, derivation, missing-data treatment, sampling limitation, and whether the value is observed, calculated, estimated, unavailable, or unverified. Report reviewed-set counts or proportions rather than production rates when the eligible population or sampling probability is unknown. Use percentile, cost, version-comparison, or causal claims only when the supplied sample and definitions support them. Treat correlations as hypotheses and name the cheapest safe discriminating check.
+
+Required deliverable
+Return concise markdown with the following task-specific sections.
+
+1. Boundary and evidence status
+State the objective, decisions, analysis unit, period, population, sampling method, environments, versions, supplied sources, privacy limits, missing inputs, conflicts, assumptions, authorized analyses, and definition of done. Include a source ledger with evidence status and limitations.
+
+2. Telemetry coverage map
+Provide a table with: stage or component; owner; telemetry source; identifiers and links; schema or convention version; sampling and retention; sensitive-content handling; authoritative outcome source; known gap.
+
+3. Sample and denominator ledger
+Provide a table with: population or stratum; eligible cases; reviewed cases; exclusions or missing cases; sampling method; weighting; denominator; permitted inference; limitation. Do not estimate unavailable counts unless an approved method is supplied.
+
+4. Trace reconstruction register
+Provide a table with: case; ordered stage or event; time or duration; parent, link, branch, or retry; observable evidence; derived or inferred detail; tool-operation status where relevant; outcome; contradiction or missing evidence. Include a trace-completeness assessment for each case.
+
+5. Versioned failure taxonomy
+Provide a table with: code; parent; stage; mechanism; definition; inclusion criteria; exclusions; positive example; near-miss; default owner; related evaluation; version status.
+
+6. Classified case register
+Provide a table with: case; outcome status; first observable divergence; primary mechanism or unresolved hypotheses; contributing factors; symptoms; severity; recoverability; detection source; confidence; reviewer status; supporting evidence. Do not report calibration or consensus unless reviewer evidence is supplied.
+
+7. Pattern and impact findings
+For each material pattern, report observed count, valid denominator, supported rate, severity, affected segments and versions, latency or cost evidence, detection and recovery evidence, sampling and observability limitations, confidence, alternative explanations, and discriminating check. Distinguish association from causation.
+
+8. Observability gap register
+Provide a table with: priority; blind spot; affected cases or stages; diagnostic consequence; privacy risk; minimum necessary telemetry; proposed owner; approval required; observable acceptance condition.
+
+9. Prevention and evaluation backlog
+Provide a table with: priority; supporting finding and cases; proposed intervention; expected mechanism; proposed owner; sanitized evaluation case; telemetry required; success and failure measures; regression risk; approval gate; rollback or retirement path. Label all unperformed work Proposed.
+
+10. Governance and verification plan
+State the proposed taxonomy version, owner and reviewers, calibration requirement, unresolved disagreements, mapping and backfill policy, historical-label preservation rule, privacy and access controls, review cadence, drift triggers, and retest plan.
+
+End with:
+- the smallest safe next action;
+- accountable owner or owner decision required;
+- evidence expected;
+- completion condition;
+- current status using the defined work-status terms.
+
+Acceptance checks
+Before finalizing, verify from the supplied material that:
+- the analysis unit, population, sample, exclusions, and denominator are explicit;
+- sampling, missingness, and trace loss constrain every rate;
+- schemas, environments, and component versions are recorded when available;
+- chronology preserves links, branches, retries, fallbacks, asynchronous work, and clock uncertainty;
+- observations are separated from inference and hidden reasoning is neither requested nor inferred;
+- tool response, external side effect, and reconciliation are distinct;
+- stage, mechanism, symptom, impact, owner, and recovery are not conflated;
+- unresolved mechanisms remain unresolved;
+- taxonomy codes have inclusion and exclusion rules and a version history;
+- severe rare cases are not hidden by aggregate frequency;
+- each recommendation cites affected cases, an owner, approval gate, verification method, observable acceptance condition, and rollback or retirement path;
+- no inspection, calculation, classification, test, approval, deployment, fix, or outcome is described as completed without supplied evidence.
+
+If any acceptance check cannot be met, mark it Unavailable, Unverified, or Owner decision required and explain the narrowest evidence needed.