Production Feedback-to-Evaluation Gap Analysis

Analyze whether material production feedback is represented accurately in the evaluations used to approve and monitor an AI system. Convert observed failures and corrections into a prioritized, privacy-safe evaluation maintenance decision.

Provide:
- Current system, release claims, user populations, task slices, and decisions supported by evaluation: [System release and evaluation scope]
- Sanitized incidents, complaints, reviewer corrections, escalations, overrides, support tickets, outcome signals, and trace summaries: [Production feedback and incident evidence]
- Evaluation datasets, test cases, rubrics, graders, slices, thresholds, run history, and known exclusions: [Current evaluation assets and results]
- Feedback sampling, severity, taxonomy, deduplication, triage, and follow-up methods: [Sampling taxonomy and triage rules]
- Consent, privacy, retention, rights, security, review capacity, and evaluation/release owners: [Privacy constraints and accountable owners]

Do not equate feedback frequency with prevalence unless collection supports that inference. Do not copy sensitive production content into tests without authorization and sanitization. Distinguish user preference, product defect, model failure, retrieval failure, policy issue, workflow issue, and measurement artifact. Do not claim that an evaluation would reproduce a failure unless execution evidence exists.

Analysis:

1. Bound the feedback population.
   Describe collection channels, period, exposed traffic, sampling biases, missing populations, deduplication, and severity rules. Identify feedback that cannot support prevalence estimates.

2. Normalize production signals.
   Group evidence into stable failure or correction patterns while preserving important context such as task, user, language, model, prompt, tool, retrieval, policy, and downstream consequence. Keep ambiguous cases unresolved.

3. Map signals to evaluation assets.
   For each material pattern, find the closest test case, slice, rubric criterion, grader behavior, and threshold. Classify representation as Direct and current, Partial, Proxy, Stale, Misweighted, Mislabelled, or Missing.

4. Diagnose the gap mechanism.
   Determine whether the gap comes from feedback capture, triage, privacy constraints, taxonomy, sampling, dataset curation, rubric design, grader behavior, test execution, thresholding, or release decision use.

5. Assess decision impact.
   Explain which release or monitoring claims become weaker because of each gap. Identify failures that were present in evaluation but hidden by aggregation or accepted thresholds.

6. Design safe evaluation updates.
   Specify the smallest representative case, slice, rubric change, grader calibration sample, threshold change, or monitoring link needed. Include provenance, sanitization, consent/rights, label ownership, and expiry or refresh rules.

7. Prioritize and assign.
   Rank by harm, recurrence evidence, claim impact, coverage absence, and feasible remediation. Do not manufacture a numeric priority score when inputs do not support one.

8. Define closure.
   Require a trace from production evidence to approved sanitized case, executable result, release interpretation, and recurrence monitoring.

The privacy or data owner must approve use of production-derived feedback, the evaluation owner approves benchmark changes, and the release owner retains authority over deployment gates. Do not copy sensitive records outside the authorized evidence boundary or treat this analysis as approval to change production evaluation policy.

Required deliverable:

# Production Feedback-to-Evaluation Gap Analysis

## Feedback Population and Limitations
- Period and traffic represented:
- Channels:
- Sampling/deduplication method:
- Biases and missing populations:

## Production Signal Taxonomy
| Pattern | Evidence count/status | Context/slice | Consequence | Confidence | Prevalence claim allowed? |
|---|---|---|---|---|---|

## Feedback-to-Evaluation Map
| Production pattern | Existing asset | Representation | Gap mechanism | Release/monitoring impact | Owner |
|---|---|---|---|---|---|

## Evaluation Maintenance Backlog
| Priority | Update | Source evidence | Sanitization/rights check | Acceptance test | Decision affected | Owner |
|---|---|---|---|---|---|---|

## Decision Summary
- Evaluation claims still supported:
- Claims narrowed or suspended:
- Updates required before next release:
- Signals retained for monitoring only:
- Unresolved uncertainty:

Completion requires every high-severity production pattern to be mapped or explicitly excluded with rationale, and every new evaluation asset to retain safe provenance back to the production learning it represents.