Regression Acceptance Threshold Calibration

Calibrate acceptance thresholds for an AI regression comparison before the threshold is used to approve a model, prompt, retrieval, routing, policy, or workflow change.

Provide:
- Exact release decision, protected behaviors, unacceptable failures, and risk tier: [Release decision and risk boundary]
- Metrics, rubrics, denominators, slices, aggregation, directionality, and scorer definitions: [Metrics slices and scoring rules]
- Current production or approved-baseline results, repeat runs, sampling uncertainty, scorer variance, and historical drift: [Baseline results and variance evidence]
- Candidate results if available, or the planned comparison design and sample allocation: [Candidate evidence or planned comparison]
- User, safety, compliance, operational, latency, and financial consequences of false acceptance and false rejection: [Operational impact and error costs]
- Sample, time, compute, policy, and deployment constraints plus evaluation and release owners: [Constraints owners and review cadence]

Do not choose thresholds merely because they match a convenient round number or a single historical score. Do not invent variance, confidence intervals, effect sizes, sample sizes, or error costs. Separate statistical detectability from practical acceptability. A strong aggregate result must not compensate for a material safety or protected-slice regression unless policy explicitly permits that trade.

Calibration method:

1. Define the decision contract.
   State what the threshold authorizes, the baseline being protected, acceptable degradation if any, non-compensable failures, and who owns the release decision.

2. Audit metric fitness.
   Check construct alignment, scoring reliability, denominator stability, missingness, boundedness, class imbalance, aggregation effects, and susceptibility to gaming. Mark metrics that cannot support a release threshold.

3. Characterize the baseline.
   Summarize supplied central tendency, dispersion, run-to-run variance, scorer disagreement, slice sample size, and temporal stability. Identify baselines too weak or unstable for calibration.

4. Price decision errors.
   Describe the consequence of accepting a harmful regression and rejecting a useful improvement. Use qualitative bands when credible monetary or impact estimates are unavailable.

5. Propose threshold types.
   Compare absolute floor, relative degradation budget, non-inferiority margin, confidence-bound rule, paired-case win/loss rule, zero-tolerance event rule, and operational stop rule. Use different rules for different metrics or slices when justified.

6. Check feasibility and power.
   Determine whether planned sample and repeated runs can distinguish the proposed margin from noise using supplied evidence. If not, recommend more evidence, a wider but risk-acceptable margin, a paired design, or a hold—not false precision.

7. Resolve aggregation and slice gates.
   Specify which gates are global, per slice, non-compensable, or monitored during canary. Address multiple comparisons and rare critical failures proportionately.

8. Produce a calibration decision.
   Recommend thresholds, rationale, evidence limits, re-estimation triggers, and an owner-approved exception path. Label provisional thresholds clearly.

The evaluation owner defines the evidence set, while the release owner approves any threshold used as a production gate; this analysis does not authorize release. For every proposed threshold, provide acceptance evidence with the expected observation, actual observation when available, sampling uncertainty, and reconciliation of false-accept and false-reject consequences. Do not claim calibration was executed unless the underlying results are supplied.

Record approval from the evaluation owner for the evidence set and approval from the release owner before the threshold becomes a production gate.

Required deliverable:

# Regression Acceptance Threshold Calibration

## Decision and Risk Contract
- Change being gated:
- Baseline protected:
- Non-compensable failures:
- False-acceptance consequence:
- False-rejection consequence:

## Metric Fitness Review
| Metric/slice | Intended construct | Reliability evidence | Noise/drift | Gaming risk | Fit for gate? |
|---|---|---|---|---|---|

## Baseline Variability Record
| Metric/slice | Baseline | Sample/repeats | Variance evidence | Stability | Limitation |
|---|---|---|---|---|---|

## Calibrated Gates
| Metric/slice | Gate type | Threshold/margin | Rationale | Sample rule | Compensable? | Owner |
|---|---|---|---|---|---|---|

## Release Interpretation
| Outcome | Decision | Canary/monitoring condition | Exception authority |
|---|---|---|---|

## Recalibration Triggers
List changes in traffic, rubric, judge, prevalence, model, prompt, retrieval, policy, or impact that invalidate the thresholds.

Completion requires a threshold for every material protected behavior, explicit treatment of uncertainty and slices, and a release interpretation that cannot hide a critical regression behind an aggregate score.