Model Fallback Failure Analysis

Analyze a production failure in which traffic was routed from a primary model or configuration to a fallback and the fallback did not preserve the required service contract.

Evidence to provide:
- Routing rules, triggers, priorities, circuit breakers, and fallback chain: [Routing and fallback policy]
- Required and observed capabilities, context limits, schemas, tool behavior, safety controls, and response contracts: [Primary and fallback model contracts]
- Timestamped routing decisions, requests, responses, errors, retries, and downstream effects: [Incident traces and outputs]
- Relevant offline evaluations, canary results, service metrics, user feedback, and accepted thresholds: [Evaluation and operational evidence]
- Data residency, privacy, security, policy, tool-access, and contractual constraints: [Data tool and compliance constraints]
- Repair, restriction, disablement, and communication options with accountable owners: [Recovery options and owners]

Do not assume that a fallback is a drop-in substitute because it accepts the same request shape. Separate routing evidence, model behavior, integration behavior, and downstream handling. Do not claim a live reproduction or test unless results are supplied. Label missing route decisions, hidden provider behavior, and unobserved failures as unknown.

Analysis:

1. Restate the service contract.
   Define the minimum quality, safety, structured-output, tool-use, latency, availability, data-boundary, and observability requirements that every fallback route must preserve. Distinguish mandatory invariants from degradable features.

2. Reconstruct the fallback path.
   Identify the trigger, routing decision, model/config selected, request transformation, context truncation, tool or schema adaptation, response validation, retries, and final downstream action. Mark the first point where evidence diverges from expected behavior.

3. Classify incompatibilities.
   Examine capability gaps, prompt/config differences, unsupported tools, schema drift, context loss, safety-policy differences, modality gaps, tokenizer or stop behavior, latency budget exhaustion, data-location conflicts, and error-normalization problems. Distinguish confirmed causes from plausible contributors.

4. Assess blast radius.
   Determine affected traffic slices, time window, user groups, task types, regions, and downstream systems using supplied evidence. Identify silent failures, incorrect success signals, unsafe outputs, repeated side effects, and records needing revalidation.

5. Evaluate controls.
   Review pre-route eligibility, health checks, compatibility tests, response validation, confidence or abstention rules, circuit breaking, observability, and disablement controls. Explain which control should have detected or contained the failure and why it did not.

6. Compare dispositions.
   Assess Repair and re-enable, Restrict fallback to compatible slices, Add an intermediate degraded mode, Route to manual handling, or Disable fallback. For each, specify evidence prerequisites, residual risk, user impact, and rollback.

7. Define proof before re-enable.
   Create task- and slice-specific compatibility tests, negative tests, schema/tool checks, canary gates, monitoring thresholds, and an owner decision. Avoid claiming equivalence beyond tested slices.

Treat this as model-routing incident analysis and fallback-design evidence for release readiness, not as permission to alter routing. The service owner and release owner must approve any re-enablement or production change. Require regression-testing acceptance evidence that names each fallback contract check, its expected observation, the actual observation when supplied, and any unreconciled reliability gap. Proposed repairs and tests remain not executed unless their results are provided.

Required deliverable:

# Model Fallback Failure Analysis

## Service Contract
| Requirement | Mandatory/degradable | Primary behavior | Required fallback behavior | Evidence |
|---|---|---|---|---|

## Fallback Reconstruction
| Sequence | Routing or transformation event | Expected | Observed | Evidence | Confidence |
|---:|---|---|---|---|---|

## First Divergence and Cause Tree
- First evidenced divergence:
- Confirmed causes:
- Contributing conditions:
- Unresolved hypotheses:

## Compatibility and Control Gaps
| Gap | Affected slice | Consequence | Existing control | Control failure | Remediation |
|---|---|---|---|---|---|

## Blast Radius
| Slice/window | Exposure evidence | Failure mode | Downstream action | Revalidation needed |
|---|---|---|---|---|

## Fallback Disposition
- Decision: Re-enable / Restrict / Degraded mode / Manual route / Disable
- Allowed scope:
- Preconditions:
- Accountable service and release owners:
- Rollback trigger:
- Residual uncertainty:

## Re-Enablement Gate
| Test or control | Slice | Acceptance threshold | Evidence owner | Status |
|---|---|---|---|---|

Completion requires a trace-supported first divergence, bounded blast radius, an explicit fallback disposition, and compatibility evidence for every slice proposed for re-enablement.