Adversarial Evaluation Coverage Review

Review whether adversarial evaluation evidence covers the credible abuse cases and failure modes of an AI system's actual production boundary. Focus on threat-to-test traceability, not the volume or novelty of prompts.

Evidence:
- Intended users, capabilities, actions, data, deployment context, and release claims: [System scope and release claims]
- Threat actors, assets, trust boundaries, misuse cases, failure hypotheses, and severity rationale: [Threat model and abuse hypotheses]
- Test cases, mutations, datasets, harness settings, results, failures, and reviewer notes: [Evaluation cases and execution records]
- Models, prompts, retrieval, memory, tools, identities, interfaces, filters, monitoring, and containment: [Architecture surfaces and controls]
- Incidents, near misses, abuse reports, support feedback, red-team findings, and observed attack variants: [Production incidents and feedback]
- Accepted risk, release threshold, policy constraints, security owner, evaluation owner, and release owner: [Risk tolerance and accountable owners]

Do not treat a test title as proof of coverage. Do not invent an attack result or claim that a control was bypassed unless execution evidence is supplied. Separate test design coverage from executed coverage, successful defense, failure detection, and containment. Avoid producing novel exploit instructions beyond what is necessary to describe a bounded test objective.

Coverage review:

1. Bound the evaluated system.
   Identify protected assets, allowed behavior, consequential actions, interfaces, user roles, data classes, and release claims. Flag architecture surfaces omitted from the supplied threat model.

2. Normalize threat hypotheses.
   Express each credible threat as actor/precondition, attack or failure path, target asset or invariant, expected harm, and observable success/failure condition. Consolidate duplicates without losing materially different preconditions.

3. Build a threat-to-test matrix.
   Map each hypothesis to test cases, variants, environments, system surfaces, evaluator/oracle, and execution evidence. Distinguish Direct, Partial, Proxy-only, Design-only, and Missing coverage.

4. Assess test strength.
   Review realism, adaptiveness, multi-turn behavior, encoded/indirect inputs, tool and retrieval paths, identity contexts, state persistence, rate effects, chained failures, and negative controls where relevant. Identify tests that exercise only a mock or obsolete configuration.

5. Evaluate oracle and control evidence.
   Determine whether the test can tell prevention, safe refusal, detection, containment, and recovery apart. Flag weak graders, ambiguous expected behavior, silent side effects, and controls whose operation is not observable.

6. Incorporate production learning.
   Map supplied incidents and feedback to existing hypotheses and tests. Identify recurring production failures absent from the evaluation or sanitized regression bank.

7. Prioritize gaps.
   Rank gaps by credible exposure, severity, control weakness, test feasibility, and release claim—not by arbitrary numeric scoring. Name the minimum test or evidence needed to close each gap.

8. Make a release-coverage decision.
   Choose Adequate for stated scope, Adequate with restrictions, Conditional on named tests, or Inadequate. State excluded claims and untested surfaces.

Keep testing within the authorized evaluation environment and supplied data boundary; do not probe production systems or real users. The security reviewer approves threat coverage and the release owner approves residual-risk acceptance. For each material threat, require acceptance evidence showing the expected observation, actual observation when tested, coverage status, and unresolved gap. Proposed adversarial tests are not executed results.

Required deliverable:

# Adversarial Evaluation Coverage Review

## System and Threat Boundary
| Asset/invariant | Surface | Actor/precondition | Consequence | Evidence source |
|---|---|---|---|---|

## Threat-to-Test Traceability Matrix
| Threat hypothesis | Test IDs | Variants/surfaces | Oracle | Execution evidence | Coverage status | Gap |
|---|---|---|---|---|---|---|

## Test-Strength Findings
| Test/group | Realism limitation | Obsolete/missing surface | Oracle weakness | Decision impact |
|---|---|---|---|---|

## Production Feedback Coverage
| Incident/feedback pattern | Existing test | Representation quality | Regression case needed | Owner |
|---|---|---|---|---|

## Priority Coverage Backlog
| Priority | Missing coverage | Minimum safe test | Required evidence | Owner | Release consequence |
|---|---|---|---|---|---|

## Coverage Decision
- Decision:
- Release scope supported:
- Restricted or unsupported scope:
- Tests required before release:
- Residual threat uncertainty:

Completion requires traceability for every material threat hypothesis, explicit separation of designed and executed coverage, and a release scope no broader than the tested system surfaces and oracles support.