Validate A/B Test Assignment Integrity

Required inputs

Have these details available before following the usage instructions.

• Experiment name, hypothesis, variants, expected allocation ratios, randomization unit, targeting rules, and eligibility criteria.
• Assignment, exposure, and analysis counts by variant, time period, platform, geography, user segment, and key filters when available.
• Experiment start and end times, ramp schedule, traffic allocation changes, holdouts, exclusions, and concurrent experiments.
• Telemetry and pipeline context: event names, joins, identity stitching, deduplication rules, bot/internal traffic handling, and late-arriving data behavior.
• Observed SRM or imbalance metrics, statistical test outputs if available, and any dashboards or query definitions used.
• Recent changes to experiment configuration, SDKs, routing, logging, consent handling, app versions, or data processing.
• Decision context: whether results are being used for launch, rollback, further analysis, or rerun planning.
• Statistical assumptions and analysis settings, including significance level, power or minimum detectable effect, multiple-testing treatment, variance assumptions, and planned analysis method.
• Predeclared stopping rules, sample or duration target, peeking policy, sequential design if any, and the accountable experiment owner and decision owner.

How to use this Skill

When to use:
- An A/B test or randomized experiment shows sample ratio mismatch, unexpected traffic imbalance, missing exposure events, or inconsistent variant counts.
- Experiment results are consequential and need assignment-integrity review before launch, rollback, or product decisions.
- You need to localize imbalance across assignment, exposure, eligibility filtering, identity resolution, logging, bot handling, analysis windows, or data pipeline joins.
- You need a decision gate for whether to trust, reanalyze, rerun, pause, or invalidate an experiment.

When not to use:
- Do not use for non-randomized observational analyses unless the task is explicitly to assess assignment-like cohort integrity.
- Do not use when the user only wants effect-size interpretation and there is no concern about allocation or telemetry validity.
- Do not use as a replacement for a statistician, experimentation platform owner, or data owner on high-stakes launch decisions.
- Do not use when no assignment counts, exposure logs, expected split, or analysis pipeline evidence can be provided.

Instructions:
1. Use the linked source prompt AMO-P-000266 as the structured investigation guide; provide the experiment evidence and ask the model to walk through allocation, assignment, exposure, telemetry, identity, and analysis stages without claiming to run platform checks.
2. Start with an evidence ledger that separates supplied facts, calculations, assumptions, inferences, missing data, and uncertainty.
3. Confirm the intended experimental design before diagnosing: randomization unit, allocation ratio, eligibility, exposure definition, analysis window, statistical assumptions, multiplicity treatment, stopping rule, and primary decision use.
4. Compare expected versus observed counts at multiple stages: targeting, assignment, exposure, event ingestion, identity resolution, filtering, metric computation, and final analysis population.
5. Localize the first divergence by slicing counts over time, platform, app version, geography, acquisition source, user type, consent state, and other plausible routing or logging boundaries.
6. Assess likely causes such as bucketing defects, ramp misconfiguration, bot filtering, event loss, duplicate exposure, identity merge errors, cache or CDN routing, late data, incompatible filters, or query bugs.
7. State whether the experiment evidence is usable, usable with limits, requires reanalysis, requires rerun, or should be held for investigation, with rationale and conditions. The accountable experiment owner must record the trust, launch, rollback, rerun, or invalidation decision.
8. Define follow-up checks, owner actions, monitoring, and prevention controls such as assignment/exposure reconciliation, SRM alerts, preflight QA, and invariant dashboards.

Expected output:
An experiment integrity review containing the design summary, expected-versus-observed allocation table, divergence localization, ranked SRM root-cause hypotheses, decision-use disposition, required reanalysis or rerun criteria, and prevention controls.

Constraints and boundaries:
- Do not treat a statistically significant treatment effect as trustworthy until assignment and exposure integrity are adequately addressed.
- Do not assert a root cause without tying it to counts, logs, configuration evidence, query logic, or clearly labeled inference.
- Do not recommend a launch, rollback, or irreversible product decision without human review when experiment evidence is materially compromised or consequential.
- Avoid exposing user-level data; use aggregated slices or anonymized examples unless authorized and necessary.
- Do not overfit to post-hoc segment patterns; label exploratory slices and require confirmatory checks.

Powered by an Amo.ng Prompt

Experiment Sample Ratio Mismatch Investigation

Open the linked prompt to use the instructions that power this Skill.

Completion criteria

Complete when:
- The output identifies the earliest supported stage where expected and observed counts diverge, or states that evidence is insufficient to locate it.
- All conclusions distinguish measured evidence from hypotheses and missing checks.
- The decision-use disposition is explicit: trust, trust with limits, reanalyze, rerun, pause, or invalidate.
- Recommended follow-up checks are concrete enough for data, product, or experimentation owners to execute.
- Human approval is required for consequential launch, rollback, customer-impacting, or public reporting decisions.