Retrieval Chunking and Metadata Experiment Design

Design a controlled retrieval experiment to isolate how chunking choices and metadata choices affect retrieval and answer quality. Keep the scope limited to corpus configuration decisions: do not redesign the full RAG architecture, diagnose the whole pipeline, change model selection, or introduce unrelated retrieval changes unless they are explicitly controlled constants.

Context to use:
- Corpus description and sample documents: [Corpus description and sample documents]
- Current retrieval setup: [Current retrieval setup]
- Candidate chunking options: [Candidate chunking options]
- Candidate metadata fields: [Candidate metadata fields]
- Query logs or benchmark questions: [Query logs or benchmark questions]
- Answer evaluation standard: [Answer evaluation standard]
- Decision owners and constraints: [Decision owners and constraints]

Evidence discipline:
- Separate observations supported by the provided inputs from inferences and assumptions.
- Do not claim that experiments, tests, retrieval runs, indexing jobs, or evaluations were completed unless results are provided.
- If required information is missing, state the missing evidence and design the smallest reasonable way to obtain it.
- Preserve uncertainty where the available evidence does not justify a firm conclusion.
- Treat the retrieval owner as accountable for experiment execution, the data owner as accountable for corpus and metadata validity, and the product owner or domain owner as accountable for acceptance criteria tied to user impact.

Produce the following deliverable:

1. Experiment objective and decision boundary
- State the configuration decision this experiment will support.
- Define what is in scope: chunking strategy, overlap, boundary rules, parent-child or hierarchical chunking if relevant, metadata extraction, metadata normalization, and metadata use in filtering or ranking.
- Define what is out of scope and should be held constant: embedding model, reranker, generation prompt, answer model, UI behavior, permissions, and production rollout unless the inputs require otherwise.
- Identify the baseline configuration and the candidate configurations to compare.

2. Controlled variables and constants
Create a table with:
- Variable under test
- Candidate levels
- Why it matters
- Expected retrieval effect
- Required implementation evidence
- What must remain constant

Include at minimum:
- Chunk size or token range
- Chunk overlap
- Chunk boundary rule
- Structural handling of headings, tables, lists, or sections
- Metadata fields to attach
- Metadata normalization rules
- Metadata use pattern: display only, filter, boost, grouping, or attribution

3. Factorial experiment matrix
Build a practical factorial or fractional-factorial matrix that isolates chunking and metadata effects without creating unnecessary runs.
For each experiment arm include:
- Arm ID
- Chunking configuration
- Metadata configuration
- Retrieval settings held constant
- Indexing requirements
- Expected comparison value
- Risk of confounding
- Minimum evidence needed before execution

If the full factorial design is too large, propose a staged design:
- Screening stage to remove weak options
- Focused comparison stage for finalists
- Confirmation stage against the baseline

4. Dataset and query slices
Define the evaluation dataset and query slices needed to detect meaningful differences.
Include:
- Document sample selection rules
- Minimum corpus coverage requirements
- Query source: logs, expert-written questions, synthetic-but-reviewed questions, or known support cases
- Query slice taxonomy
- Required number of queries per slice if enough information is available; otherwise give a sizing rule

Use query slices such as:
- Exact lookup questions
- Multi-section synthesis questions
- Long-tail entity questions
- Recent or version-specific questions
- Table or list extraction questions
- Procedure or policy questions
- Ambiguous terminology questions
- Metadata-dependent questions
- Queries where the answer should not be in the corpus

For each slice, specify the retrieval failure mode it is intended to expose.

5. Relevance judgments and answer evaluation setup
Define how ground truth or reference judgments should be created.
Include:
- Who should judge relevance and answer correctness
- What evidence they need
- How to label relevant, partially relevant, irrelevant, stale, duplicate, and misleading chunks
- How to handle multiple valid source passages
- How to record uncertainty and disagreements
- How to prevent evaluators from seeing the tested configuration when practical

6. Retrieval metrics
Specify retrieval metrics that directly measure chunking and metadata effects.
Include:
- Recall@k
- Precision@k or context precision
- MRR or nDCG where graded relevance exists
- Source coverage for multi-source answers
- Duplicate or near-duplicate rate in top-k
- Metadata filter precision and filter fallout
- Freshness or version correctness when metadata includes time or version fields
- No-answer retrieval behavior for queries outside the corpus

For each metric, define:
- Formula or scoring method in plain language
- Required inputs
- Which query slices it applies to
- What kind of configuration failure it reveals

7. Answer-level metrics
Define answer metrics only as downstream checks of retrieval configuration, not as a full generation evaluation.
Include:
- Answer correctness against the evaluation standard
- Citation/source support
- Missing critical fact rate
- Unsupported claim rate
- Wrong-version or wrong-jurisdiction rate if relevant
- Refusal or no-answer correctness where the corpus lacks the answer

Explain how to attribute answer failures to retrieval versus generation when the top-k evidence is sufficient but the answer is wrong.

8. Error attribution rules
Create a concrete error taxonomy with decision rules. Include at minimum:
- Chunk boundary split error
- Chunk too broad or noisy
- Chunk too narrow or missing context
- Overlap redundancy error
- Metadata absent
- Metadata incorrect
- Metadata too coarse
- Metadata filter excludes relevant evidence
- Metadata boost overpromotes irrelevant evidence
- Duplicate chunk crowding
- Stale or wrong-version retrieval
- Relevant evidence not indexed
- Relevant evidence indexed but not retrieved
- Answer failure despite sufficient retrieved evidence

For each error type, define:
- Observable evidence
- How to distinguish it from similar errors
- Likely configuration implication
- Whether it should count against chunking, metadata, indexing, retrieval settings, or answer generation

9. Analysis plan by query slice
Define how results should be compared across arms and slices.
Include:
- Primary metric and secondary metrics
- Minimum practical improvement threshold
- Regression checks where a configuration improves one slice but harms another
- Treatment of ties
- Treatment of small sample sizes
- Required confidence or stability checks if repeated runs are possible
- How to summarize tradeoffs for decision owners without hiding slice-level failures

10. Configuration acceptance decision
Create a decision framework the retrieval owner, data owner, and product owner can use after the experiment is run.
Include:
- Acceptance criteria for adopting a new chunking and metadata configuration
- Rejection criteria
- Conditional acceptance criteria requiring remediation
- Required evidence package before approval
- Rollback or re-indexing considerations if adopted
- Open questions that must be resolved before production use

11. Completion checklist
End with a checklist that confirms whether the experiment design is ready to execute. Include checks for:
- Baseline defined
- Candidate arms defined
- Constants identified
- Query slices complete
- Relevance judgment process defined
- Metrics mapped to slices
- Error attribution rules defined
- Acceptance criteria agreed by named accountable owners
- Missing evidence listed
- No unsupported claims of completed execution