Amo.ng curated workflow
Assess Enterprise Knowledge and RAG Readiness
Assess knowledge freshness, retrieval configuration, authorization boundaries, and sensitive-context propagation before expanding or releasing an enterprise RAG capability.
# Assess Enterprise Knowledge and RAG Readiness Workflow ID: AMO-W-000016 Workflow URL: https://amo.ng/workflows/enterprise-knowledge-and-rag-readiness-review ## Outcome A readiness package containing a corpus freshness policy and revalidation queue, retrieval experiment or evidence boundary, authorization exposure assessment, sensitive-context lineage, and a Ready, Restricted, Revalidate, or Hold decision. ## Before you begin - Knowledge use case, decision risk, users, owners, corpus scope, and current RAG architecture - Source inventory, ownership, effective dates, review history, ingestion records, and change evidence - Representative documents, query logs or benchmark questions, retrieval configuration, and answer standards - Identity, tenant, purpose, region, document ACL, cache, citation, and data-flow evidence - Sensitive data classifications, memory and tool boundaries, retention rules, release constraints, and incident history ## Step 1 — Assess corpus freshness and knowledge decay **Prompt** Enterprise Knowledge Corpus Freshness and Decay Review **Instructions** Map decision-critical knowledge assets to source change, effective date, review evidence, usage and retrieval exposure, accountable ownership, decay risk, and revalidation triggers. **Input for this step** Supply the source inventory, content and decision uses, ownership, authoritative systems, ingestion dates, source changes, review history, usage or retrieval exposure, and known stale-content incidents. **Carry forward** Carry the freshness policy map, decay-risk register, revalidation queue, source authority gaps, and owner assignments into retrieval design review. **Review note** The knowledge owner and domain owner confirm which sources are authoritative and which stale or unowned assets must be restricted pending revalidation. **Prompt ID** AMO-P-000282 **Prompt URL** https://amo.ng/prompts/enterprise-knowledge-corpus-freshness-and-decay-review **Prompt content** Determine where the enterprise knowledge corpus has become stale, time-sensitive, or decision-unsafe by relating source authority, change events, usage, review evidence, and retrieval exposure. Focus on freshness decay and revalidation evidence, not a broad knowledge-base quality audit. Context and inputs to provide: - Knowledge corpus name: [Knowledge corpus name] - Corpus export or representative content set: [Corpus export or representative content set] - Source inventory and authority rules: [Source inventory and authority rules] - Known change events and effective dates: [Known change events and effective dates] - Usage and retrieval exposure data: [Usage and retrieval exposure data] - Existing review evidence and owner list: [Existing review evidence and owner list] - Decision horizon and risk tolerance: [Decision horizon and risk tolerance] Evidence rules: - Separate observed evidence from inference. Label each conclusion as Observed, Inferred, or Unknown. - Do not claim that a source, log, review, owner approval, or retrieval result was inspected unless it appears in the provided material. - Preserve uncertainty where evidence is incomplete, contradictory, old, or sampled. - Prefer source-effective dates, documented review records, owner attestations, change logs, and actual retrieval or usage evidence over assumptions. - If the corpus export is incomplete, treat the result as a scoped review and state the coverage limits. Authority boundaries: - Do not make final compliance, legal, medical, financial, security, or customer-impact determinations unless an accountable owner has provided the relevant authority rule in the inputs. - Assign proposed decisions to accountable roles such as knowledge owner, source authority, product owner, policy owner, compliance owner, data owner, search owner, or support operations owner. - Where the correct owner is unclear, mark Owner Unknown and specify the evidence needed to assign accountability. Review method: 1. Establish the freshness policy map. - Identify content classes that require freshness controls, such as policy, product behavior, pricing, regulated guidance, operational runbooks, customer-facing answers, security procedures, data definitions, and historical reference material. - For each class, map source authority, expected review cadence, change triggers, acceptable age, required evidence of review, and decision owner. - If no explicit policy exists, infer a provisional freshness expectation from risk, source type, and decision use; mark it as Inferred. 2. Detect decay signals. - Compare corpus claims against known source changes, effective dates, product or policy updates, review timestamps, ownership changes, usage patterns, and retrieval exposure. - Identify assets with missing source authority, expired review windows, superseded references, conflicting versions, orphaned ownership, high retrieval exposure, or recent source changes without revalidation. - Distinguish content that is merely old from content that is decision-unsafe. 3. Build the decay-risk register. For each material risk, provide: - Asset or content group - Decision or workflow it may affect - Decay signal observed - Source authority status - Last known review evidence - Retrieval or usage exposure - Potential consequence if used as-is - Confidence level and evidence basis - Accountable owner - Recommended disposition: Retain, Revalidate, Update, Restrict, Retire, or Split/Merge 4. Build the source-change impact matrix. - List each known source change or effective-date event. - Map the affected corpus assets, topics, audiences, workflows, and retrieval surfaces. - Identify whether the corpus reflects the change, partially reflects it, conflicts with it, or lacks enough evidence to determine impact. - Prioritize changes that affect high-exposure retrieval, customer-facing guidance, regulated decisions, operational safety, financial terms, security controls, or contractual commitments. 5. Produce the revalidation queue. - Rank assets by decision risk, retrieval exposure, source-change proximity, review age, and owner availability. - For each item, specify the minimum evidence needed to clear or confirm the risk. - Assign an accountable owner and a practical next step. - Separate urgent restrictions from routine revalidation work. 6. Make retire, restrict, update, or revalidate decisions. - Recommend a disposition for each priority item. - Use Restrict when content could cause harmful or materially wrong decisions before validation is complete. - Use Retire when content is superseded, duplicate, ownerless with no defensible authority, or no longer tied to a valid business use. - Use Update when authoritative replacement information is available. - Use Revalidate when the content may still be valid but lacks current review evidence. - Use Retain only when freshness requirements, source authority, and review evidence are adequate for the stated decision horizon. Output format: A. Scope and evidence coverage - Corpus reviewed - Materials provided - Materials not provided but needed - Coverage limits - Assumptions and uncertainty B. Freshness policy map Create a table with columns: Content class | Source authority | Review cadence or trigger | Acceptable age | Required review evidence | Decision owner | Basis: Observed/Inferred/Unknown. C. Decay-risk register Create a table with columns: Priority | Asset or group | Affected decision/workflow | Decay signal | Source authority status | Review evidence | Retrieval/usage exposure | Consequence | Confidence | Owner | Recommended disposition. D. Source-change impact matrix Create a table with columns: Source change | Effective date | Affected assets/topics | Exposure surface | Alignment status | Risk level | Required action | Owner. E. Revalidation queue Create a table with columns: Queue rank | Asset or group | Reason for queueing | Minimum evidence needed | Proposed verifier | Target decision | Urgency | Dependency. F. Retire/restrict/update decisions Create a table with columns: Asset or group | Decision | Rationale | Evidence basis | Owner to confirm | Completion check. G. Missing evidence and unresolved questions List the specific missing logs, source records, review attestations, authority rules, owner assignments, or retrieval evidence needed to convert Unknown or Inferred judgments into Observed judgments. H. Completion criteria State whether this review is complete enough to support immediate restriction, update, retirement, or revalidation planning. Identify which decisions require confirmation by the relevant knowledge owner, source authority, product owner, compliance owner, data owner, or search owner before implementation. ## Step 2 — Validate chunking and metadata choices when material **Prompt** Retrieval Chunking and Metadata Experiment Design **Instructions** Define a controlled experiment when chunking or metadata choices lack decision-grade evidence or are changing. If current configuration is already supported by current representative evidence, record that evidence and mark a new experiment Not applicable. **Input for this step** Provide the freshness findings, representative documents, current and candidate chunking or metadata configuration, query logs or benchmark questions, answer standards, source-version fields, and constraints. **Carry forward** Carry the experiment plan or verified configuration boundary, query slices, metrics, guardrails, acceptance criteria, and untested assumptions into authorization review. **Review note** The retrieval owner, data owner, and domain owner approve the evidence standard and any configuration selected for further testing or release. **Prompt ID** AMO-P-000284 **Prompt URL** https://amo.ng/prompts/retrieval-chunking-and-metadata-experiment-design **Prompt content** Design a controlled retrieval experiment to isolate how chunking choices and metadata choices affect retrieval and answer quality. Keep the scope limited to corpus configuration decisions: do not redesign the full RAG architecture, diagnose the whole pipeline, change model selection, or introduce unrelated retrieval changes unless they are explicitly controlled constants. Context to use: - Corpus description and sample documents: [Corpus description and sample documents] - Current retrieval setup: [Current retrieval setup] - Candidate chunking options: [Candidate chunking options] - Candidate metadata fields: [Candidate metadata fields] - Query logs or benchmark questions: [Query logs or benchmark questions] - Answer evaluation standard: [Answer evaluation standard] - Decision owners and constraints: [Decision owners and constraints] Evidence discipline: - Separate observations supported by the provided inputs from inferences and assumptions. - Do not claim that experiments, tests, retrieval runs, indexing jobs, or evaluations were completed unless results are provided. - If required information is missing, state the missing evidence and design the smallest reasonable way to obtain it. - Preserve uncertainty where the available evidence does not justify a firm conclusion. - Before defining experiment arms, treat the baseline retrieval configuration, candidate changes, representative corpus and query evidence, and answer-evaluation standard as blocking when absent or materially conflicting. Request all blocking items in one consolidated clarification and stop the affected design decision until they are supplied. Record other gaps as Unknown with their effect on sizing, confounding, metrics, and acceptance criteria. - Treat the retrieval owner as accountable for experiment execution, the data owner as accountable for corpus and metadata validity, and the product owner or domain owner as accountable for acceptance criteria tied to user impact. Produce the following deliverable: 1. Experiment objective and decision boundary - State the configuration decision this experiment will support. - Define what is in scope: chunking strategy, overlap, boundary rules, parent-child or hierarchical chunking if relevant, metadata extraction, metadata normalization, and metadata use in filtering or ranking. - Define what is out of scope and should be held constant: embedding model, reranker, generation prompt, answer model, UI behavior, permissions, and production rollout unless the inputs require otherwise. - Identify the baseline configuration and the candidate configurations to compare. 2. Controlled variables and constants Create a table with: - Variable under test - Candidate levels - Why it matters - Expected retrieval effect - Required implementation evidence - What must remain constant Include at minimum: - Chunk size or token range - Chunk overlap - Chunk boundary rule - Structural handling of headings, tables, lists, or sections - Metadata fields to attach - Metadata normalization rules - Metadata use pattern: display only, filter, boost, grouping, or attribution 3. Factorial experiment matrix Build a practical factorial or fractional-factorial matrix that isolates chunking and metadata effects without creating unnecessary runs. For each experiment arm include: - Arm ID - Chunking configuration - Metadata configuration - Retrieval settings held constant - Indexing requirements - Expected comparison value - Risk of confounding - Minimum evidence needed before execution If the full factorial design is too large, propose a staged design: - Screening stage to remove weak options - Focused comparison stage for finalists - Confirmation stage against the baseline 4. Dataset and query slices Define the evaluation dataset and query slices needed to detect meaningful differences. Include: - Document sample selection rules - Minimum corpus coverage requirements - Query source: logs, expert-written questions, synthetic-but-reviewed questions, or known support cases - Query slice taxonomy - Required number of queries per slice if enough information is available; otherwise give a sizing rule Use query slices such as: - Exact lookup questions - Multi-section synthesis questions - Long-tail entity questions - Recent or version-specific questions - Table or list extraction questions - Procedure or policy questions - Ambiguous terminology questions - Metadata-dependent questions - Queries where the answer should not be in the corpus For each slice, specify the retrieval failure mode it is intended to expose. 5. Relevance judgments and answer evaluation setup Define how ground truth or reference judgments should be created. Include: - Who should judge relevance and answer correctness - What evidence they need - How to label relevant, partially relevant, irrelevant, stale, duplicate, and misleading chunks - How to handle multiple valid source passages - How to record uncertainty and disagreements - How to prevent evaluators from seeing the tested configuration when practical 6. Retrieval metrics Specify retrieval metrics that directly measure chunking and metadata effects. Include: - Recall@k - Precision@k or context precision - MRR or nDCG where graded relevance exists - Source coverage for multi-source answers - Duplicate or near-duplicate rate in top-k - Metadata filter precision and filter fallout - Freshness or version correctness when metadata includes time or version fields - No-answer retrieval behavior for queries outside the corpus For each metric, define: - Formula or scoring method in plain language - Required inputs - Which query slices it applies to - What kind of configuration failure it reveals 7. Answer-level metrics Define answer metrics only as downstream checks of retrieval configuration, not as a full generation evaluation. Include: - Answer correctness against the evaluation standard - Citation/source support - Missing critical fact rate - Unsupported claim rate - Wrong-version or wrong-jurisdiction rate if relevant - Refusal or no-answer correctness where the corpus lacks the answer Explain how to attribute answer failures to retrieval versus generation when the top-k evidence is sufficient but the answer is wrong. 8. Error attribution rules Create a concrete error taxonomy with decision rules. Include at minimum: - Chunk boundary split error - Chunk too broad or noisy - Chunk too narrow or missing context - Overlap redundancy error - Metadata absent - Metadata incorrect - Metadata too coarse - Metadata filter excludes relevant evidence - Metadata boost overpromotes irrelevant evidence - Duplicate chunk crowding - Stale or wrong-version retrieval - Relevant evidence not indexed - Relevant evidence indexed but not retrieved - Answer failure despite sufficient retrieved evidence For each error type, define: - Observable evidence - How to distinguish it from similar errors - Likely configuration implication - Whether it should count against chunking, metadata, indexing, retrieval settings, or answer generation 9. Analysis plan by query slice Define how results should be compared across arms and slices. Include: - Primary metric and secondary metrics - Minimum practical improvement threshold - Regression checks where a configuration improves one slice but harms another - Treatment of ties - Treatment of small sample sizes - Required confidence or stability checks if repeated runs are possible - How to summarize tradeoffs for decision owners without hiding slice-level failures 10. Configuration acceptance decision Create a decision framework the retrieval owner, data owner, and product owner can use after the experiment is run. Include: - Acceptance criteria for adopting a new chunking and metadata configuration - Rejection criteria - Conditional acceptance criteria requiring remediation - Required evidence package before approval - Rollback or re-indexing considerations if adopted - Open questions that must be resolved before production use 11. Completion checklist End with a checklist that confirms whether the experiment design is ready to execute. Include checks for: - Baseline defined - Candidate arms defined - Constants identified - Query slices complete - Relevance judgment process defined - Metrics mapped to slices - Error attribution rules defined - Acceptance criteria agreed by named accountable owners - Missing evidence listed - No unsupported claims of completed execution ## Step 3 — Test retrieval authorization boundaries **Prompt** RAG Access-Control Leakage Investigation **Instructions** Investigate whether identity, tenant, purpose, region, document ACL, ingestion, embedding, cache, retrieval, citation, or response behavior can cross authorized boundaries. **Input for this step** Use the corpus and retrieval findings with authorization models, ACL exports, query and response traces, ingestion records, cache evidence, tenant or region design, and known leakage scenarios. **Carry forward** Carry the identity-to-document map, confirmed and potential leakage paths, affected scope, containment needs, regression matrix, and unresolved authorization evidence into context-lineage review. **Review note** The security reviewer, data owner, and product owner decide which content, users, queries, caches, or environments must remain restricted. **Prompt ID** AMO-P-000283 **Prompt URL** https://amo.ng/prompts/rag-access-control-leakage-investigation **Prompt content** Investigate whether the RAG system described below retrieved, embedded, cached, cited, or exposed content outside the requesting identity, tenant, purpose, region, or document authorization boundary. Do not perform a general RAG quality review, connector security review, model safety review, or relevance evaluation unless it directly affects authorization leakage evidence. Use only the evidence provided. Do not claim that a log, policy, source, command, system, test, approval, or remediation was inspected or completed unless it is present in the supplied materials. Context and inputs: - RAG system and deployment context: [RAG system and deployment context] - Identity, tenant, purpose, region, and document authorization model: [Identity tenant purpose region and document authorization model] - Evidence bundle, including any retrieval logs, citation traces, document ACLs, embedding ingestion records, cache records, query logs, response samples, tickets, diagrams, or policy excerpts: [Evidence bundle] - Suspected leakage scenarios or query samples: [Suspected leakage scenarios or query samples] - Time window and affected environments: [Time window and affected environments] - Accountable owners and decision deadline: [Accountable owners and decision deadline] Investigation rules: 1. Separate observations from inference. Label each material statement as Evidence, Inference, Gap, or Decision. 2. Preserve uncertainty. If evidence is missing, conflicting, sampled, redacted, or outside the time window, say so plainly. Missing-input gate: - Treat the requesting identity, applicable authorization model, affected environment and time window, and sufficient retrieval or exposure traces as blocking for any confirmed leakage conclusion. Request all blocking items in one consolidated clarification and stop the affected conclusion until they are supplied. - Continue with non-blocking gaps only when they are marked Unknown and their effect on path confidence, affected scope, containment, and regression coverage is explicit. 3. Trace authorization from requesting identity to document exposure. Include retrieval, ranking, embedding, cache, citation, prompt assembly, generated response, export, telemetry, and feedback paths where evidence exists. 4. Treat access-control boundaries independently: identity, tenant, purpose, region, document-level authorization, group membership, delegated access, service account access, and temporal entitlement changes. 5. Distinguish content that was indexed, embedded, retrieved, ranked, cached, cited, included in context, generated in the answer, logged, or externally exposed. 6. Do not assume that document-level authorization at source ingestion remains valid at query time. Look for entitlement drift, stale indexes, overbroad service accounts, cache reuse, citation leakage, and post-filter bypass. 7. Do not recommend broad rewrites. Recommend the smallest containment and verification actions that fit the observed risk. 8. Assign owner accountability by role where needed: security reviewer, data owner, product owner, release owner, search/retrieval owner, platform owner, or legal/privacy owner. Deliverable: 1. Investigation scope - State the system, environments, time window, suspected boundary, and query/document populations covered. - State what is explicitly out of scope. - List the supplied evidence types and any major missing evidence. 2. Evidence register Create a table with columns: - Evidence ID - Evidence type - Source or artifact name - Time range - What it shows - Authorization boundary relevance - Limitations or uncertainty 3. Identity-to-document authorization map Create a map or table showing: - Requesting identity or role - Tenant or account - Purpose or workflow - Region or data residency boundary - Entitlement source - Allowed document classes or IDs - Disallowed document classes or IDs - Query-time enforcement point - Retrieval/index/cache/citation enforcement point - Evidence supporting the mapping - Gaps requiring owner confirmation 4. Leakage-path reconstruction For each suspected or observed leakage path, reconstruct the sequence from request to exposure: - Query or trigger - Requesting identity and authorization state at the time - Retrieval/index candidate set - Filter or authorization check expected - Filter or authorization check observed - Document, chunk, citation, cache entry, embedding, or response involved - Exposure mode: retrieved, ranked, cached, cited, placed in prompt context, generated in response, logged, exported, or visible in UI/API - Boundary crossed: identity, tenant, purpose, region, document authorization, or timing/entitlement drift - Evidence supporting the path - Alternative explanations - Confidence level: confirmed, probable, possible, or not supported 5. Affected corpus and query scope Estimate the likely scope without overstating certainty: - Affected tenants, identities, groups, regions, document classes, indexes, embeddings, caches, and environments - Query patterns likely to trigger the issue - Whether exposure appears isolated, systemic, configuration-specific, connector-specific, cache-specific, or time-bound - Known false-positive or false-negative risks in the evidence - Additional evidence needed to narrow scope 6. Containment decision Provide a decision-ready containment assessment for the accountable owners: - Immediate risk level and rationale - Recommended containment action: no action, monitor, disable affected queries, invalidate cache, remove affected index, restrict service account, re-ingest with corrected ACLs, disable citation/source display, tenant isolate, region isolate, or pause release - Why this is the smallest safe containment action supported by the evidence - Operational impact - Data owner, security reviewer, product owner, and release owner decisions required - Conditions for lifting containment 7. Regression test matrix Create a test matrix that can be used by engineering and security reviewers. Include: - Test ID - Boundary tested - Test identity or role - Allowed corpus - Forbidden corpus - Query pattern - Expected retrieval behavior - Expected citation behavior - Expected cache behavior - Expected response behavior - Required logs or traces - Pass/fail criteria - Owner responsible for verification Include tests for at least: - Same-tenant allowed document retrieval - Same-tenant forbidden document exclusion - Cross-tenant exclusion - Region boundary exclusion - Purpose-boundary exclusion where applicable - Stale group membership or entitlement removal - Cache reuse across identities or tenants - Citation/source leakage without answer leakage - Embedding/index rebuild after ACL correction - Service account or delegated access path 8. Open questions and evidence requests List only the missing items that materially affect the containment decision, scope estimate, or regression coverage. Assign each request to a specific owner role. 9. Completion checks End with a concise checklist confirming whether the current evidence is sufficient to: - Determine if leakage occurred - Identify the likely leakage path - Estimate affected corpus and query scope - Make a containment decision - Define regression tests - Proceed to remediation planning If any check cannot be completed from the supplied evidence, mark it as incomplete and explain the exact missing evidence. ## Step 4 — Trace sensitive context beyond retrieval **Prompt** Sensitive Context Propagation and Cross-Agent Contamination Audit **Instructions** Trace sensitive context from source and retrieval through agent handoffs, prompts, memory, tools, logs, shared workspaces, and downstream outputs. Reconcile propagation findings with freshness and authorization evidence. **Input for this step** Provide data classifications, RAG and agent flow, representative payloads, memory and tool configuration, logs, retention rules, tenant and purpose boundaries, and prior-step findings. **Carry forward** Produce the final Ready, Restricted, Revalidate, or Hold record with freshness work, retrieval evidence, authorization and context-lineage findings, owners, restrictions, regression criteria, and re-review triggers. **Review note** The knowledge owner, data owner, security and privacy reviewers, product owner, and release owner approve the readiness disposition and residual risk. **Prompt ID** AMO-P-000286 **Prompt URL** https://amo.ng/prompts/sensitive-context-propagation-and-cross-agent-contamination-audit **Prompt content** Conduct an evidence-based audit of sensitive context propagation across the specified agent workflow. Trace actual context movement through handoffs, memory, retrieval, tools, logs, outputs, and shared workspaces. Do not produce a generic sensitive-data checklist. Separate observed evidence from inference, expose missing information, and preserve uncertainty. Context to provide: - System or workflow under review: [System or workflow under review] - Authorized purposes and tenant boundaries: [Authorized purposes and tenant boundaries] - Sensitive context categories: [Sensitive context categories] - Evidence package: [Evidence package] - Known incidents or concerns: [Known incidents or concerns] - Accountable owners: [Accountable owners] - Review date range: [Review date range] Evidence rules: - Use only the evidence provided in [Evidence package] and clearly identified user-supplied context. - Do not claim that a source, system, log, tool, workspace, approval, command, or deletion was inspected or completed unless evidence is present. - Label each material statement as one of: Observed, Inferred, Not evidenced, or Requires owner confirmation. - Distinguish sensitive context from ordinary workflow context. - Distinguish authorized propagation from unauthorized propagation, excessive retention, purpose drift, and cross-tenant or cross-task contamination. - If evidence is incomplete, state the missing artifact and why it matters. Audit scope: Trace sensitive context across these surfaces where evidence exists: 1. User inputs, uploaded files, tickets, records, conversations, or task instructions. 2. Agent-to-agent handoffs, delegation messages, intermediate reasoning summaries, or task state objects. 3. Short-term memory, long-term memory, vector stores, retrieval indexes, embeddings, caches, and session stores. 4. Tool calls, API payloads, browser sessions, database queries, SaaS integrations, webhooks, automations, and background jobs. 5. Logs, traces, analytics events, evaluation datasets, transcripts, error reports, monitoring records, and support workspaces. 6. Shared folders, project workspaces, collaboration tools, exported artifacts, generated documents, and downstream notifications. 7. Human review queues, escalation paths, approval records, and operator notes. Required deliverable: 1. Audit Boundary and Evidence Inventory Create a concise table with: - Evidence item - Source or owner if known - Date range covered - Context surfaces covered - Reliability limits - Material gaps 2. Context Lineage Map Create a lineage table that traces each sensitive context category through the workflow: - Context item or category - Origin - Initial authorized purpose - Receiving agent, service, tool, memory store, log, workspace, or person - Transfer mechanism - Transformation or summarization performed - Retention location and retention duration if evidenced - Tenant, customer, project, task, or workspace boundary crossed - Evidence reference - Status: authorized, questionable, unauthorized, excessive, contaminated, or not evidenced Then provide a short narrative explaining the highest-risk propagation paths. Do not infer a path merely because it is technically possible; identify it as a hypothesis if not evidenced. 3. Sensitivity and Purpose Register Create a register with: - Sensitive context category - Sensitivity rationale - Data subject, tenant, customer, project, or task boundary affected - Authorized purpose from [Authorized purposes and tenant boundaries] - Actual observed use - Purpose alignment: aligned, narrowed, expanded, drifted, unrelated, or not evidenced - Minimum context needed for the task - Excess context observed - Owner accountable for purpose decision from [Accountable owners], or owner not identified 4. Cross-Agent and Cross-Boundary Contamination Findings For each finding, include: - Finding title - Evidence basis - Contamination type: cross-agent, cross-tenant, cross-task, cross-customer, cross-project, memory reuse, retrieval bleed, logging exposure, tool propagation, workspace exposure, or purpose drift - Affected context - Affected boundary - How the propagation occurred or is suspected to occur - Impact on confidentiality, integrity, compliance, customer trust, operational safety, or decision quality - Likelihood rating: evidenced, plausible, weakly supported, or unknown - Severity rating: critical, high, medium, low, or informational - Confidence level and reason - Missing evidence that would change the rating 5. Minimization and Containment Controls Propose controls tied to the observed lineage, not generic policy slogans. For each control, include: - Propagation point addressed - Control objective - Specific change to inputs, prompts, handoff schema, memory policy, retrieval filtering, tool payloads, logging, access control, workspace permissions, retention, or operator procedure - Expected reduction in sensitive context exposure - Owner responsible for implementation - Verification method - Residual risk Prioritize smallest effective controls that reduce propagation without breaking the authorized workflow. Avoid broad rewrites unless the evidence shows the workflow design itself is unsafe. 6. Deletion, Quarantine, and Revalidation Plan Create an action plan with: - Artifact or location requiring deletion, quarantine, redaction, re-indexing, access review, or retention change - Reason action is needed - Required owner approval: data owner, security reviewer, legal/privacy owner, product owner, platform owner, or customer account owner as appropriate - Preconditions before action - Execution evidence needed after action - Revalidation test or sampling method - Rollback or exception handling if deletion would impair legal hold, auditability, customer support, or service reliability Do not state that deletion, quarantine, redaction, or re-indexing has been completed. State only the plan and the evidence needed to verify completion. 7. Open Questions and Owner Decisions List unresolved questions that materially affect risk or remediation. For each, identify: - Question - Why it matters - Evidence needed - Accountable owner from [Accountable owners], or owner not identified - Decision deadline if inferable from [Known incidents or concerns] 8. Completion Check End with a completion check stating whether the audit is ready for owner review. Include: - Whether every sensitive context category in [Sensitive context categories] was traced or marked not evidenced - Whether every material propagation path has an evidence reference or uncertainty label - Whether contamination findings are tied to actual lineage evidence - Whether minimization controls map to specific propagation points - Whether deletion and revalidation actions identify accountable owners and verification evidence - Remaining blockers before security reviewer, data owner, product owner, or platform owner decision ## Completion criteria The workflow is complete when: - Decision-critical knowledge has a freshness status, owner, review trigger, and revalidation priority. - Retrieval configuration is supported by measured evidence or an explicit controlled experiment; unchanged verified configuration may be marked Not applicable. - Authorization and sensitive-context findings are traceable across ingestion, retrieval, cache, citation, memory, tools, logs, and handoffs. - The final readiness state is Ready, Restricted, Revalidate, or Hold with measurable conditions and owners. - Uninspected systems, unrun experiments, and unavailable evidence remain explicit. # Assess Enterprise Knowledge and RAG Readiness Workflow ID: AMO-W-000016 Workflow URL: https://amo.ng/workflows/enterprise-knowledge-and-rag-readiness-review Use this Amo.ng workflow with your preferred AI tool. Complete the steps in order and carry the specified output forward. Outcome: A readiness package containing a corpus freshness policy and revalidation queue, retrieval experiment or evidence boundary, authorization exposure assessment, sensitive-context lineage, and a Ready, Restricted, Revalidate, or Hold decision. Required inputs: - Knowledge use case, decision risk, users, owners, corpus scope, and current RAG architecture - Source inventory, ownership, effective dates, review history, ingestion records, and change evidence - Representative documents, query logs or benchmark questions, retrieval configuration, and answer standards - Identity, tenant, purpose, region, document ACL, cache, citation, and data-flow evidence - Sensitive data classifications, memory and tool boundaries, retention rules, release constraints, and incident history ## Step 1 — Assess corpus freshness and knowledge decay **Instructions** Map decision-critical knowledge assets to source change, effective date, review evidence, usage and retrieval exposure, accountable ownership, decay risk, and revalidation triggers. **Input for this step** Supply the source inventory, content and decision uses, ownership, authoritative systems, ingestion dates, source changes, review history, usage or retrieval exposure, and known stale-content incidents. **Carry forward** Carry the freshness policy map, decay-risk register, revalidation queue, source authority gaps, and owner assignments into retrieval design review. **Review note** The knowledge owner and domain owner confirm which sources are authoritative and which stale or unowned assets must be restricted pending revalidation. **Prompt** Enterprise Knowledge Corpus Freshness and Decay Review **Prompt ID** AMO-P-000282 **Prompt URL** https://amo.ng/prompts/enterprise-knowledge-corpus-freshness-and-decay-review ## Step 2 — Validate chunking and metadata choices when material **Instructions** Define a controlled experiment when chunking or metadata choices lack decision-grade evidence or are changing. If current configuration is already supported by current representative evidence, record that evidence and mark a new experiment Not applicable. **Input for this step** Provide the freshness findings, representative documents, current and candidate chunking or metadata configuration, query logs or benchmark questions, answer standards, source-version fields, and constraints. **Carry forward** Carry the experiment plan or verified configuration boundary, query slices, metrics, guardrails, acceptance criteria, and untested assumptions into authorization review. **Review note** The retrieval owner, data owner, and domain owner approve the evidence standard and any configuration selected for further testing or release. **Prompt** Retrieval Chunking and Metadata Experiment Design **Prompt ID** AMO-P-000284 **Prompt URL** https://amo.ng/prompts/retrieval-chunking-and-metadata-experiment-design ## Step 3 — Test retrieval authorization boundaries **Instructions** Investigate whether identity, tenant, purpose, region, document ACL, ingestion, embedding, cache, retrieval, citation, or response behavior can cross authorized boundaries. **Input for this step** Use the corpus and retrieval findings with authorization models, ACL exports, query and response traces, ingestion records, cache evidence, tenant or region design, and known leakage scenarios. **Carry forward** Carry the identity-to-document map, confirmed and potential leakage paths, affected scope, containment needs, regression matrix, and unresolved authorization evidence into context-lineage review. **Review note** The security reviewer, data owner, and product owner decide which content, users, queries, caches, or environments must remain restricted. **Prompt** RAG Access-Control Leakage Investigation **Prompt ID** AMO-P-000283 **Prompt URL** https://amo.ng/prompts/rag-access-control-leakage-investigation ## Step 4 — Trace sensitive context beyond retrieval **Instructions** Trace sensitive context from source and retrieval through agent handoffs, prompts, memory, tools, logs, shared workspaces, and downstream outputs. Reconcile propagation findings with freshness and authorization evidence. **Input for this step** Provide data classifications, RAG and agent flow, representative payloads, memory and tool configuration, logs, retention rules, tenant and purpose boundaries, and prior-step findings. **Carry forward** Produce the final Ready, Restricted, Revalidate, or Hold record with freshness work, retrieval evidence, authorization and context-lineage findings, owners, restrictions, regression criteria, and re-review triggers. **Review note** The knowledge owner, data owner, security and privacy reviewers, product owner, and release owner approve the readiness disposition and residual risk. **Prompt** Sensitive Context Propagation and Cross-Agent Contamination Audit **Prompt ID** AMO-P-000286 **Prompt URL** https://amo.ng/prompts/sensitive-context-propagation-and-cross-agent-contamination-audit Completion criteria: The workflow is complete when: - Decision-critical knowledge has a freshness status, owner, review trigger, and revalidation priority. - Retrieval configuration is supported by measured evidence or an explicit controlled experiment; unchanged verified configuration may be marked Not applicable. - Authorization and sensitive-context findings are traceable across ingestion, retrieval, cache, citation, memory, tools, logs, and handoffs. - The final readiness state is Ready, Restricted, Revalidate, or Hold with measurable conditions and owners. - Uninspected systems, unrun experiments, and unavailable evidence remain explicit.Copy workflow includes every step and the full linked Prompt content. Use with AI copies a shorter guide with Prompt links; neither action runs the Workflow.
Outcome
A readiness package containing a corpus freshness policy and revalidation queue, retrieval experiment or evidence boundary, authorization exposure assessment, sensitive-context lineage, and a Ready, Restricted, Revalidate, or Hold decision.
Before you begin
Have all or some of the following available before you start. The more relevant context you can provide, the stronger the workflow output will be.
- Knowledge use case, decision risk, users, owners, corpus scope, and current RAG architecture
- Source inventory, ownership, effective dates, review history, ingestion records, and change evidence
- Representative documents, query logs or benchmark questions, retrieval configuration, and answer standards
- Identity, tenant, purpose, region, document ACL, cache, citation, and data-flow evidence
- Sensitive data classifications, memory and tool boundaries, retention rules, release constraints, and incident history
Ordered sequence
Workflow steps
Complete the steps in order. For each step, provide the listed context, carry its result into the next step, and pause wherever a review note is shown.
-
Step 1 Assess corpus freshness and knowledge decay
Map decision-critical knowledge assets to source change, effective date, review evidence, usage and retrieval exposure, accountable ownership, decay risk, and revalidation triggers.
Prompt: Enterprise Knowledge Corpus Freshness and Decay ReviewDetermine where the enterprise knowledge corpus has become stale, time-sensitive, or decision-unsafe by relating source authority, change events, usage, review evidence, and retrieval exposure. Focus on freshness decay and revalidation evidence, not a broad knowledge-base quality audit. Context and inputs to provide: - Knowledge corpus name: [Knowledge corpus name] - Corpus export or representative content set: [Corpus export or representative content set] - Source inventory and authority rules: [Source inventory and authority rules] - Known change events and effective dates: [Known change events and effective dates] - Usage and retrieval exposure data: [Usage and retrieval exposure data] - Existing review evidence and owner list: [Existing review evidence and owner list] - Decision horizon and risk tolerance: [Decision horizon and risk tolerance] Evidence rules: - Separate observed evidence from inference. Label each conclusion as Observed, Inferred, or Unknown. - Do not claim that a source, log, review, owner approval, or retrieval result was inspected unless it appears in the provided material. - Preserve uncertainty where evidence is incomplete, contradictory, old, or sampled. - Prefer source-effective dates, documented review records, owner attestations, change logs, and actual retrieval or usage evidence over assumptions. - If the corpus export is incomplete, treat the result as a scoped review and state the coverage limits. Authority boundaries: - Do not make final compliance, legal, medical, financial, security, or customer-impact determinations unless an accountable owner has provided the relevant authority rule in the inputs. - Assign proposed decisions to accountable roles such as knowledge owner, source authority, product owner, policy owner, compliance owner, data owner, search owner, or support operations owner. - Where the correct owner is unclear, mark Owner Unknown and specify the evidence needed to assign accountability. Review method: 1. Establish the freshness policy map. - Identify content classes that require freshness controls, such as policy, product behavior, pricing, regulated guidance, operational runbooks, customer-facing answers, security procedures, data definitions, and historical reference material. - For each class, map source authority, expected review cadence, change triggers, acceptable age, required evidence of review, and decision owner. - If no explicit policy exists, infer a provisional freshness expectation from risk, source type, and decision use; mark it as Inferred. 2. Detect decay signals. - Compare corpus claims against known source changes, effective dates, product or policy updates, review timestamps, ownership changes, usage patterns, and retrieval exposure. - Identify assets with missing source authority, expired review windows, superseded references, conflicting versions, orphaned ownership, high retrieval exposure, or recent source changes without revalidation. - Distinguish content that is merely old from content that is decision-unsafe. 3. Build the decay-risk register. For each material risk, provide: - Asset or content group - Decision or workflow it may affect - Decay signal observed - Source authority status - Last known review evidence - Retrieval or usage exposure - Potential consequence if used as-is - Confidence level and evidence basis - Accountable owner - Recommended disposition: Retain, Revalidate, Update, Restrict, Retire, or Split/Merge 4. Build the source-change impact matrix. - List each known source change or effective-date event. - Map the affected corpus assets, topics, audiences, workflows, and retrieval surfaces. - Identify whether the corpus reflects the change, partially reflects it, conflicts with it, or lacks enough evidence to determine impact. - Prioritize changes that affect high-exposure retrieval, customer-facing guidance, regulated decisions, operational safety, financial terms, security controls, or contractual commitments. 5. Produce the revalidation queue. - Rank assets by decision risk, retrieval exposure, source-change proximity, review age, and owner availability. - For each item, specify the minimum evidence needed to clear or confirm the risk. - Assign an accountable owner and a practical next step. - Separate urgent restrictions from routine revalidation work. 6. Make retire, restrict, update, or revalidate decisions. - Recommend a disposition for each priority item. - Use Restrict when content could cause harmful or materially wrong decisions before validation is complete. - Use Retire when content is superseded, duplicate, ownerless with no defensible authority, or no longer tied to a valid business use. - Use Update when authoritative replacement information is available. - Use Revalidate when the content may still be valid but lacks current review evidence. - Use Retain only when freshness requirements, source authority, and review evidence are adequate for the stated decision horizon. Output format: A. Scope and evidence coverage - Corpus reviewed - Materials provided - Materials not provided but needed - Coverage limits - Assumptions and uncertainty B. Freshness policy map Create a table with columns: Content class | Source authority | Review cadence or trigger | Acceptable age | Required review evidence | Decision owner | Basis: Observed/Inferred/Unknown. C. Decay-risk register Create a table with columns: Priority | Asset or group | Affected decision/workflow | Decay signal | Source authority status | Review evidence | Retrieval/usage exposure | Consequence | Confidence | Owner | Recommended disposition. D. Source-change impact matrix Create a table with columns: Source change | Effective date | Affected assets/topics | Exposure surface | Alignment status | Risk level | Required action | Owner. E. Revalidation queue Create a table with columns: Queue rank | Asset or group | Reason for queueing | Minimum evidence needed | Proposed verifier | Target decision | Urgency | Dependency. F. Retire/restrict/update decisions Create a table with columns: Asset or group | Decision | Rationale | Evidence basis | Owner to confirm | Completion check. G. Missing evidence and unresolved questions List the specific missing logs, source records, review attestations, authority rules, owner assignments, or retrieval evidence needed to convert Unknown or Inferred judgments into Observed judgments. H. Completion criteria State whether this review is complete enough to support immediate restriction, update, retirement, or revalidation planning. Identify which decisions require confirmation by the relevant knowledge owner, source authority, product owner, compliance owner, data owner, or search owner before implementation.Input for this step
Supply the source inventory, content and decision uses, ownership, authoritative systems, ingestion dates, source changes, review history, usage or retrieval exposure, and known stale-content incidents.
Carry forward
Carry the freshness policy map, decay-risk register, revalidation queue, source authority gaps, and owner assignments into retrieval design review.
Review note
The knowledge owner and domain owner confirm which sources are authoritative and which stale or unowned assets must be restricted pending revalidation.
-
Step 2 Validate chunking and metadata choices when material
Define a controlled experiment when chunking or metadata choices lack decision-grade evidence or are changing. If current configuration is already supported by current representative evidence, record that evidence and mark a new experiment Not applicable.
Prompt: Retrieval Chunking and Metadata Experiment DesignDesign a controlled retrieval experiment to isolate how chunking choices and metadata choices affect retrieval and answer quality. Keep the scope limited to corpus configuration decisions: do not redesign the full RAG architecture, diagnose the whole pipeline, change model selection, or introduce unrelated retrieval changes unless they are explicitly controlled constants. Context to use: - Corpus description and sample documents: [Corpus description and sample documents] - Current retrieval setup: [Current retrieval setup] - Candidate chunking options: [Candidate chunking options] - Candidate metadata fields: [Candidate metadata fields] - Query logs or benchmark questions: [Query logs or benchmark questions] - Answer evaluation standard: [Answer evaluation standard] - Decision owners and constraints: [Decision owners and constraints] Evidence discipline: - Separate observations supported by the provided inputs from inferences and assumptions. - Do not claim that experiments, tests, retrieval runs, indexing jobs, or evaluations were completed unless results are provided. - If required information is missing, state the missing evidence and design the smallest reasonable way to obtain it. - Preserve uncertainty where the available evidence does not justify a firm conclusion. - Before defining experiment arms, treat the baseline retrieval configuration, candidate changes, representative corpus and query evidence, and answer-evaluation standard as blocking when absent or materially conflicting. Request all blocking items in one consolidated clarification and stop the affected design decision until they are supplied. Record other gaps as Unknown with their effect on sizing, confounding, metrics, and acceptance criteria. - Treat the retrieval owner as accountable for experiment execution, the data owner as accountable for corpus and metadata validity, and the product owner or domain owner as accountable for acceptance criteria tied to user impact. Produce the following deliverable: 1. Experiment objective and decision boundary - State the configuration decision this experiment will support. - Define what is in scope: chunking strategy, overlap, boundary rules, parent-child or hierarchical chunking if relevant, metadata extraction, metadata normalization, and metadata use in filtering or ranking. - Define what is out of scope and should be held constant: embedding model, reranker, generation prompt, answer model, UI behavior, permissions, and production rollout unless the inputs require otherwise. - Identify the baseline configuration and the candidate configurations to compare. 2. Controlled variables and constants Create a table with: - Variable under test - Candidate levels - Why it matters - Expected retrieval effect - Required implementation evidence - What must remain constant Include at minimum: - Chunk size or token range - Chunk overlap - Chunk boundary rule - Structural handling of headings, tables, lists, or sections - Metadata fields to attach - Metadata normalization rules - Metadata use pattern: display only, filter, boost, grouping, or attribution 3. Factorial experiment matrix Build a practical factorial or fractional-factorial matrix that isolates chunking and metadata effects without creating unnecessary runs. For each experiment arm include: - Arm ID - Chunking configuration - Metadata configuration - Retrieval settings held constant - Indexing requirements - Expected comparison value - Risk of confounding - Minimum evidence needed before execution If the full factorial design is too large, propose a staged design: - Screening stage to remove weak options - Focused comparison stage for finalists - Confirmation stage against the baseline 4. Dataset and query slices Define the evaluation dataset and query slices needed to detect meaningful differences. Include: - Document sample selection rules - Minimum corpus coverage requirements - Query source: logs, expert-written questions, synthetic-but-reviewed questions, or known support cases - Query slice taxonomy - Required number of queries per slice if enough information is available; otherwise give a sizing rule Use query slices such as: - Exact lookup questions - Multi-section synthesis questions - Long-tail entity questions - Recent or version-specific questions - Table or list extraction questions - Procedure or policy questions - Ambiguous terminology questions - Metadata-dependent questions - Queries where the answer should not be in the corpus For each slice, specify the retrieval failure mode it is intended to expose. 5. Relevance judgments and answer evaluation setup Define how ground truth or reference judgments should be created. Include: - Who should judge relevance and answer correctness - What evidence they need - How to label relevant, partially relevant, irrelevant, stale, duplicate, and misleading chunks - How to handle multiple valid source passages - How to record uncertainty and disagreements - How to prevent evaluators from seeing the tested configuration when practical 6. Retrieval metrics Specify retrieval metrics that directly measure chunking and metadata effects. Include: - Recall@k - Precision@k or context precision - MRR or nDCG where graded relevance exists - Source coverage for multi-source answers - Duplicate or near-duplicate rate in top-k - Metadata filter precision and filter fallout - Freshness or version correctness when metadata includes time or version fields - No-answer retrieval behavior for queries outside the corpus For each metric, define: - Formula or scoring method in plain language - Required inputs - Which query slices it applies to - What kind of configuration failure it reveals 7. Answer-level metrics Define answer metrics only as downstream checks of retrieval configuration, not as a full generation evaluation. Include: - Answer correctness against the evaluation standard - Citation/source support - Missing critical fact rate - Unsupported claim rate - Wrong-version or wrong-jurisdiction rate if relevant - Refusal or no-answer correctness where the corpus lacks the answer Explain how to attribute answer failures to retrieval versus generation when the top-k evidence is sufficient but the answer is wrong. 8. Error attribution rules Create a concrete error taxonomy with decision rules. Include at minimum: - Chunk boundary split error - Chunk too broad or noisy - Chunk too narrow or missing context - Overlap redundancy error - Metadata absent - Metadata incorrect - Metadata too coarse - Metadata filter excludes relevant evidence - Metadata boost overpromotes irrelevant evidence - Duplicate chunk crowding - Stale or wrong-version retrieval - Relevant evidence not indexed - Relevant evidence indexed but not retrieved - Answer failure despite sufficient retrieved evidence For each error type, define: - Observable evidence - How to distinguish it from similar errors - Likely configuration implication - Whether it should count against chunking, metadata, indexing, retrieval settings, or answer generation 9. Analysis plan by query slice Define how results should be compared across arms and slices. Include: - Primary metric and secondary metrics - Minimum practical improvement threshold - Regression checks where a configuration improves one slice but harms another - Treatment of ties - Treatment of small sample sizes - Required confidence or stability checks if repeated runs are possible - How to summarize tradeoffs for decision owners without hiding slice-level failures 10. Configuration acceptance decision Create a decision framework the retrieval owner, data owner, and product owner can use after the experiment is run. Include: - Acceptance criteria for adopting a new chunking and metadata configuration - Rejection criteria - Conditional acceptance criteria requiring remediation - Required evidence package before approval - Rollback or re-indexing considerations if adopted - Open questions that must be resolved before production use 11. Completion checklist End with a checklist that confirms whether the experiment design is ready to execute. Include checks for: - Baseline defined - Candidate arms defined - Constants identified - Query slices complete - Relevance judgment process defined - Metrics mapped to slices - Error attribution rules defined - Acceptance criteria agreed by named accountable owners - Missing evidence listed - No unsupported claims of completed executionInput for this step
Provide the freshness findings, representative documents, current and candidate chunking or metadata configuration, query logs or benchmark questions, answer standards, source-version fields, and constraints.
Carry forward
Carry the experiment plan or verified configuration boundary, query slices, metrics, guardrails, acceptance criteria, and untested assumptions into authorization review.
Review note
The retrieval owner, data owner, and domain owner approve the evidence standard and any configuration selected for further testing or release.
-
Step 3 Test retrieval authorization boundaries
Investigate whether identity, tenant, purpose, region, document ACL, ingestion, embedding, cache, retrieval, citation, or response behavior can cross authorized boundaries.
Prompt: RAG Access-Control Leakage InvestigationInvestigate whether the RAG system described below retrieved, embedded, cached, cited, or exposed content outside the requesting identity, tenant, purpose, region, or document authorization boundary. Do not perform a general RAG quality review, connector security review, model safety review, or relevance evaluation unless it directly affects authorization leakage evidence. Use only the evidence provided. Do not claim that a log, policy, source, command, system, test, approval, or remediation was inspected or completed unless it is present in the supplied materials. Context and inputs: - RAG system and deployment context: [RAG system and deployment context] - Identity, tenant, purpose, region, and document authorization model: [Identity tenant purpose region and document authorization model] - Evidence bundle, including any retrieval logs, citation traces, document ACLs, embedding ingestion records, cache records, query logs, response samples, tickets, diagrams, or policy excerpts: [Evidence bundle] - Suspected leakage scenarios or query samples: [Suspected leakage scenarios or query samples] - Time window and affected environments: [Time window and affected environments] - Accountable owners and decision deadline: [Accountable owners and decision deadline] Investigation rules: 1. Separate observations from inference. Label each material statement as Evidence, Inference, Gap, or Decision. 2. Preserve uncertainty. If evidence is missing, conflicting, sampled, redacted, or outside the time window, say so plainly. Missing-input gate: - Treat the requesting identity, applicable authorization model, affected environment and time window, and sufficient retrieval or exposure traces as blocking for any confirmed leakage conclusion. Request all blocking items in one consolidated clarification and stop the affected conclusion until they are supplied. - Continue with non-blocking gaps only when they are marked Unknown and their effect on path confidence, affected scope, containment, and regression coverage is explicit. 3. Trace authorization from requesting identity to document exposure. Include retrieval, ranking, embedding, cache, citation, prompt assembly, generated response, export, telemetry, and feedback paths where evidence exists. 4. Treat access-control boundaries independently: identity, tenant, purpose, region, document-level authorization, group membership, delegated access, service account access, and temporal entitlement changes. 5. Distinguish content that was indexed, embedded, retrieved, ranked, cached, cited, included in context, generated in the answer, logged, or externally exposed. 6. Do not assume that document-level authorization at source ingestion remains valid at query time. Look for entitlement drift, stale indexes, overbroad service accounts, cache reuse, citation leakage, and post-filter bypass. 7. Do not recommend broad rewrites. Recommend the smallest containment and verification actions that fit the observed risk. 8. Assign owner accountability by role where needed: security reviewer, data owner, product owner, release owner, search/retrieval owner, platform owner, or legal/privacy owner. Deliverable: 1. Investigation scope - State the system, environments, time window, suspected boundary, and query/document populations covered. - State what is explicitly out of scope. - List the supplied evidence types and any major missing evidence. 2. Evidence register Create a table with columns: - Evidence ID - Evidence type - Source or artifact name - Time range - What it shows - Authorization boundary relevance - Limitations or uncertainty 3. Identity-to-document authorization map Create a map or table showing: - Requesting identity or role - Tenant or account - Purpose or workflow - Region or data residency boundary - Entitlement source - Allowed document classes or IDs - Disallowed document classes or IDs - Query-time enforcement point - Retrieval/index/cache/citation enforcement point - Evidence supporting the mapping - Gaps requiring owner confirmation 4. Leakage-path reconstruction For each suspected or observed leakage path, reconstruct the sequence from request to exposure: - Query or trigger - Requesting identity and authorization state at the time - Retrieval/index candidate set - Filter or authorization check expected - Filter or authorization check observed - Document, chunk, citation, cache entry, embedding, or response involved - Exposure mode: retrieved, ranked, cached, cited, placed in prompt context, generated in response, logged, exported, or visible in UI/API - Boundary crossed: identity, tenant, purpose, region, document authorization, or timing/entitlement drift - Evidence supporting the path - Alternative explanations - Confidence level: confirmed, probable, possible, or not supported 5. Affected corpus and query scope Estimate the likely scope without overstating certainty: - Affected tenants, identities, groups, regions, document classes, indexes, embeddings, caches, and environments - Query patterns likely to trigger the issue - Whether exposure appears isolated, systemic, configuration-specific, connector-specific, cache-specific, or time-bound - Known false-positive or false-negative risks in the evidence - Additional evidence needed to narrow scope 6. Containment decision Provide a decision-ready containment assessment for the accountable owners: - Immediate risk level and rationale - Recommended containment action: no action, monitor, disable affected queries, invalidate cache, remove affected index, restrict service account, re-ingest with corrected ACLs, disable citation/source display, tenant isolate, region isolate, or pause release - Why this is the smallest safe containment action supported by the evidence - Operational impact - Data owner, security reviewer, product owner, and release owner decisions required - Conditions for lifting containment 7. Regression test matrix Create a test matrix that can be used by engineering and security reviewers. Include: - Test ID - Boundary tested - Test identity or role - Allowed corpus - Forbidden corpus - Query pattern - Expected retrieval behavior - Expected citation behavior - Expected cache behavior - Expected response behavior - Required logs or traces - Pass/fail criteria - Owner responsible for verification Include tests for at least: - Same-tenant allowed document retrieval - Same-tenant forbidden document exclusion - Cross-tenant exclusion - Region boundary exclusion - Purpose-boundary exclusion where applicable - Stale group membership or entitlement removal - Cache reuse across identities or tenants - Citation/source leakage without answer leakage - Embedding/index rebuild after ACL correction - Service account or delegated access path 8. Open questions and evidence requests List only the missing items that materially affect the containment decision, scope estimate, or regression coverage. Assign each request to a specific owner role. 9. Completion checks End with a concise checklist confirming whether the current evidence is sufficient to: - Determine if leakage occurred - Identify the likely leakage path - Estimate affected corpus and query scope - Make a containment decision - Define regression tests - Proceed to remediation planning If any check cannot be completed from the supplied evidence, mark it as incomplete and explain the exact missing evidence.Input for this step
Use the corpus and retrieval findings with authorization models, ACL exports, query and response traces, ingestion records, cache evidence, tenant or region design, and known leakage scenarios.
Carry forward
Carry the identity-to-document map, confirmed and potential leakage paths, affected scope, containment needs, regression matrix, and unresolved authorization evidence into context-lineage review.
Review note
The security reviewer, data owner, and product owner decide which content, users, queries, caches, or environments must remain restricted.
-
Step 4 Trace sensitive context beyond retrieval
Trace sensitive context from source and retrieval through agent handoffs, prompts, memory, tools, logs, shared workspaces, and downstream outputs. Reconcile propagation findings with freshness and authorization evidence.
Prompt: Sensitive Context Propagation and Cross-Agent Contamination AuditConduct an evidence-based audit of sensitive context propagation across the specified agent workflow. Trace actual context movement through handoffs, memory, retrieval, tools, logs, outputs, and shared workspaces. Do not produce a generic sensitive-data checklist. Separate observed evidence from inference, expose missing information, and preserve uncertainty. Context to provide: - System or workflow under review: [System or workflow under review] - Authorized purposes and tenant boundaries: [Authorized purposes and tenant boundaries] - Sensitive context categories: [Sensitive context categories] - Evidence package: [Evidence package] - Known incidents or concerns: [Known incidents or concerns] - Accountable owners: [Accountable owners] - Review date range: [Review date range] Evidence rules: - Use only the evidence provided in [Evidence package] and clearly identified user-supplied context. - Do not claim that a source, system, log, tool, workspace, approval, command, or deletion was inspected or completed unless evidence is present. - Label each material statement as one of: Observed, Inferred, Not evidenced, or Requires owner confirmation. - Distinguish sensitive context from ordinary workflow context. - Distinguish authorized propagation from unauthorized propagation, excessive retention, purpose drift, and cross-tenant or cross-task contamination. - If evidence is incomplete, state the missing artifact and why it matters. Audit scope: Trace sensitive context across these surfaces where evidence exists: 1. User inputs, uploaded files, tickets, records, conversations, or task instructions. 2. Agent-to-agent handoffs, delegation messages, intermediate reasoning summaries, or task state objects. 3. Short-term memory, long-term memory, vector stores, retrieval indexes, embeddings, caches, and session stores. 4. Tool calls, API payloads, browser sessions, database queries, SaaS integrations, webhooks, automations, and background jobs. 5. Logs, traces, analytics events, evaluation datasets, transcripts, error reports, monitoring records, and support workspaces. 6. Shared folders, project workspaces, collaboration tools, exported artifacts, generated documents, and downstream notifications. 7. Human review queues, escalation paths, approval records, and operator notes. Required deliverable: 1. Audit Boundary and Evidence Inventory Create a concise table with: - Evidence item - Source or owner if known - Date range covered - Context surfaces covered - Reliability limits - Material gaps 2. Context Lineage Map Create a lineage table that traces each sensitive context category through the workflow: - Context item or category - Origin - Initial authorized purpose - Receiving agent, service, tool, memory store, log, workspace, or person - Transfer mechanism - Transformation or summarization performed - Retention location and retention duration if evidenced - Tenant, customer, project, task, or workspace boundary crossed - Evidence reference - Status: authorized, questionable, unauthorized, excessive, contaminated, or not evidenced Then provide a short narrative explaining the highest-risk propagation paths. Do not infer a path merely because it is technically possible; identify it as a hypothesis if not evidenced. 3. Sensitivity and Purpose Register Create a register with: - Sensitive context category - Sensitivity rationale - Data subject, tenant, customer, project, or task boundary affected - Authorized purpose from [Authorized purposes and tenant boundaries] - Actual observed use - Purpose alignment: aligned, narrowed, expanded, drifted, unrelated, or not evidenced - Minimum context needed for the task - Excess context observed - Owner accountable for purpose decision from [Accountable owners], or owner not identified 4. Cross-Agent and Cross-Boundary Contamination Findings For each finding, include: - Finding title - Evidence basis - Contamination type: cross-agent, cross-tenant, cross-task, cross-customer, cross-project, memory reuse, retrieval bleed, logging exposure, tool propagation, workspace exposure, or purpose drift - Affected context - Affected boundary - How the propagation occurred or is suspected to occur - Impact on confidentiality, integrity, compliance, customer trust, operational safety, or decision quality - Likelihood rating: evidenced, plausible, weakly supported, or unknown - Severity rating: critical, high, medium, low, or informational - Confidence level and reason - Missing evidence that would change the rating 5. Minimization and Containment Controls Propose controls tied to the observed lineage, not generic policy slogans. For each control, include: - Propagation point addressed - Control objective - Specific change to inputs, prompts, handoff schema, memory policy, retrieval filtering, tool payloads, logging, access control, workspace permissions, retention, or operator procedure - Expected reduction in sensitive context exposure - Owner responsible for implementation - Verification method - Residual risk Prioritize smallest effective controls that reduce propagation without breaking the authorized workflow. Avoid broad rewrites unless the evidence shows the workflow design itself is unsafe. 6. Deletion, Quarantine, and Revalidation Plan Create an action plan with: - Artifact or location requiring deletion, quarantine, redaction, re-indexing, access review, or retention change - Reason action is needed - Required owner approval: data owner, security reviewer, legal/privacy owner, product owner, platform owner, or customer account owner as appropriate - Preconditions before action - Execution evidence needed after action - Revalidation test or sampling method - Rollback or exception handling if deletion would impair legal hold, auditability, customer support, or service reliability Do not state that deletion, quarantine, redaction, or re-indexing has been completed. State only the plan and the evidence needed to verify completion. 7. Open Questions and Owner Decisions List unresolved questions that materially affect risk or remediation. For each, identify: - Question - Why it matters - Evidence needed - Accountable owner from [Accountable owners], or owner not identified - Decision deadline if inferable from [Known incidents or concerns] 8. Completion Check End with a completion check stating whether the audit is ready for owner review. Include: - Whether every sensitive context category in [Sensitive context categories] was traced or marked not evidenced - Whether every material propagation path has an evidence reference or uncertainty label - Whether contamination findings are tied to actual lineage evidence - Whether minimization controls map to specific propagation points - Whether deletion and revalidation actions identify accountable owners and verification evidence - Remaining blockers before security reviewer, data owner, product owner, or platform owner decisionInput for this step
Provide data classifications, RAG and agent flow, representative payloads, memory and tool configuration, logs, retention rules, tenant and purpose boundaries, and prior-step findings.
Carry forward
Produce the final Ready, Restricted, Revalidate, or Hold record with freshness work, retrieval evidence, authorization and context-lineage findings, owners, restrictions, regression criteria, and re-review triggers.
Review note
The knowledge owner, data owner, security and privacy reviewers, product owner, and release owner approve the readiness disposition and residual risk.
Completion criteria
The workflow is complete when:
- Decision-critical knowledge has a freshness status, owner, review trigger, and revalidation priority.
- Retrieval configuration is supported by measured evidence or an explicit controlled experiment; unchanged verified configuration may be marked Not applicable.
- Authorization and sensitive-context findings are traceable across ingestion, retrieval, cache, citation, memory, tools, logs, and handoffs.
- The final readiness state is Ready, Restricted, Revalidate, or Hold with measurable conditions and owners.
- Uninspected systems, unrun experiments, and unavailable evidence remain explicit.
Related Workflows
Browse WorkflowsInvestigate an AI Agent Security Incident
Reconstruct an AI agent incident, trace delegated authority and sensitive context, conditionally investigate memory or RAG authorization, and prepare evidence-based containment and recovery gates.
Review Production AI Evaluation Reliability
Determine whether evaluation data, automated judges, production comparability, and retrieval experiments provide reliable evidence for an AI release or operating decision.
Plan a SaaS Portfolio Consolidation Decision
Reconcile application usage and ownership, evaluate true portfolio redundancy and dependencies, prepare vendor renewal or exit paths, and issue an evidence-backed consolidation decision.
Was this useful?