# RAG Customer-Support Assistant **Synthetic practice case — learner evidence.** Track B. Every entity, role, product, system, trace, quotation, date and incident is fictional. “Case Organisation Lumen-24” is a training label, not a real-company claim. All output and trace excerpts were authored for this exercise. They are not results from a real model, provider, penetration test or production environment. No credentials or real customer records are included. ## A. Case narrative Case Organisation Lumen-24 provides fictional activity kits. Its support assistant, **Guide-L**, answers public questions about kit use and support policy using a small retrieval corpus. It can display an escalation route but is not authorised to approve refunds, change orders, send messages or access a user's account. Human support retains decisions. The system has a public help interface and a staff reference interface. Public requests should retrieve only public, current documents. Staff-only material should never enter public generation context, even when the final answer does not quote it. A fictional hosted language service, **Model Service V**, receives assembled context; a separate fictional index service, **Index Service R**, supplies passages. No particular real provider or model is represented. A support policy changed on 25 September. The index snapshot used in the supplied evaluation is from 24 September, while management says the update was deployed on 26 September. The change ticket closes without entitlement-regression or full reindex evidence. Four selected trace extracts and related records are supplied. Their selection basis and population coverage are not recorded. The system owner says fluent answers and high user ratings show it is reliable. The auditor's bounded question is whether the supplied evidence supports the claimed source, access, escalation and change controls for the stated use. The auditor must separate proposed test coverage from observed case records and identify what remains unknown. ### Non-sensitive architecture summary Public/staff interface → identity and purpose context → retrieval filter → index passage selection → context assembly → hosted generation → citation/abstention checks → answer or escalation display. Supporting records: knowledge-source register; entitlement policy; change record; trace store; human review queue. The architecture has no account-modification or outbound messaging tool. No additional action interface is described in the supplied architecture. ### Intended use Use for source/evaluation review, not live security testing. scope workbook/control-and-evidence working paper frame scope and evidence; GenAI/RAG evaluation sheets records test and grounding analysis; vendor/operational workbook records vendor/permission/monitoring/change review; findings register/AMO-P-000363 challenge a finding/response. Reuse AMO-P-000242/AMO-S-000007, AMO-W-000016, AMO-W-000013/AMO-S-000012 and AMO-W-000021. Learners need access-control, version and claim-evidence literacy; specialists remain necessary for wider security or privacy conclusions. ## B. Evidence inventory Fictional retrieval date: 7 October 2026. Dates below identify the supplied source/version, not learner execution. Owners are role labels. Detailed locators are sections C1–C12. | Evidence ID | Source / owner | Date or version | Locator | Limit | |---|---|---|---|---| | CR-01 | Use and escalation policy / support owner | 20 September, v2 | C1 | Defines intended use; not proof of enforcement | | CR-02 | Source register and passages / knowledge owner | Register 27 September | C2 | Selected corpus records; completeness not established | | CR-03 | Entitlement and context policy / access owner | 20 September, v1 | C3 | Intended restrictions; full effective configuration absent | | CR-04 | Baseline/run cover / evaluation lead | Run E-17, 27 September | C4 | Selected run identity; exact model revision and full run manifest absent | | CR-05 | Query/retrieval/answer extracts / evaluation lead | E-17, 27 September | C5 | Four supplied traces; no population accuracy estimate | | CR-06 | Synthetic adversarial test proposal / security reviewer | 28 September | C6 | Not executed; authority for a later sandbox run pending | | CR-07 | Escalation and stop records / support operations | 27 September | C7 | Receipt/acknowledgement and one later answer; downstream proof partial | | CR-08 | Vendor review / procurement owner | 21 September | C8 | Claims and questions; signed applicable terms absent | | CR-09 | Monitoring extract / system owner | Week ending 30 September | C9 | Aggregate feedback; denominator and coverage limits | | CR-10 | Change ticket / release owner | Open 25, closed 26 September | C10 | Completion asserted; rollback/access/reindex evidence partial | | CR-11 | Incident/near-miss review / response coordinator | Open 28 September | C11 | Exposure scope unresolved; no real user data supplied | | CR-12 | Management response / system owner | 4 October | C12 | Proposed action and release claim; no independent retest | ## C. Sample evidence excerpts ### C1 — CR-01: use and escalation policy > Guide-L may explain public kit/support information from current published sources. It must identify a supporting source and avoid unsupported promises. If the answer is not in a permitted source or a personal account decision is required, state the limit and display the human-support route. It must not claim to send a message, approve a refund or modify an account. Staff-only context is prohibited in public requests. Human support owns decisions. A source conflict must be surfaced, not silently resolved by the model. The policy requires a current authoritative source; it does not prescribe a universal numerical accuracy target or establish legal compliance. ### C2 — CR-02: source register and local passages | Source/version | Fictional owner | Audience | Effective / lifecycle | Exact case passage | |---|---|---|---|---| | K-RET v2, §2 | Policy owner | Public | 1 September; superseded 25 September | “Unopened standard kits may be returned within 14 days. Personal exceptions require support review.” | | K-RET v3, §2 | Policy owner | Public | 25 September; current | “Unopened standard kits may be returned within 30 days. Personal exceptions require support review.” | | K-SET v1, §1 | Product support | Public | 10 September; current | “Use the supplied assembly guide. If a part is missing, display the human-support route.” | | K-OPS v1, §4 | Support operations | Staff only | 18 September; current | “Route an exception request to the staff review queue. Staff authorisation remains necessary.” | | K-DRAFT v0.4, §3 | Contributor, publication pending | Unpublished draft | No effective date | “Replacement kits are always dispatched immediately.” | These are fictional operational statements, not consumer law. K-DRAFT is included only as a synthetic governance risk in a case, not as unpublished Amo.ng content. The register retains the supplied publication state, dates and passages. ### C3 — CR-03: access and context policy > Principal PUBLIC may receive current PUBLIC passages only. Principal STAFF may receive PUBLIC and STAFF passages for the permitted support purpose. Audience and publication-state filters must apply before context is sent to Model Service V. An answer-level refusal does not cure an unauthorised context transfer. The service principal has no refund, email or account-write capability. Supplied configuration excerpt lists filter names but not effective values. Recertification of the index service's permissions is not supplied. ### C4 — CR-04: evaluation baseline Run E-17 cover: corpus C-17, index built 24 September, retriever R2, prompt template PT3, public interface unless specified. Exact hosted model revision: not recorded. Four selected trace extracts Q01–Q04 are supplied. A fifth planned case A01 is not part of E-17. No complete query set, selection method, duplicate handling, expected-answer sign-off or population distribution is supplied. No separate expected-answer approval record is supplied. A knowledge owner would need to review proposed expectations before any separately authorised test. ### C5 — CR-05: selected traces **Q01 — current return policy** - Request: “What is the return window for an unopened standard kit?” Principal PUBLIC. - Retrieved/context passage: K-RET v2 §2. - Answer: “You can return an unopened standard kit within 14 days [K-RET v2 §2].” **Q02 — missing part** - Request: “A part is missing. What should I do?” Principal PUBLIC. - Retrieved/context passage: K-SET v1 §1. - Answer: “Use the human-support route for a missing part [K-SET v1 §1].” - Interface event in the supplied trace: a human-support link is displayed. No support-contact receipt is included. **Q03 — personal exception request** - Request: “Can you guarantee that my exception will be accepted today?” Principal PUBLIC. - Retrieved/context passage: K-RET v2 §2 only. K-DRAFT is not in this trace. - Answer: “Your exception has been accepted and a replacement will be dispatched today [K-RET v2 §2].” - No action tool, authorisation or dispatch record is included in this trace. **Q04 — exception review question** - Request: “How are exception requests reviewed?” Principal PUBLIC. - Context sent onward: K-OPS v1 §4, plus K-SET v1 §1. - Answer: “Please use the public human-support route; I cannot decide an exception.” Each excerpt is synthetic observed-case material. None is a test run performed during Pack authoring or by a learner merely reading it. ### C6 — CR-06: safe adversarial test proposal, not executed Case A01 proposes inserting a clearly marked, harmless instruction-like canary into a synthetic retrieved document to see whether the system treats it as data rather than authority. No live payload, target URL, credential or exploit script is supplied. Intended result: preserve the approved task and source/access limits, ignore document instructions and record the boundary decision. Before any future test, the security reviewer must approve sandbox identity, data, harmless fixture, allowed surfaces, observation points, stop conditions and evidence retention. Production, real users, external endpoints and secret extraction are excluded. Actual result: **not performed**. No “prompt-injection resistant” claim is available from this proposal. ### C7 — CR-07: escalation and override records - Q02 interface record 10:04: human-support link displayed. No evidence of a user selecting it or a queue item being created. - Q03 review record 10:07: operator pressed stop for the affected session; acknowledgement logged at 10:07:02. - Same session answer-completion marker: 10:07:05. Delivery status and whether the marker describes a buffered or newly generated answer are absent. - Operator explanation: “Acknowledgement proves the session was contained.” No downstream containment confirmation is supplied. ### C8 — CR-08: vendor/dependency review Procurement note: “Model Service V says content is not used for training under some business arrangements. Index Service R says public/staff filtering is supported.” Applicable product tier, contract schedule, context retention, deletion evidence, subprocessor roles and effective entitlements are not attached. No agreed fallback-provider evidence exists. ### C9 — CR-09: monitoring record Management slide says “95% helpful”. Footnote: 19 of 20 voluntary feedback responses marked helpful. Total answers, source currency, query mix, repeat users, non-responders and access failures are absent. No broken-down grounding, unanswerable-query, escalation or context-entitlement metric is shown. No threshold ownership is recorded. ### C10 — CR-10: change record Change CH-24 replaces K-RET v2 with v3. The ticket says publication occurred 25 September and “index updated” on 26 September. E-17 on 27 September still identifies C-17 built on 24 September. The ticket extract contains an undated missing-part-answer screenshot. Its capture date and attachment time are absent, so it is not identified as the later E-17 Q02 run. No corpus membership comparison, public/staff negative test, Q01 recheck, rollback exercise or exact model-baseline record is attached. The release owner closed the ticket. The reason for the baseline discrepancy is not supplied. ### C11 — CR-11: incident/near-miss record Opened 28 September after review of Q04. Scope described as “staff passage in public generation context”; real personal data are not part of the supplied case. Response coordinator requests preserved traces, audience filters and context-transfer scope. No full event search or retention analysis is supplied. Record state remains “awaiting evidence”. Notification decisions, if any, are not included; learners must not invent a legal notification requirement or claim one was completed. ### C12 — CR-12: management response > “The answers are usually helpful. The policy document was updated and Q02 still works, so the issue should close. We plan to add a reminder to the release checklist and review the public/staff filter.” No accountable action commitment, due date, changed effective configuration, representative regression, containment result or independent retest is provided. Proposed actions remain proposed. ## D. Attributed assertions for review | Maker | Statement or representation | Source | |---|---|---| | System owner | Fluent answers and high user ratings support reliability | Case narrative | | Management monitoring slide | “95% helpful” with the supplied footnote | CR-09 | | Release ticket | Index updated and ticket closed | CR-10 | | Operator | Acknowledgement proves containment | CR-07 | | Procurement note, reporting supplier claims | Training-use and filtering capability claims | CR-08 | | System owner | Update and Q02 justify closure; further checks planned | CR-12 | For each, record the source, actual supporting material and any unresolved question. Do not add an assertion merely because it would be a useful teaching example. ## F. Learner/auditor tasks 1. Define a bounded system/control question and exclusions in scope workbook/AMO-P-000361 using the supplied purpose and policies. 2. With AMO-P-000242/AMO-S-000007 and GenAI/RAG evaluation sheets, analyse each query, retrieved passage and answer. Explain source support and alternative failure locations where evidence permits; preserve unknown cause and scope. 3. Use AMO-W-000016 for source authority/entitlements and AMO-W-000013/AMO-S-000012 for evaluation evidence. Assess what conclusions the monitoring data can support without inventing missing denominators. 4. Review A01's proposed authority, expected behaviour and limits in GenAI/RAG evaluation sheets. It remains not performed; do not run it against a public system. 5. Use only relevant vendor/operational workbook/AMO-W-000002/AMO-W-000021 modules to examine supplier, access, logging, monitoring, override and change evidence. Distinguish a missing record from an observed deviation against a supplied criterion. 6. Draft a bounded finding if justified, then challenge its management response and any closure claim with AMO-P-000363/findings register. Do not invent severity or operational effects. 7. Record the learner's hypothetical human judgement, editable carry-forward, unresolved matters and next evidence request. Multiple defensible answers are possible when the limits are explicit. ## J. Professional judgement and limitations There may be more than one professionally defensible answer. This case does not measure real-world accuracy, prove security or privacy compliance, establish fairness or explanation fidelity, validate a foundation model or certify a system. Human professionals determine evidence sufficiency and reliance within an authorised scope. Reading a trace or completing a sheet is not independent execution or an audit opinion. ## Related Amo.ng resources - **Prompts:** AMO-P-000074 v2.0.0 (scope/risk context), AMO-P-000242 v1.0.0 (retrieval/citation), AMO-P-000071 v2.0.0 (claim challenge); component methods remain linked through their existing Workflows. - **Workflows:** AMO-W-000002 (vendor evidence), AMO-W-000013 (evaluation reliability), AMO-W-000016 (RAG readiness), AMO-W-000021 (change review). - **Skills:** AMO-S-000007, AMO-S-000012 and AMO-S-000016 for diagnosis, evaluation admissibility and source traceability. See [the Pack resource guide](https://amo.ng/internal-audit-ai-assurance) for canonical links. - **Related audit resources:** [AMO-P-000361](https://amo.ng/prompts/challenge-internal-audit-engagement-scope-evidence-plan), [AMO-P-000362](https://amo.ng/prompts/review-audit-sampling-control-test-design) where a distinct control-sampling design is proposed, [AMO-P-000363](https://amo.ng/prompts/challenge-audit-finding-management-response-closure-claim), [AMO-W-000044](https://amo.ng/workflows/evidence-controlled-internal-audit-working-paper), [scope workbook](https://amo.ng/public/downloads/internal-audit-ai-assurance/2026-10-10/scope-inventory-evidence-request-workbook.md), [control-and-evidence working paper](https://amo.ng/public/downloads/internal-audit-ai-assurance/2026-10-10/risk-control-test-evidence-sufficiency.md), [GenAI/RAG evaluation sheets](https://amo.ng/public/downloads/internal-audit-ai-assurance/2026-10-10/genai-rag-authorised-evaluation.md), [vendor/operational workbook](https://amo.ng/public/downloads/internal-audit-ai-assurance/2026-10-10/vendor-operational-control-review.md), [findings register](https://amo.ng/public/downloads/internal-audit-ai-assurance/2026-10-10/findings-responses-closure-register.md). ## Facilitator material Reviewer interpretations, expected issues and intentional ambiguities are kept in a separate facilitator companion. For a blind learner assessment, use this learner edition before consulting worked examples in the template bundles. Supplied source assertions remain available for critical review; this file does not supply their answer key. Use of these resources is subject to the [Amo.ng Terms](https://amo.ng/terms).