LLM Governance: Audit trails, approvals and explainability
A practical guide to governing LLM applications - audit trails, human-in-the-loop approval gates, explainability and reproducibility, and data retention you could show an auditor.
Most of the LLMOps stack is about making a system work. Governance is about being able to answer for it - to a customer, a regulator, or your own incident review. It’s the layer teams notice last and need most once an LLM makes decisions that matter.
The two questions
Prototype: “Does it give a good answer?” Production: “Who approved this, what exactly did it do, and can you prove it after the fact?”
A demo is judged on output quality. A governed system is judged on accountability: a record of what happened and why, a human in the loop where the stakes demand one, and the ability to reproduce any decision.
What breaks without it
- No audit trail. A customer disputes an outcome and you have no record of the input, the context, or what the model returned. You can’t investigate, let alone defend it.
- Unexplainable decisions. “The AI decided” is not an answer a regulator accepts. Without recorded rationale and sources, every decision is a black box.
- Unreviewed high-risk actions. The model takes an irreversible action - a refund, an account change, a clinical suggestion - with no human sign-off.
- Silent drift. A provider updates the model under you; last month’s decisions are no longer reproducible because nothing was version-pinned.
- Retention and privacy breaches. Sensitive inputs sit in logs forever with no policy - a liability waiting for a subject-access request.
- No owner. When something goes wrong, no one is accountable per layer.
The controls that matter
- Audit trail - capture inputs, retrieved context, outputs and decisions immutably, with who/what/when.
- Human-in-the-loop & approval gates - a designed review step for high-risk actions, not an afterthought.
- Explainability & reproducibility - record the rationale and sources behind a decision, and pin model + prompt versions so it can be reproduced.
- Data retention & privacy - a defined, enforced policy for how long records live and who can read them.
- Incident process - a path to triage, contain and learn from model failures.
- Accountability - a named owner for each layer of the stack.
Frameworks like the NIST AI Risk Management Framework and the OWASP Top 10 for LLM Applications (both cited below) are the reference points to map these controls against.
Instrument it: the decision record
Governance becomes real when every consequential decision writes an immutable record:
{
"decision_id": "dec_8842",
"request_id": "req_9931",
"actor": "support-agent-v3",
"model": "sonnet-4.6",
"prompt_version": "v12-locked",
"inputs_redacted": true,
"sources": ["policy/refunds#window", "ticket/55831"],
"rationale": "Within 30-day window; auto-approved per policy R-12",
"human_review": "not_required",
"outcome": "refund_approved",
"retention_class": "7y",
"timestamp": "2026-06-08T09:14:02Z"
}The prompt_version, model and sources fields are what make the decision
reproducible and explainable; human_review and retention_class are what
make it accountable and compliant.
Instrument it: the approval gate
For high-risk actions, separate deciding from acting - and require a gate:
def execute(decision):
if decision["risk"] == "high" and not decision.get("approved_by"):
return enqueue_for_human_review(decision) # nothing happens yet
record_audit(decision) # immutable log first
return perform_action(decision)Nothing high-risk takes effect until a human (or an explicit policy) approves, and the audit record is written before the action, not after.
Minimal vs mature
| Aspect | Minimal | Production-grade |
|---|---|---|
| Audit | Basic logs | Immutable, access-controlled trail |
| Review | Ad-hoc | Mandatory sign-off for high-risk |
| Explainability | None | Rationale + sources per decision |
| Versioning | Latest model | Pinned model + prompt versions |
| Retention | Default / forever | Policy-driven retention & deletion |
| Ownership | Unclear | Named owner per layer |
Where this lives in a real system
Governance isn’t a bolt-on - it’s wired through the architecture. See the regulated LLM workflow reference architecture for where the policy gate, human-review queue and audit store sit, and the governance items in the Production Checklist for the bar to clear. For the bigger picture, what is LLMOps? frames how governance relates to the rest of the stack.