AI Governance That Ships

AI governance is often described as a collection of policies, committees, and approval forms. That definition is incomplete. Good governance sits inside delivery, shaping how an AI system is selected, designed, tested, released, observed, and improved.

The real objective is to make risk visible, assign ownership, establish acceptable boundaries, and create evidence that a system operates as intended, rather than simply to eliminate risk. When governance works, teams move faster because they know the path to production. Leaders can make informed decisions because benefits and risks are measured in the same operating model. Customers and employees receive systems that are more reliable, more secure, and easier to challenge when something goes wrong.

The production paradox

Enterprise adoption is no longer the hard part. Converting adoption into durable business value is.

The adoption figure comes from McKinsey’s 2025 global survey. The P&L finding comes from MIT NANDA’s 2025 GenAI Divide research, which examined interviews, employee surveys, and public AI deployments. It should be read as a finding about that research dataset, not as a universal law that 95% of all AI work fails. The agent cancellation and enterprise application forecasts come from Gartner’s agentic AI research and enterprise application forecast.

Together, these figures describe the central enterprise challenge: governance is the bridge between widespread experimentation and repeatable return on investment.

This article presents a practical approach to enterprise AI governance, with particular attention to generative AI, retrieval augmented generation, and agents that can use tools or take actions.

Governance begins with the use case, not the model

Organizations often start with a model decision: Which provider should we use? Which orchestration framework is best? Should we build a multi-agent system?

Those are implementation questions. Governance starts earlier:

1. What business outcome is the system expected to improve?
2. Who will use it, and who may be affected by it?
3. What information will it receive, create, retain, or disclose?
4. Will it recommend, decide, communicate, or act?
5. What happens when it is wrong?
6. Who owns the result and has authority to stop the system?

A read-only assistant that helps employees find an expense policy is not equivalent to an agent that approves reimbursements. They may use the same model and the same knowledge base, but their authority, consequences, controls, and evidence requirements should be different.

This leads to the first rule of practical AI governance: govern the complete use case, not the model in isolation.

Build an AI inventory before building a committee

An organization cannot govern systems it cannot see. The first durable control is an AI inventory that includes purchased products, embedded features, internal experiments, production applications, models, agents, and significant automated decisions.

Each record should identify:

The inventory should be a living system, not an annual spreadsheet. Model changes, new tools, expanded user populations, new data sources, and greater autonomy can all change the risk profile after the initial approval.

Use risk tiers to scale governance

Applying the same process to every use case creates two predictable failures. Low-risk experiments become unnecessarily slow, while high-risk systems receive superficial reviews because the process cannot afford sufficient depth.

A simple tiering model works well:

Risk tiering should consider both severity and exposure. A rare but catastrophic failure may require strong controls. So may a modest error repeated across millions of decisions.

The NIST AI Risk Management Framework provides a useful structure through four functions: Govern, Map, Measure, and Manage. The NIST Generative AI Profile extends that structure for risks specific to generative systems. ISO/IEC 42001 complements this approach by defining requirements for an organization-wide AI management system and continual improvement.

These frameworks are most valuable when translated into engineering and operating controls rather than copied into a policy document.

A hub-and-spoke operating model

Central governance teams should define the minimum standards, reusable controls, risk taxonomy, and evidence requirements. Product and business teams should own their use cases and remain responsible for outcomes.

In practice, this produces a hub-and-spoke model:

The central team should not become an implementation bottleneck or absorb accountability from delivery teams. Its job is to create paved roads, make expectations explicit, and concentrate scrutiny where the consequences are highest.

Context is a governed product

Many enterprise AI failures attributed to the model are actually context failures. The system retrieved the wrong policy, used stale content, combined sources with different permissions, or received a chunk that removed a critical exception.

A shared context library should separate three kinds of information:

• Centrally owned content, such as enterprise policy, approved product definitions, and regulated guidance
• Department-owned content that follows shared metadata, quality, and review standards
• Ephemeral project or session context with explicit expiration and retention rules

Every source should have an owner, effective date, review date, access classification, version, jurisdiction where relevant, and retirement status. Retrieval permissions must be enforced before content reaches the model. In other words, the model must not become an alternative path around normal authorization.

A reliable retrieval pipeline

A strong default for retrieval augmented generation is:

1. Rewrite or expand the query when appropriate.
2. Combine semantic and keyword retrieval.
3. Apply metadata, permission, freshness, and jurisdiction filters.
4. Rerank candidates for relevance.
5. Assemble a bounded context with source identifiers.
6. Generate the answer.
7. Test whether the answer is supported by the retrieved evidence.
8. Refuse, clarify, or escalate when evidence is insufficient.

The implementation choices depend on scale, operating model, and existing infrastructure. Common production options include:

Qdrant supports dense, sparse, and multivector fusion. Milvus provides BM25-based full-text search. Pinecone supports combined dense and sparse retrieval. pgvector can be paired with PostgreSQL full-text search and combined through rank fusion or a cross-encoder. Naming a product is not an architecture decision. Teams still need to test recall, precision, authorization, freshness, latency, operating cost, and recovery behavior against their own corpus.

Semantic chunking often improves retrieval because boundaries follow headings and topic changes rather than arbitrary token counts. Hybrid retrieval helps when a query contains exact product names, policy identifiers, codes, or uncommon terms that dense retrieval may underweight. Reranking adds cost and latency, but it can materially improve the final context. The decision should be based on measured quality and response-time objectives.

Long context windows do not remove the need for retrieval. Sending every available document increases cost and latency, can introduce conflicting instructions, and makes authorization and source traceability harder. A governed system uses a deliberate context budget for standing instructions, retrieved evidence, working memory, and task input.

Agent architecture should make control visible

An agent is a chatbot with a longer prompt only in the loosest sense. An agent may plan, select tools, call systems, maintain state, delegate tasks, and take actions. Every additional capability increases the number of paths the system can follow and the number of controls it needs.

The first architecture question should be whether an agent is needed at all. A deterministic workflow with a model used for one bounded step is often easier to test, secure, and operate than an open-ended agent. Multi-agent designs should be used when specialized roles or parallel work create measurable value, not because the pattern is fashionable.

When a framework is appropriate, evaluate it against the use case:

• Explicit state and transition control
• Persistence, checkpointing, and resumability
• Human approval and interruption points
• Retry, timeout, and compensation behavior
• Tool authorization and isolation
• Traceability across model and tool calls
• Testability and deterministic replay
• Deployment maturity, community health, and upgrade risk

LangGraph is attractive when explicit state, checkpointing, and human interruption are central requirements. AutoGen supports conversational and multi-agent patterns. CrewAI can make role-oriented prototypes approachable. LlamaIndex Workflows fits naturally when retrieval is the center of the application. These tools evolve quickly, so an enterprise standard should specify evaluation criteria and supported patterns, not permanently declare one winner.

Orchestration framework scorecard

The risk tier belongs to the deployed use case, not to the library. No framework makes an action-taking system safe by itself. The table is a starting recommendation that must be revisited as frameworks, capabilities, and organizational controls change.

Tools need contracts and boundaries

Every production capability should be exposed through an explicit tool contract with typed inputs, typed outputs, documented errors, authorization requirements, idempotency behavior, and audit expectations.

The Model Context Protocol can reduce bespoke integration by standardizing how applications expose tools and contextual resources. It does not remove the need for security design. The current MCP specification includes authorization for HTTP-based transports and calls for appropriate consent, access control, and data protection. Its authorization security considerations are essential reading for production implementations.

For each tool, ask:

• Does the agent need this function at all?
• Can read and write operations be separated?
• Can access be limited to the current user, tenant, record type, or transaction?
• Does a consequential action require confirmation?
• Can the call be safely retried?
• What prevents prompt content from becoming an executable instruction?
• Are credentials short lived and scoped?
• Are sensitive inputs and outputs excluded or redacted from logs?

An agent should never receive broad permissions merely because the underlying integration exposes them. Excessive agency is a security design failure, not an unavoidable property of AI. The OWASP GenAI Security Project provides current guidance for both LLM applications and agentic systems, including prompt injection, sensitive information disclosure, supply-chain risk, and excessive autonomy.

Map threats to enforceable controls

The OWASP categories provide a useful threat vocabulary, but the mitigation must exist outside the model’s discretion.

A prompt that says, “Do not perform this action unless the user is an administrator,” is advisory. The API gateway and downstream service must independently authenticate the user and enforce the required OAuth scope or policy. The same principle applies to MCP tools. MCP can carry a secure authorization flow, but the MCP server and the underlying system remain responsible for enforcing least privilege.

OWASP released both an updated LLM Top 10 for 2026 and an Agentic Applications Top 10 for 2026. Control libraries should record the release used, because category names and numbering change over time.

Evaluation is the evidence layer of governance

A successful demonstration alone doesn’t make an AI system production ready. Readiness comes once the organization has defined acceptable behavior, tested representative and adversarial cases, measured the results, and established a way to detect regression.

A practical evaluation program contains several layers.

Deterministic checks

Use conventional tests wherever possible: schema validity, required citations, allowed tool selection, authorization rules, calculation accuracy, response format, prohibited content, latency, and cost limits.

Golden-set regression

Maintain a versioned set of representative inputs and expected properties. Include normal cases, edge cases, ambiguous requests, stale or conflicting sources, missing information, and expected refusals. Run it when prompts, models, retrieval, tools, policies, or orchestration logic change.

Model-based evaluation

Some qualities, such as completeness, tone, and groundedness, require rubric-based judgment. A separate model can assist, but its scores must be calibrated against human reviewers. It should not be treated as an objective oracle or allowed to grade itself without independent checks.

Human and domain review

Subject-matter experts should assess factual correctness and operational usefulness. Security and risk reviewers should test misuse and abuse. People who may be affected by the system should be represented when the use case has meaningful human consequences.

Adversarial testing

Test prompt injection, indirect injection through retrieved documents, data extraction, privilege escalation, tool misuse, cross-tenant access, unsafe delegation, runaway loops, and denial-of-wallet behavior. Agentic systems also need tests for compromised tools, poisoned memory, misleading intermediate results, and unsafe action sequences.

Shadow evaluation and controlled release

Before receiving full authority, run the system on real inputs without exposing outputs or actions. Compare it with the current process. Then release to a limited population, with constrained authority and clear exit criteria, before expanding.

Most importantly, connect technical metrics to business outcomes. Retrieval precision is useful, but it does not prove that employees resolve questions faster. A high faithfulness score is useful, but it does not prove that the underlying policy is current. Governance needs both system measures and outcome measures.

A concrete evaluation toolchain

Teams can build a common harness around open-source and commercial tools such as Ragas, DeepEval, Arize Phoenix, or TruLens. The selection matters less than four operating properties: datasets are versioned, evaluators are calibrated, results are reproducible, and failures block or constrain release when required.

The following targets are a useful starting service profile for a bounded, user-facing RAG assistant. They are not universal industry thresholds and must be calibrated against domain risk, judge reliability, and the current baseline.

Scores produced by different models, prompts, and frameworks are not automatically comparable. Store the evaluator version, rubric, model, dataset, and confidence information with every reported result.

Review gates should be explicit and economical

The review process should apply inexpensive checks before expensive ones:

1. Confirm the use case, owner, users, and risk tier.
2. Validate architecture, tool schemas, and system boundaries.
3. Review data handling, permissions, privacy, retention, and vendors.
4. Review security threats and mitigations.
5. Run quality, safety, and business evaluations.
6. Verify observability, incident response, fallback, and rollback.
7. Complete shadow or limited release.
8. Approve, approve with conditions, return for changes, or reject.

The evidence for each gate should be stored with the system record. Exceptions should name an accountable approver, compensating controls, expiration date, and remediation plan.

Production governance is continuous

Approval is the beginning of production governance, not the end. AI behavior can change because a provider updates a model, a prompt changes, documents are revised, retrieval indexes are rebuilt, tools are added, users behave differently, or external conditions shift.

Every production system needs:

• Versioned prompts, models, policies, retrieval configurations, and tool definitions
• Separate development, test, and production credentials and limits
• End-to-end trace identifiers across model, retrieval, and tool calls
• Quality, safety, latency, cost, refusal, escalation, and outcome metrics
• Data minimization and redaction in telemetry
• Alerts tied to service and risk thresholds
• A tested kill switch, rollback path, and degraded mode
• Scheduled and event-driven reassessment

OpenTelemetry’s emerging generative AI conventions provide a useful foundation for consistent traces, metrics, and events across providers. Observability should record enough information to reconstruct a failure without turning the telemetry platform into an uncontrolled repository of prompts, personal information, proprietary documents, or credentials.

Use W3C distributed trace context, including traceparent, to connect the complete execution path:

At minimum, capture provider and model identifiers, operation name, input and output token counts, latency, retries, finish reason, tool name and outcome, retrieval identifiers and scores, cache use, user or tenant cost allocation, and evaluation results. Current OpenTelemetry naming uses gen_ai.usage.input_tokens and gen_ai.usage.output_tokens; the earlier prompt-token and completion-token attributes are deprecated. Recording full prompts, retrieved text, tool arguments, or results should be opt-in and governed because those fields may contain confidential or personal information. See the OpenTelemetry GenAI observability guidance for the evolving standard.

Change control should be based on materiality. A copy edit may need regression tests. A new model, new data source, expanded user population, changed decision authority, or new write-capable tool may require formal reassessment and renewed approval.

Incident response must include AI-specific failures

Traditional operational response remains necessary, but AI incidents have additional dimensions. The response plan should cover:

• Harmful, discriminatory, deceptive, or materially incorrect outputs
• Unauthorized data disclosure or cross-tenant retrieval
• Prompt injection and tool manipulation
• Actions taken beyond the intended authority
• Systematic citation or grounding failures
• Unexpected model or vendor behavior changes
• Cost spikes, loops, and resource exhaustion
• Inability to reconstruct why an action occurred

Response should include containment, evidence preservation, stakeholder notification, impact analysis, correction where possible, root-cause analysis, and changes to tests or controls. The evaluation suite should learn from every significant incident.

Vendor governance is part of system governance

Buying an AI feature does not transfer accountability to the vendor. Procurement and architecture reviews should examine:

• Whether customer data is used for training or product improvement
• Retention, deletion, residency, encryption, and subprocessors
• Model and feature change notification
• Security testing and vulnerability response
• Service levels, rate limits, and continuity options
• Audit evidence and regulatory commitments
• Intellectual-property and content ownership terms
• Options to export data, evaluations, and logs
• The ability to disable AI features or terminate cleanly

Vendor claims should be translated into controls the organization can verify. Where a system has high impact, contractual assurance alone is not a substitute for independent evaluation.

Regulatory alignment without checkbox governance

Legal obligations depend on jurisdiction, industry, role, and use case. Governance teams should maintain a control map that connects one operating control to multiple obligations.

For example, an AI inventory supports ownership, risk classification, impact assessment, and regulatory reporting. A strong evaluation record supports internal approval, customer assurance, and evidence of risk management. Access controls and audit trails support security, privacy, and accountability requirements.

The European Union’s AI Act uses a risk-based structure and became broadly applicable on August 2, 2026, with specified exceptions and extended timelines for some high-risk systems. Organizations operating in or serving the EU should determine whether they act as a provider, deployer, importer, or distributor and map obligations accordingly. Legal counsel should validate that analysis.

Compliance is a floor. A system can satisfy a documentation requirement and still frustrate users, produce poor results, or create operational risk. Effective governance asks both, “Are we permitted to do this?” and “Should we do it this way?”

A concrete example: an employee expense-policy assistant

Consider a finance team that wants an assistant to answer employee questions about expense policy.

The first decision is scope. If the system only answers questions, it is an assistive or advisory use case. If it can approve expenses, change records, or initiate payments, it becomes an action-taking system with significantly stronger requirements.

For a read-only assistant, the approved architecture could include:

• Retrieval from current policy documents with ownership and effective dates
• Permission filtering before retrieval
• Citations in every substantive answer
• A refusal or escalation path when policies conflict or evidence is weak
• A narrowly scoped lookup tool for the current employee’s submitted reports
• Golden-set questions drawn from historical support cases
• Tests for policy exceptions, missing receipts, jurisdiction differences, and indirect prompt injection
• Logging of sources, outcomes, escalations, latency, and user feedback

Success should be measured with more than answer quality. The team should track time to resolution, escalation rate, repeated questions, incorrect-answer reports, employee satisfaction, and whether policy owners are learning where the documentation itself is unclear.

If the team later proposes automatic approval, that is a material scope change. It requires a new risk assessment, authorization design, transaction limits, human approval rules, action reconciliation, and a much stronger evidence package.

A practical 90-day implementation path

Organizations do not need to design the entire governance program before learning from real use cases. They do need enough structure to learn safely.

Days 1 through 30: establish visibility and ownership

• Appoint an executive sponsor and cross-functional working group.
• Create the initial AI inventory and identify unregistered use.
• Define risk tiers, prohibited uses, and interim escalation rules.
• Select three representative use cases, including one with tool use.
• Establish minimum privacy, security, legal, and human-oversight controls.

Days 31 through 60: create the paved road

• Define the intake form and evidence requirements.
• Create approved architecture patterns for assistants, RAG, and agents.
• Establish context metadata, retrieval, and access standards.
• Build a shared evaluation harness and initial adversarial test library.
• Standardize observability and incident classification.

Days 61 through 90: prove and improve the system

• Run the pilot use cases through review and shadow evaluation.
• Measure review time, remediation effort, quality, safety, and business outcomes.
• Convert repeated findings into templates, automated checks, and reusable components.
• Train product, engineering, risk, and business teams using actual pilot lessons.
• Publish the operating model and schedule the first program review.

The program should be judged by evidence: inventory coverage, time from intake to decision, first-pass approval rate, evaluation coverage, incident rate, time to containment, exception aging, user outcomes, and realized business value.

Governance should earn its place

The strongest AI governance programs are technically credible, proportional to risk, and connected to business outcomes. They do not confuse more paperwork with more control. They create visibility, clarify accountability, establish reusable engineering standards, and demand evidence at the points where mistakes matter.

That is how governance earns its place: by making responsible scale possible, without slowing AI down.