A successful AI deployment depends on far more than model quality. The production system must control data access, route requests safely, manage cost, recover from failures, and show how every important decision was made. These concerns become more important when the system handles health, education, financial, or customer data.
This guide presents a practical reference architecture for cloud, private cloud, and on-premises deployments. It also explains how to select models, build retrieval and agent workflows, evaluate changes, and decide when self-hosting is worth the operational effort.
Start with the workload
Model selection should begin with the business task and its risk. A model that is excellent for complex reasoning may be wasteful for ticket classification. A small classifier may be more accurate, faster, and easier to operate for a narrow task. The right choice depends on quality, latency, data sensitivity, volume, cost, and the ability to operate the model reliably.
| Workload | Typical model | Good starting point | Measure first |
|---|---|---|---|
| Classification and routing | Small encoder or compact language model | BERT or DistilBERT class models; compact instruction models | F1 score, false positives, p95 latency |
| Search and RAG | Embedding model plus reranker plus generator | Domain-tested embeddings; cross-encoder reranker; grounded generator | Recall@k, answer faithfulness, citation accuracy |
| Summarization and extraction | Small or medium instruction model | Structured output model with schema validation | Field accuracy, omission rate, cost per document |
| Coding and complex reasoning | Frontier or strong open-weight reasoning model | Provider API or carefully sized self-hosted model | Task success, review effort, latency, total cost |
| Agents and tool use | Model with reliable tool calling | Model plus bounded controller and explicit permissions | End-to-end success, tool errors, unsafe actions, steps per task |
| Speech, image, and document processing | Specialized multimodal model | ASR, OCR, vision, or document model selected by domain | Domain error rate and human correction time |
A reference production architecture
A useful architecture separates application logic, policy enforcement, model execution, and data services. This prevents each application team from implementing security, logging, and provider integration in a different way.
- Application layer: chat, search, workflow, API, and agent experiences.
- AI gateway: authentication, authorization, routing, quotas, request filtering, audit events, and provider isolation.
- Model layer: managed APIs, private endpoints, or self-hosted inference pools.
- Data and tool layer: retrieval indexes, operational databases, approved APIs, and MCP servers.
- Operations layer: evaluation, tracing, cost reporting, incident response, model registry, and release controls.
Applications should usually call a stable internal AI service rather than a provider endpoint directly. The gateway can map a logical name such as clinical-summary or support-triage to an approved model version. This allows the platform team to change providers or deployment locations without changing application code. Some small systems can begin with a simpler shared client library, but the control point should become a service before many teams or regulated workloads depend on it.
Cloud, private cloud, or on-premises
Regulation does not automatically require on-premises deployment. HIPAA-regulated workloads can use cloud services when the organization completes the required risk analysis, signs an appropriate business associate agreement, configures the service correctly, and maintains the required safeguards. FERPA decisions also depend on the role of the provider, contractual controls, consent or an applicable exception, and the institution’s recordkeeping duties.
The practical question is where the data may be processed and stored, who can access it, how long it is retained, and whether the organization can prove that its controls work. A private network boundary alone does not answer those questions.
| Deployment | Best fit | Main advantages | Main tradeoffs |
|---|---|---|---|
| Managed model API | Fast experiments, variable demand, access to frontier capabilities | Low startup effort, rapid model improvements, elastic capacity | Usage pricing, provider dependency, data and retention review |
| Private managed endpoint | Regulated or enterprise workloads needing stronger isolation | Private networking, enterprise identity, managed operations | Higher cost and some provider constraints |
| Self-hosted in cloud or data center | Stable high volume, strict control, custom models, offline needs | Control of weights, data path, retention, and scheduling | GPU capacity, upgrades, security patching, reliability ownership |
| Hybrid | Mixed sensitivity or bursty demand | Routes each workload to the right boundary | More policy, testing, and operational complexity |
Model families and where they fit
The examples below are a selection, not a ranking. Model availability, licensing, pricing, and performance change often. Confirm the current model card and license, then run an evaluation set built from your own data before committing to a platform.
| Model | Access | Often a good fit for | Important note |
|---|---|---|---|
| OpenAI GPT family | Commercial API | General reasoning, coding, multimodal applications, structured tool use | Choose the smallest model that meets the quality target; check current API lifecycle and data controls |
| Anthropic Claude family | Commercial API and cloud platforms | Long documents, coding, analysis, and agent workflows | Capabilities and availability vary by model and hosting platform |
| Google Gemini family | Commercial API and Google Cloud | Multimodal work, long context, search-connected and real-time experiences | Select a stable model version for production and test media handling |
| Meta Llama | Open-weight under Meta license; commonly distributed through Hugging Face | Private deployment, fine-tuning, broad ecosystem support | License terms, safety controls, and hardware needs vary by release and size |
| Qwen | Open-weight models and hosted services; many releases on Hugging Face | Multilingual work, coding, reasoning, and compact local deployment | Evaluate the exact checkpoint and quantization on domain data |
| Mistral and Mixtral | Commercial APIs and open-weight releases | Efficient general use, coding, European deployment options | Open and commercial releases have different licenses and capabilities |
| DeepSeek | Hosted and open-weight releases | Reasoning and coding when cost or self-hosting is important | Large mixture-of-experts models can still require substantial serving capacity |
| OpenAI gpt-oss | Open-weight releases available through Hugging Face | Local or private text generation and experimentation | Validate the license, safety behavior, memory footprint, and serving stack |
| BERT and DistilBERT | Public checkpoints on Hugging Face | Classification, tagging, search features, and compact CPU inference | Often better than a generative model for narrow prediction tasks |
| Sentence Transformers | Public checkpoints on Hugging Face | Embeddings, semantic search, clustering, and duplicate detection | Embedding quality is domain-specific; pair with a reranker when precision matters |
| Whisper | Public checkpoints on Hugging Face and related hosted options | Speech recognition and transcription | Measure word error rate for accents, noise, vocabulary, and language mix |
| CLIP and vision encoders | Public checkpoints on Hugging Face | Image search, zero-shot image classification, and multimodal retrieval | Specialized medical or industrial images need domain validation |
Hugging Face is a distribution and collaboration platform, not a quality certification. Check the model card, publisher identity, license, training data description, security implications of custom code, and recent community activity. Pin the exact revision in production. Do not load unreviewed remote code or serialized artifacts into a trusted environment.
Routing and inference performance
Routing can improve quality and control cost. A straightforward request may go to a compact model, while a difficult request goes to a stronger reasoning model. Sensitive requests can be restricted to approved endpoints. Failover should consider compatibility because two models may use different tool schemas, safety behavior, context limits, and output formats.
Self-hosted systems commonly use vLLM, NVIDIA TensorRT-LLM, NVIDIA Triton, or similar serving software. Useful techniques include continuous batching, paged key-value caches, prefix caching, quantization, and request scheduling. Their benefit depends on prompt length, output length, concurrency, hardware, and model architecture, so benchmark with representative traffic.
Speculative decoding uses a faster draft process to propose tokens that a larger model verifies. It can reduce latency when the draft predictions are accepted often enough. The speedup is workload-dependent. Output equivalence also depends on the algorithm and decoding settings, so avoid promising a fixed multiplier or identical output without testing the exact implementation.
Secure retrieval for RAG
Retrieval security begins before the model sees any document text. The system should resolve the user’s identity and permissions, apply tenant and record filters during retrieval, and pass only authorized content into prompt construction. If access control is applied after retrieval, sensitive content may already have entered logs, caches, or model context.
- Carry a verified user and tenant identity from the application to retrieval.
- Enforce permissions in the system of record or retrieval service. Do not rely on the model to decide access.
- Use metadata filters, separate collections, database row-level security, or a policy service as appropriate. Products differ, so do not assume every vector database provides native row-level security.
- Log the requester, policy decision, document identifiers, purpose, model version, and result. Limit the log content to what is necessary.
- Protect source documents, embeddings, caches, backups, and evaluation datasets. Embeddings can still reveal sensitive information.
Prompt injection and tool security
Prompt injection can arrive through a user message or through content retrieved from files, websites, email, databases, and tool responses. Filters can help, but they are not a complete defense. The most reliable controls limit what the system is allowed to do.
- Treat retrieved text and tool output as untrusted data, not instructions.
- Use allowlisted tools, narrow permissions, schema validation, and explicit network and file boundaries.
- Require human approval for irreversible, high-value, or high-risk actions.
- Separate planning from execution and recheck authorization at the moment a tool is called.
- Limit steps, time, tokens, spending, and repeated retries. Record every tool call and result.
- Test direct and indirect injection attacks as part of release evaluation.
Guardrail products such as NeMo Guardrails, Guardrails AI, and safety classifiers can support these controls. They should supplement identity, authorization, isolation, and approval workflows rather than replace them.
Agents and MCP
A production agent is best treated as a controlled workflow. The model can select or propose actions, while application code owns state, permissions, timeouts, retries, idempotency, and exit conditions. Independent tool calls may run in parallel, but dependent calls must preserve ordering and transaction rules.
The Model Context Protocol provides a standard way for AI applications to discover and call tools and access resources. Current MCP implementations commonly use local standard input and output or Streamable HTTP for remote connections. Older implementations may use the earlier HTTP plus Server-Sent Events transport. Transport security does not replace authorization: each tool call still needs a verified identity, a narrow scope, input validation, output handling, and an audit event.
Cost and capacity
Cloud APIs are often economical during experimentation and variable demand. Self-hosting can become attractive when utilization is steady, the model fits the workload, and the organization can operate GPUs efficiently. Compare total cost rather than token price alone.
| Cost area | What to include | Useful measure |
|---|---|---|
| Inference | Input, output, reasoning, cached tokens, GPU hours, idle capacity | Cost per successful task |
| Data | Vector database, object storage, backups, network transfer, retention | Cost per document or tenant |
| Platform | Gateway, observability, secrets, registry, CI/CD, security tools | Monthly fixed and variable platform cost |
| People and operations | On-call, upgrades, model evaluation, incident response, capacity planning | Engineering hours per model per month |
| Quality | Human review, correction, failed tasks, rework, business risk | Human minutes and error cost per outcome |
A basic break-even model compares the monthly managed-service cost with hardware amortization, power, facilities, platform licenses, engineering labor, expected utilization, and a risk reserve. Run sensitivity analysis for traffic growth, model changes, GPU availability, and the cost of unused capacity.
Evaluation, observability, and release control
Production monitoring needs both system metrics and outcome metrics. Track time to first token, total latency, queue time, token use, cache hit rate, GPU utilization, tool errors, provider errors, and cost. Pair them with task success, groundedness, retrieval quality, schema validity, safety violations, human corrections, and user outcomes.
Before changing a model, prompt, retrieval policy, or tool, run a versioned evaluation set. Use shadow traffic when privacy rules allow it, then a limited canary rollout with clear rollback thresholds. Keep the model or endpoint ID, prompt version, retrieval configuration, tool version, and policy decision in the trace. For high-stakes use, define where human review is mandatory and what happens when confidence is low.
A practical rollout sequence
- Define the task, users, expected outcome, risk level, and unacceptable failures.
- Create a representative evaluation set before comparing models.
- Choose the simplest model and deployment boundary that meet the target.
- Add identity, authorization, data handling, retention, and audit controls.
- Instrument quality, latency, cost, and tool behavior end to end.
- Load test normal and peak traffic, including provider and tool failures.
- Release through shadow or canary stages and rehearse rollback.
- Review quality, cost, security, and model lifecycle on a regular schedule.
Conclusion
The best production architecture is rarely the one with the largest model or the most components.
It is the one that meets a measured quality target, keeps data within an approved boundary, limits what the system can do, and can be operated at a predictable cost. Begin with the workload, evaluate with real data, and add complexity only when the evidence supports it.