LLMs in Production

Moving a large language model (LLM) feature from demo to dependable product is an engineering discipline of its own. A prototype that answers impressively in a notebook faces different constraints in production: non-deterministic outputs, adversarial and out-of-distribution inputs, evaluation without a single ground truth, and unit economics measured in tokens. This article summarizes the established architecture choices and operational practices. For adjacent topics see MLAIA's posts on enterprise LLM integration, agentic AI in production, and why most agentic pilots never ship.

Prompting, RAG, or fine-tuning

Three mechanisms adapt a general model to a task, and they solve different problems:

• Prompt engineering (instructions, few-shot examples, structured output constraints) changes behavior. It is the cheapest lever, requires no training data, and should be exhausted first.
• Retrieval-augmented generation (RAG) changes knowledge: relevant documents are retrieved — typically via embedding similarity plus keyword search and a reranker — and inserted into the context. RAG suits fast-changing or proprietary corpora, provides citations, and allows per-user access control, since retrieval can respect document permissions.
• Fine-tuning (usually parameter-efficient, e.g. LoRA) changes form and skill: consistent style, domain jargon, structured-output reliability, or narrow-task accuracy on a smaller, cheaper model. It is the wrong tool for injecting facts — updated knowledge still requires retrieval or retraining.

The approaches compose: a fine-tuned small model with RAG and careful prompting is a common production endpoint. The pragmatic sequencing is prompting → RAG (if the failure is missing knowledge) → fine-tuning (if the failure is form, cost, or latency).

Evaluation

LLM systems require layered evaluation because outputs are open-ended. Standard practice combines: a golden set of curated input–expected-output pairs run on every change (prompt edits are code changes and regress like code); automatic metrics where the task permits (exact match, groundedness checks against retrieved sources, schema validity for structured output); LLM-as-judge scoring for fluency and helpfulness — useful but biased (position, verbosity, self-preference), so it should be calibrated against periodic human review; and online monitoring of user feedback, refusal rates, latency, and cost. For RAG, retrieval is evaluated separately from generation (recall of the relevant passages), since retrieval failures masquerade as model failures.

Hallucination mitigation

Hallucination — fluent, confident, false output — is a property of the technology, not a bug that disappears with scale. Mitigations that work in practice: grounding answers in retrieved sources and requiring citations; instructing and rewarding abstention ("I don't know" as a first-class answer); constraining tasks to extraction and summarization over open-ended generation where feasible; verifier passes that check claims against sources before display; structured outputs validated against schemas; and routing high-stakes outputs to human review. Measurement matters as much as mitigation: track groundedness on the golden set so regressions surface before users find them.

On-premises and small language models

Not every deployment can send data to a hosted API. Data-residency requirements, regulated data (health, defense, finance), latency floors, and predictable cost at high volume all push toward self-hosted open-weight models and small language models (SLMs) in the roughly 1–15B parameter range. Quantization (e.g. 4-bit) and efficient inference servers make capable models runnable on a single GPU, and a fine-tuned SLM frequently matches a frontier model on a narrow task at a fraction of the cost — while remaining well behind on open-ended reasoning. The realistic pattern is a portfolio: local models for sensitive or high-volume narrow tasks, frontier APIs for hard ones. The economics and Israeli enterprise context are covered in Why Israeli Enterprises Are Bringing AI On-Prem.

Cost and latency engineering

LLM serving cost is token count times per-token price (or its GPU-time equivalent), which makes cost an engineering variable. The standard toolkit: model routing (a cheap model handles easy requests; hard ones escalate — a cascade); prompt/prefix caching to avoid re-processing repeated context; context discipline (retrieve less but better, summarize history instead of resending it); batching for offline workloads; and streaming plus optimizing time-to-first-token, which dominates perceived latency in interactive use. Latency budgets should be set end-to-end — retrieval, generation, and post-processing — the same way speech systems budget theirs (see Voice AI Grows Up).

Frequently asked questions

When should I fine-tune instead of using RAG?

Fine-tune when the model's form is wrong — style, format, domain phrasing, or reliability on a narrow task — or when you need a smaller model to hit cost/latency targets. Use RAG when the model's knowledge is wrong or stale. Fine-tuning does not reliably add facts, and RAG does not fix formatting or style.

Can hallucinations be eliminated?

No. They can be reduced substantially — grounding with citations, abstention, verifier passes, and constrained tasks — and their rate can be measured and monitored. System design should assume a nonzero rate and route consequential outputs through verification or human review.

Are small language models good enough for production?

Often, for narrow tasks: classification, extraction, summarization, and domain-specific Q&A after fine-tuning. They lag frontier models on open-ended reasoning and long-horizon tasks, which is why many production systems route between a local SLM and a frontier API by request difficulty.

What does an LLM evaluation suite look like in practice?

A version-controlled golden set (typically hundreds of cases) run on every prompt or model change; automatic checks for groundedness and output schema; an LLM judge calibrated against periodic human ratings; and production dashboards for cost, latency, refusal rate, and user feedback.

MLAIA consults on LLMs in production — RAG architectures, evaluation harnesses, and on-prem/SLM deployments. See our LLM & NLP domain expertise or contact us.