Small Models, Specific Jobs: An Enterprise Architecture Beyond Parameter Counts

Retrieval changes the context; fine-tuning changes model behavior

Retrieval-augmented generation searches an approved corpus at request time and places selected passages in the model’s context. It is appropriate when facts change, citations matter or access rights differ by user. A support assistant can retrieve the current return policy rather than relying on a policy memorized during training. The engineering work lies in document parsing, chunking, metadata, ranking, authorization and citation. Retrieval can still fail: the search may miss the right passage, return an obsolete copy or combine documents the user was not permitted to see.

Fine-tuning updates model weights to encourage a behavior, format or domain pattern. It can help a compact model classify internal ticket types or emit a stable JSON schema, but it is a poor way to keep a weekly price list current. Training examples must represent the desired behavior and the edge cases, and they need rights and privacy review. Fine-tuning can also weaken general capabilities or encode sensitive strings. Many enterprise systems use both methods: tune the model for the task and response contract, while retrieving volatile facts from governed sources.

Quantization saves memory by changing numerical precision

Model weights are numbers stored at a chosen precision. Quantization represents them with fewer bits, reducing memory traffic and often enabling a model to fit on smaller hardware. A rough weight-only calculation illustrates the scale: 3.8 billion parameters at four bits require about 1.9 GB before metadata and runtime overhead. Phi-3’s report describes an approximately 1.8 GB four-bit model, consistent with that order of magnitude. Actual deployment memory also includes the runtime, temporary activations and the key-value cache used to hold conversation context.

Lower precision is not free. Quality can decline unevenly across tasks, hardware may accelerate some formats better than others, and long contexts can make cache memory dominate. A procurement team should benchmark the exact quantized artifact on the target device, not infer latency from parameter count. Measure cold start, prompt processing, token generation, peak memory, power and concurrency. A model that fits one phone in a demonstration may not meet an enterprise service-level objective when many users submit long documents at once.

Worked example one: route policy questions instead of choosing one model

Imagine an insurer building an internal policy assistant. Eighty percent of requests ask for a clause, definition or form from a controlled handbook. A small model receives retrieved passages and must answer with citations. Fifteen percent require comparing several policy versions. Five percent involve ambiguous coverage scenarios. The architecture sends the first group to the small model, routes multi-document synthesis to a larger approved model, and sends consequential ambiguity to a licensed specialist. The route is selected from observable features—document count, retrieval score, request type and policy stakes—not from the small model’s unsupported assertion that it is confident.

The small-model path can abstain when no passage clears a validated retrieval threshold, when sources conflict or when the requested output falls outside its schema. Abstention is a product behavior, not a failure to hide. The interface explains that the system could not locate sufficient support and transfers the query with its retrieved evidence. This design values correct coverage over answer volume. It also produces useful operational data: which topics escalate, which documents are missing, how often specialists overturn an answer, and whether the routing rule disadvantages a region or customer group.

Held-out evaluation must resemble the work and protect against leakage

Routing itself needs evaluation because it can create a new failure layer. Build a matrix that crosses request type with destination and expected outcome. Measure false escalations, which waste expensive capacity, and missed escalations, which expose users to an underpowered model. Include queue delay: a cheap first pass that adds two seconds before every large-model call may harm an interactive product. If a policy or specialist route is unavailable, define whether the system waits, falls back or refuses. The routing controller should be versioned and tested separately from the language models it selects.

An evaluation set should be separated from prompts used to tune the system and should contain representative ordinary cases, rare high-cost errors, adversarial wording and requests that deserve refusal. For a contract-clause classifier, accuracy alone can conceal a dangerous result if the model misses most indemnity exceptions. Report precision and recall by class, calibration, citation support, abstention quality and downstream review time. Keep a final holdout that developers do not repeatedly inspect; otherwise the team gradually optimizes to the test and mistakes familiarity for generalization.

Public benchmarks can show broad capability but cannot establish fitness for a proprietary workflow. Phi-3’s authors state that their results use a consistent internal pipeline and note that prompt details can change scores. Enterprise tests need the same candor. Record model and quantization version, prompt, retrieval index, decoding settings and evaluator rubric. Run side-by-side human review without revealing which model produced which output. After launch, monitor drift using sampled production cases while preserving a stable regression suite so apparent improvement is not created by changing the exam.

Local execution narrows some privacy exposure, not all of it

Running a model on a managed device can keep prompts away from an external inference provider and may reduce network latency. It does not make the application private by default. Inputs may still be logged by the app, copied into analytics, backed up to cloud storage or exposed through an insecure retrieval index. Model files and adapters can leak intellectual property. A compromised endpoint can read local memory. Security therefore covers identity, encryption, device management, retention, telemetry, patching and the data path around the model.

Local processing also complicates governance when thousands of devices carry different model versions. Teams need signed artifacts, update policy, revocation and evidence of which version answered a request. Highly sensitive work may be safer in a controlled data center with audited access than on a poorly managed laptop. Conversely, an offline field device may be appropriate where connectivity and data residency dominate. Privacy is an architectural property shaped by flows and controls; it cannot be inferred from the word local.

Worked example two: compare total cost, including escalation

Consider an illustrative service handling 200,000 short requests a month. A hosted large-model path costs $0.018 per request, or $3,600. A small-model service costs $900 in reserved compute, $500 in platform operations and $300 in monitoring. It sends 20 percent of requests to the larger model, adding $720, and human reviewers inspect 2,000 sampled or escalated cases at four minutes each and $48 an hour, adding $6,400. Monthly operating cost is $8,820, higher than inference alone for either option. Review dominates because quality assurance is real labor.

The comparison changes if the large-only system also needs equal review, or if the small model reduces specialist handling. Add one-time costs explicitly: $35,000 for evaluation and integration, $12,000 for fine-tuning data work and $8,000 for security assessment. Amortized over twelve months, that is $4,583 monthly, bringing first-year small-route TCO to $13,403 in this scenario. The architecture may still win for latency, data control or future volume, but the result is conditional. A fair model compares the same quality target, review policy and availability requirement on both sides.

1. Use retrieval for current governed facts and fine-tuning for stable behavior or format.
2. Benchmark the exact quantized artifact, context length, hardware and concurrency planned for production.
3. Reserve held-out cases for final evaluation and report high-cost errors separately from aggregate accuracy.
4. Give the small model an explicit abstain path and route difficult or consequential work upward.
5. Calculate TCO with evaluation, data preparation, review, operations, security and escalation—not inference price alone.

Sources

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• NIST AI Risk Management Framework
• NIST Privacy Framework