The AI that
answers from the
rule.
Not from its
assumptions.

Five numbers that summarise the whole story — the rest of the page explains how they were achieved.

All evaluation ran on local infrastructure — a controlled environment that allows full experimentation without engaging public APIs or exposing data to external services. Production will run on provisioned servers sized from these measurements.

A public e-learning platform needed an AI tutor that answers driving-theory questions from the UK Highway Code. Students ask about roundabouts, overtaking rules, and right-of-way — the system must answer only from the regulatory text, not from the model's assumptions. Public platform = thousands of concurrent users, real revenue, real liability. Three questions: Which model? How reliable is it? What infrastructure does it need?

The grounding problem is brutal. UK drives on left, rest of world on right. LLMs trained on right-hand-traffic data carry conflicting prior knowledge. When student asks about roundabouts, model might answer with right-hand-traffic logic. Model's intelligence is a liability. System must answer only from regulatory text.

Infrastructure problem: public platform = real servers, RAM, token throughput, cost model. 0.5B model needs fraction of hardware of 7B model, but can it be reliable? Answer determines server spec, hosting budget, unit economics.

This was not a model experiment. It was the design phase of a production AI system — the same methodology I'd apply to any regulated, document-grounded domain.

The knowledge base is the UK Highway Code — a single, well-defined regulatory document. I built a domain benchmark from the Code itself: over 800 questions were generated by the candidate models, validated by two independent LLM judges and a human audit, and reduced to 137 clean benchmark questions. Six models were screened. Three survived on measured grounds. The surviving three were tested across four adaptation strategies: ungrounded, grounded, fine-tuned, and fine-tuned + grounded. All evaluation ran on local consumer hardware — a controlled, reproducible test environment.

AI System Design · Model Evaluation · LLM Strategy · Infrastructure Planning

• Defined the evaluation framework: 6 models × 4 strategies × 137 questions
• Screened 6 open-weight models: eliminated 3 on measured grounds
• Built the domain benchmark: from the UK Highway Code with dual-LLM + human validation
• Designed the strict-prompt grounding protocol: drives hallucination below 5%
• Measured RAM and throughput: produces the CPU infrastructure spec
• Validated LLM judges: 30-question human audit (87.9% agreement)
• Delivered the three-model architecture: answering, question-generation, and quiz-scoring roles — all CPU
• Maintained full data confidentiality: via local evaluation throughout

• The model's intelligence is not the asset. The document is. For document-grounded domains, the model's general knowledge is a liability, not an advantage. The system must answer from the text, not from the weights.
• Hallucination is a protocol problem, not a model problem. The same model hallucinates 77% without a strict protocol and 5% with one. The prompt is the safety mechanism. Design the protocol before you pick the model.
• The minimum viable model is a cost decision, not a technical preference. Two models cleared the reliability bar — the choice between them is a business decision, derived from data and driven by hardware budget and the tighter constraint (accuracy vs safety).
• Always validate your judges. A parser bug initially showed the smallest model (0.5B) as the best performer — 92.5% accuracy. After fixing the bug, the ranking inverted: 36.1%. The smallest model was not the best. A human audit caught it. In any LLM system, the evaluation pipeline is as important as the model itself.

Based on the findings, here are the specific recommendations for the production platform:

1. Deploy a three-model architecture, not a single model. gemma-2-2b answers student questions via strict RAG — it's both the most accurate and the most grounded model in the study, and it runs comfortably on CPU. Phi-3.5-mini runs offline to generate and expand the question bank — it's the strongest CPU-capable generator, marginally ahead of gemma (which is capable of the same job as a fallback), but not the model students interact with directly. A fine-tuned Qwen2-0.5B handles quiz scoring, where it reaches 82.5% accuracy at minimal cost. No GPU hosting tier is required for this configuration. Choose gemma as the answerer because its safety is free — RAG alone gets it to the 5.1% hallucination rate, whereas phi only matches that level by adding fine-tuning that sacrifices accuracy (74.5% → 54.0%).
2. Implement the strict-prompt protocol as a non-negotiable safety mechanism. "Answer ONLY from the provided text. If the answer is not in the text, say I DON'T KNOW." This is what drives hallucination below 5%. Without it, even the grounded models drift.
3. Build a question-generation pipeline using a larger model (3.8B+). The smallest model (0.5B) cannot reliably generate multiple-choice questions — it produces ambiguous or overlapping options. A 3.8B model generates valid MCQs (8/10 pass rate). Use it for offline question bank building.
4. Consider fine-tuning for the constrained quiz task. Fine-tuning on 194 open-form QA pairs did not improve accuracy. But fine-tuning on 40 multiple-choice pairs lifted the smallest model from 60% to 82.5% — a statistically significant improvement. If the platform uses MCQs, fine-tuning the smallest model for that specific task is cost-effective.
5. Plan the infrastructure from the measured numbers. gemma-2-2b needs ~3 GB RAM and runs on CPU at ~17 tok/s. Phi-3.5-mini is also CPU-viable (~13 tok/s). The fine-tuned Qwen2-0.5B is the cheapest to run (~27 tok/s). All three map to a single CPU server spec — not a guess, and no GPU tier is required. GPU hosting would further speed up gemma if available, but it is not needed for the recommended configuration.
6. Validate the evaluation pipeline before trusting the results. A parser bug initially inverted the entire model ranking. Always validate LLM judges against a human audit. In this project, the 30-question human audit (87.9% agreement) confirmed the ranking was stable.

How I would approach designing an LLM system for any document-grounded, regulated domain:

1. Audit the sourceIdentify the authoritative text, its structure, and where the model's general knowledge conflicts with it.
2. Build the benchmarkCreate domain questions from the source. Validate with dual judges + human audit.
3. Screen the modelsTest candidate models on a pilot set. Eliminate on measured grounds: accuracy, hallucination, throughput.
4. Design the protocolStrict grounding, abstention rules, safety filters. The prompt is the product.
5. Measure the infrastructureRAM, tok/s, latency. The model choice is a cost decision.
6. Deploy & validateProduction on provisioned servers. Continuous monitoring. Periodic human audit.

And this is where I work best — at the intersection of business needs and data-driven evidence.

A generic AI that hallucinates 75% of the time, designed into a grounded, three-model system that hallucinates 5% — with a CPU-only production architecture, specific server specs, and a specific cost model for each role. The research is done. The platform is ready to build.

Have a process, a problem or an idea worth analysing? Write to me and we will turn it into a concrete next step.

Client confidentiality: client identity and commercially sensitive information have been anonymised. Model names are shown as generic references (Model A/B/C). All evaluation performed on local infrastructure for controlled, reproducible testing.