LLM evaluation is the practice of measuring how well a large language model, or an application built on one, performs against clear quality standards before and after release. It combines a fixed test set, scoring methods, and pass thresholds you agree up front. It is the first layer of solid MLOps and model monitoring.
Key Takeaways: LLM Evaluation
- Evaluate the application, not just the model. Public benchmarks help you shortlist models. Your own test set decides if a system ships.
- Build the test set from real cases. Fifty to two hundred examples from your own workflow beat thousands of generic ones.
- Use three scoring methods together: code-based checks, LLM-as-a-judge, and human review.
- Set release gates before you build. A number agreed in advance stops “it looks fine” launches.
- Keep evaluating after launch. Prompts, models and data drift, so every change needs a regression run.

Why LLM Evaluation Is Different From Normal Software Testing
Traditional software gives the same output for the same input. A unit test either passes or fails. LLMs do not work that way. The same question can produce different wording each time, and there may be many acceptable answers.
That makes LLM testing a measurement problem, not a pass or fail check. You are asking how often the system is right, how badly it fails when it is wrong, and whether a change made things better or worse.
Risk frameworks now expect this work. The NIST Generative AI Profile, published in July 2024, sets out 12 risks that generative AI creates or makes worse, including confabulation, data privacy and information security. Its suggested actions include checking sources and citations in outputs before launch and in operation.
Model Evaluation vs Application Evaluation
Teams often mix up two kinds of LLM evaluation. They answer different questions.
| Type | Question it answers | Typical method | When to use |
|---|---|---|---|
| Model evaluation | Which base model is strongest at general skills? | Public benchmarks and leaderboards | Shortlisting models |
| Application evaluation | Does our system do our task well enough to ship? | Custom test set, task metrics, human review | Every build and every change |
| Online evaluation | Is it still working with real users? | Sampled production traffic, feedback, alerts | After launch, continuously |
Public suites are useful for the first row. The Stanford HELM benchmark project compares models across many scenarios and metrics, not only accuracy. But no public benchmark knows your contracts, your support tickets or your tone of voice. Only your own test set can tell you that.
The LLM Evaluation Metrics That Matter
Pick metrics that match the job. A summarisation tool and a customer-facing agent need different scores.
| Metric | What it measures | How to score it |
|---|---|---|
| Task accuracy | Is the answer correct? | Match against a reference answer or rubric |
| Faithfulness | Is every claim supported by the source text? | LLM judge plus human spot checks |
| Citation accuracy | Do citations point to the right passage? | Code check against retrieved IDs |
| Format compliance | Is the output valid JSON, the right length, the right fields? | Code-based checks |
| Safety and policy | Does it refuse what it should and avoid leaks? | Attack prompts and policy rubrics |
| Latency | How long does each answer take? | System logs |
| Cost per task | What does each answer cost to produce? | Token and tool usage logs |
For retrieval-based systems, score retrieval on its own too. If the right passage never reaches the model, no prompt can save the answer. Our post on measured retrieval for RAG explains why.
Three Ways to Score LLM Outputs
Code-based checks
Use plain code wherever the answer can be checked by rules. Is the JSON valid? Does the reply include a required disclaimer? Is the extracted invoice total equal to the real total? These checks are fast, cheap and never change their mind.
LLM-as-a-judge
For open-ended answers, a strong model can grade another model’s output against a written rubric. Research supports this when done carefully. The LLM judge study from LMSYS researchers found that GPT-4 as a judge reached over 80% agreement with human preferences, about the same level humans reach with each other. The same study also flagged biases, such as favouring longer answers and answers in the first position.
To use judges well:
- Write a specific rubric with examples of a 1, a 3 and a 5.
- Ask for a reason before the score, so you can audit its thinking.
- Swap answer order when comparing two outputs, to cancel position bias.
- Check the judge against humans on a sample before trusting it at scale.
Human review
People catch subtle errors, tone problems and domain mistakes that automated scores miss. Use subject experts for a sample of every test run, and for every disagreement between the judge and the code checks.

How to Build an LLM Evaluation Plan in Seven Steps
- Define the task and the user. Write one sentence on what a good answer does for that person.
- Collect real examples. Pull 50 to 200 cases from tickets, emails or documents your team already handles.
- Write reference answers or rubrics with a subject expert, not just an engineer.
- Add hard cases on purpose: ambiguous questions, missing data, out-of-scope requests and attack prompts.
- Choose metrics and thresholds. For example, 95% format compliance, 90% faithfulness, zero data leaks.
- Automate the run so every prompt, model or retrieval change triggers the full test set.
- Log results over time so you can see if each change helped or hurt.
Most teams skip steps 3 and 5. That is why so many pilots never get a clear yes or no. Our guide on why AI pilots stall covers the wider pattern.
Warning Signs Your LLM Testing Is Too Weak
- The test set was written by the people who built the prompt. It will mirror their blind spots.
- Scores only go up. If the test set never catches a regression, it is too easy.
- There is no safety section. No prompt injection, no data leak tests, no refusal checks.
- Results live in a spreadsheet nobody reruns. Evaluation has to be repeatable.
- Launch was decided by a demo. A good demo shows the best case, not the typical one.
LLM Evaluation After Launch
Shipping is not the end of LLM evaluation. Model providers update models, users ask new kinds of questions, and source documents change.
Keep three habits in place:
- Sample live traffic each week and score it with the same rubric.
- Collect user feedback with a simple thumbs up or down plus an optional reason.
- Rerun the full test set before any prompt, model or retrieval change goes live.
Watch for silent drift. A provider can update a hosted model without changing its name, and answers can shift in tone or accuracy overnight. Pin model versions where you can, and treat any version change like a code release: run the test set, compare scores against the last baseline, and only switch when the results hold.
Teams in regulated sectors also need records. Evaluation logs, rubrics and results become evidence for audits. Our AI governance consulting team helps map that evidence to frameworks your auditors already know.
LLM Evaluation FAQs
What are LLM evaluations?
LLM evaluations, often called evals, are structured tests that score what a model or AI application produces against a defined standard. That standard can be a reference answer, a rubric, a format rule or a safety policy. Evals run before launch to decide if a system is ready and after launch to catch regressions.
What is LLM-as-a-judge evaluation?
LLM-as-a-judge means using a strong language model to grade another model’s outputs against a written rubric. It scales human-style judgement to thousands of examples. It works best with clear rubrics, a reason before each score, swapped answer order, and regular checks against human reviewers to catch bias.
What is LLM testing?
LLM testing is the wider practice of checking a language model system for correctness, safety, format, speed and cost. It includes unit-style code checks, scored evaluation sets, attack testing such as prompt injection, and monitoring of live traffic after release.
What are the top LLM evaluation tools?
Popular options include open-source frameworks for tracing and scoring, cloud provider evaluation features, and dedicated eval platforms. The tool matters less than the test set and rubric. Choose one that runs your own examples automatically, supports LLM judges and code checks, and keeps a history of results.
How many test cases do we need to start?
Fifty to two hundred real, well-labelled cases is enough to make a first release decision for one workflow. Add new cases every time you find a failure in production, so the set grows to reflect what your users actually ask.
Turning LLM Evaluation Into a Release Gate
A model is ready for production when it passes a test set you trust, at thresholds you agreed before building. Anything less is a guess.
Miniml is a senior team of engineers and researchers with backgrounds at GE, Amazon, Stanford, the University of Edinburgh, and the Alan Turing Institute. We build evaluation sets, judges and release gates your team keeps after handover, often starting with a two-week Production-Readiness Review. Call +1 415-941-8345 or book a Miniml consultation to review your LLM evaluation plan. Office hours are Monday to Friday, 8am to 5pm.
San Francisco: 50 California St #1500, San Francisco, CA 94111. Phone: +1 415-941-8345. Edinburgh: 93 George St, Edinburgh EH2 3ES. Find us on our Edinburgh Google Business Profile.
About Miniml: Miniml is an AI consultancy with offices in San Francisco and Edinburgh that helps enterprise teams take AI systems past the pilot and into production.