Testing and Evaluating LLM-Based Code
Code that calls an LLM needs two kinds of tests.
The first kind is the usual unit test. Most of an LLM application is plain code: text chunking, prompt templates, reply parsing, retrieval math, and storage. This code is deterministic, so a unit test can check that a given input gives an exact output. Unit tests should run in seconds with no network, so we replace each model call with a fake.
The second kind is the evaluation, or eval. Here we run the real system with real models over a fixed set of questions and score the answers. The output changes from run to run, a correct answer can be worded a hundred ways, and no single score tells the whole story. An eval answers a question that unit tests can’t: did my last change make the answers better or worse?
In this chapter we build llm-eval, a small library for both kinds of tests, and use it on the agentic RAG system from the earlier RAG chapter. We cover:
- Unit tests with the FiveAM framework, including a test that runs the whole RAG pipeline with fake models.
- A golden dataset of questions and reference answers, stored as Lisp data.
- Three scores that need no model: exact match, token F1, and keyword recall.
- LLM-as-judge, where a second model grades each answer.
- A run history in SQLite, so we can find regressions between runs.
Everything runs on local models through Ollama. You need three models:
1 ollama pull nomic-embed-text # embeddings for RAG retrieval
2 ollama pull llama3.2:3b # the RAG system's chat model
3 ollama pull qwen3.5:4b # the judge
Note: The source code is in the directory src/llm-eval.
Project Structure
| File | Purpose |
|---|---|
package.lisp |
Package definition and exports |
dataset.lisp |
Loading golden datasets |
scoring.lisp |
Exact match, token F1, and keyword recall |
judge.lisp |
LLM-as-judge |
harness.lisp |
Runs a system over a dataset and scores it |
store.lisp |
Run history in SQLite |
rag-eval.lisp |
Evaluates the RAG example |
tests.lisp |
FiveAM unit tests |
examples.lisp |
Runs three evals and compares them |
data/rag-golden.lisp |
Golden dataset for the RAG example |
llm-eval.asd |
ASDF system definitions |
1 ;;;; llm-eval.asd -- testing and evaluating LLM-based code.
2
3 (asdf:defsystem #:llm-eval
4 :description "Evaluation harness for LLM applications: golden datasets, scoring, LLM-as-judge, and run history in SQLite."
5 :author "Mark Watson"
6 :license "Apache 2"
7 :depends-on (#:litelm #:sqlite)
8 :serial t
9 :components ((:file "package")
10 (:file "dataset")
11 (:file "scoring")
12 (:file "judge")
13 (:file "harness")
14 (:file "store"))
15 :in-order-to ((asdf:test-op (asdf:test-op #:llm-eval/tests))))
16
17 (asdf:defsystem #:llm-eval/rag
18 :description "Evaluate the agentic RAG example from the RAG chapter."
19 :depends-on (#:llm-eval #:rag)
20 :components ((:file "rag-eval")))
21
22 (asdf:defsystem #:llm-eval/tests
23 :description "FiveAM unit tests. They run offline: every model call is faked."
24 :depends-on (#:llm-eval #:rag #:fiveam)
25 :components ((:file "tests"))
26 :perform (asdf:test-op (op c)
27 (declare (ignore op c))
28 (uiop:symbol-call :fiveam :run! (uiop:find-symbol* :all-tests :llm-eval-tests))))
We split the code into three systems. The core system llm-eval knows nothing about RAG, so you can use it to evaluate any LLM application. The :in-order-to and :perform options connect the tests to ASDF, so (asdf:test-system :llm-eval) runs them. The :perform method uses uiop:symbol-call and uiop:find-symbol* because the test package does not exist when ASDF reads the .asd file.
The Golden Dataset
A golden dataset is a list of questions, each with a correct answer written by a person. I wrote twelve cases from the three text files in src/RAG/data. Each case is a plist in a plain Lisp file:
1 (:id "co2-preindustrial"
2 :question "What was the pre-industrial concentration of atmospheric CO2?"
3 :reference "About 280 parts per million (ppm)."
4 :keywords ("280"))
5
6 (:id "solid-state-companies"
7 :question "Which companies are leading the development of solid-state batteries?"
8 :reference "Toyota, Samsung SDI, and QuantumScape."
9 :keywords ("toyota" "samsung" "quantumscape"))
10
11 (:id "storage-roles"
12 :question "What is the most common form of grid-scale energy storage, and what role does it play next to batteries in a hybrid storage system?"
13 :reference "Pumped-storage hydroelectricity. It provides bulk, long-duration storage over hours or days, while batteries handle short, fast, frequent charge cycles."
14 :keywords ("pumped"))
The :id stays the same across runs, so we can match a case in one run with the same case in another. :keywords lists the facts a correct answer must contain.
Lisp data files are easy to write and need no parser, but the Lisp reader can run code: #.(form) evaluates form at read time. load-cases turns that off:
1 (defstruct eval-case
2 id ; short stable name, used to match results across runs
3 question
4 reference ; a correct answer, written by a person
5 keywords) ; facts a correct answer must mention
6
7 (defun load-cases (pathname)
8 "Read eval cases from PATHNAME, a file of plists. *READ-EVAL* is NIL so
9 a #. form in a data file can't run code."
10 (with-open-file (in pathname)
11 (with-standard-io-syntax
12 (let ((*read-eval* nil))
13 (loop for form = (read in nil in)
14 until (eq form in)
15 collect (destructuring-bind (&key id question reference keywords)
16 form
17 (unless (and (stringp id) (stringp question) (stringp reference))
18 (error "Bad eval case, needs string :id, :question, and :reference: ~S"
19 form))
20 (make-eval-case :id id :question question
21 :reference reference :keywords keywords)))))))
with-standard-io-syntax resets the reader to its standard settings, so a change to *read-base* or *package* elsewhere in your image can’t change how the file reads. Passing the stream itself as the end-of-file value is a common idiom: no form in the file can be eq to the stream, so until (eq form in) can’t stop early.
Twelve cases is enough to show the method. For a real application, aim for at least fifty, and add a case each time a user reports a bad answer.
Scores That Need No Model
scoring.lisp has three scores. Each returns a number from 0 to 1. All three start from the same tokenizer:
1 (defun tokens (text)
2 "Lowercase word and number tokens from TEXT. Commas inside numbers are
3 dropped and periods kept, so \"$1,100\" gives \"1100\" and \"1.1°C\" gives
4 \"1.1\" and \"c\"."
5 (let ((result '())
6 (current (make-string-output-stream)))
7 (flet ((finish-token ()
8 (let ((token (string-trim "." (get-output-stream-string current))))
9 (when (plusp (length token))
10 (push token result)))))
11 (loop for previous = nil then char
12 for char across (string-downcase text)
13 for index from 0
14 do (cond ((alphanumericp char) (write-char char current))
15 ((char= char #\.) (write-char char current))
16 ((and (char= char #\,)
17 previous (digit-char-p previous)
18 (< (1+ index) (length text))
19 (digit-char-p (char text (1+ index))))
20 nil)
21 (t (finish-token))))
22 (finish-token))
23 (nreverse result)))
Numbers matter in this dataset, so the tokenizer keeps decimal points and drops thousands separators. string-trim removes the period at the end of a sentence, so "1.1." becomes "1.1". In the loop, previous holds the character before char: the clauses step in order, so previous takes the old value of char before char moves on.
Exact Match and Token F1
1 (defun exact-match (answer reference)
2 "1 if ANSWER and REFERENCE have the same tokens, else 0."
3 (if (equal (tokens answer) (tokens reference)) 1 0))
4
5 (defun token-f1 (answer reference)
6 "F1 score of the token overlap between ANSWER and REFERENCE, the measure
7 used by the SQuAD question answering benchmark. Each token counts as many
8 times as it appears in both texts."
9 (let ((answer-tokens (tokens answer))
10 (reference-tokens (tokens reference)))
11 (if (or (null answer-tokens) (null reference-tokens))
12 0
13 (let ((counts (make-hash-table :test #'equal))
14 (common 0))
15 (dolist (token reference-tokens)
16 (incf (gethash token counts 0)))
17 (dolist (token answer-tokens)
18 (when (plusp (gethash token counts 0))
19 (decf (gethash token counts))
20 (incf common)))
21 (if (zerop common)
22 0
23 (let ((precision (/ common (length answer-tokens)))
24 (recall (/ common (length reference-tokens))))
25 (/ (* 2 precision recall) (+ precision recall))))))))
Precision is the share of answer tokens found in the reference. Recall is the share of reference tokens found in the answer. F1 is their harmonic mean. Common Lisp’s rational numbers keep the result exact, so (token-f1 "about 280 ppm" "About 280 parts per million (ppm).") returns 2/3, not 0.6666667, which makes unit tests simple.
Keyword Recall
1 (defun keyword-recall (answer keywords)
2 "Fraction of KEYWORDS found in ANSWER as whole tokens. With no keywords
3 the score is 1."
4 (if (null keywords)
5 1
6 (let ((text (format nil " ~{~A~^ ~} " (tokens answer))))
7 (/ (count-if (lambda (keyword)
8 (search (format nil " ~{~A~^ ~} " (tokens keyword)) text))
9 keywords)
10 (length keywords)))))
Both strings are joined with spaces and padded with a space at each end, so "280" matches the token 280 but not 1280.
LLM-as-Judge
A judge is a second model that reads the question, the reference answer, and the system’s answer, and decides whether the answer is correct. It handles what the other scores can’t: different wording, extra detail, and answers that contain the right number but say something false about it.
1 (defvar *judge-model* "ollama/qwen3.5:4b")
2
3 (defvar *judge-extra-body* '(("reasoning_effort" . "none"))
4 "Extra request fields for the judge. qwen3.5 thinks before it answers,
5 which takes about 50 seconds per call on my Mac; turning thinking off
6 brings that under one second.")
7
8 (defun judge-prompt (question reference answer)
9 (format nil "You are grading an answer from a question answering system.
10
11 Question: ~A
12
13 Reference answer: ~A
14
15 Candidate answer: ~A
16
17 Does the candidate answer agree with the reference answer on the facts the
18 question asks for? Extra correct detail is fine. Wrong numbers, wrong names,
19 or a missing answer are not.
20
21 Reply with exactly two lines:
22 SCORE: 1 if the candidate agrees with the reference, or SCORE: 0 if not
23 REASON: one sentence"
24 question reference answer))
A pass/fail score works better than a 1 to 10 scale with small judge models. A small model has no steady sense of the gap between a 6 and a 7, but it can tell whether two numbers match. The REASON line costs little and helps a great deal when you read the failures.
Choosing the Judge Model
I tried three local models on one test: the reference says 280 ppm and the candidate says “roughly 300 parts per million”. A good judge must fail this answer.
| Model | Verdict | Time |
|---|---|---|
llama3.2:3b |
SCORE: 1, “300 ppm is approximately equal to 280 ppm” | 0.5 s |
qwen3.5:4b |
SCORE: 0, “a significant numerical discrepancy” | 52 s |
qwen3.5:4b, thinking off |
SCORE: 0, “a factual disagreement” | 0.8 s |
llama3.2:3b is too lenient to judge. That is also the model our RAG system uses, and a model grading its own output is a poor check in any case. qwen3.5:4b gave the right verdict, but it is a reasoning model that writes a long hidden chain of thought before it answers. Ollama’s OpenAI-style endpoint accepts "reasoning_effort": "none" to turn that off, and the verdict stayed correct.
The litelm library had no way to send a field like that, so I added an :extra-body keyword to litelm:completion: an alist that completion appends to the JSON request. The litelm chapter describes it.
Test your judge before you trust it. A few known-bad answers like the one above make a good unit test for a judge, run by hand whenever you change the judge model or prompt.
Parsing the Verdict
1 (defun parse-judge-reply (reply)
2 "Return the score (0 or 1) and the reason from a judge REPLY. The score
3 is NIL when the reply has no SCORE line with a 0 or 1."
4 (let* ((score-start (search "SCORE:" reply :test #'char-equal))
5 (digit (and score-start
6 (find-if-not (lambda (char) (member char '(#\Space #\Tab)))
7 reply :start (+ score-start 6))))
8 (reason-start (search "REASON:" reply :test #'char-equal))
9 (reason (if reason-start
10 (string-trim '(#\Space #\Tab #\Newline #\Return)
11 (subseq reply (+ reason-start 7)))
12 "")))
13 (values (case digit (#\1 1) (#\0 0) (t nil))
14 reason)))
15
16 (defun litelm-judge (prompt)
17 (litelm:response-content
18 (litelm:completion *judge-model* :messages prompt
19 :temperature 0
20 :extra-body *judge-extra-body*)))
21
22 (defvar *judge-function* 'litelm-judge
23 "Function of one argument, the prompt, that returns the judge's reply.
24 Tests bind this to a fake judge.")
25
26 (defun judge (question reference answer)
27 "Ask the judge model to grade ANSWER. Returns the score (0, 1, or NIL
28 if the reply could not be parsed) and the judge's reason."
29 (let ((reply (funcall *judge-function* (judge-prompt question reference answer))))
30 (multiple-value-bind (score reason) (parse-judge-reply (or reply ""))
31 (values score (if score reason (format nil "Unparsable judge reply: ~S" reply))))))
An unparsable reply gets the score nil, not 0. Counting a judge failure as a wrong answer would mix two separate problems. summarize skips nil scores when it computes the mean, and the reason column shows the raw reply so you can fix the prompt.
The judge runs at temperature 0 so that it gives the same verdict for the same input as often as the model allows.
The Harness
run-eval takes the system under test as a function of one question string that returns an answer string. This keeps the harness separate from any one application:
1 (defstruct eval-result
2 case-id
3 answer
4 exact
5 f1
6 keywords
7 judge ; 0, 1, or NIL when not judged or the judge failed
8 judge-reason
9 seconds
10 error) ; error message string when the system failed
11
12 (defun elapsed-seconds (start)
13 (float (/ (- (get-internal-real-time) start) internal-time-units-per-second)))
14
15 (defun run-eval (system cases &key (use-judge t) (verbose t))
16 "Call SYSTEM, a function of one question string that returns an answer
17 string, on each of CASES and score each answer. An error in one case is
18 recorded in its result, and the run goes on to the next case."
19 (loop for eval-case in cases
20 for index from 1
21 collect
22 (let* ((start (get-internal-real-time))
23 (error-message nil)
24 (answer (handler-case (funcall system (eval-case-question eval-case))
25 (error (condition)
26 (setf error-message (princ-to-string condition))
27 "")))
28 (seconds (elapsed-seconds start))
29 (reference (eval-case-reference eval-case)))
30 (multiple-value-bind (judge-score judge-reason)
31 (cond (error-message (values 0 "System error"))
32 (use-judge (judge (eval-case-question eval-case) reference answer))
33 (t (values nil "")))
34 (when verbose
35 (format t "~&[~D/~D] ~A: judge ~A, ~,1F s~%"
36 index (length cases) (eval-case-id eval-case)
37 (or judge-score "-") seconds))
38 (make-eval-result
39 :case-id (eval-case-id eval-case)
40 :answer answer
41 :exact (exact-match answer reference)
42 :f1 (float (token-f1 answer reference))
43 :keywords (float (keyword-recall answer (eval-case-keywords eval-case)))
44 :judge judge-score
45 :judge-reason judge-reason
46 :seconds seconds
47 :error error-message)))))
The handler-case around the system call matters. An eval runs many slow model calls, and one crash on case 40 should not throw away 39 finished results. A system error scores 0, since from the user’s view a crash is a wrong answer.
summarize and print-report compute and print the means:
1 (defun mean (numbers)
2 (let ((numbers (remove nil numbers)))
3 (if numbers
4 (float (/ (reduce #'+ numbers) (length numbers)))
5 nil)))
6
7 (defun summarize (results)
8 "Return a plist of mean scores over RESULTS."
9 (list :cases (length results)
10 :errors (count-if #'eval-result-error results)
11 :exact (mean (mapcar #'eval-result-exact results))
12 :f1 (mean (mapcar #'eval-result-f1 results))
13 :keywords (mean (mapcar #'eval-result-keywords results))
14 :judge (mean (mapcar #'eval-result-judge results))
15 :seconds (mean (mapcar #'eval-result-seconds results))))
Storing Runs in SQLite
An eval score means little alone. It means a lot next to the score from yesterday’s run. store.lisp saves every run in SQLite with the sqlite library, the same one the Knowledge Graph Navigator chapter uses:
1 (defun open-store (path)
2 "Open (and create if needed) the SQLite database at PATH. Use
3 \":memory:\" for a throwaway database."
4 (let ((db (sqlite:connect path)))
5 (sqlite:execute-non-query db
6 "CREATE TABLE IF NOT EXISTS runs (
7 id INTEGER PRIMARY KEY,
8 created TEXT NOT NULL,
9 label TEXT NOT NULL,
10 config TEXT)")
11 (sqlite:execute-non-query db
12 "CREATE TABLE IF NOT EXISTS results (
13 run_id INTEGER NOT NULL REFERENCES runs(id),
14 case_id TEXT NOT NULL,
15 answer TEXT,
16 exact REAL, f1 REAL, keywords REAL, judge REAL,
17 judge_reason TEXT,
18 seconds REAL,
19 error TEXT)")
20 db))
21
22 (defun timestamp ()
23 (multiple-value-bind (second minute hour day month year) (get-decoded-time)
24 (format nil "~4,'0D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D"
25 year month day hour minute second)))
26
27 (defun save-run (db label config results)
28 "Store RESULTS under a new run and return the run id. LABEL names the run
29 and CONFIG describes the settings, for example the model name."
30 (sqlite:with-transaction db
31 (sqlite:execute-non-query db
32 "INSERT INTO runs (created, label, config) VALUES (?, ?, ?)"
33 (timestamp) label (prin1-to-string config))
34 (let ((run-id (sqlite:last-insert-rowid db)))
35 (dolist (result results)
36 (sqlite:execute-non-query db
37 "INSERT INTO results (run_id, case_id, answer, exact, f1, keywords,
38 judge, judge_reason, seconds, error)
39 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
40 run-id
41 (eval-result-case-id result)
42 (eval-result-answer result)
43 (eval-result-exact result)
44 (eval-result-f1 result)
45 (eval-result-keywords result)
46 (eval-result-judge result)
47 (eval-result-judge-reason result)
48 (eval-result-seconds result)
49 (eval-result-error result)))
50 run-id)))
with-transaction writes the run and its results as one unit: either all rows land or none do. It also makes the inserts much faster, since SQLite then syncs to disk once instead of once per row. A nil value binds as SQL NULL, which is what we want for a missing judge score.
The run’s config column holds the settings as a printed plist. Always record what you changed between runs; a score history without the settings is hard to use a month later.
Comparing Runs
The mean hides the details. Two runs can have the same mean judge score while one case got fixed and another broke. compare-runs lines up the cases of two runs by id:
1 (defun compare-runs (db old-run new-run &key (metric "judge") (threshold 0))
2 "Compare METRIC, a column name of the results table, between two runs.
3 Returns two lists of (case-id old-score new-score): regressions, where the
4 score dropped by more than THRESHOLD, and improvements, where it rose by
5 more than THRESHOLD."
6 (unless (member metric '("exact" "f1" "keywords" "judge") :test #'string=)
7 ;; The column name goes into the SQL text, so only allow known names.
8 (error "Unknown metric ~S" metric))
9 (let ((rows (sqlite:execute-to-list db
10 (format nil "SELECT a.case_id, a.~A, b.~A
11 FROM results a JOIN results b ON a.case_id = b.case_id
12 WHERE a.run_id = ? AND b.run_id = ?
13 ORDER BY a.case_id" metric metric)
14 old-run new-run)))
15 (flet ((change (row) (- (or (third row) 0) (or (second row) 0))))
16 (values (remove-if-not (lambda (row) (< (change row) (- threshold))) rows)
17 (remove-if-not (lambda (row) (> (change row) threshold)) rows)))))
SQL placeholders (?) work for values but not for column names, so the metric name goes into the SQL text with format. The check against a fixed list of names keeps that safe. The self-join on case_id pairs each case in the old run with the same case in the new run.
The threshold exists because of what I saw in the real runs below: F1 moves a little on every run, and a list of tiny changes hides the real ones.
Unit Tests with FiveAM
FiveAM is a widely used Common Lisp test framework. You group tests into suites with def-suite, write each test with test, and check values with is. When an is check fails, FiveAM reports the form and the values it saw. Here are the scoring tests:
1 (defpackage #:llm-eval-tests
2 (:use #:cl #:llm-eval #:fiveam)
3 (:export #:all-tests))
4
5 (in-package #:llm-eval-tests)
6
7 (def-suite all-tests :description "All llm-eval tests.")
8 (def-suite scoring :in all-tests)
9 (def-suite judging :in all-tests)
10 (def-suite datasets :in all-tests)
11 (def-suite harness :in all-tests)
12
13 ;;; Scoring
14
15 (in-suite scoring)
16
17 (test tokens
18 (is (equal '("hello" "world") (tokens "Hello, world!")))
19 (is (equal '("1100" "per" "kwh") (tokens "$1,100 per kWh")))
20 (is (equal '("1.1" "c") (tokens "1.1°C.")))
21 (is (null (tokens " ... "))))
22
23 (test exact-match
24 (is (= 1 (exact-match "About 73 miles." "about 73 miles")))
25 (is (= 0 (exact-match "73 miles" "About 73 miles."))))
26
27 (test token-f1
28 ;; 3 answer tokens, 6 reference tokens, 3 in common:
29 ;; precision 1, recall 1/2, F1 2/3.
30 (is (= 2/3 (token-f1 "about 280 ppm" "About 280 parts per million (ppm).")))
31 (is (= 1 (token-f1 "Over 50%." "over 50")))
32 (is (= 0 (token-f1 "no idea" "About 16%.")))
33 (is (= 0 (token-f1 "" "About 16%."))))
34
35 (test keyword-recall
36 (is (= 2/3 (keyword-recall "Toyota and QuantumScape lead." '("toyota" "samsung" "quantumscape"))))
37 (is (= 1 (keyword-recall "It costs $250 to $600." '("250" "600"))))
38 (is (= 0 (keyword-recall "1280 ppm" '("280"))) "Keywords match whole tokens only")
39 (is (= 1 (keyword-recall "anything" '()))))
signals checks that a form signals a condition. This test writes a data file with a #. form and checks that loading it fails in the reader instead of running the code:
1 (test read-eval-is-off
2 (uiop:with-temporary-file (:stream out :pathname path :direction :output)
3 (write-string "(:id #.(error \"code ran\") :question \"q\" :reference \"r\")" out)
4 :close-stream
5 (signals reader-error (load-cases path))))
If *read-eval* were true, the reader would run (error "code ran"), which signals a simple-error, not a reader-error, and the test would fail. In uiop:with-temporary-file, the :close-stream marker separates the forms that write the file from the forms that use it.
Faking Model Calls
Every model call in llm-eval goes through a special variable: *judge-function* for the judge. The RAG code from the RAG chapter does the same with rag::*embedding-fn* for embeddings and rag::*generate-fn* for chat. A test binds these variables to fakes with let, and the binding ends when the let ends, even on error.
This style of fake is simple in Lisp. There is no mocking library and no HTTP server to start; the test replaces the function that would make the HTTP call. The cost is a small design rule for your own code: route each model call through a variable or a parameter.
The most useful fake-model test runs the whole RAG pipeline: chunking, embedding, vector search, the query rewriter, the sufficiency check, and the synthesis agent. Only the two model calls are fake:
1 (defun fake-embedding (text)
2 "A tiny bag-of-words embedding over a fixed vocabulary."
3 (let ((words (tokens text)))
4 (map 'simple-vector
5 (lambda (word) (if (member word words :test #'string=) 1.0 0.0))
6 '("co2" "ppm" "wind" "denmark" "battery"))))
7
8 (defun fake-chat (prompt &key model)
9 (declare (ignore model))
10 (cond ((search "Sufficient Context Agent" prompt)
11 (format nil "VERDICT: SUFFICIENT~%REASON: ok~%MISSING: NONE"))
12 ((search "Synthesis Agent" prompt)
13 (if (search "280 ppm" prompt)
14 "About 280 ppm (facts.txt)."
15 "I don't know."))
16 (t "pre-industrial co2 ppm")))
17
18 (test rag-pipeline-offline
19 (uiop:with-temporary-file (:stream out :pathname path :direction :output :type "txt")
20 (write-string "Pre-industrial CO2 was 280 ppm. Denmark gets half its power from wind." out)
21 :close-stream
22 (let ((rag:*rag-verbose* nil)
23 (rag::*embedding-fn* #'fake-embedding)
24 (rag::*embedding-cache* (make-hash-table :test 'equal))
25 (rag::*generate-fn* #'fake-chat)
26 (*judge-function* #'exact-judge))
27 (let ((corpus (rag:make-corpus :name "test" :description "test")))
28 (rag:add-document corpus path)
29 (let ((results (run-eval (lambda (question) (rag:query corpus question))
30 (list (make-eval-case :id "co2"
31 :question "Pre-industrial CO2?"
32 :reference "About 280 ppm (facts.txt)."
33 :keywords '("280")))
34 :verbose nil)))
35 (is (= 1 (eval-result-keywords (first results))))
36 (is (eql 1 (eval-result-judge (first results)))))))))
The fake chat model answers each RAG agent by looking for words from that agent’s prompt. The synthesis agent’s fake answers correctly only when the retrieved passage that reached its prompt contains “280 ppm”, so the test fails if retrieval breaks. The test also binds a fresh rag::*embedding-cache*; without that, cached embeddings from an earlier run could hide a bug.
exact-judge is a fake judge that passes an answer only if it repeats the reference:
1 (defun exact-judge (prompt)
2 "A fake judge that passes an answer only if it repeats the reference."
3 (let* ((reference (subseq prompt (+ (search "Reference answer: " prompt) 18)))
4 (reference (subseq reference 0 (position #\Newline reference)))
5 (candidate (subseq prompt (+ (search "Candidate answer: " prompt) 18)))
6 (candidate (subseq candidate 0 (position #\Newline candidate))))
7 (format nil "SCORE: ~D~%REASON: fake" (if (string= reference candidate) 1 0))))
The store tests use an in-memory SQLite database, ":memory:", so they leave no files behind:
1 (test store-finds-regressions
2 (let ((*judge-function* #'exact-judge)
3 (db (open-store ":memory:")))
4 (unwind-protect
5 (let* ((old (save-run db "old" '(:version 1)
6 (run-eval (lambda (q) (cond ((string= q "qa") "alpha")
7 ((string= q "qb") "beta")
8 (t "wrong")))
9 (fake-cases) :verbose nil)))
10 (new (save-run db "new" '(:version 2)
11 (run-eval (lambda (q) (cond ((string= q "qa") "alpha")
12 ((string= q "qc") "gamma")
13 (t "wrong")))
14 (fake-cases) :verbose nil))))
15 (multiple-value-bind (regressions improvements) (compare-runs db old new)
16 (is (equal '(("b" 1.0d0 0.0d0)) regressions))
17 (is (equal '(("c" 0.0d0 1.0d0)) improvements)))
18 (is (equal '(nil nil)
19 (multiple-value-list (compare-runs db old new :threshold 1)))
20 "Changes no larger than the threshold are ignored")
21 (is (= 2 (length (run-summaries db)))))
22 (sqlite:disconnect db))))
SQLite returns REAL columns as double floats, hence 1.0d0 in the expected values.
Run the tests with:
1 (asdf:test-system :llm-eval)
1 Running test suite ALL-TESTS
2 Running test suite SCORING
3 Running test TOKENS ....
4 Running test EXACT-MATCH ..
5 Running test TOKEN-F1 ....
6 Running test KEYWORD-RECALL ....
7 Running test suite JUDGING
8 Running test PARSE-JUDGE-REPLY ....
9 Running test JUDGE-PROMPT-HAS-ALL-PARTS ...
10 Running test JUDGE-WITH-FAKE-MODEL ....
11 Running test suite DATASETS
12 Running test GOLDEN-DATASET-LOADS ..
13 Running test READ-EVAL-IS-OFF .
14 Running test suite HARNESS
15 Running test RUN-EVAL-SURVIVES-A-FAILING-CASE ...
16 Running test STORE-FINDS-REGRESSIONS ....
17 Running test RAG-PIPELINE-OFFLINE ..
18 Did 37 checks.
19 Pass: 37 (100%)
20 Skip: 0 ( 0%)
21 Fail: 0 ( 0%)
Evaluating the RAG System
rag-eval.lisp connects the harness to the RAG system. It points RAG’s embeddings at the local nomic-embed-text model, loads the three sample documents, and wraps rag:query as a function of one question:
1 (defparameter *rag-cases-file*
2 (asdf:system-relative-pathname :llm-eval "data/rag-golden.lisp"))
3
4 (defparameter *store-file*
5 (asdf:system-relative-pathname :llm-eval "eval-runs.db"))
6
7 (defun build-rag-corpus ()
8 (setf rag:*embedding-model* "ollama/nomic-embed-text"
9 rag:*rag-verbose* nil)
10 (let ((corpus (rag:make-corpus :name "sample-docs"
11 :description "Energy, EVs, and climate")))
12 (dolist (file '("renewable-energy.txt" "electric-vehicles.txt" "climate-science.txt"))
13 (rag:add-document corpus (asdf:system-relative-pathname :rag (format nil "data/~A" file))))
14 corpus))
15
16 (defun make-rag-system (corpus &key (model "ollama/llama3.2:3b") (top-k 3) (max-iterations 3))
17 "Return a function of one question that answers it with agentic RAG."
18 (lambda (question)
19 (rag:query corpus question :model model :top-k top-k :max-iterations max-iterations)))
20
21 (defun run-rag-eval (label &key (model "ollama/llama3.2:3b") (top-k 3) (max-iterations 3)
22 (corpus (build-rag-corpus)))
23 "Run the RAG golden dataset with the given settings, print a report, and
24 store the run. Returns the run id."
25 (let* ((cases (load-cases *rag-cases-file*))
26 (results (run-eval (make-rag-system corpus :model model :top-k top-k
27 :max-iterations max-iterations)
28 cases))
29 (db (open-store (namestring *store-file*))))
30 (unwind-protect
31 (progn
32 (print-report results)
33 (save-run db label
34 (list :model model :top-k top-k :max-iterations max-iterations
35 :judge *judge-model*)
36 results))
37 (sqlite:disconnect db))))
examples.lisp runs three evals and compares them. The first two use the same settings, which shows how much the scores move from noise alone. The third gives the RAG system less to work with: one retrieved chunk per query and one round of retrieval.
1 (let* ((corpus (build-rag-corpus))
2 (baseline (run-rag-eval "baseline" :corpus corpus))
3 (baseline-again (run-rag-eval "baseline again" :corpus corpus))
4 (less-context (run-rag-eval "top-k 1, 1 iteration" :corpus corpus
5 :top-k 1 :max-iterations 1))
6 (db (open-store (namestring *store-file*))))
7 (unwind-protect
8 (progn
9 (terpri)
10 (print-run-summaries db)
11 (terpri)
12 (print-comparison db baseline baseline-again)
13 (terpri)
14 (print-comparison db baseline less-context))
15 (sqlite:disconnect db)))
1 (ql:quickload :llm-eval/rag)
2 (load "examples.lisp")
All three runs took under two minutes on my Mac. Here is the report for the third run:
1 case exact f1 keys judge
2 co2-preindustrial 0 0.13 1.00 1
3 co2-2024 0 0.30 1.00 0
4 warming-so-far 0 0.23 1.00 1
5 greenland 0 0.06 1.00 1
6 dac-cost 0 0.18 1.00 1
7 leaf-range 0 0.08 1.00 1
8 battery-manufacturing-co2 0 0.17 1.00 1
9 solid-state-companies 0 0.02 0.00 0
10 storage-cost-drop 0 0.53 1.00 1
11 denmark-wind 0 0.06 1.00 1
12 hydro-share 0 0.09 1.00 1
13 storage-roles 0 0.13 1.00 1
14 mean 0.00 0.16 0.92 0.83
15 12 cases, 0 errors, 1.2 seconds per case
And the summary and comparisons:
1 id label created cases f1 keys judge
2 1 baseline 2026-11-19 17:08:10 12 0.13 1.00 1.00
3 2 baseline again 2026-11-19 17:08:55 12 0.12 1.00 1.00
4 3 top-k 1, 1 iteration 2026-11-19 17:09:18 12 0.16 0.92 0.83
5
6 Run 1 -> run 2, metric judge
7 Regressions: none
8 Improvements: none
9
10 Run 1 -> run 3, metric judge
11 Regressions:
12 co2-2024: 1.00 -> 0.00
13 solid-state-companies: 1.00 -> 0.00
14 Improvements: none
What the Scores Tell Us
Exact match is useless for generated answers. It scored 0 on all 36 answers, including the correct ones. The RAG system writes full sentences and cites its sources, so its answers never match a short reference word for word. Exact match only suits tasks with short, fixed answers, such as classification labels.
Token F1 does not track correctness. The baseline’s answers were all correct, yet their mean F1 was 0.13. Worse, F1 rose to 0.16 in the third run while two answers broke. Shorter answers simply share a larger share of their tokens with a short reference. Here is the F1 comparison between runs 1 and 3, with a threshold of 0.1 to hide the smallest changes:
1 (print-comparison db 1 3 :metric "f1" :threshold 0.1)
1 Run 1 -> run 3, metric f1
2 Regressions:
3 co2-preindustrial: 0.26 -> 0.13
4 Improvements:
5 storage-cost-drop: 0.20 -> 0.53
6 warming-so-far: 0.05 -> 0.23
F1 flagged a case the judge passed and missed both real failures. Without the threshold, the list had twelve entries, most of them noise.
Keyword recall caught one failure but not the other. With one chunk per query, retrieval never found the passage that names the solid-state battery companies, and the answer said so. Keyword recall dropped to 0 for that case. The co2-2024 failure is different. You can see why by reading the answer in SQLite:
1 sqlite3 eval-runs.db "SELECT run_id, answer, judge_reason FROM results WHERE case_id = 'co2-2024'"
In run 3 the answer says CO2 “has reached over 420 parts per million”, then ends with “However, I couldn’t find any information on the specific level of atmospheric CO2 as of 2024.” The number is there, so keyword recall gave it full marks. The judge failed it: “The candidate answer contains a direct contradiction.”
The judge is the most useful score, and it is not consistent. The run 1 answer for the same case had almost the same contradiction (“the passage does not provide information on the exact level of atmospheric CO2 as of 2024”), and the judge passed it. The judge is a model too, so it makes mistakes. Read the answers and reasons behind every regression before you act on it. Don’t treat a one-case change as proof.
Two runs with the same settings agreed on the judge score. Runs 1 and 2 both scored 12 of 12, and their F1 means differed by 0.01. Before you trust a difference between two configurations, run the same configuration twice to see how much the scores move on their own.
The cheap setting was faster and worse. The third run took 1.2 seconds per case against about 3 seconds for the baseline, and it lost two answers. That is the kind of trade-off an eval should show you before your users find it.
Applying the Harness to Other Code
run-eval takes any function from a question string to an answer string, so it works with any question answering code in this book. For the document QA chapter it would be:
1 (run-eval (lambda (question) (docs-qa:qa question t))
2 (load-cases "my-docs-qa-cases.lisp"))
I have not run this one: docs-qa uses OpenAI embeddings by default, and you would first need to write a golden dataset for its sample documents.
Wrap Up
LLM applications need both unit tests and evals:
- Unit tests check the deterministic code. Route each model call through a special variable or parameter, and tests can replace it with a fake and run offline in seconds.
- Evals check the answers. Keep a golden dataset with stable case ids, store every run with its settings, and compare runs case by case rather than by the mean alone.
- Of the scores we tried, keyword recall is a cheap first check and an LLM judge is the best single score. Exact match and token F1 did not track correctness for this RAG system.
- Test the judge before you trust it, keep it a different model from the system under test, and read the answers behind each regression.
A good next step is to grow the golden dataset with questions the documents can’t answer. A good RAG system should say it doesn’t know, and a case with the reference answer “The documents do not say.” tests that.