Evaluating and Operating LLM Applications
Earlier chapters in this book showed how to call language models from OpenAI, Google Gemini, Perplexity, and local Ollama servers, how to build a local vector RAG system with zvec, and how to construct a web-scraping agent with Agno. Those programs print useful answers in an interactive session, yet a single successful terminal transcript does not prove that an application works reliably. When you change a system prompt, swap qwen3:30b for qwen3:1.7b, or alter chunk sizes in a retriever, you need a repeatable way to measure whether accuracy improved or regressed, how much each query costs, and whether an attacker can hijack your tools.
The code for this chapter is in the directory source_code_for_examples/llm_eval.
What You Will Build
In this chapter, I build hy-eval, a self-contained evaluation, tracing, structured-output, and operations toolkit written in Hy 1.3.1. We use hy-eval to compare a keyword-only RAG baseline (modeled on the RAG_zvec chapter) against a graph-enhanced RAG pipeline (modeled on the knowledge graph chapters) and to test an agent trajectory modeled on the agents_agno web-scraping agent. Everything runs offline by default using a deterministic FakeModel, with an optional flag to run against a local Ollama server.
1 +-----------------------------------------------------------------------+
2 | hy-eval Toolkit Flow |
3 +-----------------------------------------------------------------------+
4 | |
5 | JSONL Dataset (data/eval_cases.jsonl + data/corpus.jsonl) |
6 | | |
7 | v |
8 | Declarative Hy DSL: (defeval suite (case "id" (expect ...))) |
9 | | |
10 | v |
11 | ModelRouter ---> [1. Prompt Injection Guardrail] |
12 | ---> [2. Deterministic ResponseCache] |
13 | ---> [3. USD & Token BudgetTracker] |
14 | ---> [4. Primary Model (OllamaModel) -> FallbackModel] |
15 | ---> [5. Tracer (PII/Secret Redaction -> JSONL Sink)] |
16 | | |
17 | +---> RAG Evaluators (Recall@k, Precision@k, Groundedness, |
18 | | Citation Accuracy) |
19 | +---> Structured Output Loop (Pydantic + Self-Healing Retry) |
20 | +---> Agent Evaluators (Tool Sequence, Argument Correctness, |
21 | | ToolAllowlist SSRF Gate) |
22 | +---> Metrics & Judges (Exact, Token F1, BoW Cosine, |
23 | LLMJudge, Debiased Pairwise Judge) |
24 | | |
25 | v |
26 | ASCII Comparison Tables + Failure Inspector + JSON Report |
27 +-----------------------------------------------------------------------+
Why It Matters
Demos fail in production for four reasons:
- Nondeterminism and prompt sensitivity. A minor wording change in a system prompt can fix one question while breaking three others, or cause a model to wrap JSON output in markdown prose that crashes a parser.
- Silent retrieval failures in RAG. When a RAG pipeline gives a bad answer, you cannot tell from the final text alone whether the retriever missed the relevant document chunk or the generator ignored the context it received.
- Unbounded agent actions and prompt injection. An agent equipped with a web scraper or shell tool can be tricked by untrusted text into reading internal network metadata (
http://169.254.169.254) or leaking secrets into log files. - Uncontrolled latency and token spend. Calling a large model repeatedly on identical inputs wastes money and time without a response cache, token budget, and fallback router.
The Hy Angle
Lisp macros let us turn repetitive evaluation boilerplate into a compact specification language. In hy-eval, three macros (defeval, case, and expect) capture both the runtime value and the unevaluated Hy source expression at compile time. When a check fails, the report knows which expression produced the value without requiring bytecode decompilation or heavy test-runner plugins.
Python Equivalent and Macro Expansion
In plain Python, recording the source expression alongside a check result requires either repeating the expression as a string label or writing a custom pytest AST rewriter:
1 # Python: manual ceremony to time a case and label each assertion
2 def smoke():
3 def _case_c1():
4 return run_expect(ask(q), expr="ask(q)", contains="Krampton")
5 return summarize_suite("smoke", [execute_case("c1", _case_c1)])
In Hy, we write the same evaluation suite declaratively:
1 (require hy_eval [defeval case expect])
2
3 (defeval smoke
4 (case "c1"
5 (expect (ask q) :contains "Krampton")))
We can inspect what expect expands to at compile time with hy.macroexpand-1:
1 '(hy.I.hy_eval/harness.run-expect (ask q) :expr "'(ask q)" :contains "Krampton")
Running uv run hy2py on the defeval snippet shows the exact Python code that Hy generates:
1 import hy
2 hy.macros.require('hy_eval', None, target_module_name='<stdin>', assignments=[['defeval', 'defeval'], ['case', 'case'], ['expect', 'expect']], prefix='')
3
4 def smoke():
5
6 def _hy_gensym_case_fn_1():
7 return hy.I.hyx_hy_evalXsolidusXharness.run_expect(ask(q), expr="'(ask q)", contains='Krampton')
8 return hy.I.hyx_hy_evalXsolidusXharness.summarize_suite('smoke', [hy.I.hyx_hy_evalXsolidusXharness.execute_case('c1', _hy_gensym_case_fn_1)])
Input Data Format
Before looking at the code, let us inspect the two JSON Lines (.jsonl) files in data/. Keeping evaluation data in line-oriented JSON files lets us track changes in Git and add regression cases whenever we find a bug.
The fixture corpus in data/corpus.jsonl holds six document chunks spanning the economics examples from the RAG_zvec chapter, the Hy compiler, the NSK query engine, and the Agno agent chapter:
1 {"id": "econ-1", "title": "Microeconomics vs Macroeconomics", "text": "Microeconomics studies the decisions of individuals, households, and firms in specific markets, focusing on supply, demand, pricing, and resource allocation. Macroeconomics analyzes the economy as a whole, examining aggregate indicators such as inflation, national income, gross domestic product (GDP), unemployment, and fiscal or monetary policy.", "entities": ["Microeconomics", "Macroeconomics", "GDP"]}
2 {"id": "econ-2", "title": "Blendergast on Economic Models", "text": "Pauli Blendergast, a senior economist at the University of Krampton Ohio, famously argued in his 2024 lecture that 'economics is bullshit' when mathematical equilibrium models ignore institutional friction and human irrationality.", "entities": ["Pauli Blendergast", "University of Krampton Ohio"]}
3 {"id": "econ-3", "title": "Krampton Department Periodicals", "text": "The Krampton Ohio Department of Economics edits the quarterly Journal of Skeptical Economics, studying empirical market failures and behavioral anomalies.", "entities": ["University of Krampton Ohio", "Journal of Skeptical Economics"]}
Notice how econ-2 connects Pauli Blendergast to University of Krampton Ohio, while econ-3 connects the Krampton department to the Journal of Skeptical Economics without repeating the phrase "Pauli Blendergast" or "University". A multi-hop question that asks which journal is published where Blendergast teaches will test whether our retriever can follow entity links across chunks.
The evaluation dataset in data/eval_cases.jsonl pairs each question with gold document IDs, a reference answer, required and forbidden keywords, and a flag indicating whether the model should refuse to answer:
1 {"id": "q-blendergast", "question": "Who says economics is bullshit and at which university does he teach?", "relevant_docs": ["econ-2"], "reference_answer": "Pauli Blendergast, an economist at the University of Krampton Ohio, stated that economics is bullshit [econ-2].", "required_keywords": ["Pauli Blendergast", "Krampton"], "forbidden_keywords": [], "expect_refusal": false}
2 {"id": "q-krampton-journal", "question": "Which journal is published by the university where Pauli Blendergast teaches?", "relevant_docs": ["econ-2", "econ-3"], "reference_answer": "Pauli Blendergast teaches at the University of Krampton Ohio [econ-2], which publishes the Journal of Skeptical Economics [econ-3].", "required_keywords": ["Journal of Skeptical Economics", "Krampton"], "forbidden_keywords": [], "expect_refusal": false}
3 {"id": "q-chemistry-refusal", "question": "What titration procedures are performed in organic chemistry labs?", "relevant_docs": [], "reference_answer": "The provided context does not contain information about chemistry lab procedures.", "required_keywords": ["does not contain"], "forbidden_keywords": ["burette", "phenolphthalein", "molarity"], "expect_refusal": true}
Including q-chemistry-refusal (inspired by the unanswerable chemistry query in the RAG_zvec chapter) verifies that our system abstains when the corpus lacks evidence instead of inventing laboratory procedures.
Run It
Synchronize the environment and run the evaluation demo:
1 cd source_code_for_examples/llm_eval
2 uv sync
3 uv run hy run_evals.hy
The default run uses FakeModel so it finishes in under a second with no API keys or local daemon required. Passing uv run hy run_evals.hy --live-ollama attempts to query a local qwen3:1.7b model via Ollama and falls back to FakeModel with a note if Ollama is not running.
Here is the real output from uv run hy run_evals.hy:
1 === 1. RAG System Comparison: Baseline Keyword vs Graph-Enhanced ===
2 +-------------------------+----------------+----------------+------------+
3 | Metric | Baseline RAG | Graph RAG | Delta |
4 +-------------------------+----------------+----------------+------------+
5 | Retrieval Recall@2 | 0.900 | 1.000 | +0.100 |
6 | Context Precision@2 | 1.000 | 1.000 | +0.000 |
7 | Answer Token F1 | 0.763 | 0.892 | +0.129 |
8 | BoW Cosine Similarity | 0.790 | 0.905 | +0.115 |
9 | Groundedness | 0.877 | 0.924 | +0.047 |
10 | Citation Accuracy | 0.200 | 1.000 | +0.800 |
11 | LLM Judge Score | 0.830 | 0.950 | +0.120 |
12 | Suite Pass Rate | 0.200 | 1.000 | +0.800 |
13 +-------------------------+----------------+----------------+------------+
14
15 === 2. Pairwise Position-Debiased Judge (Multi-Hop Krampton Case) ===
16 Winner: System B (order-consistent=True, AB=B, BA=A)
17
18 === 3. Per-Case Suite Reports ===
19 Suite: baseline-keyword-rag (1/5 passed, pass_rate=20.0%, mean_score=0.706)
20 +------------------------+--------+---------+------------+----------------------------+
21 | Case ID | Status | Score | Latency ms | Failed Checks |
22 +------------------------+--------+---------+------------+----------------------------+
23 | q-blendergast | FAIL | 0.790 | 0.12 | citation_accuracy==1.0 |
24 | q-krampton-journal | FAIL | 0.170 | 0.05 | retrieval_recall==1.0, req |
25 | q-micro-macro | FAIL | 0.790 | 0.04 | citation_accuracy==1.0 |
26 | q-chemistry-refusal | PASS | 0.990 | 0.03 | - |
27 | q-hy-nsk-macros | FAIL | 0.790 | 0.03 | citation_accuracy==1.0 |
28 +------------------------+--------+---------+------------+----------------------------+
29
30 Suite: graph-enhanced-rag (5/5 passed, pass_rate=100.0%, mean_score=0.990)
31 +------------------------+--------+---------+------------+----------------------------+
32 | Case ID | Status | Score | Latency ms | Failed Checks |
33 +------------------------+--------+---------+------------+----------------------------+
34 | q-blendergast | PASS | 0.990 | 0.03 | - |
35 | q-krampton-journal | PASS | 0.990 | 0.04 | - |
36 | q-micro-macro | PASS | 0.990 | 0.03 | - |
37 | q-chemistry-refusal | PASS | 0.990 | 0.02 | - |
38 | q-hy-nsk-macros | PASS | 0.990 | 0.03 | - |
39 +------------------------+--------+---------+------------+----------------------------+
40
41 === 4. Failure Inspection (Baseline RAG) ===
42 - [q-blendergast] citation_accuracy==1.0 (score=0.0): value=0.0
43 - [q-krampton-journal] retrieval_recall==1.0 (score=0.0): value=0.5
44 - [q-krampton-journal] required_keywords (score=0.5): missing=['Journal of Skeptical Economics'], forbidden=[]
45 - [q-krampton-journal] groundedness>=0.75 (score=0.0): value=0.6667
46 - [q-krampton-journal] citation_accuracy==1.0 (score=0.0): value=0.0
47 - [q-krampton-journal] llm_judge (score=0.35): Misses the specific journal name Journal of Skeptical Economics because chunk econ-3 was not retrieved.
48 - [q-micro-macro] citation_accuracy==1.0 (score=0.0): value=0.0
49 - [q-hy-nsk-macros] citation_accuracy==1.0 (score=0.0): value=0.0
50
51 === 5. Operational Readiness Suite (Structured Retry, Agent, Guardrails, Cache) ===
52 Suite: operational-readiness-suite (4/4 passed, pass_rate=100.0%, mean_score=1.000)
53 +------------------------+--------+---------+------------+----------------------------+
54 | Case ID | Status | Score | Latency ms | Failed Checks |
55 +------------------------+--------+---------+------------+----------------------------+
56 | structured-output-self | PASS | 1.000 | 0.66 | - |
57 | agno-agent-trajectory | PASS | 1.000 | 0.08 | - |
58 | guardrails-injection-a | PASS | 1.000 | 0.04 | - |
59 | cache-and-pii-redactio | PASS | 1.000 | 0.10 | - |
60 +------------------------+--------+---------+------------+----------------------------+
61
62 === 6. Trace & Budget Telemetry Summary ===
63 {
64 "trace_id": "demo0001",
65 "total_spans": 13,
66 "total_tokens": 1075,
67 "total_cost_usd": 0.000518,
68 "cache_hits": 1,
69 "total_redactions": 4,
70 "budget_remaining_usd": 0.049482
71 }
Interpreting the Results: Avoid Metric Worship
Look closely at the numbers for Baseline RAG in Section 1 of the output above. Its average Context Precision@2 is 1.000, its Retrieval Recall@2 is 0.900, its Groundedness is 0.877, and its average LLM Judge Score is 0.830. If a team only tracked aggregate averages on a dashboard, they might ship Baseline RAG thinking it is 85% to 90% accurate.
Yet the Suite Pass Rate is 0.200 (only 1 of 5 cases passed). Section 4 (Failure Inspection) shows why aggregate averages can mislead you:
- Multi-hop retrieval failure on
q-krampton-journal. Keyword retrieval foundecon-2(because the question mentionsPauli Blendergast), giving a 50% recall on that question and a 90% recall average across the five questions. Because it missedecon-3, the generator did not know the title Journal of Skeptical Economics and guessed"an economics review". That single missing chunk caused four downstream checks to fail:retrieval_recall,required_keywords,groundedness, andllm_judge. - Missing source citations. On
q-blendergast,q-micro-macro, andq-hy-nsk-macros, the baseline prose was factually right, so token F1 and the LLM judge scored it high, but it omitted bracketed[doc-id]citations. Only the chemistry refusal case passed all five checks.
The lesson is simple: do not worship aggregate metrics. Prefer a small, well-understood evaluation set where you inspect every failed check over a large, noisy benchmark that reduces system behavior to a single number.
How It Works
The hy_eval package is organized into nine focused modules. Let us walk through each component.
1. Model Abstraction: hy_eval/models.hy
To run fast, deterministic evaluations in continuous integration (CI) while supporting live local models when needed, hy_eval/models.hy defines a uniform ModelResponse dataclass and two model classes: FakeModel and OllamaModel.
1 (defclass [(dataclass :frozen True)] ToolCall []
2 "A structured tool call emitted by a model or agent."
3 #^ str name
4 #^ dict arguments)
5
6
7 (defclass [dataclass] ModelResponse []
8 "Standardized model response with token, latency, cost, and tool telemetry."
9 #^ str text
10 #^ str model
11 (setv #^ int prompt-tokens 0
12 #^ int completion-tokens 0
13 #^ float latency-ms 0.0
14 #^ float cost-usd 0.0
15 #^ bool cached False
16 #^ list tool-calls (field :default-factory list)))
FakeModel matches substrings in the combined system and user prompt against a dictionary of recorded responses. An entry in responses can be a string, a dictionary containing tool_calls, a callable, or a list of responses consumed in sequence across successive calls. Returning a sequence of replies lets us test retry loops where the first model call returns malformed JSON and the second call succeeds:
1 (defclass FakeModel []
2 "Deterministic, offline model for tests and reproducible evaluations.
3
4 `responses` maps a prompt substring to one of:
5 - a plain string reply
6 - a dict with keys 'text', optional 'tool_calls', optional 'latency_ms'
7 - a list of replies consumed in order (useful for testing retries)
8 - a callable (fn [prompt system] ...) returning a string or dict"
9
10 (defn __init__ [self
11 [name "fake-qwen3:1.7b"]
12 [responses None]
13 [default-response "The provided context does not contain enough information."]
14 [cost-per-1k-tokens 0.0005]
15 [simulated-latency-ms 4.2]]
16 (setv self.name name
17 self.responses (if (is responses None) {} (dict responses))
18 self.default-response default-response
19 self.cost-per-1k-tokens cost-per-1k-tokens
20 self.simulated-latency-ms simulated-latency-ms
21 self.history []
22 self._cursors {}))
23
24 (defn _resolve-entry [self key entry prompt system]
25 (cond
26 (isinstance entry list)
27 (do
28 (setv idx (.get self._cursors key 0))
29 (setv chosen (get entry (min idx (- (len entry) 1))))
30 (setv (get self._cursors key) (+ idx 1))
31 chosen)
32
33 (callable entry)
34 (entry prompt system)
35
36 True
37 entry))
OllamaModel uses Python’s standard urllib.request module (just like the RAG_zvec chapter) to call /api/chat on a local Ollama daemon, and falls back to self.fallback if the daemon is unreachable.
2. Metrics and Position-Debiased Judges: hy_eval/metrics.hy
A practical evaluation suite mixes fast deterministic checks with rubric-based model judges:
exact-match: normalizes case, punctuation, and whitespace before comparing strings.token-f1: computes multiset token precision, recall, and F1 overlap usingcollections.Counter.keyword-checkandregex-check: verify that required domain terms appear and forbidden hallucination terms do not.bow-cosine-similarity: computes bag-of-words cosine similarity over lightly stemmed tokens using standard-librarymath.sqrt, avoiding heavy neural embedding dependencies in unit tests.LLMJudgeandpairwise-judge: grade open-ended answers against a rubric.
1 (defn token-f1 [#^ str actual #^ str reference]
2 "Compute token-level precision, recall, and F1 between actual and reference."
3 (setv pred-tokens (tokenize actual)
4 ref-tokens (tokenize reference))
5 (when (and (not pred-tokens) (not ref-tokens))
6 (return {"precision" 1.0 "recall" 1.0 "f1" 1.0}))
7 (when (or (not pred-tokens) (not ref-tokens))
8 (return {"precision" 0.0 "recall" 0.0 "f1" 0.0}))
9
10 (setv overlap (& (Counter pred-tokens) (Counter ref-tokens))
11 num-same (sum (.values overlap)))
12 (when (= num-same 0)
13 (return {"precision" 0.0 "recall" 0.0 "f1" 0.0}))
14
15 (setv precision (/ num-same (len pred-tokens))
16 recall (/ num-same (len ref-tokens))
17 f1 (/ (* 2.0 precision recall) (+ precision recall)))
18 {"precision" (round precision 4)
19 "recall" (round recall 4)
20 "f1" (round f1 4)})
When using an LLM to compare two system outputs (A and B), models exhibit position bias: weak judges often favor whichever candidate is listed first. Our pairwise-judge function calls the judge model twice, first in order (A, B) and then in swapped order (B, A). It declares a winner only if both passes agree on the same underlying answer:
1 (defn pairwise-judge [model #^ str question #^ str answer-a #^ str answer-b
2 [rubric "Prefer answers that are accurate, grounded in context, and cite sources."]]
3 "Compare two candidate answers in both orders (A, B) and (B, A) to detect position bias."
4 (setv sys-prompt
5 (+ "You are a pairwise evaluation judge. Compare Candidate 1 and Candidate 2. "
6 "Return JSON: {\"winner\": \"A\" if Candidate 1 is better, \"B\" if Candidate 2 is better, or \"TIE\", \"reason\": string}.\n"
7 f"Rubric: {rubric}"))
8 (setv prompt-ab
9 f"[PAIRWISE_ORDER:AB]\nQuestion: {question}\nCandidate 1 (A): {answer-a}\nCandidate 2 (B): {answer-b}"
10 prompt-ba
11 f"[PAIRWISE_ORDER:BA]\nQuestion: {question}\nCandidate 1 (A): {answer-b}\nCandidate 2 (B): {answer-a}")
12
13 (setv resp-ab (.generate model prompt-ab :system sys-prompt)
14 resp-ba (.generate model prompt-ba :system sys-prompt)
15 win-ab (_parse-winner resp-ab.text)
16 win-ba (_parse-winner resp-ba.text))
17
18 ;; When order is swapped (B first, A second), "A" in the second call means answer-b won,
19 ;; and "B" in the second call means answer-a won.
20 (setv mapped-ba (cond
21 (= win-ba "A") "B"
22 (= win-ba "B") "A"
23 True "TIE"))
24 (if (and (= win-ab mapped-ba) (in win-ab ["A" "B"]))
25 {"winner" win-ab "consistent" True "first_pass" win-ab "swapped_pass" win-ba}
26 {"winner" "TIE" "consistent" (= win-ab mapped-ba) "first_pass" win-ab "swapped_pass" win-ba}))
3. The Evaluation Harness and Macro DSL: hy_eval/harness.hy
hy_eval/harness.hy provides JSONL dataset helpers (load-jsonl and save-jsonl), the runtime check dispatcher run-expect, and the three macros expect, case, and defeval.
A small stack (_collector-stack) lets expect work both as a standalone assertion in a REPL or test function and as a check collector inside a case block:
1 (defn execute-case [#^ str case-id thunk [metadata None]]
2 "Run a zero-arg `thunk`, collecting all `expect` checks executed inside it."
3 (setv checks []
4 start (time.perf-counter))
5 (.append _collector-stack checks)
6 (try
7 (thunk)
8 (finally
9 (.pop _collector-stack)))
10 (setv elapsed (* (- (time.perf-counter) start) 1000.0)
11 passed (if checks (all (gfor c checks c.passed)) True)
12 score (if checks
13 (round (/ (sum (gfor c checks c.score)) (len checks)) 4)
14 1.0))
15 (CaseResult
16 :case-id case-id
17 :passed passed
18 :score score
19 :checks checks
20 :latency-ms (round elapsed 2)
21 :metadata (if (is metadata None) {} (dict metadata))))
22
23
24 (defmacro expect [actual-form #* options]
25 "Declarative expectation macro that captures the source form and runs `run-expect`."
26 `(hy.I.hy_eval/harness.run-expect
27 ~actual-form
28 :expr ~(hy.repr actual-form)
29 ~@options))
30
31
32 (defmacro case [case-id #* body]
33 "Define a single evaluation case that times `body` and collects all `expect` checks."
34 (setv fn-sym (hy.gensym "case_fn"))
35 `(do
36 (defn ~fn-sym []
37 ~@body)
38 (hy.I.hy_eval/harness.execute-case ~case-id ~fn-sym)))
39
40
41 (defmacro defeval [suite-name #* cases]
42 "Define a reusable evaluation suite function that returns a SuiteReport."
43 `(defn ~suite-name []
44 (hy.I.hy_eval/harness.summarize-suite
45 ~(str suite-name)
46 [~@cases])))
Notice the hygiene in case: (hy.gensym "case_fn") creates a unique temporary function symbol so nested or adjacent cases never collide with user variables.
4. Tracing and PII Redaction: hy_eval/tracing.hy
When an LLM application misbehaves in production, you need the exact prompt, response, token count, latency, and cost written to an inspectable log. At the same time, user prompts often contain email addresses, phone numbers, Social Security numbers, or accidentally pasted API keys that must never land in plain text on disk.
hy_eval/tracing.hy runs every prompt and response through redact-with-stats before storing a SpanRecord in memory or appending it to a .jsonl trace file:
1 (setv _REDACTION_PATTERNS
2 [#("API_KEY"
3 (re.compile r"(?:sk-[A-Za-z0-9_-]{12,}|Bearer\s+[A-Za-z0-9._-]{12,}|(?:OPENAI|ANTHROPIC|GEMINI)_API_KEY\s*=\s*\S+)"))
4 #("EMAIL"
5 (re.compile r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b"))
6 #("SSN"
7 (re.compile r"\b\d{3}-\d{2}-\d{4}\b"))
8 #("PHONE"
9 (re.compile r"(?<!\d)(?:\+1[-.\s]?)?\(?\d{3}\)?[-.\s]\d{3}[-.\s]\d{4}(?!\d)"))])
10
11
12 (defn redact-with-stats [#^ str text]
13 "Replace secrets and PII tokens in `text`. Return #(redacted_text count)."
14 (when (not text)
15 (return #("" 0)))
16 (setv out text
17 total-count 0)
18 (for [[label pattern] _REDACTION_PATTERNS]
19 (setv [out n] (.subn pattern f"[REDACTED_{label}]" out))
20 (+= total-count n))
21 #(out total-count))
5. Structured Output Validation and Retry Loop: hy_eval/structured.hy
Downstream code often expects a typed structure rather than free-form prose. hy_eval/structured.hy defines Pydantic models (CitedAnswer and ExtractedTriple) and a generate-structured function. If the model wraps its JSON in markdown code fences, extract-json-block strips the fences; if json.loads or Pydantic’s .model_validate raises an error, generate-structured feeds the exact validation error message back to the model in a repair prompt up to max-retries times:
1 (defclass CitedAnswer [BaseModel]
2 "Structured RAG answer with required source citations and confidence score."
3 #^ str answer
4 (setv #^ (get list str) citations (Field :default-factory list)
5 #^ float confidence (Field :ge 0.0 :le 1.0)
6 #^ bool abstained False)
7
8 (defn [(field_validator "answer")] non-empty-answer [cls v]
9 (when (not (.strip v))
10 (raise (ValueError "answer must not be empty")))
11 v))
12
13
14 (defn generate-structured [model schema-cls #^ str prompt [system ""] [max-retries 2]]
15 "Generate a response validated against `schema-cls` (a Pydantic BaseModel).
16
17 On invalid JSON or schema validation failure, feeds the error back to the
18 model up to `max-retries` times."
19 (setv schema-json (json.dumps (.model_json_schema schema-cls))
20 full-system
21 (+ (if system (+ system "\n\n") "")
22 "Respond ONLY with valid JSON conforming to this JSON Schema:\n"
23 schema-json)
24 current-prompt prompt
25 errors []
26 last-raw "")
27
28 (for [attempt (range (+ max-retries 1))]
29 (setv resp (.generate model current-prompt :system full-system)
30 last-raw resp.text)
31 (try
32 (setv candidate (extract-json-block resp.text)
33 parsed (json.loads candidate)
34 validated (.model_validate schema-cls parsed))
35 (return {"valid" True
36 "data" validated
37 "attempts" (+ attempt 1)
38 "errors" errors
39 "raw_response" last-raw})
40 (except [e #(json.JSONDecodeError ValidationError ValueError)]
41 (setv err-msg (str e))
42 (.append errors err-msg)
43 (setv current-prompt
44 (+ prompt
45 f"\n\n[VALIDATION_RETRY_{attempt}]\n"
46 f"Your previous output was invalid:\n{resp.text}\n"
47 f"Validation error:\n{err-msg}\n"
48 "Return ONLY corrected JSON matching the schema.")))))
49
50 {"valid" False
51 "data" None
52 "attempts" (+ max-retries 1)
53 "errors" errors
54 "raw_response" last-raw})
6. Evaluating RAG Retrieval and Groundedness: hy_eval/rag.hy
In the RAG_zvec chapter, we built an offline vector RAG pipeline over local text files. Evaluating a RAG system requires measuring the retriever and the generator separately:
- Retrieval Recall@k (
retrieval-recall-at-k): what fraction of the gold relevant document IDs appear in the topkretrieved chunks? - Context Precision@k (
context-precision-at-k): are the relevant chunks ranked near the top of the retrieved list? - Groundedness (
groundedness-score): what fraction of content words in the answer are supported by the retrieved chunks (or did the model properly abstain)? - Citation Accuracy (
citation-accuracy): do the bracketed[doc-id]citations in the answer refer to chunks that were actually retrieved, with zero hallucinated IDs?
hy_eval/rag.hy also compares keyword-retrieve against graph-retrieve, which expands one hop across entity edges (DEFAULT_GRAPH_EDGES) to pull in connected evidence chunks:
1 (defn graph-retrieve [corpus #^ str query [graph-edges None] [top-k 2]]
2 "Retrieve initial chunks by keyword overlap, then expand 1 hop over entity edges."
3 (setv edges (if (is graph-edges None) DEFAULT_GRAPH_EDGES graph-edges)
4 initial (keyword-retrieve corpus query :top-k top-k)
5 by-id (dfor doc corpus (get doc "id") doc)
6 seen-ids (set (gfor d initial (get d "id")))
7 results (list initial)
8 seed-entities (set))
9
10 (for [doc initial]
11 (for [ent (.get doc "entities" [])]
12 (.add seed-entities (.lower ent))))
13 (setv q-lower (.lower query))
14 (for [edge edges]
15 (when (in (.lower (get edge "source")) q-lower)
16 (.add seed-entities (.lower (get edge "source"))))
17 (when (in (.lower (get edge "target")) q-lower)
18 (.add seed-entities (.lower (get edge "target")))))
19
20 ;; Expand one hop along matching graph edges
21 (for [edge edges]
22 (setv src (.lower (get edge "source"))
23 tgt (.lower (get edge "target"))
24 target-doc-id (get edge "doc_id"))
25 (when (and (or (in src seed-entities) (in tgt seed-entities))
26 (in target-doc-id by-id)
27 (not (in target-doc-id seen-ids))
28 (< (len results) top-k))
29 (.add seen-ids target-doc-id)
30 (.append results (get by-id target-doc-id))))
31
32 (cut results 0 top-k))
7. Agent Trajectory Evaluation and Guardrails: hy_eval/agent_eval.hy and hy_eval/guardrails.hy
In the agents_agno chapter, an agent called scrape-website-content on https://markwatson.com to answer a question about musical instruments. Evaluating an agent requires checking how it reached its answer:
- Did it invoke the expected sequence of tools (
tool-sequence-score)? - Did it pass the right arguments (
argument-correctness)? - Did it stay within the step budget (
evaluate-trajectory)?
Before executing any tool call or model prompt, hy_eval/guardrails.hy enforces two defenses:
detect-prompt-injectionscans untrusted inputs for instruction overrides ("Ignore all previous instructions"), system prompt extraction attempts, role hijacks ("You are now DAN"), and delimiter escapes.ToolAllowlistimplements a deny-by-default permission gate. Even whenscrape_website_contentis on the allowlist,ToolAllowlistinspects anyurlargument to blockfile://schemes, loopback addresses (localhost,127.0.0.1), cloud metadata endpoints (169.254.169.254), and domains outsideallowed_domains:
1 (defclass ToolAllowlist []
2 "Deny-by-default permission gate for agent tools and URL arguments."
3
4 (defn __init__ [self allowed-tools [allowed-domains None]]
5 (setv self.allowed-tools (set allowed-tools)
6 self.allowed-domains (if (is allowed-domains None)
7 None
8 (set (gfor d allowed-domains (.lower d))))))
9
10 (defn validate-call [self #^ str tool-name [arguments None]]
11 "Return {\"allowed\": bool, \"reason\": str} for a proposed tool call."
12 (setv args (if (is arguments None) {} (dict arguments)))
13 (when (not (in tool-name self.allowed-tools))
14 (return {"allowed" False
15 "reason" f"Tool '{tool-name}' is not in the allowlist."}))
16
17 (when (in "url" args)
18 (setv raw-url (str (get args "url"))
19 parsed (urlparse raw-url)
20 scheme (.lower (or parsed.scheme ""))
21 host (.lower (or parsed.hostname "")))
22 (when (not (in scheme #{"http" "https"}))
23 (return {"allowed" False
24 "reason" f"Disallowed URL scheme '{scheme}' in {(repr raw-url)}."}))
25 (when (or (not host) (_is-private-host host))
26 (return {"allowed" False
27 "reason" f"Blocked private or loopback host '{host}'."}))
28 (when (not (is self.allowed-domains None))
29 (setv domain-ok (any (gfor d self.allowed-domains
30 (or (= host d) (.endswith host (+ "." d))))))
31 (when (not domain-ok)
32 (setv allowed-list (sorted self.allowed-domains))
33 (return {"allowed" False
34 "reason" f"Host '{host}' is outside allowed domains {allowed-list}."}))))
35
36 {"allowed" True "reason" "ok"}))
8. Operations: Cache, Budgets, and Fallback Routing: hy_eval/ops.hy
ModelRouter wraps a primary model and a list of fallback models with detect-prompt-injection, ResponseCache, BudgetTracker, and Tracer. When a request arrives:
- The router checks the prompt against the injection guardrail.
- It checks
ResponseCache. On a cache hit, it returns a copy of the cachedModelResponsewithcached=Trueandcost_usd=0.0. - It checks
BudgetTrackerto ensure cumulative USD spend and token usage remain below their hard ceilings. - It calls the primary model. If the primary model raises a network or timeout error, it iterates through
self.fallbacksand records the fallback index in the trace tags.
Putting Evaluations in Continuous Integration (CI)
A good evaluation workflow splits tests into two tiers:
- Fast smoke evaluations on every pull request (under 5 seconds). Run
uv run pytest -qanduv run hy run_evals.hyusingFakeModeland recorded fixtures. Every PR verifies that prompt templates, structured-output Pydantic schemas, retrieval logic, citation parsers, guardrails, PII redaction, and agent trajectory checks pass 100% deterministically with zero API cost. - Scheduled nightly evaluations against live models. On a nightly cron job (or before cutting a release), run the full dataset against your live local Ollama model (
uv run hy run_evals.hy --live-ollama) or cloud provider, export the JSON report withexport-report-json, and alert when retrieval recall, groundedness, or pass rate drops below your baseline threshold.
Tests
The project includes four offline test suites (test_metrics.hy, test_harness.hy, test_structured.hy, and test_guardrails.hy) containing 18 tests covering deterministic metrics, debiased pairwise judges, macro expansion, JSONL round-tripping, self-healing Pydantic retries, agent trajectories, SSRF blocking, PII redaction, caching, and budget enforcement.
Run the test suite with pytest:
1 $ uv run pytest -q
2 .................. [100%]
3 18 passed in 0.11s
Wrap Up
In this chapter, we moved from one-off LLM demos to an observable, testable engineering workflow. We built hy-eval in Hy 1.3.1 using macros (defeval, case, expect) for readable evaluation suites, combined deterministic metrics with a position-debiased pairwise judge, validated structured outputs with Pydantic retry loops, measured RAG retrieval separately from generation, enforced deny-by-default tool allowlists, and traced every call with automatic PII redaction. Most importantly, our side-by-side comparison of keyword RAG and graph-enhanced RAG showed why engineers must inspect individual failure cases rather than trusting high aggregate metric averages.
Optional Practice Problems
- Add a Latency SLA Expectation (Small): Extend
run-expectinhy_eval/harness.hyto support a:max-latency-mskeyword option when passed aModelResponse. Add a test case intest_harness.hyasserting that a cached response fromModelRouterfinishes in under1.0millisecond. - Evaluate Live
zvecand Agno Pipelines (Open-Ended): Importhy_evalinto theRAG_zvecoragents_agnoexample directory. Record five real responses from your local Ollama model into a.jsonlfixture file, write adefevalsuite that checks groundedness and tool argument safety, and compareqwen3:1.7bagainst another local model usingformat-comparison-table.
Further Reading
- Chip Huyen, AI Engineering (O’Reilly, 2025), Chapters 3-6 and 10 on evaluation methodology, RAG evaluation, guardrails, and observability.
- Paul Iusztin and Maxime Labonne, LLM Engineer’s Handbook (Packt, 2024), on end-to-end RAG evaluation and monitoring pipelines.
- Tomaž Bratanič and Oskar Hane, Essential GraphRAG (Manning, 2025), on combining structured graph traversal with text retrieval for traceable answers.