AutoContext: Prepare Effective Prompts with Context for LLM Queries
Dear reader, given a large corpus of text documents, and given a query, how do we create a combined one-shot prompt that contains a relatively small context to include in the prompt?
We start by processing our large text corpus into small two or three sentence “chunks.” We combine BM25 (lexical search) and vector similarity (semantic search) into hybrid search that is used to identify, given a user’s question, a small number of chunks that we use to construct a one-shot prompt with this small tailored context and the user’s original question.
The example in this chapter uses the magicl matrix library, which is tested to run on SBCL and is available from Quicklisp — either install it explicitly with (ql:quickload :magicl), or let the system’s declared dependency pull it in. What magicl is, and why a matrix library earns its place in a text retrieval example, is explained in the section on embeddings below.
The purpose of this example is allowing the use of small models supporting small contexts and still take advantage of huge text datasets.
Implementing the BM25 Algorithm
The following Common Lisp code in bm25.lisp provides a complete self-contained implementation of the Okapi BM25 ranking function, a powerful algorithm widely used in information retrieval and search engine technology. Unlike simpler term frequency-inverse document frequency (TF-IDF) models, BM25 is a probabilistic model that scores the relevance of documents to a given search query by considering not only the frequency of query terms within a document but also the document’s length relative to the average length in the entire corpus. This allows it to penalize overly long documents and account for term saturation, where the relevance score doesn’t increase proportionally after a term appears a certain number of times. This implementation encapsulates the logic within a bm25-index class, which pre-calculates and stores necessary statistics about the text corpus such as document frequencies, document lengths, and the average document length, to enable efficient scoring and retrieval of the most relevant text chunks from documents for a user’s query. In our application we use this code to process individual small text chunks and not entire documents. I used Google Gemini command line tool gemini-cli to write and debug this code:
1 ;; bm25.lisp NOTE: Generated by Gemini 2.5 Pro in Research Mode and corrected by gemini-cli.
2 ;; A self-contained implementation of the BM25Okapi ranking function.
3
4 (defpackage #:bm25
5 (:use #:cl)
6 (:export #:bm25-index
7 #:make-bm25-index
8 #:get-top-n))
9
10 (in-package #:bm25)
11
12 (defclass bm25-index ()
13 ((doc-freqs :initarg :doc-freqs :reader doc-freqs)
14 (doc-lengths :initarg :doc-lengths :reader doc-lengths)
15 (avg-doc-length :initarg :avg-doc-length :reader avg-doc-length)
16 (corpus-size :initarg :corpus-size :reader corpus-size)
17 (corpus :initarg :corpus :reader corpus)
18 (k1 :initarg :k1 :reader k1)
19 (b :initarg :b :reader b)))
20
21 (defun make-bm25-index (tokenized-corpus &key (k1 1.5) (b 0.75))
22 "Creates and initializes a BM25 index from a corpus of tokenized documents."
23 (let* ((corpus-size (length tokenized-corpus))
24 (doc-lengths (mapcar #'length tokenized-corpus))
25 (avg-doc-length (/ (reduce #'+ doc-lengths) corpus-size))
26 (doc-freqs (make-hash-table :test 'equal)))
27 ;; Calculate document frequencies for each term
28 (dolist (doc tokenized-corpus)
29 (dolist (term (remove-duplicates doc :test #'string=))
30 (incf (gethash term doc-freqs 0))))
31 (make-instance 'bm25-index
32 :doc-freqs doc-freqs
33 :doc-lengths doc-lengths
34 :avg-doc-length avg-doc-length
35 :corpus-size corpus-size
36 :corpus tokenized-corpus
37 :k1 k1
38 :b b)))
39
40 (defmethod get-top-n ((index bm25-index) query-tokens n)
41 "Returns up to N (score . document-index) pairs for a given query,
42 sorted by descending score. The index is the document's position in
43 the tokenized corpus that built this index, which is also its position
44 in the caller's own chunk list. Returning the index (rather than the
45 token list) lets callers map a hit straight back to the original text
46 and deduplicate hits against other retrievers by identity."
47 (let ((scores (loop for i from 0 below (length (corpus index))
48 collect (cons (score-doc index query-tokens i) i))))
49 ;; Sort by score descending
50 (let* ((sorted-scores (sort scores #'> :key #'car))
51 (top-scores (subseq sorted-scores 0 (min n (length sorted-scores)))))
52 top-scores)))
53
54 ;;; Internal methods
55
56 (defmethod inverse-document-frequency ((index bm25-index) term)
57 "Calculates the IDF for a given term."
58 (let* ((doc-freq (gethash term (doc-freqs index) 0))
59 (corpus-size (corpus-size index)))
60 (log (/ (+ (- corpus-size doc-freq) 0.5) (+ doc-freq 0.5)) 10)))
61
62 (defmethod score-doc ((index bm25-index) query-tokens doc-index)
63 "Calculates the BM25 score for a single document."
64 (let* ((k1 (k1 index))
65 (b (b index))
66 (doc-length (nth doc-index (doc-lengths index)))
67 (doc (nth doc-index (corpus index)))
68 (avg-dl (avg-doc-length index))
69 (score 0.0))
70 (dolist (term query-tokens)
71 (let* ((term-freq (count term doc :test #'string=))
72 (idf (inverse-document-frequency index term)))
73 (incf score (* idf (/ (* term-freq (+ k1 1))
74 (+ term-freq (* k1 (+ (- 1 b) (* b (/ doc-length avg-dl))))))))))
75 score))
The constructor function, make-bm25-index, handles the heavy lifting of tokenizing the corpus, calculating the document frequency for every term, and determining the length of each document. This upfront processing makes subsequent searches very efficient. The primary public-facing function, get-top-n takes an index, a list of query tokens, and the desired number of results n. It scores every document against the query, sorts them in descending order of relevance, and returns the top n matching documents, providing a simple and effective interface for search.
The core of the relevance calculation is found in the score-doc method, that implements the BM25 formula. For each term in the user’s query, it calculates a score based on three main components: the inverse document frequency (IDF), the term’s frequency within the specific document, and the length of that document. The helper method inverse-document-frequency computes the IDF, which gives higher weight to rarer terms across the corpus. The main score-doc method combines this with a term frequency component that is tunable via the k1 and b parameters. The k1 parameter controls term frequency saturation, while b controls the degree to which document length normalizes the score, making this a flexible and powerful tool for document ranking.
Implementing Vectorization of Text and Semantic Similarity
We will take advantage of Python’s library support for deep learning and use a Python command line tool generate_embeddings.py to calculate vector embeddings for text input using a deep learning transformer model and returns the results in JSON format. Here is an example of calling this script manually:
1 echo "some text" | uv run generate_embeddings.py
This script will be called from our Common Lisp code.
For completeness, here is the Python script in full:
1 # generate_embeddings.py
2 #
3 # This script reads text lines from stdin, generates embeddings using
4 # sentence-transformers, and prints the result as a JSON array to stdout.
5 # This allows the Common Lisp program to easily get embeddings without
6 # needing a complex foreign function interface.
7 #
8 # Usage (from Lisp):
9 # echo "some text" | python3 generate_embeddings.py
10
11 import sys
12 import json
13 from sentence_transformers import SentenceTransformer
14
15 def main():
16 # Use a pre-trained model, which will be downloaded on first run.
17 model_name = 'all-MiniLM-L6-v2'
18 try:
19 model = SentenceTransformer(model_name)
20 except Exception as e:
21 print(f"Error loading SentenceTransformer model: {e}", file=sys.stderr)
22 sys.exit(1)
23
24 # Read all lines from standard input.
25 lines = [line.strip() for line in sys.stdin if line.strip()]
26
27 if not lines:
28 print("[]") # Output empty JSON array if no input
29 return
30
31 # Generate embeddings and normalize them for cosine similarity.
32 embeddings = model.encode(
33 lines,
34 normalize_embeddings=True,
35 show_progress_bar=False # Keep stdout clean for the Lisp process
36 )
37
38 # Convert the numpy array to a list of lists for JSON serialization.
39 embeddings_list = embeddings.tolist()
40
41 # Print the JSON output to stdout.
42 json.dump(embeddings_list, sys.stdout)
43
44 if __name__ == '__main__':
45 main()
The following function runs the Python script and is found in the file main.lisp that we discuss in the next section.
1 (defun generate-embeddings (text-list)
2 "Calls the Python script to generate embeddings for a list of strings.
3 The command is passed as a list of arguments (it never goes through a
4 shell), and :DIRECTORY pins uv's working directory so it finds both
5 pyproject.toml and generate_embeddings.py."
6 (let* ((input-string (format nil "~{~a~%~}" text-list))
7 (command (list "uv" "run" "generate_embeddings.py"))
8 (json-output (uiop:run-program command
9 :directory *system-directory*
10 :input (make-string-input-stream input-string)
11 :output :string)))
12 (let* ((parsed (yason:parse json-output))
13 (num-embeddings (length parsed))
14 (embedding-dim (if (> num-embeddings 0) (length (first parsed)) 0))
15 (flat-data (apply #'append parsed)))
16 (magicl:from-list flat-data (list num-embeddings embedding-dim) :type 'double-float))))
Why a matrix library?
magicl — Matrix Algebra proGrams In Common Lisp — is a CLOS-based linear algebra library, backed by BLAS/LAPACK where those are available. It earns its place here because the embedding step does not produce one vector; it produces a matrix, with one row per chunk and one column per embedding dimension.
The transformer we call, all-MiniLM-L6-v2, emits 384-dimensional vectors, so the 22 chunks of the sample corpus arrive from Python as a nested JSON array of 22 lists of 384 numbers. generate-embeddings flattens that nested list and hands it to magicl:from-list together with the (rows columns) shape, which builds a 22×384 double-float matrix in a single call. After that the rest of the code reads naturally: magicl:nrows tells us how many chunks we have, magicl:tref reaches a single element, and get-row-vector turns one row back into a vector so that magicl:dot and magicl:norm can compute a cosine similarity.
Two details are worth keeping in mind as you read the code:
- The Python script is called with
normalize_embeddings=True, so every row of the matrix is already scaled to unit length. Cosine similarity then reduces to the dot product, and the divisions by the two norms inside cosine-similarity are redundant — harmless, and they keep the function correct if you ever feed it un-normalized vectors. - Nothing here requires a matrix: for a corpus this size, plain lists of numbers would work. What the matrix buys is an explicit shape, and a single place to change if you later want BLAS-backed batch scoring, a different embedding dimension, or a corpus far larger than 22 chunks.
Implementation of Main Program
This listing shows main.lisp, the core of our hybrid Retrieval-Augmented Generation (RAG) system written in Common Lisp that also constructs a one-shot prompt for later input into a LLM.
The following diagram shows the high-level architecture of the AutoContext system developed in this chapter:
The auto-context class manages the entire lifecycle of loading, processing, and querying a local document collection. On instantiation, it scans a directory for text files, breaks them down into smaller, semantically coherent chunks, and then builds two parallel retrieval indices. The first is a classic BM25 sparse index for efficient keyword-based search, and the second is a dense index of vector embeddings for capturing semantic similarity. A key feature of this implementation is its pragmatic approach to interoperability; it generates embeddings by calling an external Python script that leverages the sentence-transformers library, communicating via standard I/O and JSON. The primary entry point for users is the get-prompt method, which takes a query, performs searches against both indices, merges the results into a unified context, and formats a complete prompt ready to be processed by a Large Language Model.
1 ;; main.lisp
2 ;; The core AutoContext class and application logic.
3
4 (defpackage #:autocontext
5 (:use #:cl #:bm25)
6 (:export #:auto-context
7 #:get-prompt
8 #:run-example
9 #:ask-with-context
10 #:ask-example
11 #:*chat-model*
12 #:*default-data-directory*))
13
14 (in-package #:autocontext)
15
16 (defclass auto-context ()
17 ((chunks :reader chunks :initarg :chunks :documentation "A list of original text chunks.")
18 (bm25-index :reader bm25-index :initarg :bm25-index :documentation "The BM25 sparse index.")
19 (chunk-embeddings :reader chunk-embeddings :initarg :chunk-embeddings :documentation "A magicl matrix of dense embeddings.")))
20
21 ;;; Configuration
22
23 (defparameter *system-directory*
24 (asdf:system-relative-pathname :autocontext "")
25 "Directory holding this system's source files. Resolving it through ASDF
26 means the Python helper is found no matter what the REPL's current
27 directory is.")
28
29 (defparameter *default-data-directory*
30 (asdf:system-relative-pathname :autocontext "../data/")
31 "Directory of sample .txt files used by RUN-EXAMPLE.")
32
33 ;;; Initialization
34
35 (defmethod initialize-instance :after ((ac auto-context) &key directory-path)
36 "Constructor for the auto-context class. Loads data and builds indices."
37 (format t "~&Initializing AutoContext from directory: ~a" directory-path)
38 (let ((chunks (load-and-chunk-documents directory-path)))
39 (when chunks
40 (format t "~&Building sparse and dense retrievers...")
41 (let* ((tokenized-chunks (mapcar #'tokenize chunks))
42 (bm25 (make-bm25-index tokenized-chunks))
43 (embeddings (generate-embeddings chunks)))
44 ;; Using shared-initialize to set the slots after creation
45 (shared-initialize ac t :chunks chunks
46 :bm25-index bm25
47 :chunk-embeddings embeddings))
48 (format t "~&Initialization complete. AutoContext is ready."))))
49
50 (defun tokenize (text)
51 "Simple whitespace tokenizer."
52 (split-sequence:split-sequence #\Space text :remove-empty-subseqs t))
53
54 (defun list-txt-files-in-directory (dir-path)
55 "Return a list of pathnames for *.txt files in the directory given by DIR-PATH (a string or pathname)."
56 (let* ((dir (uiop:ensure-directory-pathname
57 (if (pathnamep dir-path)
58 dir-path
59 (uiop:parse-native-namestring dir-path))))
60 ;; create a wildcard pathname for *.txt under that directory
61 (pattern (uiop:merge-pathnames*
62 (make-pathname :name :wild :type "txt")
63 dir)))
64 (directory pattern)))
65
66
67 (defun split-into-sentences (text)
68 "Splits text into a list of sentences based on punctuation. This is a heuristic approach and may not be perfect. It tries to avoid splitting on abbreviations like 'e.g.'."
69 (let ((sentences '())
70 (start 0))
71 (loop for i from 0 below (length text)
72 do (when (and (member (char text i) '(#\. #\? #\!))
73 (or (= (1+ i) (length text))
74 (member (char text (1+ i)) '(#\Space #\Newline #\"))))
75 (push (string-trim '(#\Space #\Newline) (subseq text start (1+ i))) sentences)
76 (setf start (1+ i))))
77 (let ((last-part (string-trim '(#\Space #\Newline) (subseq text start))))
78 (when (plusp (length last-part))
79 (push last-part sentences)))
80 (remove-if (lambda (s) (zerop (length s))) (nreverse sentences))))
81
82 (defun chunk-text (text &key (chunk-size 3))
83 "Splits text into sentences and then groups them into chunks of chunk-size sentences."
84 (let ((sentences (split-into-sentences text)))
85 (loop for i from 0 below (length sentences) by chunk-size
86 collect (let* ((end (min (+ i chunk-size) (length sentences)))
87 (sentence-group (subseq sentences i end)))
88 (string-trim '(#\Space #\Newline) (format nil "~{~a~^ ~}" sentence-group))))))
89
90
91 (defun load-and-chunk-documents (directory-path)
92 "Loads .txt files and splits them into chunks of a few sentences."
93 ;;(format t "&load-and-chunk-documents: directory-path=~A~%" directory-path)
94 (let* ((chunks (remove-if (lambda (s) (string= s ""))
95 (loop for file in (list-txt-files-in-directory directory-path)
96 nconcing (chunk-text (uiop:read-file-string file))))))
97 (format t "~&Loaded ~d text chunks." (length chunks))
98 ;;(format t "&load-and-chunk-documents: chunks=~A~%~%" chunks)
99 chunks));;; Embedding Generation (Interface to Python)
100
101 (defun generate-embeddings (text-list)
102 "Calls the Python script to generate embeddings for a list of strings.
103 The command is passed as a list of arguments (it never goes through a
104 shell), and :DIRECTORY pins uv's working directory so it finds both
105 pyproject.toml and generate_embeddings.py."
106 (let* ((input-string (format nil "~{~a~%~}" text-list))
107 (command (list "uv" "run" "generate_embeddings.py"))
108 (json-output (uiop:run-program command
109 :directory *system-directory*
110 :input (make-string-input-stream input-string)
111 :output :string)))
112 (let* ((parsed (yason:parse json-output))
113 (num-embeddings (length parsed))
114 (embedding-dim (if (> num-embeddings 0) (length (first parsed)) 0))
115 (flat-data (apply #'append parsed)))
116 (magicl:from-list flat-data (list num-embeddings embedding-dim) :type 'double-float))))
117
118 ;;; Core Method
119
120 (defun cosine-similarity (vec1 vec2)
121 "Calculates cosine similarity between two magicl vectors."
122 (/ (magicl:dot vec1 vec2)
123 (* (magicl:norm vec1) (magicl:norm vec2))))
124
125 (defun get-row-vector (matrix row-index)
126 "Extracts a row from a matrix and returns it as a magicl vector."
127 (let* ((num-cols (magicl:ncols matrix))
128 (row-elements (loop for col-index from 0 below num-cols
129 collect (magicl:tref matrix row-index col-index))))
130 (magicl:from-list row-elements (list num-cols) :type (magicl:element-type matrix))))
131
132 (defmethod get-prompt ((ac auto-context) query &key (num-results 5))
133 "Retrieves context and formats it into a prompt for an LLM."
134 (format t "~&--- Retrieving context for query: '~a' ---" query)
135
136 ;; 1. Sparse Search (BM25)
137 (let* ((query-tokens (tokenize query))
138 (bm25-hits (bm25:get-top-n (bm25-index ac) query-tokens num-results))
139 (bm25-indices (mapcar #'cdr bm25-hits)))
140 (format t "~&BM25 found ~d keyword-based results." (length bm25-indices))
141 ;;(format t "~%~%bm25-hits:~%~A~%~%" bm25-hits)
142
143 ;; 2. Dense Search (Vector Similarity)
144 (let* ((query-embedding-matrix (generate-embeddings (list query)))
145 (query-vector (get-row-vector query-embedding-matrix 0))
146 (all-embeddings (chunk-embeddings ac))
147 (similarities (loop for i from 0 below (magicl:nrows all-embeddings)
148 collect (cons (cosine-similarity query-vector
149 (get-row-vector all-embeddings i))
150 i)))
151 (sorted-sim (sort similarities #'> :key #'car))
152 (vector-indices (mapcar #'cdr (subseq sorted-sim 0 (min num-results (length sorted-sim))))))
153 (format t "~&Vector search found ~d semantic-based results." (length vector-indices))
154 ;;(format t "~%~%vector-indices:~%~A~%~%" vector-indices)
155
156 ;; 3. Combine and deduplicate. Both retrievers index the same chunk
157 ;; list, so a chunk found by both is a single index: deduplicating on
158 ;; the index merges the two hits, where comparing their *text* could
159 ;; not (BM25 used to hand back re-joined tokens, which never string=
160 ;; the original chunk). The surviving chunks are taken from CHUNKS,
161 ;; so every passage in the prompt is the original text.
162 (let* ((unique-indices (remove-duplicates
163 (append bm25-indices vector-indices)))
164 (unique-results (mapcar (lambda (i) (nth i (chunks ac)))
165 unique-indices)))
166 (format t "~&Combined and deduplicated, we have ~d context chunks." (length unique-results))
167
168 ;; 4. Format the final prompt. ~{~a~%---~%~} puts a separator after
169 ;; every chunk; the ~^ form used previously omitted it after the
170 ;; last one, welding that chunk onto the END CONTEXT marker.
171 (format nil "Based on the following context, please answer the question:~%~A~2%--- CONTEXT ---~%~{~a~%---~%~}--- END CONTEXT ---~2%Question: ~a~%Answer:"
172 query unique-results query)))))
173
174 ;;; Example Usage
175
176
177 (defun run-example ()
178 "A simple top-level function to demonstrate the system. Prints the
179 generated prompt and returns it, so callers can pass it to a model."
180 (let* ((ac (make-instance 'auto-context
181 :directory-path *default-data-directory*))
182 (query "who says that economics is bullshit?")
183 (prompt (get-prompt ac query :num-results 2)))
184 (format t "~&~%--- Generated Prompt for LLM ---~%~a~%" prompt)
185 prompt))
186
187 ;;; End-to-end: send the prompt to a model
188
189 (defparameter *chat-model* "ollama/qwen3.5:4b"
190 "Model used by ASK-WITH-CONTEXT and ASK-EXAMPLE, written as a litelm
191 \"provider/model\" string. The default runs locally through Ollama, so no
192 API key is needed; change the provider prefix to route elsewhere.")
193
194 (defun ask-with-context (ac question &key (model *chat-model*) (num-results 5))
195 "Build a context prompt for QUESTION with GET-PROMPT, send it to MODEL
196 through litelm, and return the model's answer as a string."
197 (let ((prompt (get-prompt ac question :num-results num-results)))
198 (format t "~&~%--- Sending ~d-character prompt to ~a ---~%"
199 (length prompt) model)
200 (litelm:response-content (litelm:completion model :messages prompt))))
201
202 (defun ask-example (&key (model *chat-model*))
203 "Run the demo end to end: build the sample corpus, generate a prompt for
204 the demo query, and have MODEL answer it. Needs a reachable model -- the
205 default is a local Ollama server. Returns the answer string."
206 (let* ((ac (make-instance 'auto-context
207 :directory-path *default-data-directory*))
208 (answer (ask-with-context ac
209 "who says that economics is bullshit?"
210 :model model
211 :num-results 2)))
212 (format t "~&~%--- Answer ---~%~a~%" answer)
213 answer))
This code demonstrates a powerful hybrid search technique. By leveraging both BM25 and vector similarity the retrieval process becomes more robust. BM25 is adept at finding chunks containing specific keywords or jargon present in the query, while the dense vector search excels at uncovering conceptually related content even when the exact wording differs. This dual approach mitigates the weaknesses of each individual method, leading to a more comprehensive and relevant context for a LLM. The auto-context class neatly encapsulates this complexity, holding the text chunks, the sparse index, and the dense embedding matrix as a single cohesive unit.
From an implementation perspective the most notable detail is the seamless integration with Python for embedding generation. The generate-embeddings function wisely avoids the complexity of using a foreign function interface or trying to reimplement a transformer model in Lisp. Instead, it uses the simple and reliable method of running a command-line script and parsing its JSON output, a practical pattern for leveraging the strengths of different language ecosystems. Internally, the use of the magicl library to represent the embeddings as a matrix is also significant, as it provides an efficient, specialized data structure for the numerical calculations required for computing cosine similarity between the query and document vectors.
Example Generated Prompt with Context
Here we run run-example from the previous listing. The transcript below is a real run against the sample corpus in src/data:
1 * (autocontext:run-example)
2 Initializing AutoContext from directory: /Users/markw/GITHUB/loving-common-lisp/src/data/
3 Loaded 22 text chunks.
4 Building sparse and dense retrievers...
5 Initialization complete. AutoContext is ready.
6 --- Retrieving context for query: 'who says that economics is bullshit?' ---
7 BM25 found 2 keyword-based results.
8 Vector search found 2 semantic-based results.
9 Combined and deduplicated, we have 3 context chunks.
10
11 --- Generated Prompt for LLM ---
12 Based on the following context, please answer the question:
13 who says that economics is bullshit?
14
15 --- CONTEXT ---
16 There exists an economic problem, subject to study by economic science, when a decision (choice) is made by one or more resource-controlling players to attain the best possible outcome under bounded rational conditions. In other words, resource-controlling agents maximize value subject to the constraints imposed by the information the agents have, their cognitive limitations, and the finite amount of time they have to make and execute a decision. Economic science centers on the activities of the economic agents that comprise society.[1] They are the focus of economic analysis.[2]
17
18 The traditional concern of economic analysis is to gain an understanding of the processes that govern the production, distribution and consumption of goods and services in an exchange economy.[3] An approach to understanding these processes, through the study of agent behavior under scarcity, may go as s:
19
20 An interesting Economist is Pauli Blendergast who teaches at the University of Krampton Ohio and is famouse for saying economics is bullshit.
21 ---
22 ; Looking better can help you feel better. Schedule a realistic day. ; Avoid the tendency to schedule back-to-back appointments; allow time between appointments for a breathing spell.
23 ---
24
25 which requires that you sit at a desk all day. ; If you hate to talk
26 politics, don't associate with people who love to talk politics, etc. Learn to live one day at a time.
27 ---
28 --- END CONTEXT ---
29
30 Question: who says that economics is bullshit?
31 Answer:
Besides printing the prompt, run-example returns it — a 1611-character string in this run — so a caller can pass it directly to a model instead of scraping it back out of the transcript.
Reading the retrieved context
Look closely at what was retrieved, because it is instructive. Only one passage — the one naming Pauli Blendergast — actually answers the question. The other two are health-advice excerpts with nothing to do with economics, and they were returned by both retrievers. Nothing in the code judges whether a match is actually good: get-prompt takes the top num-results chunks from each retriever and merges them, with no score threshold and no normalization between the two.
BM25 cannot help here. It always returns its best num-results chunks however low those scores are — there is no minimum-score cut-off — so when the query matches nothing well it still fills its quota. (Its IDF term can also go negative for a word appearing in more than half the corpus, which can push a relevant chunk below an irrelevant one.) The vector side has the mirror-image problem: cosine similarity always has a nearest neighbour, so it too always returns something, relevant or not.
Note also that num-results is applied to each retriever separately, so asking for 2 results can yield up to 4 chunks. Three survived here because the two retrievers agreed on one chunk, so the merge is not bounded by a single budget. Adding a score threshold, or re-scoring the merged candidates, is the subject of practice problems 2 and 3.
Merging the two retrievers
The merge step is worth pausing on, because the obvious implementation is wrong. Both retrievers index the same chunk list — tokenized-chunks is built with mapcar over chunks, so position 7 of the BM25 corpus is position 7 of chunks — which is why get-top-n returns (score . index) pairs rather than the matching text. With an index in hand, get-prompt looks the chunk up in chunks, so every passage in the prompt is the original text with its spacing intact, and it deduplicates the two retrievers’ hits by comparing indices.
Comparing text instead does not work, and it is worth seeing why. BM25 hands back a chunk’s tokens (it splits on spaces only, so newlines and tabs survive inside tokens), and re-joining them with (format nil "~{~a~^ ~}" tokens) normalizes spacing and turns newlines at token boundaries into single spaces. The same chunk therefore comes back from the two retrievers as two different strings, string= never matches, and the passage is admitted to the prompt twice — consuming a slot in the very budget this example exists to protect. Earlier drafts of this code did exactly that, and the count read “4 context chunks” instead of 3.
Normally you would combine the code developed in this chapter in an application using a small LLM running on your laptop using Ollama or LM Studio, or huge LLMs like Gemini-2.5-pro or GPT-5 via commercial APIs.
Here, I feed the generated prompt into a tiny 3B model running on Ollama for test purposes (most output not shown):
1 $ ollama run gemma3:latest
2 >>> Based on the following context, please answer the question:
3 ... who says that economics is bullshit?
4 ...
5 ... --- CONTEXT ---
6 ...
7 ... which requires that you sit at a desk all day. ; If you hate to talk
8 ... politics, don't associate with people who love to talk politics, etc. Learn to live one day at a time.
9 ... ---
10 ... An interesting Economist is Pauli Blendergast who teaches at the University of Krampton Ohio and is famouse fo
11 ... r saying economics is bullshit.
12 ... ---
13 ... ; Looking better can help you feel better. Schedule a realistic day. ; Avoid the tendency to schedule back-to-
14 ... back appointments; allow time between appointments for a breathing spell.
15 ... ---
16 ... END CONTEXT ---
17 ...
18 ... Question: who says that economics is bullshit?
19 ... Answer:
20 Pauli Blendergast
21
22 >>>
Answering the question
Everything so far builds a prompt: the pipeline stops at Answer: and leaves the last step to you, which is why the transcript above ends there. The system now also ships that last step. ask-with-context sends the generated prompt to a model through litelm and returns the answer, and ask-example is the whole demo end to end — calling (autocontext:ask-example) builds the sample corpus, retrieves context, and asks the model.
Against qwen3.5:4b running locally under Ollama, the answer comes back as:
1 --- Answer ---
2 Based on the provided context, **Pauli Blendergast** says that economics is bullshit. He is described as an interesting Economist who teaches at the University of Krampton Ohio.
The model is taken from *chat-model*, which defaults to "ollama/qwen3.5:4b" — a local model, so there is no API key and no per-token cost. Because the value is a litelm "provider/model" string, pointing the same code at a hosted model (Gemini, DeepSeek, Fireworks AI) is a one-line change, exactly as in the embeddings chapter.
Wrap Up For Generating Prompts with Contexts
Large context models like Gemini 2.5-pro have a context window of a million tokens so in principle many large documents can be used as-is for context to a prompt.
My motivation for writing this example code is my desire to mostly write applications using smaller LLMs running locally using Ollama or LM Studio. Often these small models only support a context size from 16K to 64K in size and they run slowly on my laptop when processing very long one-shot prompts.
I find that example we developed here allows me to use small models and still take advantage of huge text datasets for context.
Optional Practice Problems
BM25 Parameter Exploration: The
make-bm25-indexfunction accepts two tuning parameters:k1(term frequency saturation, default 1.5) andb(document length normalization, default 0.75). Write a functioncompare-bm25-paramsthat builds three BM25 indices from the same corpus using different(k1, b)pairs—for example(1.2, 0.5),(1.5, 0.75), and(2.0, 1.0)—and prints the top-3 results from each for the same query. Observe how the rankings change and discuss which settings favor shorter chunks versus longer ones.Chunking Strategy Comparison: The
chunk-textfunction groups sentences into chunks of 3 by default. Write a functionevaluate-chunk-sizesthat processes the same document directory with chunk sizes of 1, 3, and 5 sentences. For each chunk size, build a freshauto-contextinstance and callget-promptwith the same query. Compare the generated prompts: does a smaller chunk size produce more precise context, or does a larger chunk size provide better surrounding context? Print the number of total chunks and the final deduplicated context chunks for each setting.Weighted Hybrid Search: The current
get-promptmethod gives equal weight to BM25 and vector similarity results by simply appending and deduplicating them. Modify theget-promptmethod (or write a new methodget-weighted-prompt) that accepts keyword arguments:bm25-weightand:vector-weight(both defaulting to 0.5 and summing to 1.0). Score each chunk by its normalized rank position from each retriever, compute a weighted combined score, and return context chunks sorted by combined relevance instead of the current append-and-deduplicate approach.Embedding Cache: Each call to
generate-embeddingslaunches an external Python process, which is slow. Write a simple disk caching layer: a functioncached-generate-embeddingsthat hashes the input text list (e.g., usingsxhashon the concatenated strings), checks for a cached JSON file in a.cache/directory, and returns the cached matrix if it exists. Only call the Python script when the cache misses. Integrate this into theauto-contextinitialization to speed up repeated runs on the same corpus.End-to-End RAG with Gemini: The chapter stops at generating the prompt but does not send it to an LLM programmatically. Using the
geminilibrary from the previous chapter, write a functionask-with-contextthat takes anauto-contextinstance and a question string, callsget-promptto build the prompt, and then passes it togemini:generateto get an actual answer. Print both the constructed prompt and the Gemini response. As a bonus, usegemini:count-tokensto display the token count of the prompt before sending it.Add a TF-IDF Retriever: The chapter uses BM25 for sparse retrieval. As an exercise, implement a simple TF-IDF retriever class
tfidf-indexwith the same public interface asbm25-index(i.e., a constructor and aget-top-nmethod). TF-IDF scores each term as(term-frequency) * log(N / document-frequency)without the length normalization and saturation of BM25. Integrate your new retriever as a third retrieval source inget-promptalongside BM25 and vector search, and compare whether the three-way hybrid produces more relevant context than the original two-way hybrid.