Retrieval Augmented Generation of Text Using Embeddings

Retrieval-Augmented Generation (RAG) is a framework that combines the strengths of pre-trained language models (LLMs) with retrievers. Retrievers are system components for accessing knowledge from external sources of text data. In RAG a retriever selects relevant documents or passages from a corpus, and a generator produces a response based on both the retrieved information and the input query. The process typically follows these steps that we will use in the example Racket code:

Query Encoding: The input query is encoded into a vector representation.
Document Retrieval: A retriever system uses the query representation to fetch relevant documents or passages from an external corpus.
Document Encoding: The retrieved documents are encoded into vector representations.
Joint Encoding: The query and document representations are combined, often concatenated or mixed via attention mechanisms.
Generation: A generator, usually LLM, is used to produce a response based on the joint representation.

RAG enables the LLM to access and leverage external text data sources, which is crucial for tasks that require information beyond what the LLM has been trained on. It’s a blend of retrieval-based and generation-based approaches, aimed at boosting the factual accuracy and informativeness of generated responses.

Example Implementation

In the following short Racket example program (file Racket-AI-book/source-code/embeddingsdb /embeddingsdb.rkt) I implement some ideas of a RAG architecture. At file load time the text files in the subdirectory data are read, split into “chunks”, and each chunk along with its parent file name and OpenAI text embedding is stored in a local SQLite database. When a user enters a query, the OpenAI embedding is calculated, and this embedding is matched against the embeddings of all chunks using the dot product of two 1536 element embedding vectors. The “best” chunks are concatenated together and this “context” text is passed to GPT-4 along with the user’s original query. Here I describe the code in more detail:

The provided Racket code uses a local SQLite database and OpenAI’s APIs for calculating text embeddings and for text completions.

Utility Functions:

floats->string and string->floats are utility functions for converting between a list of floats and its string representation.
read-file reads a file’s content.
join-strings joins a list of strings with a specified separator.
truncate-string truncates a string to a specified length.
interleave merges two lists by interleaving their elements.
break-into-chunks breaks a text into chunks of a specified size.
string-to-list and decode-row are utility functions for parsing and processing database rows.

Database Setup:

Database connection is established to “test.db” and a table named “documents” is created with columns for document_path, content, and embedding.

Document Management:

insert-document inserts a document and its associated information into the database.
get-document-by-document-path and all-documents are utility functions for querying documents from the database.
create-document reads a document from a file path, breaks it into chunks, computes embeddings for each chunk via a function embeddings-openai, and inserts these into the database.

Semantic Matching and Interaction:

execute-to-list and dot-product are utility functions for database queries and vector operations.
semantic-match performs a semantic search by calculating the dot product of embeddings of the query and documents in the database. It then aggregates contexts of documents with a similarity score above a certain threshold, and sends a new query constructed with these contexts to OpenAI for further processing.
QA is a wrapper around semantic-match for querying.
CHAT initiates a loop for user interaction where each user input is processed through semantic-match to generate a response, maintaining a context of the previous chat.

Test Code:

test function creates documents by reading from specified file paths, and performs some queries using the QA function.

The code uses a local SQLite database to store and manage document embeddings and the OpenAI API for generating embeddings and performing semantic searches based on user queries. Two functions are exported in case you want to use this example as a library: create-document and QA.

  1 #lang racket
  2 
  3 (require db)
  4 (require llmapis)
  5 (require racket/runtime-path)
  6 
  7 (provide create-document QA CHAT semantic-match)
  8 
  9 ; Function to convert list of floats to string representation
 10 (define (floats->string floats)
 11   (string-join (map number->string floats) " "))
 12 
 13 ; Function to convert string representation back to list of floats
 14 (define (string->floats str)
 15   (map string->number (string-split str)))
 16 
 17 
 18 (define (read-file infile)
 19   (with-input-from-file infile
 20     (lambda ()
 21       (let ((contents (read)))
 22         contents))))
 23 
 24 (define (join-strings separator list)
 25   (string-join list separator))
 26 
 27 (define (truncate-string string length)
 28   (substring string 0 (min length (string-length string))))
 29 
 30 (define (interleave list1 list2)
 31   (if (or (null? list1) (null? list2))
 32       (append list1 list2)
 33       (cons (car list1)
 34             (cons (car list2)
 35                   (interleave (cdr list1) (cdr list2))))))
 36 
 37 (define (break-into-chunks text chunk-size)
 38   (let loop ((start 0) (chunks '()))
 39     (if (>= start (string-length text))
 40         (reverse chunks)
 41         (loop (+ start chunk-size)
 42               (cons (substring text start (min (+ start chunk-size) (string-length text))) chunks)))))
 43 
 44 (define (string-to-list str)
 45   (map string->number (string-split str)))
 46 
 47 (define (decode-row row)
 48   (let ((id (vector-ref row 0))
 49         (context (vector-ref row 1))
 50         (embedding (string-to-list (vector-ref row 2))))
 51     (list id context embedding)))
 52 
 53 (define db (sqlite3-connect #:database "test.db" #:mode 'create #:use-place #t))
 54 
 55 (with-handlers ([exn:fail? (lambda (ex) (void))])
 56   (query-exec
 57    db
 58    "CREATE TABLE documents (document_path TEXT, content TEXT, embedding TEXT);"))
 59       
 60 ;; ... database setup, error handling, and queries ...
 61 
 62 (define (insert-document document-path content embedding)
 63   (printf "~%insert-document:~%  content:~a~%~%" content)
 64   (query-exec
 65    db
 66    "INSERT INTO documents (document_path, content, embedding) VALUES (?, ?, ?);"
 67    document-path content (floats->string embedding)))
 68 
 69 (define (get-document-by-document-path document-path)
 70   (map decode-row
 71        (query-rows db
 72                     "SELECT * FROM documents WHERE document_path = ?;"
 73                     document-path)))
 74 
 75 (define (all-documents)
 76   (map
 77    decode-row
 78    (query-rows
 79     db
 80     "SELECT * FROM documents;")))
 81    
 82 ;; ... remaining database query functions ...
 83 
 84 (define (create-document fpath)
 85   (let ((contents (break-into-chunks (file->string fpath) 200)))
 86     (for-each
 87      (lambda (content)
 88        (with-handlers ([exn:fail? (lambda (ex) (void))])
 89          (let ((embedding (embeddings-openai content)))
 90            (insert-document fpath content embedding))))
 91      contents)))
 92 
 93 
 94 ;; Assuming a function to fetch documents from database
 95 (define (execute-to-list db query)
 96   (query-rows db query))
 97 
 98 (define (dot-product a b) ;; dot product of two lists of floating point numbers
 99   (for/sum ([x a] [y b])
100     (* x y)))
101 
102 
103 (define (semantic-match query custom-context [cutoff 0.7])
104   (let ((emb (embeddings-openai query))
105         (ret '()))
106     (for-each
107      (lambda (doc)
108        (let* ((context (second doc))
109               (embedding (third doc))
110               (score (dot-product emb embedding)))
111          (when (> score cutoff)
112            (set! ret (cons context ret)))))
113      (all-documents))
114     (printf "~%semantic-search: ret=~a~%" ret)
115     (let* ((context (string-join (reverse ret) " . "))
116            (query-with-context (string-join (list context custom-context "Question:" query) " ")))
117       (question-openai query-with-context))))
118 
119 (define (QA query [quiet #f])
120   (let ((answer (semantic-match query "")))
121     (unless quiet
122       (printf "~%~%** query: ~a~%** answer: ~a~%~%" query answer))
123     answer))
124 
125 (define (CHAT)
126   (let ((messages '(""))
127         (responses '("")))
128     (let loop ()
129       (printf "~%Enter chat (STOP or empty line to stop) >> ")
130       (let ((string (read-line)))
131         (cond
132          ((or (string=? string "STOP") (< (string-length string) 1))
133           (list (reverse messages) (reverse responses)))
134          (else
135           (let* ((custom-context
136                   (string-append
137                    "PREVIOUS CHAT: "
138                    (string-join (reverse messages) " ")))
139                  (response (semantic-match string custom-context)))
140             (set! messages (cons string messages))
141             (set! responses (cons response responses))
142             (printf "~%Response: ~a~%" response)
143             (loop))))))))
144 
145 (define-runtime-path data-dir "data")
146 
147 (define (test)
148   "Test code for Semantic Document Search Using OpenAI GPT APIs and local vector database"
149   (create-document (path->string (simplify-path (build-path data-dir "sports.txt"))))
150   (create-document (path->string (simplify-path (build-path data-dir "chemistry.txt"))))
151   (QA "What is the history of the science of chemistry?")
152   (QA "What are the advantages of engaging in sports?"))
153 
154 (module+ main
155   ;; Uncomment below if you want to execute tests when running this module
156   ;; (test)
157   )

Let’s look at a few examples form a Racket REPL:

 1 > (QA "What is the history of the science of chemistry?")
 2 ** query: What is the history of the science of chemistry?
 3 ** answer: The history of the science of chemistry dates back thousands of years. Ancient civilizations such as the Egyptians, Greeks, and Chinese were experimenting with various substances and observing chemical reactions even before the term "chemistry" was coined.
 4 
 5 The foundations of modern chemistry can be traced back to the works of famous scholars such as alchemists in the Middle Ages. Alchemists sought to transform common metals into gold and discover elixirs of eternal life. Although their practices were often based on mysticism and folklore, it laid the groundwork for the understanding of chemical processes and experimentation.
 6 
 7 In the 17th and 18th centuries, significant advancements were made in the field of chemistry. Prominent figures like Robert Boyle and Antoine Lavoisier began to understand the fundamental principles of chemical reactions and the concept of elements. Lavoisier is often referred to as the "father of modern chemistry" for his work in establishing the law of conservation of mass and naming and categorizing elements.
 8 
 9 Throughout the 19th and 20th centuries, chemistry continued to progress rapidly. The development of the periodic table by Dmitri Mendeleev in 1869 revolutionized the organization of elements. The discovery of new elements, the formulation of atomic theory, and the understanding of chemical bonding further expanded our knowledge.
10 
11 Chemistry also played a crucial role in various industries and technologies, such as the development of synthetic dyes, pharmaceuticals, plastics, and materials. The emergence of quantum mechanics and spectroscopy in the early 20th century opened up new avenues for understanding the behavior of atoms and molecules.
12 
13 Today, chemistry is an interdisciplinary science that encompasses various fields such as organic chemistry, inorganic chemistry, physical chemistry, analytical chemistry, and biochemistry. It continues to evolve and make significant contributions to society, from developing sustainable materials to understanding biological processes and addressing global challenges such as climate change.
14 
15 In summary, the history of the science of chemistry spans centuries, starting from ancient civilizations to the present day, with numerous discoveries and advancements shaping our understanding of the composition, properties, and transformations of matter.

This output is the combination of data found in the text files in the directory Racket-AI-book/source-code/embeddingsdb/data and the data that OpenAI GPT-4 was trained on. Since the local “document” file chemistry.txt is very short, most of this output is derived from the innate knowledge GPT-4 has from its training data.

In order to show that this example is also using data in the local “document” text files, I manually edited the file data/chemistry.txt adding the following made-up organic compound:

1 ZorroOnian Alcohol is another organic compound with the formula C 6 H 10 O.

GPT-4 was never trained on my made-up data so it has no idea what the non-existent compound ZorroOnian Alcohol is. The following answer is retrieved via RAG from the local document data (for brevity, most of the output for adding the local document files to the embedding index is not shown):

 1 > (create-document
 2    "/Users/markw/GITHUB/Racket-AI-book/source-code/embeddingsdb/data/chemistry.txt")
 3 
 4 insert-document:
 5   content:Amyl alcohol is an organic compound with the formula C 5 H 12 O. ZorroOnian Alcohol is another organic compound with the formula C 6 H 10 O. All eight isomers of amyl alcohol are known.
 6 
 7   ...
 8 
 9 > (QA "what is the formula for ZorroOnian Alcohol")
10 
11 ** query: what is the formula for ZorroOnian Alcohol
12 ** answer: The formula for ZorroOnian Alcohol is C6H10O.

There is also a chat interface:

1 Enter chat (STOP or empty line to stop) >> who is the chemist Robert Boyle
2 
3 Response: Robert Boyle was an Irish chemist and physicist who is known as one of the pioneers of modern chemistry. He is famous for Boyle's Law, which describes the inverse relationship between the pressure and volume of a gas, and for his experiments on the properties of gases. He lived from 1627 to 1691.
4 
5 Enter chat (STOP or empty line to stop) >> Where was he born?
6 
7 Response: Robert Boyle was born in Lismore Castle, County Waterford, Ireland.
8 
9 Enter chat (STOP or empty line to stop) >>

Notice how the second question, “Where was he born?”, contains no name at all. The CHAT loop prepends the previous turns as custom context (“PREVIOUS CHAT: …”), so the model sees the word “he” against the earlier mention of Robert Boyle. This is the whole trick behind chat-over-documents systems: the retrieval machinery stays the same, and conversation history is just more context.

What an Embedding Actually Is

The code above treats embeddings as opaque lists of 1536 floats. It is worth building an intuition for what those numbers are, because every design decision in a RAG system follows from it.

A text embedding model is a neural network trained so that distance in its output space corresponds to meaning. Two sentences about the same topic get vectors that point in nearly the same direction, even if they share no words (“My dog ate my homework” and “The puppy chewed up the assignment”). Two sentences that share many words but are unrelated (“The bank of the river” and “The bank approved the loan”) get vectors that point far apart. The vector’s direction encodes what the text is about; its length is typically normalized to 1.

Concretely, think of each of the 1536 dimensions as a soft answer to a learned question the model found useful during training: some dimensions fire for sports text, some for chemistry, some for grammar structure, and the rest have no human-readable interpretation at all. We never pick the questions; training does.

That geometric view explains two things.

First, why we compare vectors at all: if the model pushes related texts to nearby points in space, then finding the chunks closest to the query’s point is finding the most relevant chunks, with no keyword matching anywhere in the loop. This is why a query “the formula for ZorroOnian Alcohol” can find a chunk containing the words “formula” and “ZorroOnian” even though no synonym list connects them.

Second, why the raw dot-product in embeddingsdb.rkt is subtly fragile. For two vectors a and b with angle Code Test between them:

The dot product rewards long vectors as much as aligned directions. If your embedding model does not promise unit-length vectors, a long but tangentially related chunk can outscore a short, perfectly on-topic one. text-embedding-ada-002 returns unit vectors, so for that model Code Test and the dot product equals cosine similarity. But that is a property of one model, not of embeddings in general. As soon as you swap in a local embedding model, use cosine similarity:

The new file rag_extensions.rkt in the embeddingsdb directory implements this and the other upgrades in this chapter, all runnable without an API key. First, the similarity functions:

 1 (define (magnitude v)
 2   (sqrt (for/sum ([x v]) (* x x))))
 3 
 4 (define (cosine-similarity a b)
 5   (let ([ma (magnitude a)]
 6         [mb (magnitude b)])
 7     (if (or (zero? ma) (zero? mb))
 8         0.0
 9         (/ (for/sum ([x a] [y b]) (* x y))
10            (* ma mb)))))

The zero check matters more than it looks: a chunk of text that produces an all-zero vector (it happens, for example with some models on empty or whitespace-only input) would otherwise crash the division in the middle of an indexing run. Return 0.0 instead, and that chunk simply never matches anything.

Chunking: the Most Underrated Part of RAG

Look again at how create-document in embeddingsdb.rkt prepares text for the index: break-into-chunks splits the file every 200 characters. Here is what that does to a normal paragraph, compared with sentence-aware chunking. The output below is real, produced by the code in this section:

1 old break-into-chunks style, every 60 chars:
2   [Robert Boyle was born in Ireland in 1627. He studied the beh]
3   [avior of gases under pressure. His law states that pressure ]
4   [and volume are inversely related. He also wrote The Sceptica]
5   [l Chymist, a founding text of modern chemistry. He died in L]
6   [ondon in 1691.]

Every chunk except the first starts mid-word and mid-thought. Each chunk gets embedded separately, so the vector for "avior of gases under pressure. His law states that pressure " describes a sentence that nobody ever wrote. Retrieval still half-works, because the surrounding words provide signal, but you have handed the embedding model garbage on every boundary.

The chunker in rag_extensions.rkt splits on sentence boundaries, packs whole sentences into chunks up to a target size, and then prepends the tail of the previous chunk to each new one. Overlap exists because an answer can sit right at a boundary: if the sentence “His law states that pressure and volume are inversely related” got split from the sentence that introduced “he” as Boyle, neither chunk alone answers “who was Robert Boyle.” The overlap carries that context across:

1 chunked (60 chars, 25 overlap):
2   [Robert Boyle was born in Ireland in 1627.]
3   [born in Ireland in 1627. He studied the behavior of gases under pressure.]
4   [of gases under pressure. His law states that pressure and volume are inversely related.]
5   [me are inversely related. He also wrote The Sceptical Chymist, a founding text of modern chemistry.]
6   [text of modern chemistry. He died in London in 1691.]

Chunk 2 repeats the tail of chunk 1, so the query “where was Boyle born” and the query “what did he study” each find a chunk containing both the setup and the payoff. The overlap is cheap: it duplicates a few dozen characters of storage in exchange for covering the boundary cases.

Here is the code:

 1 (define sentence-end (pregexp "[.!?]+[\"'\\)]*\\s+"))
 2 
 3 (define (split-sentences text)
 4   (let loop ([rest (string-trim text)] [acc '()])
 5     (if (zero? (string-length rest))
 6         (reverse acc)
 7         (let ([m (regexp-match-positions sentence-end rest)])
 8           (if (not m)
 9               (reverse (cons rest acc))
10               (let* ([end (cdar m)]
11                      [sentence (string-trim (substring rest 0 end))])
12                 (loop (string-trim (substring rest end))
13                       (cons sentence acc))))))))
14 
15 (define (chunk-by-sentences text
16                             #:chunk-size [chunk-size 500]
17                             #:overlap [overlap 40])
18   (define sentences (split-sentences text))
19   (define (with-overlap chunk prev-chunk)
20     (if (and prev-chunk (> overlap 0))
21         (let ([tail (substring prev-chunk
22                                (max 0 (- (string-length prev-chunk) overlap)))])
23           (string-trim (string-append tail " " chunk)))
24         chunk))
25   ;; First pass: pack whole sentences into raw chunks of at most
26   ;; CHUNK-SIZE characters (a sentence longer than CHUNK-SIZE stands alone).
27   (define raw-chunks
28     (let loop ([todo sentences] [current ""] [chunks '()])
29       (cond
30         [(null? todo)
31          (if (zero? (string-length current))
32              (reverse chunks)
33              (reverse (cons current chunks)))]
34         [else
35          (define sentence (car todo))
36          (define candidate
37            (if (zero? (string-length current))
38                sentence
39                (string-append current " " sentence)))
40          (cond
41            [(not (> (string-length candidate) chunk-size))
42             (loop (cdr todo) candidate chunks)]
43            [(zero? (string-length current))
44             ;; Single oversized sentence gets its own chunk.
45             (loop (cdr todo) "" (cons sentence chunks))]
46            [else
47             (loop todo "" (cons current chunks))])])))
48   ;; Second pass: prepend the trailing OVERLAP characters of each chunk to
49   ;; the next, so context survives chunk boundaries.
50   (if (null? raw-chunks)
51       '()
52       (cons (car raw-chunks)
53             (for/list ([prev raw-chunks] [cur (cdr raw-chunks)])
54               (with-overlap cur prev)))))

The structure is deliberately simple: one pass packs sentences, a second pass adds overlap. The edge case to handle is a single sentence longer than the target chunk size, which gets its own chunk rather than looping forever. (A production system would split such a sentence at a clause boundary or fall back to character splitting; for this chapter, keeping it whole is honest.)

How do you pick chunk-size and overlap? Smaller chunks retrieve more precisely (each vector is about one thing) but carry less context; larger chunks are more forgiving but blur several topics into one vector. Around 300 to 1000 characters with a 10 to 20 percent overlap is a good default for prose. The only real rule: treat chunking as an experiment you rerun against your own documents, not a constant you set once.

Retrieval, Without Thresholds

semantic-match filters with a hard similarity cutoff of 0.7. Any chunk scoring above it goes into the prompt, everything else vanishes. That means one bad day can return zero context (and the LLM answers from training alone), while a query where twenty chunks all score 0.71 drowns the model in context and blows the token budget.

Ranking instead of thresholding fixes both: always take the top k chunks, whatever their scores. The generalization in rag_extensions.rkt also lets the caller swap in any embedding function with the #:embed keyword, so the same code ranks with OpenAI embeddings in production and with the deterministic local embedder (below) in tests:

 1 (define (rank-chunks query chunks
 2                      #:embed [embed hash-embed]
 3                      #:top-k [top-k (length chunks)])
 4   "Rank CHUNKS (list of strings) against QUERY by embedding similarity.
 5    Returns a list of (score . chunk) pairs, best first, at most TOP-K."
 6   (define q-emb (embed query))
 7   (define scored
 8     (for/list ([chunk chunks])
 9       (cons (cosine-similarity q-emb (embed chunk)) chunk)))
10   (take (sort scored > #:key car) (min top-k (length scored))))
11 
12 (define (assemble-prompt contexts custom-context query)
13   "Build the exact string sent to the LLM: retrieved context, any extra
14    context the caller supplies, then the question."
15   (string-join (list (string-join contexts " . ")
16                      custom-context
17                      "Question:" query)
18                " "))

Seeing the Whole Pipeline Offline

The problem with developing a RAG system is that every experiment costs API calls. The last piece of rag_extensions.rkt is a tiny deterministic embedder that lets you run and test the entire pipeline (chunk, embed, rank, assemble) offline. It is a stand-in with the same shape as a real embedder: text in, unit-length vector of floats out. It hashes each word into one of 256 buckets and normalizes, so texts that share words score well together. Never use it in production; do use it to test your plumbing:

 1 (define vocab-dim 256)
 2 
 3 (define (tokenize text)
 4   (regexp-split #px"[^a-z0-9]+" (string-downcase text)))
 5 
 6 (define (hash-embed text)
 7   "Deterministic unit-length embedding of TEXT as a list of VOCAB-DIM floats."
 8   (define v (make-vector vocab-dim 0.0))
 9   (for ([tok (tokenize text)])
10     (when (> (string-length tok) 0)
11       (define h (modulo (equal-hash-code tok) vocab-dim))
12       (vector-set! v h (+ 1.0 (vector-ref v h)))))
13   (define m (magnitude (vector->list v)))
14   (if (zero? m)
15       (vector->list v)
16       (map (lambda (x) (/ x m)) (vector->list v))))

The demo at the bottom of the file runs three queries against four miniature documents with a top-k of 2. Real output:

 1 $ racket rag_extensions.rkt
 2 
 3 == Retrieval over 4 tiny documents (hash embedder) ==
 4 
 5 Query: what is the formula for ZorroOnian Alcohol?
 6   score 0.428  Amyl alcohol is an organic compound with the formula C 5 H 12 O. Zorro...
 7   score 0.3904  Robert Boyle is known as one of the pioneers of modern chemistry. He i...
 8 
 9 Query: who is Robert Boyle?
10   score 0.4743  Robert Boyle is known as one of the pioneers of modern chemistry. He i...
11   score 0.1849  Amyl alcohol is an organic compound with the formula C 5 H 12 O. Zorro...
12 
13 Query: tell me about team sports and exercise
14   score 0.3571  Playing sports improves cardiovascular health, builds muscle, and teac...
15   score 0.0845  Dmitri Mendeleev published the periodic table in 1869, organizing the ...
16 
17 == Assembled RAG prompt for the top match ==
18 
19 Robert Boyle is known as one of the pioneers of modern chemistry. He is famous for Boyle's Law, which describes the inverse relationship between the pressure and volume of a gas.  Question: who is Robert Boyle?

Each query’s best match lands on the right document, and the scores show the useful failure mode too: the sports query’s runner-up scores 0.0845, far below the winner. The score gap between first and second place is itself signal worth watching when you tune a RAG system. A query whose top two scores are close and low usually means the answer is not in your documents at all.

The last block shows the honest output of assemble-prompt: retrieved context, a blank custom context, then the question. The word “Question:” is not decoration. It gives the LLM a stable separator between evidence to read and the task to do, and prompts that keep that structure consistent across every call get more consistent answers.

Testing the Pipeline

tests.rkt in the same directory runs the whole stack with rackunit, still offline. A few of the properties it pins down:

 1 (test-case "cosine never throws on a zero vector"
 2   (check-equal? (cosine-similarity '(0 0 0) '(1 2 3)) 0.0))
 3 
 4 (test-case "chunk-by-sentences keeps whole sentences"
 5   (define text
 6     "Alpha beta gamma delta. Epsilon zeta eta theta. Iota kappa lambda mu.")
 7   (define chunks (chunk-by-sentences text #:chunk-size 30 #:overlap 0))
 8   (check-true (> (length chunks) 1))
 9   ;; no chunk ends mid-word
10   (for ([c chunks])
11     (check-true (regexp-match? #px"[.!?]\\s*$" (string-trim c)))))
12 
13 (test-case "hash-embed distinguishes related and unrelated text"
14   (define chem (hash-embed "chemistry atoms molecules elements"))
15   (define chem2 (hash-embed "the chemistry of molecules and atoms"))
16   (define sports (hash-embed "sports players teams and goals"))
17   (check-true (> (cosine-similarity chem chem2)
18                  (cosine-similarity chem sports))))
19 
20 (test-case "rank-chunks orders best first and honors top-k"
21   (define test-docs
22     '("The periodic table organizes elements by atomic weight."
23       "Boyle's Law relates gas pressure and volume."
24       "Sports improve cardiovascular health."))
25   (define ranked (rank-chunks "gas pressure law" test-docs #:top-k 2))
26   (check-equal? (length ranked) 2)
27   (check-true (> (car (first ranked)) (car (second ranked))))
28   (check-true (string-contains? (cdr (first ranked)) "Boyle")))

1 $ raco test tests.rkt
2 raco test: "tests.rkt"
3 
4 All tests passed.
5 11 tests passed

Production Notes

A few hard-won notes if you carry this design further:

Embedding calls dominate indexing cost. create-document calls the embedding API once per chunk, and re-running test re-inserts every document. Cache by file content (a hash) and skip unchanged files.
The vector column is a TEXT string. floats->string stores 1536 numbers as space-separated text, and every query parses all of them back with string->floats. For a few thousand chunks that is fine. Beyond that, keep vectors as blobs or move to a database with a vector index.
Linear scans do not scale. Every query walks every row. Approximate nearest neighbor indexes (HNSW is the usual choice in SQLite-adjacent tooling like sqlite-vec) turn the scan into a near-constant lookup.
Duplicates pollute prompts. If two documents repeat the same paragraph, both copies can rank in the top k and waste context. Deduplicate by chunk text before indexing, and drop near-duplicate retrieved chunks before assembling the prompt.

The following diagram shows the high-level architecture of the RAG pipeline developed in this chapter:

Retrieval Augmented Generation Wrap Up

Retrieval Augmented Generation (RAG) is one of the best use cases for semantic search. Another way to write RAG applications is to use a web search API to get context text for a query, and add this context data to whatever context data you have in a local embeddings data store.

Optional Practice Problems

Wire the Upgrades In: Modify create-document in embeddingsdb.rkt to use chunk-by-sentences instead of break-into-chunks, and semantic-match to use cosine-similarity and top-k ranking instead of dot-product with a cutoff. Run the chemistry and sports queries before and after and compare answers.
Measure Chunking: Write a utility that reports, for a given document and chunker settings, the number of chunks, their min/max/mean lengths, and how many chunks end mid-word. Use it to compare the old and new chunkers on data/chemistry.txt.
Hybrid Retrieval: Some questions are pure keyword (“what is the formula for ZorroOnian Alcohol”) and some are conceptual (“why is teamwork valuable”). Add a keyword score (a simple word-overlap count) alongside the embedding score, and combine them with a weight. Show a query where hybrid beats embeddings alone, and one where it does not.
Extend Database Operations: Add a function delete-document that deletes all chunks and vector representations associated with a given file path from the SQLite database, and a test that proves re-adding a modified file does not leave stale chunks behind.
Score-Gap Detection: Add a check to rank-chunks callers that detects “no good match”: top score below a floor and a small gap between first and second place. When detected, have the prompt tell the LLM explicitly that the local documents may not contain the answer, and observe how its answers change.

A Racket Coding Agent

Up next

Agentic RAG Using the Gemini API