Agentic RAG Using the Gemini API
Standard RAG systems, like the one we built in the earlier chapter on embeddings, do one thing: embed a question, retrieve a few text chunks, and hand them to an LLM to write an answer. That is fine for simple questions, but it falls apart when a question needs facts from several documents, or when the first retrieval misses part of the answer. In this chapter we build an agentic RAG system in Racket where four specialized agents cooperate: a query rewriter, a search fanout, a sufficiency checker, and an answer synthesizer. The design follows Google’s research on agentic RAG.
The key idea is a feedback loop. Instead of retrieving once and hoping for the best, the system asks “do I have enough context to answer this question?” If not, it rewrites the question, searches again, and only then writes the answer. We implement this loop with a configurable iteration bound so cost stays predictable.
Architecture
The pipeline has four agents, each with a single responsibility:
1 User Query
2 |
3 v
4 +------------------+
5 | Query Rewriter | Decomposes complex questions into
6 | Agent | 1-3 focused sub-queries
7 +---------+--------+
8 |
9 v
10 +------------------+
11 | Search Fanout | Embeds each sub-query, searches
12 | Agent | across multiple corpora
13 +---------+--------+
14 |
15 v
16 +------------------+ +-----------------+
17 | Sufficient Context|---->| Refine Queries |
18 | Agent | NO | (iterate) |
19 +---------+--------+ +---------+--------+
20 | YES |
21 v |
22 +------------------+ |
23 | Synthesis Agent |<----------+
24 | (final answer) |
25 +------------------+
The critical piece is the Sufficient Context Agent. After retrieval it scores the chunks against the original question and decides whether they cover every part of it. If they do, we synthesize; if not, it tells us what is missing, and a second rewriter produces follow-up queries. This is the difference between a search engine and a system that answers questions.
The code is in the directory Racket-AI-book/source-code/RAG and is split into four modules plus tests:
| File | Purpose |
|---|---|
embeddings.rkt |
Gemini embedding API, batching, caching, retry, vector math |
vector-store.rkt |
In-memory store: chunking, save/load, cosine similarity search |
agents.rkt |
The four agents and the orchestrating pipeline |
main.rkt |
Public API, the test demo, and an interactive REPL |
tests.rkt |
15 offline unit tests (no network needed) |
Before we walk through the code, a note on configuration. The system uses one environment variable, GOOGLE_API_KEY, and nothing else. It calls gemini-embedding-001 for embeddings and gemini-3-flash-preview for all LLM calls.
Embeddings and Vector Math
The embeddings.rkt module handles all contact with the Gemini embedding API. Documents are split into chunks of about 500 characters with 50 characters of overlap, each chunk is embedded once, and the vectors are kept in a hash table so repeated runs do not re-embed text. The cache is bounded by *embedding-cache-cap* and clears itself when full; this is deliberately simple, and a production system would use a proper LRU cache.
The public entry points are get-embedding for one string and get-embeddings for a list. The list version is the one that matters for speed: all cache misses are sent in batchEmbedContents requests of at most 100 texts each, which is the API limit. This is why the demo below takes seconds instead of minutes.
1 (define (get-embeddings texts)
2 (when (eq? (*embedding-fn*) %fetch-embedding)
3 (define misses
4 (remove-duplicates
5 (filter (lambda (text)
6 (not (hash-has-key? (*embedding-cache*) (embedding-cache-key text))))
7 texts)))
8 (unless (null? misses)
9 (for ([text misses]
10 [vec (%fetch-embeddings-batch misses)])
11 (%cache-put text
12 (if (vector? vec)
13 vec
14 (list->vector (map exact->inexact vec)))))))
15 (map get-embedding texts))
Network calls are wrapped by call-with-retries, which retries HTTP 429 and 5xx errors with exponential backoff and fails fast on 4xx errors. This keeps a flaky network from throwing away a whole query.
The vector math is three small functions. dot-product checks that both vectors have the same length, because a length mismatch almost always means the embedding model was changed after the corpus was built, and silently truncating would corrupt every score. vector-magnitude computes the L2 norm, and cosine-similarity divides the dot product by the two magnitudes.
Chunk vectors are stored normalized (unit length), so scoring a query against every chunk is a dot product divided by the query norm. The query’s own norm appears only once per search, in the denominator:
1 (define (score-chunks chunks query-embedding #:query-norm [query-norm 1.0])
2 (map (lambda (chunk)
3 (cons (/ (dot-product query-embedding (document-chunk-embedding chunk))
4 query-norm)
5 chunk))
6 chunks))
The Vector Store
vector-store.rkt defines the data structures and the retrieval logic. Two structs hold state:
1 (struct document-chunk (text source embedding norm)
2 #:methods gen:custom-write
3 [(define write-proc print-document-chunk)])
4
5 (struct corpus (name description [chunks #:mutable])
6 #:methods gen:custom-write
7 [(define write-proc print-corpus)])
Both have custom printers. Printing a corpus shows its name, description, and chunk count; printing a chunk shows its text and source but only the first 10 floats of the embedding, followed by the dimension. Without this, printing an embedded corpus at the REPL would dump thousands of numbers.
Splitting text into chunks looks for a sentence end (period or newline) within 80 characters of the target boundary, and falls back to a hard cut if none is found. A forward-progress guard ensures that we can never loop forever on input whose only sentence break is at the start of the text:
1 (define (split-into-chunks text
2 #:chunk-size [chunk-size *default-chunk-size*]
3 #:overlap [overlap *chunk-overlap*])
4 (define len (string-length text))
5 (let loop ([start 0] [chunks '()])
6 (if (>= start len)
7 (reverse chunks)
8 (let* ([end (min (+ start chunk-size) len)]
9 [break-pos
10 (if (>= end len)
11 end
12 (let* ([search-from (max start (- end 80))]
13 [window (substring text search-from end)]
14 [period-idx (last-index-of-char window #\.)]
15 [newline-idx (last-index-of-char window #\newline)]
16 [idx (max (or period-idx -1) (or newline-idx -1))])
17 (if (>= idx 0)
18 (+ search-from idx)
19 end)))]
20 [actual-end* (if (< break-pos end) (+ break-pos 1) end)])
21 (define actual-end
22 (if (<= actual-end* start)
23 (min (+ start chunk-size) len)
24 actual-end*))
25 ...
26 (loop new-start new-chunks)))))
Corpora can be saved to disk and loaded back. save-corpus writes one s-expression with the name, description, and every chunk with its embedding as a plain list of numbers. load-corpus reads it back, validates that every chunk has non-empty text and source plus a numeric embedding, and re-normalizes the vectors so files written by older versions of the code still work.
Retrieval scores every chunk in a corpus against a query embedding and returns the top k as (score . chunk) pairs sorted by descending score. search-corpora does the same across several corpora, which is what makes cross-corpus questions possible.
The Agents
The interesting logic is in agents.rkt. Every LLM call goes through rag-generate, which applies the retry wrapper and signals an error if the model returns nothing. The model is stored in the parameter *rag-model* and can be overridden per call with a keyword argument.
Agent 1: Query Rewriter
The rewriter asks Gemini to break the user’s question into 1 to 3 short search queries. The prompt is strict: one query per line, no numbering, no bullets, under 15 words each. The raw response often comes back with markdown formatting, so parse-query-lines strips leading -, *, +, and numbered prefixes like 4. while preserving digits that are part of the query text itself (2024 lithium prices stays intact; 4. fourth query loses its prefix).
The original question is always appended to the rewritten list as a fallback, so the fanout always searches for exactly what the user asked.
1 (define (rewrite-queries user-query #:model [model (*rag-model*)])
2 (define prompt
3 (format (string-append
4 "You are a search query rewriter for a RAG system. "
5 "Your job is to break a complex user question into "
6 "1-3 simple, focused search queries that will help "
7 "retrieve relevant information from a document collection.\n"
8 "\nRules:\n"
9 "- Output ONLY the queries, one per line\n"
10 "- No numbering, bullets, or extra text\n"
11 "- Each query should target a specific fact or concept\n"
12 "- Keep queries concise (under 15 words each)\n"
13 "\nUser question: ~a")
14 user-query))
15 (define queries (parse-query-lines (rag-generate prompt #:model model)))
16 (remove-duplicates (append queries (list user-query))))
Agent 2: Search Fanout
The fanout embeds every sub-query in one batched API call, searches all corpora with each embedding, and deduplicates the results by (source . text). Keeping the source in the key matters: the same text in a different file is a different chunk and must not be collapsed. Results are sorted by descending score before being returned.
1 (define (search-fanout corpora sub-queries #:top-k [top-k 3])
2 (define query-embeddings (get-embeddings sub-queries))
3 (define seen-keys (make-hash))
4 (define all-results '())
5 (for ([query sub-queries]
6 [query-embedding query-embeddings])
7 (for ([result (search-corpora corpora query-embedding #:top-k top-k)])
8 (define key (document-chunk-key (cdr result)))
9 (unless (hash-ref seen-keys key #f)
10 (hash-set! seen-keys key #t)
11 (set! all-results (cons result all-results)))))
12 (sort all-results > #:key car))
Agent 3: Sufficient Context Agent
This agent separates agentic RAG from vanilla RAG. The prompt gives the model the question and every retrieved passage, and asks it to answer in exactly three lines: VERDICT, REASON, and MISSING. The parser is deliberately defensive. If the model forgets the format, we treat the context as sufficient. This bounds cost, because the only other safeguard is the iteration limit, and a wrong “insufficient” verdict wastes API calls without improving the answer.
1 (define (parse-verdict-response response)
2 (define lines (regexp-split #rx"\n" (or response "")))
3 (define verdict-line
4 (findf (lambda (line) (string-contains-ci? line "VERDICT:")) lines))
5 (define missing-line
6 (findf (lambda (line) (string-contains-ci? line "MISSING:")) lines))
7 ...
8 (cond
9 [(and verdict-word (string-contains-ci? verdict-word "INSUFFICIENT"))
10 (values #f feedback)]
11 [(and verdict-word (string-contains-ci? verdict-word "SUFFICIENT"))
12 (values #t feedback)]
13 [else
14 (debug-log "WARNING parse-verdict-response: unparseable verdict ~s; treating as SUFFICIENT~%"
15 verdict-word)
16 (values #t feedback)]))
Agent 4: Synthesis Agent
The last agent writes the answer. The prompt is simple: use only the retrieved passages, cite the source file for each fact, and if the passages do not cover the question, say what is missing.
The Orchestrator
agentic-rag ties the four agents together. It runs the rewriter, then the fanout, then enters the sufficiency loop. On each iteration it assesses the current context; if sufficient, it synthesizes and returns. If not, it refines the queries using the feedback from the sufficiency agent, searches again, merges the new chunks with the old ones (deduplicating by (source . text)), and tries once more.
Two details matter. First, max-iterations bounds the loop, and at the last iteration we skip the sufficiency check entirely: at that point “synthesize with what we have” is the only sensible move, so the check would be wasted. Second, max-context-chunks caps how many passages are sent to the LLM no matter how many iterations have accumulated; without this, a hard question could grow the prompt without bound.
1 (define (agentic-rag corpora user-query
2 #:max-iterations [max-iterations 3]
3 #:top-k [top-k 3]
4 #:model [model (*rag-model*)]
5 #:max-context-chunks [max-context-chunks 8])
6 (define sub-queries (rewrite-queries user-query #:model model))
7 (define initial-chunks (search-fanout corpora sub-queries #:top-k top-k))
8 (call/ec
9 (lambda (return)
10 (let loop ([iteration 1] [all-chunks initial-chunks])
11 (when (null? all-chunks)
12 (return "I could not find any relevant information in the available documents."))
13 (define context-chunks (cap-context all-chunks max-context-chunks))
14 (when (>= iteration max-iterations)
15 (return (synthesize-answer user-query context-chunks #:model model)))
16 (define-values (sufficient? feedback)
17 (assess-sufficiency user-query context-chunks #:model model))
18 (when sufficient?
19 (return (synthesize-answer user-query context-chunks #:model model)))
20 (define refined-queries (refine-queries user-query feedback #:model model))
21 (define new-chunks (search-fanout corpora refined-queries #:top-k top-k))
22 ...
23 (loop (+ iteration 1)
24 (sort accumulated > #:key car))))))
Running the Demo
The demo in main.rkt builds three corpora from the sample texts in data/ (renewable energy, electric vehicles, climate science), loads them, and asks three questions in order of difficulty. Load the module and call (test):
1 (require "main.rkt")
2 (test)
With *rag-verbose* at its default of #t, we see each agent report its decisions. Here is the start of the run, with most of the trace removed; the first line of each sub-query shows what the rewriter produced:
1 ============================================
2 Agentic RAG Demo -- Loading Documents
3 ============================================
4
5 DEBUG add-document: loading .../data/renewable-energy.txt
6 DEBUG add-document: split into 9 chunks
7 DEBUG get-embeddings: batch-fetching 9 embeddings
8 DEBUG add-document: added 9 chunks from renewable-energy.txt
9 DEBUG add-document: loading .../data/electric-vehicles.txt
10 DEBUG add-document: split into 9 chunks
11 DEBUG get-embeddings: batch-fetching 9 embeddings
12 DEBUG add-document: added 9 chunks from electric-vehicles.txt
13 DEBUG add-document: loading .../data/climate-science.txt
14 DEBUG add-document: split into 9 chunks
15 DEBUG get-embeddings: batch-fetching 9 embeddings
16 DEBUG add-document: added 9 chunks from climate-science.txt
17
18 Loaded 27 total chunks across 3 corpora.
19
20 ===== TEST QUERY 1 (single topic) =====
21
22 ========================================
23 AGENTIC RAG PIPELINE
24 Query: What is the current cost of lithium-ion battery storage per kilowatt-hour?
25 ========================================
26
27 DEBUG rewrite-queries: decomposing query...
28
29 DEBUG rewrite-queries: generated 2 sub-queries:
30 - current cost lithium-ion battery storage per kWh
31 - lithium-ion battery storage price trend
32 DEBUG search-fanout: searching 3 corpora with 3 queries
33 DEBUG search-fanout: searching with: "current cost lithium-ion battery storage per kWh"
34 DEBUG search-fanout: searching with: "lithium-ion battery storage price trend"
35 DEBUG search-fanout: searching with: "What is the current cost of lithium-ion battery storage per kilowatt-hour?"
36 DEBUG search-fanout: found 9 unique chunks
37
38 --- Iteration 1/3 ---
39
40 DEBUG assess-sufficiency: evaluating 8 chunks
41 DEBUG assess-sufficiency response:
42 VERDICT: SUFFICIENT
43 REASON: The passages state the price has fallen to under $140 per kilowatt-hour.
44 MISSING: NONE
45 DEBUG assess-sufficiency: verdict=SUFFICIENT
46
47 DEBUG synthesize-answer: generating answer from 8 chunks
48 DEBUG synthesize-answer: generated response (342 chars)
49
50 ANSWER 1:
51 The current cost of lithium-ion battery storage is under $140 per kilowatt-hour. This price has fallen by approximately 90% since 2010, when it was over $1,100 per kilowatt-hour, according to renewable-energy.txt.
Two things are worth noticing in that trace. The original user question appears verbatim in the search fanout, which is the fallback the rewriter adds so the search always covers the exact wording the user typed. And the sufficiency agent answered SUFFICIENT on the first pass, so the pipeline went straight to synthesis.
The second question is a multi-hop question and needs two corpora:
1 ===== TEST QUERY 2 (multi-hop, cross-corpus) =====
2 ... ANSWER 2:
3 Manufacturing an EV battery produces approximately 75-100 kg of CO2 per kilowatt-hour of battery capacity (electric-vehicles.txt). In return, an EV charged from renewable energy produces zero operational emissions (electric-vehicles.txt), and even on the average US grid mix it produces roughly 50-60% fewer lifecycle greenhouse gas emissions than a comparable gasoline vehicle (electric-vehicles.txt). Transitioning to electric vehicles powered by renewable energy is one of the most effective strategies for reducing transportation emissions (climate-science.txt).
The answer cites electric-vehicles.txt for the battery manufacturing emissions and climate-science.txt for the transportation context, confirming that the fanout crossed corpus boundaries.
The third question asks about two technologies that live in the same document but in different chunks:
1 ===== TEST QUERY 3 (complex, iterative) =====
2 ... ANSWER 3:
3 Solid-state batteries and pumped-storage hydroelectricity play complementary roles in solving the intermittency of wind and solar. Solid-state designs respond within milliseconds to smooth output when clouds pass over a solar farm or wind speeds drop, handle the fast, frequent charge cycles, and shift solar generation into the evening demand peak (renewable-energy.txt). Pumped-storage hydroelectricity provides bulk, long-duration storage that moves large amounts of energy across many hours or days, handling the deep, infrequent discharges (renewable-energy.txt). Together they let wind and solar installations deliver firm, dispatchable power around the clock (renewable-energy.txt).
test returns the list of corpora, so we can keep them and start an interactive session without re-embedding:
1 (define *corpora* (test))
2 (interactive-demo *corpora*)
This prints a RAG> prompt and answers questions until you type quit.
Testing Without the Network
Everything that talks to the outside world is behind a parameter. *embedding-fn* produces embedding vectors, *batch-request-fn* posts one batch of texts to Gemini, and *generate-fn* calls the LLM. In tests.rkt all three are replaced with stubs using parameterize, so the entire suite runs offline:
1 (parameterize ([*embedding-fn* (lambda (text) '(1.0 0.0 0.0))]
2 [*embedding-cache* (make-hash)]
3 [*rag-verbose* #f])
4 ...)
The suite has 15 tests. They cover the chunker (including the forward-progress guard), the query-line parser (digits inside queries survive; list prefixes of any number are stripped), vector math (including the error on dimension mismatch), retrieval ranking, deduplication across sources, batched query embedding, batch splitting at the 100-text API cap, cache eviction, retry behavior, verdict parsing, save/load round-trip with corruption checks, and the full pipeline with a stubbed LLM (sufficient on the first try, insufficient then sufficient, and the skipped sufficiency check at the last iteration). Run them with:
1 racket tests.rkt
You should see:
1 15 success(es) 0 failure(s) 0 error(s) 15 test(s) run
Wrap Up
The agentic pipeline costs more API calls than vanilla RAG: one call to rewrite the query, at least one to assess sufficiency, and one more to synthesize. In exchange you get answers that hold together for multi-hop questions, with citations that point back to the source files. The iteration bound and the context cap keep the cost predictable even on hard questions.
Optional Practice Problems
Persist the cache:
*embedding-cache*lives in memory and is lost when the program exits. Changeget-embeddingso the cache is written to a file after each batch call and read back at startup. What happens to startup time for the demo?Score-weighted context: The synthesis agent receives chunks in descending score order but no scores. Modify
format-retrieved-chunksand the synthesis prompt so each passage shows its relevance score, and experiment with instructing the model to prefer higher-scoring passages when they conflict.Async fanout:
search-fanoutsearches each sub-query in turn. Useracket/async-channelor futures to search all sub-queries in parallel and measure the speedup on a corpus of a few hundred chunks.