RAG Using zvec Vector Datastore and Local Model

The zvec library implements a lightweight, lightning-fast, in-process vector database. Allibaba released zvec in February 2026. We will see how to use zvec and then build a high performance RAG system. We will use the tiny model qwen3:1.7b running localling using Ollama as part of the application.

Note: The source code for this example can be found in hy-lisp-python-book/source_code_for_examples/RAG_zvec/app.hy. Not all the code in this file is listed here.

Note: This chapter was derived from a chapter with the same name from my Python book Ollama in Action.

Introduction and Architecture

Building a Retrieval-Augmented Generation (RAG) pipeline entirely locally ensures absolute data privacy, eliminates API latency costs, and provides full control over the embedding and generation models. In this chapter, we construct a fully offline RAG system utilizing Ollama for both embeddings (embeddinggemma) and inference (qwen3:1.7b), paired with zvec, a lightweight, high-performance local vector database.

The architecture follows a classic two-phase RAG pattern, adding an additional third step to improve the user experience:

  • Ingestion: Parse local text files, chunk the content, generate embeddings via Ollama, and index them into zvec.
  • Retrieval & Generation: Embed the user query, perform a similarity search in zvec, and save the retrieved top-k chunks for processing by a local Ollama chat model.
  • Use a small LLM model (qwen3:1.7b) to process the retrieved chunks and taking into account the user’s original query and then format a subset of the text in the returned chunks for the user to read.
RAG pipeline architecture using zvec vector datastore and local Ollama model

Design Analysis: Dependency Minimization

A notable design choice in our implementation is the reliance on Python’s standard library for network calls. By utilizing urllib.request instead of third-party libraries like requests or the official ollama-python client library, the dependency footprint is minimized exclusively to zvec. This reduces virtual environment overhead and potential version conflicts, prioritizing a lean deployment.

Implementation Walkthrough

Here we look at some of the code in the source file app.hy.

Embedding and Chunking Strategy

The ingestion phase relies on a fixed-size overlapping window strategy. Here are the functions for talking to the Ollama server, getting embeddings, and chunking text:

 1 (defn ollama-post [path payload]
 2   "POST PAYLOAD as JSON to the local Ollama server and return the parsed reply."
 3   (setv req (urllib.request.Request f"{OLLAMA-BASE}{path}"
 4               :data (.encode (json.dumps payload) "utf-8")
 5               :headers {"Content-Type" "application/json"}))
 6   (with [res (urllib.request.urlopen req)]
 7     (json.loads (.decode (.read res) "utf-8"))))
 8 
 9 
10 (defn get-embeddings [texts]
11   "Embed a list of TEXTS with one request to Ollama's /api/embed endpoint."
12   (setv model (get config "embedding_model"))
13   (try
14     (get (ollama-post "/api/embed" {"model" model "input" texts}) "embeddings")
15     (except [e OSError]
16       ;; Silently using placeholder vectors would make every search return
17       ;; arbitrary chunks, so stop with a helpful message instead.
18       (raise (RuntimeError
19                f"Could not get embeddings from Ollama ({e}). Is Ollama running, and did you run: ollama pull {model}?")))))
20 
21 
22 (defn get-embedding [text]
23   "Embed a single TEXT."
24   (get (get-embeddings [text]) 0))
25 
26 
27 (defn chunk-text [text [chunk-size 500] [overlap 50]]
28   "Split text into overlapping chunks."
29   (setv chunks [])
30   (setv start 0)
31   (while (< start (len text))
32     (setv end (+ start chunk-size))
33     (.append chunks (cut text start end))
34     (setv start (- end overlap)))
35   chunks)

Analysis of code:

  • Chunk Size (500 chars): This relatively small chunk size yields high-granularity embeddings. It reduces the risk of retrieving “diluted” context where a single chunk contains multiple disparate concepts.
  • Overlap (50 chars): Crucial for preventing context loss at the boundaries of chunks. It ensures that a semantic concept bisected by a hard character limit is still captured cohesively in at least one chunk.
  • Embedding Model: The system uses embeddinggemma by default (set the environment variable EMBEDDING_MODEL to use another model such as nomic-embed-text). The function ollama-post sends a JSON request to the local Ollama server; get-embeddings calls Ollama’s /api/embed endpoint, which accepts a list of texts, so all chunks are embedded with a single request instead of one request per chunk.
  • Failing Loudly: If the Ollama server is not running or the embedding model has not been pulled, get-embeddings raises an exception with a message telling you what to check. An earlier version of this example returned a vector of zeros instead; that kept the program running, but every search then returned arbitrary chunks, which is a much harder problem to diagnose than an error message.

Vector Storage with zvec

The zvec integration demonstrates a strictly typed, schema-driven approach to local vector storage.

Design notes for code:

  • Dimensionality Matching: The vector schema must declare the number of dimensions in each embedding (FP32 values). Instead of hard-coding the 768 dimensions produced by embeddinggemma, build-index embeds the chunks first and uses the length of the first embedding, so changing the embedding model does not require a code change.
  • Storage Path: The database is initialized locally at ./zvec_example. The implementation includes a defensive teardown (shutil.rmtree) of existing databases on startup. This is excellent for testing and iterative development, though destructive in a persistent production environment.

The following function builds the index using an embedding model for the local Ollama server:

 1 (defn read-chunks []
 2   "Return [file-name chunk-number chunk-text] for every chunk of every text
 3   file under the data directory."
 4   (setv results [])
 5   (for [[root _ files] (os.walk (get config "data_dir"))
 6         file (sorted files)
 7         :if (.endswith (.lower file) (get config "extensions"))]
 8     (try
 9       (setv content (.read-text (/ (Path root) file) :encoding "utf-8"))
10       (except [e OSError]
11         (print f"Skipping {file}: {e}")
12         (continue)))
13     (for [[i chunk] (enumerate (chunk-text content))]
14       (.append results [file i chunk])))
15   results)
16 
17 
18 (defn build-index []
19   "Index all text files from the data directory into zvec."
20   (setv data-dir (get config "data_dir"))
21   (setv chunks (read-chunks))
22   (setv embeddings (if chunks (get-embeddings (lfor [_ _ text] chunks text)) []))
23   ;; Size the vectors to match whatever the embedding model produces
24   ;; (embeddinggemma and nomic-embed-text both produce 768 numbers).
25   (setv dimension (if embeddings (len (get embeddings 0)) 768))
26   (setv schema
27     (zvec.CollectionSchema
28       :name    "example"
29       :vectors (zvec.VectorSchema "embedding" zvec.DataType.VECTOR-FP32 dimension)
30       :fields  (zvec.FieldSchema  "text"      zvec.DataType.STRING)))
31 
32   (setv db-path "./zvec_example")
33   (when (os.path.exists db-path)
34     (shutil.rmtree db-path))
35   (setv collection (zvec.create-and-open :path db-path :schema schema))
36 
37   (when chunks
38     (.insert collection
39              (lfor [[file i text] embedding] (zip chunks embeddings)
40                    (zvec.Doc :id      f"{file}_{i}"
41                              :vectors {"embedding" embedding}
42                              :fields  {"text" text}))))
43   (print f"Indexed {(len chunks)} chunks from {data-dir}")
44   collection)

The function read-chunks walks the data directory (configured with the environment variable DATA_DIR) and returns every chunk of every .txt file along with its file name and chunk number. A file that cannot be read is reported and skipped. The function build-index then initializes a local vector database and populates it with document embeddings. Specifically, it executes three main operations:

  • Embedding: Gets the embeddings for all chunks with one call to get-embeddings.
  • Schema & Storage Initialization: Defines a strict schema for zvec (FP32 vectors sized to match the embedding model, and a string metadata field) and destructively recreates the local database directory (./zvec_example).
  • Batch Insertion: Pairs each chunk with its embedding using zip, builds a zvec.Doc for each pair, and performs a single bulk insert into the zvec collection.

Retrieval and LLM Synthesis

The synthesis phase bridges the vector database and the Generative LLM. Function search identifies matching text chunks in the vector database:

 1 (defn search [collection query [topk 5]]
 2   "Search the zvec collection for chunks relevant to the query."
 3   (setv query-vector (get-embedding query))
 4   (setv results
 5     (.query collection
 6       (zvec.Query "embedding" :vector query-vector)
 7       :topk topk))
 8   (setv chunks [])
 9   (for [res results]
10     (setv text
11       (if res.fields
12         (.get res.fields "text" "")
13         ""))
14     (when text
15       (.append chunks text)))
16   chunks)

Function search performs a Top-K retrieval. The default topk=5 retrieves roughly 2,500 characters of context. This easily fits within the context window of modern small models like qwen3:1.7b without causing attention dilution (“lost in the middle” syndrome).

System Prompt Engineering and Using a Small LLM to Prepare Output for a User

The ask-ollama function utilizes strict prompt constraints: “Answer the user’s question using ONLY the context provided below. If the context does not contain enough information, say so.” This significantly mitigates hallucination by forcing the model to ground its response exclusively in the retrieved data.

 1 (defn ask-ollama [question context-chunks]
 2   "Send retrieved chunks + user question to the Ollama chat model."
 3   (setv context (.join "\n\n---\n\n" context-chunks))
 4   (setv system-prompt
 5     (.join ""
 6       ["You are a helpful assistant. Answer the user's question using ONLY "
 7        "the context provided below. If the context does not contain enough "
 8        "information, say so. Be concise and accurate.\n\n"
 9        f"Context:\n{context}"]))
10   (try
11     (setv body (ollama-post "/api/chat"
12                             {"model"    (get config "chat_model")
13                              "stream"   False
14                              "messages" [{"role" "system" "content" system-prompt}
15                                          {"role" "user"   "content" question}]}))
16     (get body "message" "content")
17     (except [e Exception]
18       f"Error calling Ollama chat: {e}")))

Function ask-ollama uses stateless execution: The /api/chat call sets “stream”: False and does not maintain a conversation history array across loop iterations. This makes it a pure Q&A interface rather than a continuous chat, ensuring each answer is cleanly tied to a fresh zvec retrieval.

Example Run

To run the pipeline, ensure the Ollama daemon is running locally on port 11434 and that both models (embeddinggemma and qwen3:1.7b) have been pulled. Place your .txt files in the ../data directory and execute the script. The system will build the index and immediately drop you into a REPL loop for interactive querying.

Here is an example run:

 1  $ uv run hy app.hy 
 2 Building zvec index from text files
 3 Indexed 9 chunks from ../data
 4 
 5 RAG chat ready  (model: qwen3:1.7b)
 6 Type your question, or 'quit' to exit.
 7 
 8 You> who says economics is bullshit?
 9 
10 Assistant> The context mentions Pauli Blendergast, an economist at the University of Krampton Ohio, who is noted for stating that "economics is bullshit." No other individuals are explicitly cited in the provided text.
11 
12 You> what procedures are performed in chemistry labs?
13 
14 Assistant> The provided context does not contain information about procedures performed in chemistry labs. It focuses on economic concepts, microeconomics, macroeconomics, and related topics, but does not mention chemistry or laboratory procedures.
15 
16 You> how do microeconomics and macroeconomics differ?
17 
18 Assistant> Microeconomics focuses on individual agents (e.g., households, firms) and specific markets, analyzing decisions like pricing, resource allocation, and consumer behavior. Macroeconomics examines the entire economy, addressing broader issues such as unemployment, inflation, growth, and fiscal/money policy. While microeconomics deals with "how" resources are used, macroeconomics focuses on "what" the economy produces and "how collectively" it functions.
19 
20 You> quit
21 Goodbye!

Dear reader, notice that there was no information in the indexed text to answer the second example query and this program correctly refused to hallucinate (or make up) an answer.

Wrap Up for RAG Using zvec Vector Datastore and Local Model

In this chapter, we built a completely offline, privacy-preserving RAG architecture by bridging Alibaba’s recently released in-process vector database, zvec, with local Ollama inference. By intentionally minimizing external dependencies and utilizing a strictly typed, schema-driven datastore, we eliminated the network overhead and deployment bloat typical of client-server vector databases. The fixed-size overlapping chunking strategy, combined with the 768-dimensional embeddinggemma model, ensures high-fidelity semantic retrieval. Simultaneously, the compact qwen3:1.7b model demonstrates that a heavily constrained, prompt-engineered generation phase can effectively synthesize retrieved context without hallucination.

The resulting pipeline serves as a robust, lightweight foundation for edge-deployable AI applications. Because the entire storage and inference stack executes locally within the same process, the pattern is exceptionally portable, fast, and secure. Moving forward, this baseline implementation can be extended to handle more complex retrieval requirements, such as integrating dynamic semantic chunking, implementing Reciprocal Rank Fusion (RRF) for hybrid multi-vector queries, or introducing multi-turn conversational memory. Ultimately, combining embedded vector storage with small-parameter LLMs proves that high-performance, domain-specific RAG does not require massive cloud infrastructure.

Optional Practice Problems

  1. Source Attribution & Metadata Retrieval: Modify the schema in build-index to include a new string field source_file. Update the ingestion logic to populate this field with the base filename of the text file being processed. Then, update the search and ask-ollama functions so that when the assistant answers a question, it also prints or appends a list of the unique source files from which the retrieved chunks originated.

  2. Semantic Paragraph Chunking: The current chunk-text function splits documents blindly by character limits, which can cut sentences in half. Rewrite the chunking logic to split the input text by paragraphs (e.g., using \n\n as a delimiter). Ensure that chunks are grouped so they do not exceed a certain character count (e.g., 500 characters) while still keeping paragraphs intact where possible.

  3. Reusing a Persistent Index: Each run of app.hy deletes the ./zvec_example directory and re-embeds every document. Modify build-index so that it reopens the existing collection when one exists and none of the files in the data directory have changed since it was built. Hint: store a small JSON file next to the index that records each file’s name and modification time (from os.path.getmtime), and compare it with the current files before deciding to rebuild.