Knowledge Base Navigator: Building an AI-Powered Information System

Earlier we looked the my old Knowledge Graph Navigator (KGN) project that combined symbolic Natural language Processing (NLP) with access to public knowledge graphs like Wikidata and DBPedia. Here I take a much simpler approach using an inexpensive Gemini model.

The code can be found in the directory: loving-common-lisp/src/knowledge-base-navigator.

In this chapter, we explore a practical application that combines modern AI APIs with Common Lisp to create an interactive knowledge exploration tool. The Knowledge Base Navigator demonstrates how to integrate external services, handle JSON data, and supply a user-friendly text interfaces.

This example differs from the KGN project since no knowledge base is constructed and stored. Here we use live web searches via Google’s web_search tool that is available with the Gemini APIs to dynamically construct user content. Dear reader, this is a different approach!

Project Overview

The Knowledge Base Navigator is a modern evolution of my classic Knowledge Graph Navigator (KGN). This new version uses Google’s Gemini Flash LLM API to extract entities from natural language, disambiguate them, discover semantic links between entities, and retrieve detailed encyclopedic information. This represents a paradigm shift from traditional database-backed systems to an AI-driven pipeline.

The system follows a two-stage process:

  1. Entity Extraction: User provides text, Gemini identifies potential entities (people, companies, countries, etc.) and returns them as a numbered list
  2. Deep Retrieval: User selects entities by number, Gemini provides detailed facts and analyzes relationships between entities

Project Structure

The New Knowledge Navigator consists of three files, one source file and two small configuration files:

1 knowledge-base-navigator/
2 ├── knowledge-base-navigator.asd     # ASDF system definition
3 ├── project.lisp                     # Package definition
4 └── knowledge-base-navigator.lisp    # Core application

System Definition (ASDF)

The .asd file defines the system, its metadata, dependencies, and compilation order:

 1 ;;;; knowledge-base-navigator.asd
 2 
 3 (asdf:defsystem #:knowledge-base-navigator
 4   :description "Knowledge Base Navigator using Gemini 3 Flash LLM"
 5   :author "Mark Watson <markw@markwatson.com>"
 6   :license "Apache 2"
 7   :depends-on (#:cl-json #:uiop #:alexandria)
 8   :serial t
 9   :components ((:file "project")
10                (:file "knowledge-base-navigator")))

Key elements:

  • #:prefix notation creates uninterned symbols, avoiding package conflicts
  • :depends-on specifies required libraries: cl-json for JSON encoding/decoding, uiop for system utilities, alexandria for common utilities
  • :serial t ensures files compile in order (project.lisp before knowledge-base-navigator.lisp)

Package Definition

The package file establishes the namespace and exports:

1 ;;;; project.lisp
2 ;;;; Package definition and global variables
3 
4 (defpackage #:knowledge-base-navigator
5   (:use #:cl)
6   (:export #:kbn-ui
7            #:get-gemini-chat-completion))
8 
9 (in-package #:knowledge-base-navigator)
  • defpackage creates a new namespace, isolating symbols from other packages
  • :use #:cl imports standard Common Lisp symbols
  • :export makes kbn-ui and get-gemini-chat-completion accessible externally

Core Implementation

The following code snippets are part of the file knowledge-base-navigator.lisp.

Tokenizer Utility

The tokenizer parses user input into individual tokens:

 1 (defun tokenize-string (string &key (separators '(#\Space #\Tab #\Newline #\Return #\,)))
 2   (let ((tokens '())
 3         (current-word (make-string-output-stream)))
 4     (loop for char across string
 5           if (member char separators)
 6             do (let ((word (get-output-stream-string current-word)))
 7                  (when (plusp (length word))
 8                    (push word tokens)))
 9           else
10             do (write-char char current-word))
11     (let ((word (get-output-stream-string current-word)))
12       (when (plusp (length word))
13         (push word tokens)))
14     (nreverse tokens)))

This function uses a character-by-character approach:

  1. Iterates across each character in the input string
  2. Accumulates characters into a stream until hitting a separator
  3. Pushes completed words onto the token list
  4. Returns tokens in original order via nreverse

Why implement a custom tokenizer? The standard cl-ppcre:split or uiop:split-string would work here, but this implementation demonstrates manual string processing and is dependency-free.

HTTP Communication via curl

Rather than using Common Lisp HTTP libraries (which can have SSL/Certificate issues), this project runs curl directly:

 1 (defun run-curl-command-with-json (url json-payload &optional headers)
 2   "POST JSON-PAYLOAD to URL with curl. HEADERS is a list of \"Name: value\"
 3    strings. Returns the response body string."
 4   (uiop:with-temporary-file (:pathname temp-file :type "json")
 5     (with-open-file (stream temp-file :direction :output :if-exists :supersede)
 6       (write-string json-payload stream))
 7     (uiop:run-program (append (list "curl" "-s" "-X" "POST" url
 8                                     "-H" "Content-Type: application/json")
 9                               (loop for h in headers append (list "-H" h))
10                               (list "-d" (concatenate 'string "@" (namestring temp-file))))
11                       :output :string
12                       :error-output :string
13                       :ignore-error-status t)))

Key technique: we write the JSON payload to a temporary file and pass it to curl with -d @filename. We also pass curl an argument list instead of a shell command string, so no shell ever parses the payload, the URL, or the headers. Quotes and other special characters in a prompt can not break the command.

The flow:

  1. uiop:with-temporary-file creates a temporary file and deletes it when the body exits, even after an error
  2. Write the JSON payload to the file
  3. Run curl with the @filename syntax, adding a -H argument for each extra header in HEADERS
  4. Return the response body

Gemini API Integration

The main API function constructs requests and parses responses:

 1 (defun get-gemini-chat-completion (user-prompt)
 2   "Sends a prompt to the Gemini Interactions API and returns the content of the response."
 3   (let* ((api-key (or (uiop:getenv "GEMINI_API_KEY") (uiop:getenv "GOOGLE_API_KEY")))
 4          (model-name "gemini-3-flash-preview")
 5          (base-url "https://generativelanguage.googleapis.com/v1beta/interactions")
 6          (payload
 7            `((:model . ,model-name)
 8              (:input . ,user-prompt)))
 9          (json-payload (json:encode-json-to-string payload)))
10     
11     (unless (and api-key (not (string= api-key "")))
12       (error "GEMINI_API_KEY (or GOOGLE_API_KEY) environment variable is not set."))
13 
14     (handler-case
15         ;; the API key is sent in a header rather than in the URL so that it
16         ;; does not show up in process listings:
17         (let* ((response-body
18                  (run-curl-command-with-json
19                   base-url json-payload
20                   (list (concatenate 'string "x-goog-api-key: " api-key)
21                         "Api-Revision: 2026-05-20")))
22                (parsed-response (json:decode-json-from-string response-body))
23                (steps (cdr (assoc :steps parsed-response))))
24           (if steps
25               ;; Find the last model_output step and extract its text
26               (loop for step in (reverse steps)
27                     when (string-equal (cdr (assoc :type step)) "model_output")
28                     return (let* ((content (cdr (assoc :content step)))
29                                   (first-content (first content)))
30                              (cdr (assoc :text first-content))))
31               (progn
32                 (format *error-output* "~%[Error] Gemini Interactions API response did not contain steps: ~A~%" parsed-response)
33                 nil)))
34       (error (e)
35         (format *error-output* "An unexpected error occurred: ~A~%" e)
36         nil))))

The following diagram shows the high-level architecture of the Knowledge Base Navigator system developed in this chapter:

Architecture diagram

API Structure Notes:

  • The code uses the Gemini Interactions API. The request is a JSON object with the model name and the user input.
  • The function reads the API key from GEMINI_API_KEY, or from GOOGLE_API_KEY when the first is not set.
  • The key goes in the x-goog-api-key header, not in the URL, so it does not show up in process listings.
  • Backquote (`) is used for template construction, with comma (,) for evaluation
  • json:encode-json-to-string converts Lisp data to JSON
  • json:decode-json-from-string parses the response
  • The response holds a list of steps. The loop walks the steps from last to first and returns the text of the last model_output step.
  • handler-case provides error handling, returning nil on failure

Response parsing pattern: The Gemini response has nested associative lists (alists). Navigate with:

1 (cdr (assoc :key alist))  ; Get value for :key
2 (first list)              ; Get first element
3 (cdr ...)                 ; Continue navigating

Interactive UI Loop

The kbn-ui function implements an interactive text user interface:

 1 (defun kbn-ui ()
 2   "Main user interface loop for Knowledge Base Navigator powered by Gemini."
 3   (let ((prompt ""))
 4     (loop
 5       (format t "~%============= GEMINI KNOWLEDGE BASE NAVIGATOR =============~%")
 6       (format t "~%Enter entity names separated by space or a descriptive sentence (or type 'quit' to exit):~%> ")
 7       (finish-output)
 8       (setf prompt (read-line))
 9       
10       (when (or (string-equal prompt "quit") (string-equal prompt "q"))
11         (format t "Goodbye!~%")
12         (return))
13         
14       (when (> (length prompt) 0)
15         (format t "~%[Extracting entities using Gemini 3 Flash...]~%")
16         ;; ... entity extraction and detail retrieval
17         ))))

Key loop patterns:

  • loop with (return) for controlled exit
  • read-line for user input
  • finish-output ensures prompts display immediately
  • when guards for conditional execution

The entity extraction creates a prompt that instructs Gemini to return ONLY a numbered list:

1 (format nil "Analyze the following user text: \"~A\".~%~
2 Identify potential encyclopedic entities (people, companies, countries, cities, products, concepts, etc.) mentioned.~%~
3 Categorize them if necessary. Return them as a neatly formatted numbered list (1., 2., 3., etc.) with a short 1-sentence description for each.~%~
4 DO NOT return any other conversational text, ONLY the numbered list so the user can see their options." prompt)

Prompt engineering principle: Be explicit about output format. The instruction to return ONLY the numbered list prevents verbose conversational responses.

User selection handling:

1 (let* ((tokens (tokenize-string selection-line))
2        (valid-tokens (remove-if-not #'(lambda (s) (every #'digit-char-p s)) tokens))
3        (indices (mapcar #'parse-integer valid-tokens)))
4   ;; Process indices...
5 )

This filters non-numeric input and converts valid selections to integers.

Running the Application

Load the system via Quicklisp:

 1 ;; Ensure project directory is in Quicklisp's path
 2 ;; Usually via symlink:
 3 ;;  ln -s /path/to/knowledge-base-navigator ~/quicklisp/local-projects/
 4 ;; or by adding the current directory (or a parent directory)
 5 ;; to ql:*local-project-directories*
 6 
 7 ;; Load system
 8 (ql:quickload "knowledge-base-navigator")
 9 
10 ;; Start UI
11 (knowledge-base-navigator:kbn-ui)

Example Session

 1 ============= GEMINI KNOWLEDGE BASE NAVIGATOR =============
 2 
 3 Enter entity names separated by space or a descriptive sentence (or type 'quit' to exit):
 4 > Bill Gates and Microsoft
 5 
 6 [Extracting entities using Gemini 3 Flash...]
 7 
 8 --- IDENTIFIED ENTITIES ---
 9 1. Bill Gates: An American business magnate, software developer, and philanthropist who co-founded Microsoft Corporation.
10 2. Microsoft: A multinational technology corporation that develops, manufactures, and licenses computer software.
11 ---------------------------
12 
13 Enter the numbers of the entities you want detailed information for (space separated):
14 > 1 2
15 
16 [Fetching detailed facts and relationships for selected entities...]
17 
18 === BILL GATES ===
19 * Born: October 28, 1955, Seattle, Washington
20 * Occupation: Business magnate, investor, philanthropist
21 * Net Worth: ~$120 billion (as of 2024)
22 * Founded Microsoft in 1975 with Paul Allen
23 
24 === MICROSOFT ===
25 * Founded: April 4, 1975
26 * Headquarters: Redmond, Washington
27 * Industry: Technology, Software, Cloud Computing
28 * Revenue: $211 billion (2023)
29 
30 === RELATIONSHIP ===
31 Bill Gates co-founded Microsoft with Paul Allen in 1975. He served as CEO until 2000 and remained Chairman until 2014. Microsoft was the primary source of Gates' wealth.

Key Takeaways

  1. Package Organization: Use ASDF systems with separate files for package definitions
  2. External Process Integration: uiop:run-program provides robust external command execution
  3. JSON Handling: cl-json encodes Lisp data structures to JSON and decodes responses
  4. Error Handling: Use handler-case for robust API error handling
  5. Prompt Engineering: Construct clear prompts with explicit format instructions
  6. REPL Interfaces: loop + read-line + format create simple but effective CLIs

Dependencies

Library Purpose
cl-json JSON encoding/decoding
uiop System utilities, process execution
alexandria Common Lisp utilities

Environment Setup

 1 # Set API key before starting Lisp (GOOGLE_API_KEY also works)
 2 export GEMINI_API_KEY="your_api_key_here"
 3 
 4 # Start SBCL
 5 sbcl
 6 
 7 # Load Quicklisp
 8 (quicklisp:setup)
 9 
10 # Load project
11 (ql:quickload "knowledge-base-navigator")

This example demonstrates that Common Lisp remains highly relevant for modern API-driven applications. The combination of powerful REPL development, mature libraries, and straightforward process integration makes it an excellent choice for exploratory programming and rapid prototyping.

Previous

Image Processing With Local Ollama Models

Up next

Part 7: Prompting, Retrieval, and Agentic Applications