Knowledge Base Navigator: Building an AI-Powered Information System

Earlier we looked the my old Knowledge Graph Navigator (KGN) project that combined symbolic Natural language Processing (NLP) with access to public knowledge graphs like Wikidata and DBPedia. Here I take a much simpler approach using an inexpensive Gemini model.

The code can be found in the directory: loving-common-lisp/src/knowledge-base-navigator.

In this chapter, we explore a practical application that combines modern AI APIs with Common Lisp to create an interactive knowledge exploration tool. The Knowledge Base Navigator demonstrates how to integrate external services, handle JSON data, and supply a user-friendly text interfaces.

This example differs from the KGN project since no knowledge base is constructed and stored. Here we use live web searches via Google’s web_search tool that is available with the Gemini APIs to dynamically construct user content. Dear reader, this is a different approach!

Project Overview

The Knowledge Base Navigator is a modern evolution of my classic Knowledge Graph Navigator (KGN). This new version uses Google’s Gemini Flash LLM API to extract entities from natural language, disambiguate them, discover semantic links between entities, and retrieve detailed encyclopedic information. This represents a paradigm shift from traditional database-backed systems to an AI-driven pipeline.

The system follows a two-stage process:

  1. Entity Extraction: User provides text, Gemini identifies potential entities (people, companies, countries, etc.) and returns them as a numbered list
  2. Deep Retrieval: User selects entities by number, Gemini provides detailed facts and analyzes relationships between entities

Project Structure

The New Knowledge Navigator consists of three files, one source file and two small configuration files:

1 knowledge-base-navigator/
2 ├── knowledge-base-navigator.asd     # ASDF system definition
3 ├── project.lisp                     # Package definition
4 └── knowledge-base-navigator.lisp    # Core application

System Definition (ASDF)

The .asd file defines the system, its metadata, dependencies, and compilation order:

 1 ;;;; knowledge-base-navigator.asd
 2 
 3 (asdf:defsystem #:knowledge-base-navigator
 4   :description "Knowledge Base Navigator using Gemini 3 Flash LLM"
 5   :author "Mark Watson <markw@markwatson.com>"
 6   :license "Apache 2"
 7   :depends-on (#:cl-json #:uiop #:alexandria)
 8   :serial t
 9   :components ((:file "project")
10                (:file "knowledge-base-navigator")))

Key elements:

  • #:prefix notation creates uninterned symbols, avoiding package conflicts
  • :depends-on specifies required libraries: cl-json for JSON encoding/decoding, uiop for system utilities, alexandria for common utilities
  • :serial t ensures files compile in order (project.lisp before knowledge-base-navigator.lisp)

Package Definition

The package file establishes the namespace and exports:

1 ;;;; project.lisp
2 ;;;; Package definition and global variables
3 
4 (defpackage #:knowledge-base-navigator
5   (:use #:cl)
6   (:export #:kbn-ui
7            #:get-gemini-chat-completion))
8 
9 (in-package #:knowledge-base-navigator)
  • defpackage creates a new namespace, isolating symbols from other packages
  • :use #:cl imports standard Common Lisp symbols
  • :export makes kbn-ui and get-gemini-chat-completion accessible externally

Core Implementation

The following code snippets are part of the file knowledge-base-navigator.lisp.

Tokenizer Utility

The tokenizer parses user input into individual tokens:

 1 (defun tokenize-string (string &key (separators '(#\Space #\Tab #\Newline #\Return)))
 2   (let ((tokens '())
 3         (current-word (make-string-output-stream)))
 4     (loop for char across string
 5           if (member char separators)
 6             do (let ((word (get-output-stream-string current-word)))
 7                  (when (plusp (length word))
 8                    (push word tokens)))
 9           else
10             do (write-char char current-word))
11     (let ((word (get-output-stream-string current-word)))
12       (when (plusp (length word))
13         (push word tokens)))
14     (nreverse tokens)))

This function uses a character-by-character approach:

  1. Iterates across each character in the input string
  2. Accumulates characters into a stream until hitting a separator
  3. Pushes completed words onto the token list
  4. Returns tokens in original order via nreverse

Why implement a custom tokenizer? The standard cl-ppcre:split or uiop:split-string would work here, but this implementation demonstrates manual string processing and is dependency-free.

HTTP Communication via curl

Rather than using Common Lisp HTTP libraries (which can have SSL/Certificate issues), this project invokes curl directly:

 1 (defun run-curl-command-with-json (url json-payload)
 2   (let ((temp-file (uiop:run-program "mktemp" :output :string)))
 3     (setf temp-file (string-trim '(#\Space #\Tab #\Newline #\Return) temp-file))
 4     (with-open-file (stream temp-file :direction :output :if-exists :supersede)
 5       (write-string json-payload stream))
 6     (let* ((curl-cmd
 7             (format nil "curl -s -X POST \"~A\" -H \"Content-Type: application/json\" -d @~A"
 8                     url temp-file))
 9            (response-body
10             (multiple-value-bind (output error-output exit-code)
11                 (uiop:run-program curl-cmd :output :string :error-output :string :ignore-error-status t)
12               (declare (ignore exit-code error-output))
13               output)))
14       (uiop:run-program (format nil "rm -f ~A" temp-file) :ignore-error-status t)
15       response-body)))

Key technique: JSON payload is written to a temporary file, then passed to curl with -d @filename. This avoids complex shell escaping issues with embedded quotes and special characters.

The flow:

  1. Create temp file with mktemp
  2. Write JSON payload to file
  3. Execute curl with @filename syntax
  4. Clean up temp file
  5. Return response body

Gemini API Integration

The main API function constructs requests and parses responses:

 1 (defun get-gemini-chat-completion (user-prompt)
 2   "Sends a prompt to the Gemini API and returns the content of the response."
 3   (let* ((api-key (uiop:getenv "GEMINI_API_KEY"))
 4          (model-name "models/gemini-3-flash-preview")
 5          (base-url (format nil "https://generativelanguage.googleapis.com/v1beta/~A:generateContent?key=~A" 
 6                            model-name api-key))
 7          (payload
 8            `((:contents . ,(vector
 9                             `((:parts . ,(vector `((:text . ,user-prompt)))))))))
10          (json-payload (json:encode-json-to-string payload)))
11     
12     (unless (and api-key (not (string= api-key "")))
13       (error "GEMINI_API_KEY environment variable is not set."))
14 
15     (handler-case
16         (let* ((response-body (run-curl-command-with-json base-url json-payload))
17                (parsed-response (json:decode-json-from-string response-body))
18                (candidates (cdr (assoc :candidates parsed-response))))
19           (if candidates
20               (let* ((first-candidate (first candidates))
21                      (content (cdr (assoc :content first-candidate)))
22                      (parts (cdr (assoc :parts content)))
23                      (first-part (first parts))
24                      (text (cdr (assoc :text first-part))))
25                 text)
26               (progn
27                 (format *error-output* "~%[Error] Gemini API response did not contain candidates: ~A~%" 
28                         parsed-response)
29                 nil)))
30       (error (e)
31         (format *error-output* "An unexpected error occurred: ~A~%" e)
32         nil))))

API Structure Notes:

  • Backquote (`) is used for template construction, with comma (,) for evaluation
  • Gemini expects a nested JSON structure: contents -> parts -> text
  • json:encode-json-to-string converts Lisp data to JSON
  • json:decode-json-from-string parses the response
  • handler-case provides error handling, returning nil on failure

Response parsing pattern: The Gemini response has nested associative lists (alists). Navigate with:

1 (cdr (assoc :key alist))  ; Get value for :key
2 (first list)              ; Get first element
3 (cdr ...)                 ; Continue navigating

Interactive UI Loop

The kbn-ui function implements an interactive text user interface:

 1 (defun kbn-ui ()
 2   "Main user interface loop for Knowledge Base Navigator powered by Gemini."
 3   (let ((prompt ""))
 4     (loop
 5       (format t "~%============= GEMINI KNOWLEDGE BASE NAVIGATOR =============~%")
 6       (format t "~%Enter entity names separated by space or a descriptive sentence (or type 'quit' to exit):~%> ")
 7       (finish-output)
 8       (setf prompt (read-line))
 9       
10       (when (or (string-equal prompt "quit") (string-equal prompt "q"))
11         (format t "Goodbye!~%")
12         (return))
13         
14       (when (> (length prompt) 0)
15         (format t "~%[Extracting entities using Gemini 3 Flash...]~%")
16         ;; ... entity extraction and detail retrieval
17         ))))

Key loop patterns:

  • loop with (return) for controlled exit
  • read-line for user input
  • finish-output ensures prompts display immediately
  • when guards for conditional execution

The entity extraction creates a prompt that instructs Gemini to return ONLY a numbered list:

1 (format nil "Analyze the following user text: \"~A\".~
2 Identify potential encyclopedic entities (people, companies, countries, cities, products, concepts, etc.) mentioned.~
3 Categorize them if necessary. Return them as a neatly formatted numbered list (1., 2., 3., etc.) with a short 1-sentence description for each.~
4 DO NOT return any other conversational text, ONLY the numbered list so the user can see their options." prompt)

Prompt engineering principle: Be explicit about output format. The instruction to return ONLY the numbered list prevents verbose conversational responses.

User selection handling:

1 (let* ((tokens (tokenize-string selection-line))
2        (valid-tokens (remove-if-not #'(lambda (s) (every #'digit-char-p s)) tokens))
3        (indices (mapcar #'parse-integer valid-tokens)))
4   ;; Process indices...
5 )

This filters non-numeric input and converts valid selections to integers.

Running the Application

Load the system via Quicklisp:

 1 ;; Ensure project directory is in Quicklisp's path
 2 ;; Usually via symlink:
 3 ;;  ln -s /path/to/knowledge-base-navigator ~/quicklisp/local-projects/
 4 ;; or by adding the current directory (or a parent directory)
 5 ;; to ql:*local-project-directories*
 6 
 7 ;; Load system
 8 (ql:quickload "knowledge-base-navigator")
 9 
10 ;; Start UI
11 (knowledge-base-navigator:kbn-ui)

Example Session

 1 ============= GEMINI KNOWLEDGE BASE NAVIGATOR =============
 2 
 3 Enter entity names separated by space or a descriptive sentence (or type 'quit' to exit):
 4 > Bill Gates and Microsoft
 5 
 6 [Extracting entities using Gemini 3 Flash...]
 7 
 8 --- IDENTIFIED ENTITIES ---
 9 1. Bill Gates: An American business magnate, software developer, and philanthropist who co-founded Microsoft Corporation.
10 2. Microsoft: A multinational technology corporation that develops, manufactures, and licenses computer software.
11 ---------------------------
12 
13 Enter the numbers of the entities you want detailed information for (space separated):
14 > 1 2
15 
16 [Fetching detailed facts and relationships for selected entities...]
17 
18 === BILL GATES ===
19 * Born: October 28, 1955, Seattle, Washington
20 * Occupation: Business magnate, investor, philanthropist
21 * Net Worth: ~$120 billion (as of 2024)
22 * Founded Microsoft in 1975 with Paul Allen
23 
24 === MICROSOFT ===
25 * Founded: April 4, 1975
26 * Headquarters: Redmond, Washington
27 * Industry: Technology, Software, Cloud Computing
28 * Revenue: $211 billion (2023)
29 
30 === RELATIONSHIP ===
31 Bill Gates co-founded Microsoft with Paul Allen in 1975. He served as CEO until 2000 and remained Chairman until 2014. Microsoft was the primary source of Gates' wealth.

Key Takeaways

  1. Package Organization: Use ASDF systems with separate files for package definitions
  2. External Process Integration: uiop:run-program provides robust external command execution
  3. JSON Handling: cl-json encodes Lisp data structures to JSON and decodes responses
  4. Error Handling: Use handler-case for robust API error handling
  5. Prompt Engineering: Construct clear prompts with explicit format instructions
  6. REPL Interfaces: loop + read-line + format create simple but effective CLIs

Dependencies

Library Purpose
cl-json JSON encoding/decoding
uiop System utilities, process execution
alexandria Common Lisp utilities

Environment Setup

 1 # Set API key before starting Lisp
 2 export GEMINI_API_KEY="your_api_key_here"
 3 
 4 # Start SBCL
 5 sbcl
 6 
 7 # Load Quicklisp
 8 (quicklisp:setup)
 9 
10 # Load project
11 (ql:quickload "knowledge-base-navigator")

This example demonstrates that Common Lisp remains highly relevant for modern API-driven applications. The combination of powerful REPL development, mature libraries, and straightforward process integration makes it an excellent choice for exploratory programming and rapid prototyping.

Up next

Interfacing with External Programs: A Lightpanda Browser Client