More Agents Using X’s Grok and Perplexity APIs
One of the joys of working in Common Lisp is how naturally it lends itself to building extensible agent systems. We can represent knowledge symbolically, apply reasoning rules, and integrate procedural code, all within one coherent runtime. Over the years, I’ve experimented with many ways of connecting Lisp-based reasoning systems to external AI services, from symbolic logic engines to modern large language models (LLMs). In this chapter, we’ll take that a step further by exploring two complementary APIs that allow our Lisp agents to both reason and learn from the world in real time: X’s Grok LLM and the Perplexity Web Search API.
Why Grok?
Grok, the LLM developed and maintained by X (formerly Twitter), provides a conversational and reasoning-capable API similar to other large language models but with a twist: it’s designed for real-time access to context and access to current data through X’s ecosystem. While Grok is still an evolving platform, it’s particularly interesting to Lisp developers because it can be treated as a remote reasoning component, an external “mind” that our Lisp agent can consult for pattern completion, text summarization, or general problem solving.
In the first example of this chapter, we’ll look at the simplest possible integration: a Lisp program that sends a prompt to the Grok API and uses a single, very basic tool: a Lisp function called get_current_date. The tool simply returns the current date and time in a human-readable format. While this might seem trivial, it serves an example to demonstrate how to:
- Define a Lisp-side function as an external tool the model can call.
- Serialize and pass structured information between Lisp and Grok.
- Maintain conversational context between model invocations.
This minimal setup provides a foundation for richer tool-using agents later on. Once the pattern is clear, a model prompt, tool definition, and response interpretation, then we can add more tools or swap in different LLM backends without changing the surrounding Lisp logic.
From Reasoning to Knowledge: Adding Perplexity Search
The second example expands the system into something more dynamic. Instead of relying solely on the Grok model’s internal knowledge, we connect to the Perplexity API, which acts as an intelligent web search layer. Perplexity’s model performs real-time retrieval from the web, returning concise, cited answers. Combined with Grok, this gives our Lisp agent two distinct reasoning modalities:
- Generative reasoning through Grok: language understanding, summarization, creative or speculative reasoning.
- Retrieval reasoning through Perplexity: grounded, factual responses based on live web content.
This dual setup mirrors the way human researchers work: we think abstractly, but we also look things up. By orchestrating these two APIs from Lisp, we can build an agent that decides when to “ask” Grok for interpretation versus when to query Perplexity for up-to-date information. The Lisp runtime remains the central coordinator, maintaining context and deciding when and how to merge results.
A Unified Lisp Interface for Multiple Cognitive Modes
In both examples, the Common Lisp code will share a similar structure. We’ll define a small framework for:
- Representing API requests and responses as Lisp objects.
- Managing authentication and HTTP requests.
- Logging and tracing agent conversations for debugging and reuse.
The goal isn’t to build a full tool calling agent abstraction layer for all LLM APIs, but to provide a reusable pattern for experimentation. Lisp’s macro system, combined with its symbolic data structures, makes it easy to treat prompts and API calls as first-class objects, allowing us to script agent workflows that feel like extensions of the language itself.
Agent Using X’s Grok API
The program’s architecture is centered around a few key components that manage the agent’s capabilities and state. Configuration for the Grok API is handled by the global variable X_GROK_API_KEY, while the agent’s extensible skills are stored in the tools hash table. This hash table serves as a registry, mapping tool names to their description, parameter schema, and the actual Lisp function that implements the tool’s logic. The def-tool macro provides a clean, declarative syntax for populating this registry, abstracting away the JSON schema details required by the API and making it simple for developers to add new capabilities. A small helper function, hash, is also included to simplify the creation of nested hash tables that are later serialized into the JSON format expected by the Grok service.
The agent’s operational logic resides in the run-agent function, which implements a conversational loop. It begins by constructing an initial list of messages, including the user’s query and an optional system prompt. In each iteration of the loop, it calls call-grok-chat, which sends the current conversation history and the list of available tools to the Grok API. The agent then inspects the model’s response. If the model’s finish_reason indicates it wants to call a tool, the code extracts the tool name and arguments, invokes the corresponding Lisp function via execute-tool, and appends the tool’s output to the message history. This new history is then sent back to the model in the next loop iteration. This cycle continues until the model generates a final textual answer, at which point the loop terminates and returns the result.
File agent.lisp:
1 ;;;; agent-system.lisp
2 ;;;; A Common Lisp agent system using Grok API with support for tool calling.
3 ;;;;
4 ;;;; Dependencies (load via Quicklisp):
5 ;;;; (ql:quickload '(:drakma :yason :alexandria :uiop :cl+ssl))
6 ;;;;
7 ;;;; Usage:
8 ;;;; Set *grok-api-key* to your xAI Grok API key.
9 ;;;; Define custom tools using def-tool.
10 ;;;; Run (run-agent "Your query here")
11 ;;;;
12 ;;;; Note: This assumes Grok API is compatible with OpenAI-style chat completions.
13
14 (in-package :cl-user)
15
16 (ql:quickload '(:drakma :yason :alexandria :uiop :cl+ssl))
17
18 ;; Required libraries
19 (require 'asdf)
20 (require 'uiop)
21
22 ;; Configure YASON to handle symbol keys & values
23 (setf yason:*symbol-encoder* #'yason:encode-symbol-as-string)
24
25 (defvar *grok-api-key*
26 (uiop:getenv "X_GROK_API_KEY")
27 "Your xAI Grok API key. Obtain from https://x.ai/api")
28
29 (defvar *grok-base-url* "https://api.x.ai/v1"
30 "Base URL for Grok API.")
31
32 (defvar *tools* (make-hash-table :test 'equal)
33 "Hash table of tools: name -> (description parameters lisp-function)")
34
35 (defun hash (&rest pairs)
36 "Helper to create hash-table from pairs. Converts symbol or keyword keys to lowercase strings so YASON sees only string keys."
37 (let ((ht (make-hash-table :test 'equal)))
38 (loop for (k v) on pairs by #'cddr
39 for key = (if (symbolp k)
40 (string-downcase (symbol-name k))
41 k)
42 do (setf (gethash key ht) v))
43 ht))
44
45 (defmacro def-tool (name description parameters lisp-function)
46 "Define a custom tool."
47 `(setf (gethash ,name *tools*)
48 (list ,description ,parameters ,lisp-function)))
49
50 ;; Example custom tool: get current date
51 (def-tool "get_current_date"
52 "Get the current date in YYYY-MM-DD format."
53 (hash :type "object" :properties (hash) :required #())
54 (lambda (args)
55 (declare (ignore args))
56 (multiple-value-bind (s m h d mo y) (decode-universal-time (get-universal-time))
57 (declare (ignore s m h))
58 (format nil "~4,'0d-~2,'0d-~2,'0d" y mo d))))
59
60 ;; Function to get tools in API format
61 (defun get-tools ()
62 "Return list of tool schemas for API."
63 (loop for name being the hash-keys of *tools*
64 collect (destructuring-bind (desc params fn)
65 (gethash name *tools*)
66 (declare (ignore fn))
67 (hash "type" "function"
68 "function" (hash "name" name
69 "description" desc
70 "parameters" params)))))
71
72 (defun call-grok-chat (messages &key (model "grok-4") tools)
73 (let ((body (hash "model" model
74 "messages" messages
75 "stream" yason:false)))
76 (when tools (setf (gethash "tools" body) tools))
77 (let* ((json-body (with-output-to-string (s) (yason:encode body s)))
78 (status nil) (raw nil))
79 (multiple-value-setq (raw status)
80 (drakma:http-request
81 (concatenate 'string *grok-base-url* "/chat/completions")
82 :method :post
83 :additional-headers
84 `(("Authorization" . ,(concatenate 'string "Bearer " *grok-api-key*)))
85 :content json-body
86 :content-type "application/json"
87 :verify nil))
88 (unless (= status 200)
89 (error "Grok API returned status ~a: ~a" status raw))
90 (let* ((body-str (if (vectorp raw)
91 (babel:octets-to-string raw :encoding :utf-8)
92 raw))
93 (parsed (yason:parse body-str)))
94 parsed))))
95
96 (defun execute-tool (tool-call)
97 "Execute a tool call and return the result string (or hash) from the invoked tool."
98 (let* ((function-info (gethash "function" tool-call))
99 (name (gethash "name" function-info))
100 (args-raw (gethash "arguments" function-info))
101
102 ;; Force ARGS-JSON to a true simple-string
103 (args-json
104 (cond
105 ;; Character vector → simple-string
106 ((and (vectorp args-raw) (every #'characterp args-raw))
107 (coerce args-raw 'simple-string))
108
109 ;; Already a string → coerce to simple-string to drop any adjustable/ fill‑pointer baggage
110 ((stringp args-raw)
111 (coerce args-raw 'simple-string))
112
113 ;; Octet vector → decode UTF‑8
114 ((vectorp args-raw)
115 (babel:octets-to-string args-raw :encoding :utf-8))
116
117 (t
118 (error "Unexpected arguments payload type: ~s" (type-of args-raw)))))
119
120 (tool-info (gethash name *tools*)))
121 ;; DEBUG PRINTS ----------------------------------------------------------
122 (format t "~&[execute-tool] name=~a args-raw type=~a~%" name (type-of args-raw))
123 (cond
124 ((stringp args-raw)
125 (format t "[execute-tool] first 32 chars: ~a~%"
126 (subseq args-raw 0 (min 32 (length args-raw)))))
127 ((and (vectorp args-raw) (not (stringp args-raw)))
128 (format t "[execute-tool] first 16 bytes: ~{~d~^ ~}~%"
129 (subseq args-raw 0 (min 16 (length args-raw))))))
130 (format t "[execute-tool] args-json final type=~a first 32: ~a~%"
131 (type-of args-json)
132 (subseq args-json 0 (min 32 (length args-json))))
133 ;; ----------------------------------------------------------------------
134 (let* ((args (yason:parse args-json)))
135 (if tool-info
136 (let ((fn (third tool-info)))
137 (funcall fn args))
138 (error "Unknown tool: ~s" name)))))
139
140 (defun run-agent (query &key (model "grok-4") (system-prompt "You are a helpful agent that can use tools to answer questions."))
141 "Run the agent loop for a query."
142 (let ((messages (if system-prompt
143 (list (hash "role" "system" "content" system-prompt)
144 (hash "role" "user" "content" query))
145 (list (hash "role" "user" "content" query))))
146 (tools (get-tools)))
147 (loop
148 (let ((response (call-grok-chat messages :model model :tools tools)))
149 (let* ((choice (first (gethash "choices" response)))
150 (message (gethash "message" choice))
151 (finish-reason (gethash "finish_reason" choice)))
152 (push message messages) ;; Add assistant message to history
153 (cond
154 ;; Tool invocation (either explicit finish_reason or implicit
155 ;; via presence of tool_calls)
156 ((or (member finish-reason '("tool_calls" "tool_call") :test #'equal)
157 (gethash "tool_calls" message))
158 (let ((tool-calls (gethash "tool_calls" message)))
159 (dolist (tool-call tool-calls)
160 (let* ((result (execute-tool tool-call))
161 (tool-response (hash "role" "tool"
162 "tool_call_id" (gethash "id" tool-call)
163 "name" (gethash "name" (gethash
164 "function"
165 tool-call))
166 "content" result)))
167 (push tool-response messages)))))
168
169 ;; Conversation finished
170 ((or (equal finish-reason "stop")
171 ;; finish_reason NIL/"" --> stop only if no tool_calls present
172 (and (or (null finish-reason) (equal finish-reason ""))
173 (not (gethash "tool_calls" message))))
174 (return (gethash "content" message)))
175
176 (t
177 (error "Unknown finish reason: ~s" finish-reason))))))))
178
179 (trace call-grok-chat)
180 (trace execute-tool)
181 (trace get-tools)
182
183 ;; (run-agent "what is 1 + 12?")
184 ;; (run-agent "Consultant Mark Watson has written books on AI, Lisp, and the semantic web. What musical instruments does Mark play?")
This program provides a functional and concise foundation for building intelligent agents that can take action in the world. By combining the classic strengths of Common Lisp with the modern capabilities of the Grok API, it demonstrates a powerful pattern for creating tool-augmented AI systems. The def-tool macro, in particular, offers a clear path for extension, allowing a developer to easily equip the agent with a wide array of custom functions, from interacting with databases and other APIs to controlling local system processes. This example serves mostly as a demonstration, but if customized for your agent requirements it can also be a robust starting point for developing more sophisticated and specialized AI applications in Lisp.
Let’s run the two examples at the bottom of the last listing:
1 CL-USER 1 > (load "agent.lisp")
2 CL-USER 2 > (run-agent "what is 1 + 12?")
3 0 GET-TOOLS > ...
4 0 GET-TOOLS < ...
5 << VALUE-0 : (#<EQUAL Hash Table{2} 801002623B>)
6 0 CALL-GROK-CHAT > ...
7 >> MESSAGES : (#<EQUAL Hash Table{2} 8010017C03> #<EQUAL Hash Table{2} 801001A013>)
8 >> MODEL : "grok-4"
9 >> TOOLS : (#<EQUAL Hash Table{2} 801002623B>)
10 0 CALL-GROK-CHAT < ...
11 << VALUE-0 : #<EQUAL Hash Table{7} 801009F0E3>
12 "13"
13 T
14
15 CL-USER 3 > (run-agent "Consultant Mark Watson has written books on AI, Lisp, and the semantic web.")
16 0 GET-TOOLS > ...
17 0 GET-TOOLS < ...
18 << VALUE-0 : (#<EQUAL Hash Table{2} 80100AF723>)
19 0 CALL-GROK-CHAT > ...
20 >> MESSAGES : (#<EQUAL Hash Table{2} 80100A23CB> #<EQUAL Hash Table{2} 80100A4763>)
21 >> MODEL : "grok-4"
22 >> TOOLS : (#<EQUAL Hash Table{2} 80100AF723>)
23 0 CALL-GROK-CHAT < ...
24 << VALUE-0 : #<EQUAL Hash Table{7} 80100CFF4B>
25 "Yes, that's accurate! Mark Watson is a software consultant and author with a strong background in artificial intelligence, programming languages like Common Lisp, and technologies such as the Semantic Web. Some of his notable books include:
26
27 - **Loving Common Lisp, or the Savvy Programmer's Secret Weapon** (on Lisp programming).
28 - **Practical Semantic Web and Linked Data Applications** (focusing on Semantic Web technologies).
29 - **Practical Artificial Intelligence Programming With Java** (covering AI concepts).
30
31 He's written over 20 books in total, often emphasizing practical, hands-on approaches to these subjects. If you're interested in recommendations, specific book details, or more about his work, let me know!"
32 T
33
34 CL-USER 4 >
Here we are using the innate knowledge in X’s Grok model.
Agent Using X’s Grok API and Perplexity’s Search API
Here we extend the example in the last section to use a web search tool implemented with Perplexity’s web search API.
The architecture of this example tool using agent is centered around the run-agent function, which implements the core reasoning loop. It begins by sending the user’s query and a list of available tools to the Grok API. The program then inspects the API response’s finish_reason. If Grok determines a tool is needed, the reason will be in tool_calls, and the response will contain the name of the tool to execute and the arguments to use. The execute-tool function then dispatches to the appropriate local Lisp function. The tool’s output is then packaged into a new message and sent back to Grok, continuing the loop. This cycle repeats until Grok has sufficient information and returns a finish_reason of stop, at which point it delivers its final synthesized answer to the user.
The system’s extensibility is handled by the def-tool macro that creates a simple domain-specific language for adding new tools with specified capabilities. To define a new tool, a developer provides its name, a natural language description for the LLM to understand its purpose, a JSON schema for its parameters, and the Lisp lambda function that performs the actual work. The web_search tool is an example, as it acts as a bridge to another AI service, Perplexity. Instead of performing a raw web search, it effectively asks the Perplexity Sonar model to answer the query, ensuring the result returned to Grok is a concise, relevant summary. This demonstrates a powerful pattern of chaining specialized AI models together within a single agentic framework. Communication with the external APIs is managed by the Drakma library for HTTP requests and the YASON library for handling the necessary JSON serialization and parsing.
This agent example is a work in progress and currently running the agent results in hundreds of lines of debug printout.
File agent_grok_perplexity.lisp:
1 ;;;; agent-system.lisp
2 ;;;; A Common Lisp agent system using Grok API with support for tool calling.
3 ;;;; Optionally uses Perplexity Sonar API for web search tool.
4 ;;;;
5 ;;;; Dependencies (load via Quicklisp):
6 ;;;; (ql:quickload '(:drakma :yason :alexandria :uiop :cl+ssl))
7 ;;;;
8 ;;;; Usage:
9 ;;;; Set *grok-api-key* to your xAI Grok API key.
10 ;;;; Optionally set *perplexity-api-key* for web search support.
11 ;;;; Define custom tools using def-tool.
12 ;;;; Run (run-agent "Your query here")
13 ;;;;
14 ;;;; Note: This assumes Grok API is compatible with OpenAI-style chat completions.
15
16 (in-package :cl-user)
17
18 (ql:quickload '(:drakma :yason :alexandria :uiop :cl+ssl))
19
20 ;; Required libraries
21 (require 'asdf)
22 (require 'uiop)
23
24 ;; Configure YASON to handle symbol keys & values
25 (setf yason:*symbol-encoder* #'yason:encode-symbol-as-string)
26
27 (defvar *grok-api-key*
28 (uiop:getenv "X_GROK_API_KEY")
29 "Your xAI Grok API key. Obtain from https://x.ai/api")
30
31 (defvar *perplexity-api-key* (uiop:getenv "PERPLEXITY_API_KEY")
32 "Optional Perplexity AI API key for web search. If nil, web_search tool will error.")
33
34 (defvar *grok-base-url* "https://api.x.ai/v1"
35 "Base URL for Grok API.")
36
37 (defvar *perplexity-base-url* "https://api.perplexity.ai"
38 "Base URL for Perplexity API.")
39
40 (defvar *tools* (make-hash-table :test 'equal)
41 "Hash table of tools: name -> (description parameters lisp-function)")
42
43 (defun hash (&rest pairs)
44 "Helper to create hash-table from pairs. Converts symbol or keyword keys to lowercase strings so YASON sees only string keys."
45 (let ((ht (make-hash-table :test 'equal)))
46 (loop for (k v) on pairs by #'cddr
47 for key = (if (symbolp k)
48 (string-downcase (symbol-name k))
49 k)
50 do (setf (gethash key ht) v))
51 ht))
52
53 (defun pp-hash (ht &optional (stream *standard-output*) (indent 0))
54 "Pretty-print hash table HT to STREAM, indenting by INDENT spaces."
55 (let ((keys (loop for k being the hash-keys of ht collect k)))
56 (format stream "~&~v@{~}" indent "") ; indent
57 (format stream "#HASH{~%")
58 (let ((next-indent (+ indent 2)))
59 (dolist (k keys)
60 (let ((v (gethash k ht)))
61 (format stream "~v@{~}" next-indent "")
62 (format stream "~S => ~S~%" k v)))
63 (format stream "~v@{~}" indent "")
64 (format stream "}") )
65 ht))
66
67 (defmacro def-tool (name description parameters lisp-function)
68 "Define a custom tool."
69 `(setf (gethash ,name *tools*)
70 (list ,description ,parameters ,lisp-function)))
71
72 ;; Example tools
73
74 (defvar *x* nil) ;; DEBUG
75
76 ;; Web search tool using Perplexity (optional)
77 (def-tool "web_search"
78 "Search the web for up-to-date information when needed. Use this for current events or real-time data."
79 (hash :type "object"
80 :properties (hash "query" (hash :type "string"
81 :description "The search query string."))
82 :required (list "query"))
83 (lambda (args)
84 (if *perplexity-api-key*
85 (let* ((query (gethash "query" args))
86 (messages (list (hash "role" "system"
87 "content" "You are a helpful search assistant. Provide a concise answer based on web search.")
88 (hash "role" "user"
89 "content" query)))
90 (body (hash "model" "sonar"
91 "messages" messages
92 "max_tokens" 1024
93 "temperature" 0.7))
94 (json-body (with-output-to-string (s) (yason:encode body s)))
95 (raw nil) (status nil))
96 ;; Call Perplexity
97 (multiple-value-setq (raw status)
98 (drakma:http-request
99 (concatenate 'string *perplexity-base-url* "/chat/completions")
100 :method :post
101 :additional-headers `(("Authorization" . ,(concatenate 'string "Bearer " *perplexity-api-key*))
102 ("Content-Type" . "application/json"))
103 :content json-body
104 :verify nil))
105 ;; Convert to string if octet‑vector
106 (let* ((body-str (if (vectorp raw)
107 (babel:octets-to-string raw :encoding :utf-8)
108 raw)))
109 ;; DEBUG
110 (format t "~&[web_search] Perplexity status=~a~%" status)
111 (format t "[web_search] First 8192 chars: ~a~%" (subseq body-str 0 (min 8192 (length body-str))))
112 ;; Handle non-200 errors
113 (unless (= status 200)
114 (return
115 (format nil "Web search failed (HTTP ~a): ~a" status body-str)))
116 ;; Parse JSON
117 (setf *x* (yason:parse body-str)) ;; DEBUG ONLY
118 (let* ((parsed (ignore-errors (yason:parse body-str)))
119 (choices (and (hash-table-p parsed) (gethash "choices" parsed))))
120 (format t "~%[web_search] choices=~%~A~%" choices)
121 (cond
122 ((and choices (plusp (length choices)))
123 (let* ((choice (first choices))
124 (msg (and (hash-table-p choice) (gethash "message" choice)))
125 (content (and (hash-table-p msg) (gethash "content" msg))))
126 (format t "~%[web_search] content=~%~A~%" content)
127 content)))))))))
128
129 ;; Example custom tool: get current date
130 (def-tool "get_current_date"
131 "Get the current date in YYYY-MM-DD format."
132 (hash :type "object" :properties (hash) :required #())
133 (lambda (args)
134 (declare (ignore args))
135 (multiple-value-bind (s m h d mo y) (decode-universal-time (get-universal-time))
136 (declare (ignore s m h))
137 (format nil "~4,'0d-~2,'0d-~2,'0d" y mo d))))
138
139 ;; Function to get tools in API format
140 (defun get-tools ()
141 "Return list of tool schemas for API."
142 (loop for name being the hash-keys of *tools*
143 collect (destructuring-bind (desc params fn)
144 (gethash name *tools*)
145 (declare (ignore fn))
146 (hash "type" "function"
147 "function" (hash "name" name
148 "description" desc
149 "parameters" params)))))
150
151 (defun call-grok-chat (messages &key (model "grok-4") tools)
152 (let ((body (hash "model" model
153 "messages" messages
154 "stream" yason:false)))
155 (when tools (setf (gethash "tools" body) tools))
156 (let* ((json-body (with-output-to-string (s) (yason:encode body s)))
157 (status nil) (raw nil))
158 (multiple-value-setq (raw status)
159 (drakma:http-request
160 (concatenate 'string *grok-base-url* "/chat/completions")
161 :method :post
162 :additional-headers
163 `(("Authorization" . ,(concatenate 'string "Bearer " *grok-api-key*)))
164 :content json-body
165 :content-type "application/json"
166 :verify nil))
167 (unless (= status 200)
168 (error "Grok API returned status ~a: ~a" status raw))
169 (let* ((body-str (if (vectorp raw)
170 (babel:octets-to-string raw :encoding :utf-8)
171 raw))
172 (parsed (yason:parse body-str)))
173 parsed))))
174
175 (defun execute-tool (tool-call)
176 "Execute a tool call and return the result string (or hash) from the invoked tool."
177 (let* ((function-info (gethash "function" tool-call))
178 (name (gethash "name" function-info))
179 (args-raw (gethash "arguments" function-info))
180
181 ;; Force ARGS-JSON to a true simple-string
182 (args-json
183 (cond
184 ;; Character vector --> simple-string
185 ((and (vectorp args-raw) (every #'characterp args-raw))
186 (coerce args-raw 'simple-string))
187
188 ;; Already a string --> coerce to simple-string to drop any
189 ;; adjustable/fill-pointer baggage
190 ((stringp args-raw)
191 (coerce args-raw 'simple-string))
192
193 ;; Octet vector → decode UTF‑8
194 ((vectorp args-raw)
195 (babel:octets-to-string args-raw :encoding :utf-8))
196
197 (t
198 (error "Unexpected arguments payload type: ~s" (type-of args-raw)))))
199
200 (tool-info (gethash name *tools*)))
201 ;; DEBUG PRINTS ----------------------------------------------------------
202 (format t "~&[execute-tool] name=~a args-raw type=~a~%" name (type-of args-raw))
203 (cond
204 ((stringp args-raw)
205 (format t "[execute-tool] first 32 chars: ~a~%"
206 (subseq args-raw 0 (min 32 (length args-raw)))))
207 ((and (vectorp args-raw) (not (stringp args-raw)))
208 (format t "[execute-tool] first 16 bytes: ~{~d~^ ~}~%"
209 (subseq args-raw 0 (min 16 (length args-raw))))))
210 (format t "[execute-tool] args-json final type=~a first 32: ~a~%"
211 (type-of args-json)
212 (subseq args-json 0 (min 32 (length args-json))))
213 ;; ----------------------------------------------------------------------
214 (let* ((args (yason:parse args-json)))
215 (if tool-info
216 (let ((fn (third tool-info)))
217 (funcall fn args))
218 (error "Unknown tool: ~s" name)))))
219
220 (defun run-agent (query &key (model "grok-4") (system-prompt "You are a helpful agent that can use tools to answer questions."))
221 "Run the agent loop for a query."
222 (let ((messages (if system-prompt
223 (list (hash "role" "system" "content" system-prompt)
224 (hash "role" "user" "content" query))
225 (list (hash "role" "user" "content" query))))
226 (tools (get-tools)))
227 (loop
228 (let ((response (call-grok-chat messages :model model :tools tools)))
229 (let* ((choice (first (gethash "choices" response)))
230 (message (gethash "message" choice))
231 (finish-reason (gethash "finish_reason" choice)))
232 (push message messages) ;; Add assistant message to history
233 (cond
234 ;; Tool invocation (either explicit finish_reason or implicit
235 ;; via presence of tool_calls)
236 ((or (member finish-reason '("tool_calls" "tool_call") :test #'equal)
237 (gethash "tool_calls" message))
238 (let ((tool-calls (gethash "tool_calls" message)))
239 (dolist (tool-call tool-calls)
240 (let* ((result (execute-tool tool-call))
241 (tool-response (hash "role" "tool"
242 "tool_call_id" (gethash "id" tool-call)
243 "name" (gethash "name" (gethash
244 "function"
245 tool-call))
246 "content" result)))
247 (push tool-response messages)))))
248
249 ;; Conversation finished
250 ((or (equal finish-reason "stop")
251 ;; finish_reason NIL/"" → stop only if no tool_calls present
252 (and (or (null finish-reason) (equal finish-reason ""))
253 (not (gethash "tool_calls" message))))
254 (return (gethash "content" message)))
255
256 (t
257 (error "Unknown finish reason: ~s" finish-reason))))))))
258
259 (trace call-grok-chat)
260 (trace execute-tool)
261 (trace get-tools)
262
263 ;; (run-agent "what is 1 + 12?")
264 ;; (run-agent "Consultant Mark Watson has written books on AI, Lisp, and the semantic web. What musical instruments does Mark play? Return only a list of musical instruments.")
Let’s run an example that requires a web search since Grok’s innate knowledge can’t answer the query:
1 CL-USER 1 > (load "agent_grok_perplexity.lisp")
2 CL-USER 3 > (run-agent "Consultant Mark Watson has written books on AI, Lisp, and the semantic web. What musical instruments does Mark play? Return only a list of musical instruments.")
3 0 GET-TOOLS > ...
4 0 GET-TOOLS < ...
5 << VALUE-0 : (#<EQUAL Hash Table{2} 801002345B> #<EQUAL Hash Table{2} 801002A9DB>)
6 0 CALL-GROK-CHAT > ...
7 >> MESSAGES : (#<EQUAL Hash Table{2} 8010015043> #<EQUAL Hash Table{2} 80100173DB>)
8 >> MODEL : "grok-4"
9 >> TOOLS : (#<EQUAL Hash Table{2} 801002345B> #<EQUAL Hash Table{2} 801002A9DB>)
10 0 CALL-GROK-CHAT < ...
11 << VALUE-0 : #<EQUAL Hash Table{7} 80100A46EB>
12 0 EXECUTE-TOOL > ...
13 >> TOOL-CALL : #<EQUAL Hash Table{3} 80100A625B>
14 [execute-tool] name=web_search args-raw type=(ARRAY CHARACTER (80))
15 [execute-tool] first 32 chars: {"query":"Mark Watson AI Lisp se
16 [execute-tool] args-json final type=SIMPLE-TEXT-STRING first 32: {"query":"Mark Watson AI Lisp se
17 [web_search] Perplexity status=200
18 [web_search] First 8192 chars: {"id": "ecc7aa86-4434-4d70-8dfa-e85b7c7e27d6", "model": "sonar", "created": 1759702702, "usage": {"prompt_tokens": 27, "completion_tokens": 230, "total_tokens": 257, "search_context_size": "low", "cost": {"input_tokens_cost": 0.0, "output_tokens_cost": 0.0, "request_cost": 0.005, "total_cost": 0.005}}, "citations": ["https://leanpub.com/hy-lisp-python", "https://markwatson.com/llms.txt", "https://creativecommons.org/2005/07/01/watson/", "https://www.goodreads.com/author/list/5182666.Mark_Watson", "https://github.com/mark-watson/lisp_practical_semantic_web", "https://www.chessprogramming.org/Mark_Watson", "https://markwatson.com", "https://mark-watson.blogspot.com", "https://leanpub.com/u/markwatson", "https://markwatson.com/opencontent/book_lisp.pdf", "https://github.com/mark-watson/free-older-books-and-software"], "search_results": [{"title": "A Lisp Programmer Living in\u2026 by Mark Watson [PDF/iPad/Kindle]", "url": "https://leanpub.com/hy-lisp-python", "date": "2025-08-20", "last_updated": "2025-09-27", "snippet": "A Lisp Programmer Living in Python-Land: The Hy Programming Language. Use Hy with Large Language Models, Semantic Web, Web Scraping, Web Search, ...", "source": "web"}, {"title": "https://markwatson.com/llms.txt", "url": "https://markwatson.com/llms.txt", "date": null, "last_updated": "2025-10-05", "snippet": "Mark Watson's hobbies are cooking, photography, hiking, travel, and playing the following musical instruments: guitar, didgeridoo, and American Indian flute.", "source": "web"}, {"title": "Mark Watson - Creative Commons", "url": "https://creativecommons.org/2005/07/01/watson/", "date": "2005-07-01", "last_updated": "2024-09-26", "snippet": "Mark Watson is an accomplished programmer and writer of thirteen books on various technical topics. An expert in artificial intelligence and language ...", "source": "web"}, {"title": "Books by Mark Watson (Author of Loving Common Lisp ... - Goodreads", "url": "https://www.goodreads.com/author/list/5182666.Mark_Watson", "date": "2025-10-01", "last_updated": "2025-10-05", "snippet": "Mark Watson has 31 books on Goodreads with 379 ratings. Mark Watson's most popular book is Loving Common Lisp, or the Savvy Programmer's Secret Weapon.", "source": "web"}, {"title": "Examples from the Lisp version of my semantic web book - GitHub", "url": "https://github.com/mark-watson/lisp_practical_semantic_web", "date": "2011-10-23", "last_updated": "2025-06-13", "snippet": "Examples from the Lisp version of my semantic web book. markwatson.com \u00b7 37 stars 8 forks Branches Tags Activity.", "source": "web"}, {"title": "Mark Watson - Chessprogramming wiki", "url": "https://www.chessprogramming.org/Mark_Watson", "date": "2003-03-27", "last_updated": "2025-02-12", "snippet": "Mark Watson, an American computer scientist, programmer, consultant and author of books on Artificial Intelligence, Java, Ruby, Common LISP, Semantic Web, NLP, ...", "source": "web"}, {"title": "Mark Watson: AI Practitioner and Author of 20+ AI Books | Mark ...", "url": "https://markwatson.com", "date": "2023-04-18", "last_updated": "2025-10-05", "snippet": "I am the author of 20+ books on Artificial Intelligence, Python, Common Lisp, Deep Learning, Haskell, Clojure, Java, Ruby, Hy language, and the Semantic Web ...", "source": "web"}, {"title": "Mark Watson's artificial intelligence and Lisp hacking blog", "url": "https://mark-watson.blogspot.com", "date": "2025-07-16", "last_updated": "2025-10-05", "snippet": "I am a consultant and the author of 20+ books on artificial intelligence, machine learning, and the semantic web. 55 US patents. My favorite languages are ...", "source": "web"}, {"title": "Mark Watson - Leanpub", "url": "https://leanpub.com/u/markwatson", "date": null, "last_updated": "2025-10-05", "snippet": "He is the author of 20+ published books on Artificial Intelligence, Deep Learning, Java, Ruby, Machine Learning, Common LISP, Clojure, JavaScript, Semantic Web, ...", "source": "web"}, {"title": "[PDF] Practical Semantic Web and Linked Data Applications - Mark Watson", "url": "https://markwatson.com/opencontent/book_lisp.pdf", "date": "2010-11-03", "last_updated": "2025-09-24", "snippet": "The broader purpose of this book is to provide application programming examples using AllegroGraph and. Linked Data sources on the web. This ...", "source": "web"}, {"title": "GitHub - mark-watson/free-older-books-and-software", "url": "https://github.com/mark-watson/free-older-books-and-software", "date": "2023-05-09", "last_updated": "2025-02-21", "snippet": "Mark Watson: AI Practitioner and Consultant Specializing in Large Language Models, LangChain/Llama-Index Integrations, Deep Learning, and the Semantic Web.", "source": "web"}], "object": "chat.completion", "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "Mark Watson is an AI practitioner, author, and programmer specializing in Lisp, semantic web, and large language models. He is the author of numerous books on AI, Lisp, semantic web technologies, and programming languages. He has worked extensively on semantic web and linked data applications, including a Common Lisp version of his semantic web book, and he integrates AI tools like OpenAI GPT and LangChain in his work[1][5][6].\n\nRegarding musical instruments, Mark Watson plays the **guitar, didgeridoo, and American Indian flute** as part of his hobbies[2].\n\nIn summary:\n\n| Aspect | Details |\n|----------------------------|-------------------------------------------------------|\n| Profession | AI practitioner, Lisp programmer, semantic web author |\n| Key Contributions | Books on AI, Lisp, semantic web; projects using Lisp and AI |\n| Semantic Web Work | Practical Semantic Web and Linked Data Applications (Common Lisp and others) |\n| Musical Instruments Played | Guitar, didgeridoo, American Indian flute |\n\nThis information is based on Mark Watson's personal website, books, and profiles[1][2][5][6]."}, "delta": {"role": "assistant", "content": ""}}]}
19
20 [web_search] choices=
21 (#<EQUAL Hash Table{4} 80100DD84B>)
22
23 [web_search] content=
24 Mark Watson is an AI practitioner, author, and programmer specializing in Lisp, semantic web, and large language models. He is the author of numerous books on AI, Lisp, semantic web technologies, and programming languages. He has worked extensively on semantic web and linked data applications, including a Common Lisp version of his semantic web book, and he integrates AI tools like OpenAI GPT and LangChain in his work[1][5][6].
25
26 Regarding musical instruments, Mark Watson plays the **guitar, didgeridoo, and American Indian flute** as part of his hobbies[2].
27
28 In summary:
29
30 | Aspect | Details |
31 |----------------------------|-------------------------------------------------------|
32 | Profession | AI practitioner, Lisp programmer, semantic web author |
33 | Key Contributions | Books on AI, Lisp, semantic web; projects using Lisp and AI |
34 | Semantic Web Work | Practical Semantic Web and Linked Data Applications (Common Lisp and others) |
35 | Musical Instruments Played | Guitar, didgeridoo, American Indian flute |
36
37 This information is based on Mark Watson's personal website, books, and profiles[1][2][5][6].
38 0 EXECUTE-TOOL < ...
39 << VALUE-0 : "Mark Watson is an AI practitioner, author, and programmer specializing in Lisp, semantic web, and large language models. He is the author of numerous books on AI, Lisp, semantic web technologies, and programming languages. He has worked extensively on semantic web and linked data applications, including a Common Lisp version of his semantic web book, and he integrates AI tools like OpenAI GPT and LangChain in his work[1][5][6].
40
41 Regarding musical instruments, Mark Watson plays the **guitar, didgeridoo, and American Indian flute** as part of his hobbies[2].
42
43 In summary:
44
45 | Aspect | Details |
46 |----------------------------|-------------------------------------------------------|
47 | Profession | AI practitioner, Lisp programmer, semantic web author |
48 | Key Contributions | Books on AI, Lisp, semantic web; projects using Lisp and AI |
49 | Semantic Web Work | Practical Semantic Web and Linked Data Applications (Common Lisp and others) |
50 | Musical Instruments Played | Guitar, didgeridoo, American Indian flute |
51
52 This information is based on Mark Watson's personal website, books, and profiles[1][2][5][6]."
53 0 CALL-GROK-CHAT > ...
54 >> MESSAGES : (#<EQUAL Hash Table{4} 80100DE1A3> #<EQUAL Hash Table{4} 80100A5F73> #<EQUAL Hash Table{2} 8010015043> #<EQUAL Hash Table{2} 80100173DB>)
55 >> MODEL : "grok-4"
56 >> TOOLS : (#<EQUAL Hash Table{2} 801002345B> #<EQUAL Hash Table{2} 801002A9DB>)
57 0 CALL-GROK-CHAT < ...
58 << VALUE-0 : #<EQUAL Hash Table{7} 80100EA7C3>
59 "- Guitar
60 - Didgeridoo
61 - American Indian flute"
62 T
63
64 CL-USER 4 >
This tool using agent is a work in progress so I left debug output in place.