One Interface for Brave, Tavily, and Perplexity Web Search APIs
Note: this chapter replaces the separate Brave, Tavily, and Perplexity chapters from earlier editions. The three client libraries are now a single library, search-apis, in the directory loving-common-lisp/src/search_APIs.
A web search API lets an application send a query and receive web results without crawling or indexing pages itself. Several companies offer one. Brave, Tavily, and Perplexity are the three used in this book. Each works differently:
- Brave takes a GET request with the subscription key in a header and returns ranked links with snippets.
- Tavily takes a POST request with the key in the JSON body and returns ranked links with snippets and a relevance score.
- Perplexity runs a search and then asks a language model to answer the question from the pages it found, returning an answer with citations.
Writing one function per API spreads provider details across the whole application. This chapter builds one library that hides those details. Every provider is called through a single function, websearch, and every provider returns the same search-response structure. Adding a provider means registering one function, not editing the call sites.
Two Kinds of Search API
It helps to separate the providers into two groups before looking at the code.
The first group performs pure search. Brave and Tavily belong here. You send a query and get back a list of pages, each with a title, a URL, and a short text snippet. The snippet comes from the page or from the search engine’s summary of it.
The second group performs search plus language model processing. Perplexity belongs here. You send a query, Perplexity searches, and a language model writes an answer that cites the pages it used. You get both an answer and a list of sources.
The library models both groups with one structure. The results slot always holds the list of pages. The answer slot holds the synthesized answer when the provider produces one, and is nil for pure search providers. Callers that only want links read results and ignore answer.
The Shared Data Model
The whole point of the library is that all three providers produce the same two structures:
1 (defstruct search-result
2 title
3 url
4 content
5 (score nil))
6
7 (defstruct search-response
8 provider
9 query
10 answer
11 results
12 raw)
A search-result is one page. content holds the provider’s snippet and may be nil when a provider returns links only. score holds a provider-specific relevance value and is nil when the provider does not supply one.
A search-response is the result of one call. provider records which API answered, query records the question, answer holds the synthesized answer or nil, results holds the list of search-result values, and raw holds the full decoded JSON in case you need a field the structures do not expose.
To see where these fields come from, here is a shortened Brave response:
1 {
2 "web": {
3 "results": [
4 {
5 "title": "Visit Sedona | The official site of the Sedona Tourism Bureau",
6 "url": "https://visitsedona.com/",
7 "description": "The official site of the Sedona, AZ tourism bureau."
8 }
9 ]
10 }
11 }
Here is a shortened Tavily response. It carries the same title, url, and content fields, plus a score:
1 {
2 "query": "Sedona Arizona",
3 "results": [
4 {
5 "title": "Visit Sedona",
6 "url": "https://visitsedona.com/",
7 "content": "The official site of the Sedona, AZ tourism bureau.",
8 "score": 0.98
9 }
10 ]
11 }
Here is a shortened Perplexity response. The answer sits under choices, and the sources sit under citations and search_results:
1 {
2 "choices": [
3 { "message": { "content": "Sedona is in northern Arizona." } }
4 ],
5 "citations": ["https://visitsedona.com/", "https://www.sedonaaz.gov/"]
6 }
Each provider parses a different shape, and each parser writes the same search-response.
Source Code
The library has seven files. The ASDF system lists them in load order:
1 ;;; search-apis.asd
2
3 (asdf:defsystem #:search-apis
4 :description "One common web search interface across providers (Brave, Tavily, Perplexity)."
5 :author "Mark Watson"
6 :license "MIT"
7 :version "1.0.0"
8 :serial t
9 :depends-on (#:dexador #:quri #:uiop)
10 :components ((:file "package")
11 (:file "json")
12 (:file "search-apis")
13 (:file "providers")))
The dependency list is short. dexador makes the HTTP requests, quri URL-encodes the query string, and uiop reads environment variables. Everything else, including the JSON codec, lives in the library.
package.lisp
The package exports the entry point, the accessors for the two structures, the provider registry, and the condition hierarchy:
1 ;;; package.lisp
2
3 (defpackage #:search-apis
4 (:nicknames #:search_apis)
5 (:use #:cl)
6 (:export
7 #:websearch
8 #:search-result
9 #:search-result-title
10 #:search-result-url
11 #:search-result-content
12 #:search-result-score
13 #:search-response
14 #:search-response-provider
15 #:search-response-query
16 #:search-response-answer
17 #:search-response-results
18 #:search-response-raw
19 #:search-provider
20 #:search-provider-name
21 #:search-provider-base-url
22 #:search-provider-env-keys
23 #:define-search-provider
24 #:find-search-provider
25 #:search-error
26 #:api-error
27 #:api-error-status
28 #:api-error-body
29 #:authentication-error
30 #:rate-limit-error
31 #:not-found-error
32 #:json-encode
33 #:json-decode))
The nickname search_apis matches the directory name, so both search-apis:websearch and search_apis:websearch work.
json.lisp
Every provider speaks JSON, so the library carries its own codec instead of depending on a JSON library. It encodes Lisp alists, lists, strings, numbers, and the symbols t, :false, and :null, and it decodes JSON back into alists with string keys. Keys are converted between Lisp and JSON conventions: a keyword such as :max-results becomes the JSON key max_results.
1 ;;; json.lisp
2
3 (defun json-key (key)
4 "Convert a string or keyword to a JSON object key (downcased, - -> _)."
5 (etypecase key
6 (string key)
7 (symbol (substitute #\_ #\- (string-downcase (symbol-name key))))))
8
9 (defun %write-json-string (s out)
10 (write-char #\" out)
11 (loop for ch across s do
12 (case ch
13 (#\" (write-string "\\\"" out))
14 (#\\ (write-string "\\\\" out))
15 (#\Newline (write-string "\\n" out))
16 (#\Return (write-string "\\r" out))
17 (#\Tab (write-string "\\t" out))
18 (t (if (< (char-code ch) 32)
19 (format out "\\u~4,'0x" (char-code ch))
20 (write-char ch out)))))
21 (write-char #\" out))
22
23 (defun %write-json-float (f out)
24 (if (and (<= -1.0d15 f 1.0d15) (= f (truncate f)))
25 (format out "~D.0" (truncate f))
26 (let ((s (string-right-trim " " (format nil "~G" f))))
27 (write-string s out)
28 (when (char= (char s (1- (length s))) #\.)
29 (write-char #\0 out)))))
30
31 (defun %json-alist-p (x)
32 (and (consp x)
33 (every (lambda (e) (and (consp e) (or (stringp (car e)) (symbolp (car e)))))
34 x)))
35
36 (defun write-json (x out)
37 (cond
38 ((stringp x) (%write-json-string x out))
39 ((eq x t) (write-string "true" out))
40 ((eq x :false) (write-string "false" out))
41 ((eq x :null) (write-string "null" out))
42 ((integerp x) (princ x out))
43 ((floatp x) (%write-json-float x out))
44 ((realp x) (%write-json-float (coerce x 'double-float) out))
45 ((%json-alist-p x)
46 (write-char #\{ out)
47 (loop for (k . v) in x for first = t then nil do
48 (unless first (write-char #\, out))
49 (%write-json-string (json-key k) out)
50 (write-char #\: out)
51 (write-json v out))
52 (write-char #\} out))
53 ((listp x)
54 (write-char #\[ out)
55 (loop for e in x for first = t then nil do
56 (unless first (write-char #\, out))
57 (write-json e out))
58 (write-char #\] out))
59 ((symbolp x) (%write-json-string (string-downcase (symbol-name x)) out))
60 (t (error "Cannot JSON-encode ~S" x))))
61
62 (defun json-encode (x)
63 "Encode the nested list structure X as a JSON string."
64 (with-output-to-string (out) (write-json x out)))
65
66 (defun json-decode (string)
67 "Decode a JSON string into nested alists (string keys), lists, strings,
68 numbers, t (true) and nil (false/null)."
69 (let ((pos 0) (len (length string)))
70 (labels ((peek () (and (< pos len) (char string pos)))
71 (advance () (incf pos))
72 (skip-ws ()
73 (loop while (and (peek) (member (peek) '(#\Space #\Tab #\Newline #\Return)))
74 do (advance)))
75 (expect (ch)
76 (unless (eql (peek) ch)
77 (error "JSON parse error at ~D: expected ~S in ~S" pos ch string))
78 (advance))
79 (parse-string ()
80 (expect #\")
81 (with-output-to-string (out)
82 (loop for ch = (peek) do
83 (cond ((null ch) (error "Unterminated JSON string"))
84 ((char= ch #\") (advance) (return))
85 ((char= ch #\\)
86 (advance)
87 (let ((esc (peek)))
88 (advance)
89 (case esc
90 (#\n (write-char #\Newline out))
91 (#\t (write-char #\Tab out))
92 (#\r (write-char #\Return out))
93 (#\b (write-char #\Backspace out))
94 (#\f (write-char #\Page out))
95 (#\u (write-char (code-char (parse-integer string
96 :start pos
97 :end (+ pos 4)
98 :radix 16))
99 out)
100 (incf pos 4))
101 (t (write-char esc out)))))
102 (t (write-char ch out) (advance))))))
103 (parse-number ()
104 (let ((start pos))
105 (when (eql (peek) #\-) (advance))
106 (loop while (and (peek) (digit-char-p (peek))) do (advance))
107 (let ((is-float nil))
108 (when (eql (peek) #\.)
109 (setf is-float t) (advance)
110 (loop while (and (peek) (digit-char-p (peek))) do (advance)))
111 (when (and (peek) (member (peek) '(#\e #\E)))
112 (setf is-float t) (advance)
113 (when (and (peek) (member (peek) '(#\+ #\-))) (advance))
114 (loop while (and (peek) (digit-char-p (peek))) do (advance)))
115 (let ((token (string-trim " " (subseq string start pos))))
116 (if is-float
117 (let ((*read-default-float-format* 'double-float))
118 (read-from-string token))
119 (parse-integer token))))))
120 (parse-array ()
121 (expect #\[) (skip-ws)
122 (if (eql (peek) #\])
123 (progn (advance) nil)
124 (loop collect (parse-value) into items
125 do (skip-ws)
126 (cond ((eql (peek) #\,) (advance) (skip-ws))
127 ((eql (peek) #\]) (advance) (return items))
128 (t (error "JSON array parse error at ~D" pos))))))
129 (parse-object ()
130 (expect #\{) (skip-ws)
131 (if (eql (peek) #\})
132 (progn (advance) nil)
133 (loop collect (let ((k (parse-string)))
134 (skip-ws) (expect #\:) (skip-ws)
135 (cons k (parse-value)))
136 into pairs
137 do (skip-ws)
138 (cond ((eql (peek) #\,) (advance) (skip-ws))
139 ((eql (peek) #\}) (advance) (return pairs))
140 (t (error "JSON object parse error at ~D" pos))))))
141 (parse-literal (word value)
142 (unless (and (<= (+ pos (length word)) len)
143 (string= word string :start2 pos :end2 (+ pos (length word))))
144 (error "JSON literal parse error at ~D" pos))
145 (incf pos (length word))
146 value)
147 (parse-value ()
148 (skip-ws)
149 (let ((ch (peek)))
150 (cond
151 ((null ch) (error "Unexpected end of JSON input"))
152 ((char= ch #\") (parse-string))
153 ((char= ch #\{) (parse-object))
154 ((char= ch #\[) (parse-array))
155 ((char= ch #\t) (parse-literal "true" t))
156 ((char= ch #\f) (parse-literal "false" nil))
157 ((char= ch #\n) (parse-literal "null" nil))
158 ((or (digit-char-p ch) (char= ch #\-)) (parse-number))
159 (t (error "JSON parse error at ~D: ~S" pos ch))))))
160 (parse-value))))
161
162 (defun aget (alist key)
163 "Fetch KEY (string) from a decoded JSON alist."
164 (cdr (assoc key alist :test #'equal)))
Two details matter for the provider code. The encoder writes :false as false and :null as null, which lets a provider send a JSON boolean or a JSON null. The decoder returns t for true and nil for both false and null, so a provider that needs to tell them apart reads the raw slot. The aget helper looks up a string key in a decoded object.
search-apis.lisp
This file holds the shared core: the conditions, the two structures, the provider registry, the HTTP helpers, and the websearch entry point.
1 ;;; search-apis.lisp
2
3 (in-package #:search-apis)
4
5 ;;; ---- conditions ----
6
7 (define-condition search-error (simple-error) ())
8
9 (define-condition api-error (search-error)
10 ((status :initarg :status :reader api-error-status)
11 (body :initarg :body :reader api-error-body))
12 (:report (lambda (c stream)
13 (format stream "Web search API error ~A: ~A"
14 (api-error-status c) (api-error-body c)))))
15
16 (define-condition authentication-error (api-error) ())
17 (define-condition rate-limit-error (api-error) ())
18 (define-condition not-found-error (api-error) ())
19
20 (defun %map-http-error (status body)
21 "Map an HTTP status code to the corresponding search-apis condition."
22 (let ((condition
23 (cond ((member status '(401 403)) 'authentication-error)
24 ((= status 429) 'rate-limit-error)
25 ((= status 404) 'not-found-error)
26 (t 'api-error))))
27 (error condition :status status :body body)))
28
29 ;;; ---- result and response ----
30
31 (defstruct search-result
32 "One web search hit. CONTENT is the provider's snippet/summary and may be
33 nil when the provider only returns links. SCORE is provider specific."
34 title
35 url
36 content
37 (score nil))
38
39 (defstruct search-response
40 "The result of a WEBSEARCH call. ANSWER is populated by search-plus-LLM
41 providers such as Perplexity; RESULTS is the list of SEARCH-RESULTs."
42 provider
43 query
44 answer
45 results
46 raw)
47
48 ;;; ---- provider registry ----
49
50 (defstruct search-provider
51 name
52 base-url
53 env-keys
54 (requires-key t)
55 function)
56
57 (defvar *search-providers* (make-hash-table :test 'eq))
58
59 (defun define-search-provider (name base-url &key env-keys (requires-key t) function)
60 "Register a search provider."
61 (setf (gethash name *search-providers*)
62 (make-search-provider
63 :name name
64 :base-url base-url
65 :env-keys (if (listp env-keys) env-keys (list env-keys))
66 :requires-key requires-key
67 :function function)))
68
69 (defun find-search-provider (name)
70 (or (gethash name *search-providers*)
71 (error 'search-error
72 :format-control "Unknown search provider ~S. Known providers: ~S"
73 :format-arguments
74 (list name (loop for k being the hash-keys of *search-providers*
75 collect k)))))
76
77 (defun provider-api-key (provider explicit-key)
78 (or explicit-key
79 (loop for var in (search-provider-env-keys provider)
80 for value = (uiop:getenv var)
81 when (and value (plusp (length value))) return value)
82 (if (search-provider-requires-key provider)
83 (error 'search-error
84 :format-control "No API key for provider ~S. Pass :api-key or set one of ~S"
85 :format-arguments (list (search-provider-name provider)
86 (search-provider-env-keys provider)))
87 nil)))
88
89 ;;; ---- HTTP ----
90
91 (defun %get-json (url headers)
92 "GET URL and decode the JSON body into a Lisp alist."
93 (handler-case
94 (json-decode (dex:get url :headers headers))
95 (dex:http-request-failed (e)
96 (%map-http-error (dex:response-status e) (dex:response-body e)))))
97
98 (defun %post-json (url headers payload)
99 "POST PAYLOAD (a nested alist) as JSON to URL and decode the response."
100 (handler-case
101 (json-decode (dex:post url :headers headers :content (json-encode payload)))
102 (dex:http-request-failed (e)
103 (%map-http-error (dex:response-status e) (dex:response-body e)))))
104
105 ;;; ---- main entry point ----
106
107 (defun websearch (query &key (provider :brave) api-key max-results model)
108 "Search the web for QUERY using PROVIDER, a keyword such as :brave, :tavily
109 or :perplexity. MAX-RESULTS is a hint (used by Brave and Tavily). MODEL selects
110 the search-plus-LLM model for Perplexity (default \"sonar-pro\")."
111 (let* ((p (find-search-provider provider))
112 (key (provider-api-key p api-key)))
113 (funcall (search-provider-function p) p query
114 :api-key key
115 :max-results (or max-results 5)
116 :model model)))
The provider registry follows the same pattern as the litelm library from the LLM chapter. A search-provider records the provider’s name, its endpoint, the environment variables that may hold its key, and the function that performs the search. define-search-provider stores one in the *search-providers* hash table. find-search-provider looks one up by keyword and signals a search-error for an unknown name.
provider-api-key resolves the key in the order a caller expects: an explicit :api-key first, then the environment variables in order, and finally an error when the provider needs a key and none is found.
The two HTTP helpers wrap dex:get and dex:post. Both catch Dexador’s http-request-failed condition and pass the status code to %map-http-error, which turns 401 and 403 into authentication-error, 429 into rate-limit-error, 404 into not-found-error, and everything else into api-error. The status and body travel with the condition, so a caller can log them.
websearch ties it together. It finds the provider, resolves the key, and calls the provider function with the query and the four options. Providers ignore the options that do not apply to them.
providers.lisp
Each provider has two functions. A parse function turns a decoded JSON response into a search-response. A search function builds the request, sends it, and calls the parser. Keeping the parsers separate means they can be tested offline against recorded JSON.
The following diagram shows the high-level architecture of the Brave client:
Brave uses a GET request and puts the key in the X-Subscription-Token header. The query is URL-encoded, so spaces and special characters travel safely in the query string:
1 ;;; ---- Brave: GET, key in the X-Subscription-Token header ----
2
3 (defun parse-brave-response (json query)
4 "Turn a decoded Brave web search response into a SEARCH-RESPONSE."
5 (let* ((web (aget json "web"))
6 (items (and web (aget web "results"))))
7 (make-search-response
8 :provider :brave
9 :query query
10 :results (loop for item in items
11 collect (make-search-result
12 :title (aget item "title")
13 :url (aget item "url")
14 :content (aget item "description")))
15 :raw json)))
16
17 (defun brave-search (provider query &key api-key (max-results 5) model &allow-other-keys)
18 (declare (ignore model))
19 (let* ((url (format nil "~A?q=~A&count=~D"
20 (search-provider-base-url provider)
21 (quri:url-encode query :space-to-plus t)
22 max-results))
23 (headers (list (cons "Accept" "application/json")
24 (cons "X-Subscription-Token" api-key)))
25 (json (%get-json url headers)))
26 (parse-brave-response json query)))
Brave nests its hits under web.results, and the snippet field is named description. The parser reads those fields and discards the rest. Because Brave has no synthesized answer, answer stays nil.
The following diagram shows the high-level architecture of the Tavily client:
Tavily uses a POST request and puts the key in the JSON body along with the query and the result limit. The parser also checks the top-level error field that Tavily uses for application-level errors, which arrive with an HTTP 200 status and would otherwise pass silently:
1 ;;; ---- Tavily: POST, key in the JSON body ----
2
3 (defun parse-tavily-response (json query)
4 "Turn a decoded Tavily search response into a SEARCH-RESPONSE."
5 (when (aget json "error")
6 (error 'search-error
7 :format-control "Tavily API error: ~A"
8 :format-arguments (list (aget json "error"))))
9 (make-search-response
10 :provider :tavily
11 :query query
12 :results (loop for item in (aget json "results")
13 collect (make-search-result
14 :title (aget item "title")
15 :url (aget item "url")
16 :content (aget item "content")
17 :score (aget item "score")))
18 :raw json))
19
20 (defun tavily-search (provider query &key api-key (max-results 5) model &allow-other-keys)
21 (declare (ignore model))
22 (let ((headers '(("Content-Type" . "application/json")))
23 (payload `(("api_key" . ,api-key)
24 ("query" . ,query)
25 ("max_results" . ,max-results))))
26 (parse-tavily-response
27 (%post-json (search-provider-base-url provider) headers payload)
28 query)))
Tavily names its snippet field content and supplies a score between 0 and 1, which the parser copies into the result.
The following diagram shows the high-level architecture of the Perplexity client:
Perplexity uses the OpenAI-compatible chat completions endpoint. The key travels in the Authorization header as a Bearer token, and the query becomes the single user message. The response carries the answer under choices, and the sources under search_results or citations:
1 ;;; ---- Perplexity: POST chat completion, answer plus citations ----
2
3 (defun parse-perplexity-response (json query)
4 "Turn a decoded Perplexity chat response into a SEARCH-RESPONSE. The LLM
5 answer goes in the ANSWER slot; citations/search_results become RESULTS."
6 (let* ((choice (first (aget json "choices")))
7 (message (and choice (aget choice "message")))
8 (search-results (aget json "search_results"))
9 (citations (aget json "citations")))
10 (make-search-response
11 :provider :perplexity
12 :query query
13 :answer (and message (aget message "content"))
14 :results (cond
15 (search-results
16 (loop for item in search-results
17 collect (make-search-result
18 :title (aget item "title")
19 :url (aget item "url")
20 :content (aget item "snippet")
21 :score (aget item "score"))))
22 (citations
23 (loop for url in citations
24 collect (make-search-result :url url)))
25 (t nil))
26 :raw json)))
27
28 (defun perplexity-search (provider query &key api-key model max-results &allow-other-keys)
29 (declare (ignore max-results))
30 (let ((headers (list (cons "Content-Type" "application/json")
31 (cons "Authorization" (concatenate 'string "Bearer " api-key))))
32 (payload `(("model" . ,(or model "sonar-pro"))
33 ("messages" . ((("role" . "user")
34 ("content" . ,query)))))))
35 (parse-perplexity-response
36 (%post-json (search-provider-base-url provider) headers payload)
37 query)))
Perplexity’s parser is the only one that fills the answer slot. It prefers the richer search_results array, which has titles and snippets, and falls back to the plain citations list of URLs when that array is absent. The or around the model name means the default sonar-pro applies when the caller does not choose a model.
The file ends by registering all three providers:
1 ;;; ---- registration ----
2
3 (define-search-provider :brave "https://api.search.brave.com/res/v1/web/search"
4 :env-keys '("BRAVE_SEARCH_API_KEY")
5 :function #'brave-search)
6
7 (define-search-provider :tavily "https://api.tavily.com/search"
8 :env-keys '("TAVILY_API_KEY")
9 :function #'tavily-search)
10
11 (define-search-provider :perplexity "https://api.perplexity.ai/chat/completions"
12 :env-keys '("PERPLEXITY_API_KEY")
13 :function #'perplexity-search)
Setting the API Keys
Each provider reads its key from an environment variable. Set the ones you plan to use:
1 export BRAVE_SEARCH_API_KEY=BSGhQ-Nd-......
2 export TAVILY_API_KEY=tvly-......
3 export PERPLEXITY_API_KEY=pplx-......
Get keys from the provider sites:
- Brave: https://brave.com/search/api/. The free tier allows 2000 queries a month.
- Tavily: https://tavily.com/.
- Perplexity: https://www.perplexity.ai/.
A caller can also pass a key with :api-key, which takes precedence over the environment.
Running the Code
The library is in the book repository under src/search_APIs. Point ASDF at the system file and load it. With SBCL:
1 sbcl --no-userinit --non-interactive \
2 --eval '(load "~/quicklisp/setup.lisp")' \
3 --eval '(asdf:load-asd "search-apis.asd")' \
4 --eval '(asdf:load-system :search-apis)'
For interactive use, load the system in the REPL and call websearch. This example asks Brave for three results and prints each one:
1 * (ql:quickload :search-apis)
2 * (dolist (r (search-apis:search-response-results
3 (search-apis:websearch "Sedona Arizona" :provider :brave :max-results 3)))
4 (format t "~A~% ~A~% ~A~%~%"
5 (search-apis:search-result-title r)
6 (search-apis:search-result-url r)
7 (search-apis:search-result-content r)))
8 Visit Sedona | The official site of the Sedona Tourism Bureau
9 https://visitsedona.com/
10 The official site of the Sedona, AZ tourism bureau.
11
12 City of Sedona | Home
13 https://www.sedonaaz.gov/
14 Official site for the City of Sedona, Arizona.
15
16 Sedona, Arizona - Wikipedia
17 https://en.wikipedia.org/wiki/Sedona,_Arizona
18 Sedona is a city in the northern Verde Valley region of Arizona.
Tavily returns the same fields, plus a score. The loop is identical except for the provider:
1 * (dolist (r (search-apis:search-response-results
2 (search-apis:websearch "Fun things to do in Flagstaff Arizona"
3 :provider :tavily :max-results 3)))
4 (format t "~A (~,2F)~% ~A~% ~A~%~%"
5 (search-apis:search-result-title r)
6 (search-apis:search-result-score r)
7 (search-apis:search-result-url r)
8 (search-apis:search-result-content r)))
9 Top Things to Do in Flagstaff, AZ (0.99)
10 https://www.visitflagstaff.com/things-to-do/
11 Explore the best things to do in Flagstaff, from scenic drives around the
12 San Francisco Peaks to Lowell Observatory and the historic downtown.
13
14 Downtown Flagstaff (0.97)
15 https://example.com/downtown-flagstaff
16 Heritage Square, Wheeler Park, and the Weatherford Hotel anchor a walkable
17 downtown.
18
19 Lowell Observatory (0.95)
20 https://lowell.edu/
21 An astronomy hub with stargazing and the discovery of Pluto.
Perplexity returns an answer in addition to its sources. Read search-response-answer for the text and search-response-results for the citations:
1 * (let ((resp (search-apis:websearch "Where is Sedona Arizona?" :provider :perplexity)))
2 (format t "~A~%" (search-apis:search-response-answer resp))
3 (format t "Sources:~%")
4 (dolist (r (search-apis:search-response-results resp))
5 (format t " ~A~%" (search-apis:search-result-url r))))
6 Sedona is in **northern Arizona**, in the **Verde Valley**, about 30 miles
7 south of Flagstaff and roughly 115 miles north of Phoenix. It sits on the
8 county line between Coconino and Yavapai counties, near Oak Creek Canyon.
9 Sources:
10 https://en.wikipedia.org/wiki/Sedona,_Arizona
11 https://visitsedona.com/
12 https://www.sedonaaz.gov/
Interpreting the Results
The three listings show the same call pattern and the same accessors, which is the goal of the library. What differs is what each provider returns and what that data is good for.
Brave and Tavily give you a list of pages. Use them when you need to present links, build your own context for a later model call, or check what the web says about a topic. The content field is a short snippet, not the full page. Treat it as a hint that helps a reader choose a link, or as a small amount of grounding text. If you need the full page, fetch the URL yourself, as the web scraping chapter describes.
The score field is provider-specific. Tavily’s score is a relevance value between 0 and 1, so 0.99 means Tavily considers the page a strong match. Brave does not return a comparable score in the fields this parser reads, so its results keep nil. Never compare scores across providers; they are on different scales and mean different things.
Perplexity gives you an answer plus its sources. This is useful when the question is what matters and the reader does not need every link. The answer is generated text, so it can be wrong even when the sources are correct. Always keep the citations and check the answer against them when the stakes are high. The results list from Perplexity may be shorter than a Brave or Tavily list, because it reflects the pages the model chose to cite rather than a ranked result set.
The raw slot holds the full decoded JSON for each provider. When you need a field that the shared structures omit, such as Tavily’s answer summary or Perplexity’s token usage, read it from raw. This keeps the shared interface small while leaving the full response available.
Adding a Provider
Any search API can join the library at runtime. Write a parse function and a search function, then register them:
1 (search-apis:define-search-provider :my-engine "https://api.example.com/search"
2 :env-keys '("MY_ENGINE_API_KEY")
3 :function #'my-engine-search)
The search function is called as
1 (fn provider query :api-key key :max-results n :model m &allow-other-keys)
and must return a search-response. Once registered, websearch routes :my-engine to it with no other changes.
Error Handling
Failures map onto a condition hierarchy, so callers can react to the cause instead of parsing an HTTP status:
1 (handler-case
2 (search-apis:websearch "Sedona" :provider :tavily)
3 (search-apis:rate-limit-error (c)
4 (format t "Rate limited, status ~A. Back off and retry.~%"
5 (search-apis:api-error-status c)))
6 (search-apis:authentication-error (c)
7 (format t "Bad API key, status ~A.~%" (search-apis:api-error-status c)))
8 (search-apis:api-error (c)
9 (format t "Other API error ~A: ~A~%"
10 (search-apis:api-error-status c)
11 (search-apis:api-error-body c))))
authentication-error covers 401 and 403 and usually means a missing or wrong key. rate-limit-error covers 429 and means you should slow down or wait. not-found-error covers 404 and usually means a wrong endpoint. Every api-error carries the status and body, so logging is simple.
Testing the Library
The tests.lisp file checks the JSON codec, the provider registry, and the three parsers against recorded JSON. Those checks run without network access. The file also runs a live check for each provider whose key is present in the environment. Run the tests with SBCL:
1 sbcl --no-userinit --non-interactive \
2 --eval '(load "~/quicklisp/setup.lisp")' \
3 --eval '(asdf:load-asd "search-apis.asd")' \
4 --eval '(asdf:load-system :search-apis)' \
5 --load tests.lisp
The parsers are the part most likely to break when a provider changes its response shape, so the offline checks matter. Here is how the Brave parser check looks:
1 (let ((resp (search-apis::parse-brave-response
2 (search-apis:json-decode
3 (concatenate 'string
4 "{\"web\":{\"results\":["
5 "{\"title\":\"Sedona\",\"url\":\"https://example.com/sedona\","
6 "\"description\":\"A city in Arizona\"}]}}"))
7 "Sedona")))
8 (check (eq :brave (search-apis:search-response-provider resp)))
9 (let ((r (first (search-apis:search-response-results resp))))
10 (check (string= "Sedona" (search-apis:search-result-title r)))
11 (check (string= "A city in Arizona" (search-apis:search-result-content r)))))
When all checks pass, the test run prints:
1 --- live BRAVE_SEARCH_API_KEY tests ---
2 brave => 3 results
3 --- live TAVILY_API_KEY tests ---
4 tavily => 3 results
5 --- live PERPLEXITY_API_KEY tests ---
6 perplexity => "Sedona is in northern Arizona, in the Verde Valley..."
7 0 failure(s).
A non-zero failure count exits with a non-zero status, which makes the file usable in a build script.
Wrap Up
The search-apis library turns three unrelated web search APIs into one call. Brave, Tavily, and Perplexity differ in their HTTP method, their key location, their request body, and their response shape. The library absorbs those differences behind a provider registry and two shared structures.
The design has three parts. The search-result and search-response structures define one vocabulary for search data, with results for links and answer for synthesized text. The registry maps a keyword such as :brave to a provider record, so websearch stays small and adding a provider touches one place. The condition hierarchy turns HTTP failures into named conditions, so callers handle a bad key or a rate limit without inspecting status codes.
The self-contained JSON codec keeps the dependency list short. That costs some code, but it means the library builds with dexador, quri, and uiop alone.
Two habits are worth keeping. First, keep the parser functions separate from the request functions, so you can test them offline and catch a provider’s format change before it reaches users. Second, read the raw slot when you need a provider-specific field instead of growing the shared structures for one caller.
Optional Practice Problems
Add a fourth provider: Register a provider for another search API, such as Google Custom Search or Bing, by writing a parse function and a search function and calling define-search-provider. The new provider must return a search-response and work through websearch without any change to the core file.
Sort and filter results: Write a function best-results that takes a search-response and returns only the results whose score is above a threshold, sorted from highest to lowest. Decide what to do when a provider returns nil scores, and document the choice.
Unified answer function: Write answer-question that tries a pure search provider, builds a prompt from the top results, and sends that prompt to a language model to produce an answer with citations. Compare its output with Perplexity’s answer for the same query.
Retry with backoff: Wrap websearch in a function websearch-with-retry that catches rate-limit-error and retries a fixed number of times with increasing delay. Use sleep between attempts and give up after the last one.
Result cache: Add an in-memory cache keyed by the provider and query so a repeated search returns the stored search-response instead of calling the API again. Include a way to clear the cache, and explain when a cache would return stale results.