Client Library for the Tavily Web Search APIs
Tavily Search APIs offer developers a simple way to integrate powerful, real-time search capabilities into their applications. By abstracting away the complexities of web crawling, indexing, and ranking, Tavily allows developers to quickly access high-quality, relevant search results from across the web with minimal coding effort. I also use commercial Search APIs from Google, Microsoft, and Brave. For my work I find APIs from Tavily and Brave are the simplest to use when prototyping.
The source code for this Tavily client library is in my GitHub repository https://github.com/mark-watson/tavily. As usual you want to git clone this repository in your local directory ~/quicklisp/local-projects/ so Quicklisp can find this library with (ql:quickload :tavily). We will list the code below and then look at example use.
package.lisp
1 ;;;; package.lisp
2
3 (defpackage #:tavily
4 (:use :cl)
5 (:import-from :babel :octets-to-string)
6 (:export #:websearch))
tavily.asd
1 ;;;; openai.asd
2
3 (asdf:defsystem #:tavily
4 :description "Library for using the perplexity search+LLM APIs"
5 :author "Mark Watson"
6 :license "Apache 2"
7 :depends-on (#:uiop #:cl-json #:dexador :jonathan)
8 :components ((:file "package")
9 (:file "tavily")))
tavily.lisp
The following code defines a package :tavily that interacts with the Tavily Search API. It declares two global variables: tavily-api-key, which retrieves the API key from the environment variable TAVILY_API_KEY, and tavily-api-url, set to the API’s base URL. The parse-json function utilizes the cl-json library to convert JSON strings into Lisp data structures, ensuring lists are returned as property lists. The convert-bytes-to-string function transforms byte arrays into UTF-8 encoded strings using the babel libray’s octets-to-string function. To construct the JSON payload for API requests, the function make-tavily-json-payload creates a JSON string containing the API key, search query, and a maximum result limit of five, employing jonathan:to-json for encoding. The filter-tavily-response-item function extracts specific elements from each API response item, returning a list of data. The primary function, websearch, accepts a search query and an optional API key. It sends a POST request to the Tavily API using dex:post from the dexador library. Upon receiving a response, it parses the JSON, checks for errors, and processes the results by mapping filter-tavily-response-item over the response data. Finally, it concatenates the content fields of the filtered results into a single string, separated by newlines, and returns this string.
1 (in-package :tavily)
2
3 (defvar *tavily-api-key* (uiop:getenv "TAVILY_API_KEY")
4 "Your Tavily Search API key.")
5
6 (defvar *tavily-api-url* "https://api.tavily.com/search"
7 "Base URL for Tavily Search API.")
8
9 (defun parse-json (json-string)
10 "Parses a JSON string into a Lisp data structure."
11 (json:set-decoder-simple-list-semantics) ;; required for returning plist
12 (cl-json:decode-json-from-string json-string))
13
14 ;; Define the conversion function using the imported babel function directly
15 (defun convert-bytes-to-string (bytes)
16 (octets-to-string bytes :encoding :utf-8))
17
18 (defun make-tavily-json-payload (query)
19 "Helper function to create JSON payload for Tavily API request."
20 (jonathan:to-json
21 (list :|api_key| (or *tavily-api-key* (uiop:getenv "TAVILY_API_KEY"))
22 :|query| query
23 :|max_results| 5)))
24
25 (defun filter-tavily-response-item (result)
26 "Helper function to filter Tavily API response item."
27 ;;(format t "** result: ~A~%" result)
28 (list (cdr (nth 0 result))
29 (cdr (nth 1 result))
30 (cdr (nth 2 result))))
31
32 (defun websearch (query &key (api-key *tavily-api-key*))
33 "Performs a search using the Tavily Search API."
34 (format t "~%* Calling tavily-search with query: ~A~%" query)
35 (let* ((api-key-to-use (or api-key (uiop:getenv "TAVILY_API_KEY")))
36 (api-url *tavily-api-url*)
37 (prompt-data (make-tavily-json-payload query)))
38
39 (unless api-key-to-use
40 (error "Tavily API key is not set."))
41
42 (handler-case
43 (let ((response-str
44 (dex:post api-url
45 :headers '(("Content-Type" . "application/json"))
46 :content prompt-data)))
47 (let ((response-json (parse-json response-str)))
48 (if (getf response-json :error)
49 (error "Tavily API Error: ~A" (getf response-json :error))
50 ;; Process and return search results
51 (let ((uri-title-content-list
52 (mapcar #'filter-tavily-response-item
53 (cdr (assoc :RESULTS response-json)))))
54 ;; let's just return concatenated content
55 (format nil "~{~a~%~}" (mapcar #'caddr uri-title-content-list))))))
56 (error (c)
57 (error "Error communicating with Tavily API: ~A" c)))))
58
59 ;; (tavily:websearch "Fun things to do in Sedona Arizona")
Example Use
The following listing shows the use of the Tavily client code.
1 * (ql:quickload :tavily)
2 To load "tavily":
3 Load 1 ASDF system:
4 tavily
5 ; Loading "tavily"
6 ......................
7 (:tavily)
8 * (tavily:websearch "Fun things to do in Flagstaff Arizona")
9
10 * Calling tavily-search with query: Fun things to do in Flagstaff Arizona
11 "Top things to do in Flagstaff include Walnut Canyon & Wupatki National Monuments, nature tours, and an astronomy hub with stargazing. Cliff dwellings and ancient pueblos are also popular.
12 Flagstaff offers visitors seemingly endless things to do. Whether it be outdoor adventures, cultural experiences, or simply a tranquil escape from the every
13 1. Downtown Flagstaff. Dusk on a busy downtown street in a small town. · 2. Flagstaff Brewery Trail · 3. Lowell Observatory · 4. Museum of Northern Arizona · 5.
14 Explore the best things to do in Flagstaff, AZ, with our comprehensive guide! From scenic drives around the San Francisco Peaks to the
15 1. See what's going on at Heritage Square or Wheeler Park. · 2. Enjoy the sunshine and great weather. · 3. Visit the historic Weatherford Hotel, Hotel Monte Vista
16 "