Interfacing with External Programs: A Lightpanda Browser Client

In this chapter, we build a complete Common Lisp library that interfaces with an external program: the Lightpanda headless web browser. This example demonstrates practical techniques for shell integration, string processing, and building reusable APIs. Directions for installing the Lightpanda command line tool can be found in the Lightpanda documentation.

The Problem: JavaScript-Rendered Web Content

Modern web pages often require JavaScript execution to display their content. Traditional HTTP clients like Drakma only fetch static HTML, missing the dynamic content rendered by JavaScript. Lightpanda is a headless browser that runs from the command line and outputs fully rendered pages.

Our goal: create a Common Lisp interface that:

  1. Invokes Lightpanda as a subprocess
  2. Captures its output (HTML, Markdown, or semantic tree)
  3. Provides helper functions for common operations like link extraction

Project Structure

A well-organized Common Lisp project uses ASDF for system definition:

1 ;; lightpanda.asd
2 (asdf:defsystem #:lightpanda
3   :description "Common Lisp interface to the Lightpanda headless browser"
4   :license "Apache-2.0"
5   :version "0.1.0"
6   :depends-on (#:cl-json)
7   :components ((:file "lightpanda")
8                (:file "project" :depends-on ("lightpanda"))))

Key elements:

  • #:lightpanda — Uninterned symbol naming the system
  • :depends-on — External dependencies (loaded via Quicklisp)
  • :components — Source files in load order

Package Definition:

We define a package with explicit exports, creating a clean public API:

 1 ;; lightpanda.lisp (package section)
 2 (defpackage #:lightpanda
 3   (:use #:cl)
 4   (:export
 5    ;; Config
 6    #:*lightpanda-binary*
 7    ;; Fetch interface
 8    #:fetch-url
 9    ;; Helpers
10    #:fetch-and-extract-links
11    #:demo-fetch))
12 
13 (in-package #:lightpanda)

The :use #:cl imports the standard Common Lisp symbols. #:*lightpanda-binary* and other symbols marked with #:export become part of the public API that users call with these package prefixes like (lightpanda:fetch-url "https://example.com/").

Configuration

You must have the lightpanda tool installed; here I verify the installation on my laptop:

1 $ which lightpanda
2 /Users/mark/bin/lightpanda

We use defvar for configurable settings:

1 (defvar *lightpanda-binary* "lightpanda"
2   "Path to the lightpanda binary. Can be an absolute path or a name on PATH.")

Using defvar (not defparameter) means users can setf this variable, and it won’t be reset if the file is reloaded. The earmuffs (*...*) convention marks it as a special variable.

Users can configure a new file path:

1 (setf lightpanda:*lightpanda-binary* "/usr/local/bin/lightpanda")

Running External Programs with UIOP

Common Lisp doesn’t have a built-in standard way to run subprocesses, but the UIOP (Universal Interface to OS Processes) library provides portable functions. UIOP comes bundled with ASDF, so it’s universally available.

1 (defun %run (command)
2   "Run a shell command string, return stdout as a string (or nil on error)."
3   (handler-case
4       (uiop:run-program command
5                         :output :string
6                         :error-output :string)
7     (error (e)
8       (format t "Command error: ~a~%Command: ~a~%" e command)
9       nil)))

Key UIOP features:

  • :output :string — Captures stdout as a Lisp string
  • :error-output :string — Captures stderr separately
  • handler-case — Catches errors gracefully, returning nil on failure

The %run name uses the Common Lisp convention: a leading % marks internal/private functions not meant for export.

For link extraction, we scan HTML with simple string operations—a pragmatic choice for this use case:

 1 (defun %extract-links (html)
 2   "Return a list of href strings found in <a> tags within an HTML string."
 3   (let ((links '())
 4         (pos    0)
 5         (marker "href=\""))
 6     (loop
 7       (let ((found (search marker html :start2 pos)))
 8         (unless found (return))
 9         (let* ((start (+ found (length marker)))
10                (end   (position #\" html :start start)))
11           (when end
12             (push (subseq html start end) links))
13           (setf pos (or end (+ found 1)))))))
14     (nreverse links)))

Walkthrough:

  1. search finds "href=\"" starting from position pos
  2. subseq extracts the href value between quotes
  3. push accumulates links (building a list in reverse)
  4. nreverse reverses in-place for correct order
  5. loop (without keywords) is the simple “infinite loop with return” form

The following diagram shows the high-level architecture of the Lightpanda browser client developed in this chapter:

Architecture diagram

The let* (not let) allows end to reference start. The loop form returns when found is nil.

The Main API Function

The central function combines everything:

 1 (defun fetch-url (url &key (log-level "warn") obey-robots (dump "html"))
 2   "Fetch URL using `lightpanda fetch`, returning the JS-rendered content string.
 3 DUMP controls what is written to stdout; valid values are:
 4   \"html\"               - full rendered HTML (default)
 5   \"markdown\"           - page as Markdown
 6   \"semantic_tree\"      - semantic tree
 7   \"semantic_tree_text\" - semantic tree as plain text
 8 No server process is required; lightpanda is invoked directly.
 9 
10   (lightpanda:fetch-url \"https://markwatson.com/\")
11   (lightpanda:fetch-url \"https://markwatson.com/\" :dump \"markdown\")
12 "
13   (let* ((extra (append (when obey-robots '(\"--obey_robots\"))
14                         (list "--dump" dump
15                               "--log_level" log-level
16                               "--log_format" "pretty")))
17          (args  (append (list *lightpanda-binary* "fetch")
18                         extra
19                         (list url)))
20          (cmd   (format nil "~{~a~^ ~}" args)))
21     (%run cmd)))

Key techniques:

  • &key with default values: (dump "html") makes :dump optional
  • when returns nil or the provided list—clean conditional inclusion
  • format nil "~{~a~^ ~}" args — produces "arg1 arg2 arg3" from a list

The ~{~a~^ ~} format directive:

  • ~{ — iterate over a list
  • ~a — aesthetic (human-readable) output
  • ~^ — exit iteration if no more elements (no trailing space)
  • ~} — end iteration

Helper Functions

Higher-level helpers make common operations easy:

 1 (defun fetch-and-extract-links (url)
 2   "Fetch URL with lightpanda and return a list of href link strings.
 3 
 4   (lightpanda:fetch-and-extract-links \"https://markwatson.com/\")
 5 "
 6   (let ((html (fetch-url url)))
 7     (if html
 8         (%extract-links html)
 9         (progn
10           (format t "Failed to fetch ~a~%" url)
11           nil))))
12 
13 (defun demo-fetch (url)
14   "Fetch URL, print a snippet of HTML and the discovered links.
15 
16   (lightpanda:demo-fetch \"https://markwatson.com/\")
17 "
18   (format t "~%=== Fetch demo: ~a ===~%" url)
19   (let ((html (fetch-url url)))
20     (if html
21         (progn
22           (format t "Received ~a bytes of HTML.~%" (length html))
23           (format t "First 500 chars:~%~a~%~%" (subseq html 0 (min 500 (length html))))
24           (let ((links (%extract-links html)))
25             (format t "Found ~a link(s):~%" (length links))
26             (dolist (l links)
27               (format t "  ~a~%" l))))
28         (format t "No HTML returned.~%"))))

Usage Examples

After loading with (ql:quickload :lightpanda):

 1 ;; Basic HTML fetch
 2 (lightpanda:fetch-url "https://markwatson.com/")
 3 
 4 ;; Get Markdown output (good for LLM input)
 5 (lightpanda:fetch-url "https://markwatson.com/" :dump "markdown")
 6 
 7 ;; Respect robots.txt
 8 (lightpanda:fetch-url "https://markwatson.com/" :obey-robots t)
 9 
10 ;; Extract all links
11 (lightpanda:fetch-and-extract-links "https://markwatson.com/")
12 
13 ;; Interactive demo
14 (lightpanda:demo-fetch "https://markwatson.com/")

Compatibility Package

For users who prefer a different package name, we provide an alias:

1 ;; project.lisp
2 (defpackage #:lightpanda-browser
3   (:use #:cl #:lightpanda))
4 
5 (in-package #:lightpanda-browser)

Using (:use #:cl #:lightpanda) re-exports all symbols from lightpanda, so (lightpanda-browser:fetch-url ...) works identically.

Key Code Style Takeaways

  1. UIOP:run-program — Portable subprocess execution in Common Lisp
  2. defvar with earmuffs — Configurable special variables
  3. Package exports — Design a clean public API
  4. Private function conventions — Use % prefix for internal helpers
  5. Format directives — ~{~a~^ ~} for clean command construction
  6. Key arguments with defaults — &key (param default) makes flexible APIs

This pattern—shelling out to a specialized tool and processing its output—is a powerful technique. You can wrap any command-line tool this way: databases, image processors, compilers, or your own scripts. The result is a Lisp API that hides the implementation details while providing access to the tool’s capabilities.

Up next

Using a Local Document Embeddings Vector Database With OpenAI GPT-5 APIs for Semantically Querying Your Own Data