I have used Lisp languages since the late 1970s, Common Lisp and Clojure professionally and other Lisp languages, mostly various Scheme implementations for writing utilities, network programming, and various AI experiments. This book is specifically about effectively using Gerbil Scheme for writing software to solve practical problems.

Both the source code examples and the manuscript files for this book are maintained in a single GitHub repository: https://github.com/mark-watson/gerbil_scheme_book. I recommend that you keep a local copy:

The source code examples are provided under the LGPL (Lesser General Public License), a “business-friendly” license. The manuscript files are released under a Creative Commons Attribution–ShareAlike, Non-Commercial license.

My goal, dear reader, is that you find value in this material and freely reuse it in your own projects.

A few key points about the LGPL:

• Compatibility: LGPL code can be combined with code under other popular licenses such as Apache 2.0, MIT, etc.
• Commercial use: You are free to use LGPL code in commercial applications without restrictions.
• Modifications: If you modify the LGPL portions of the code, you must make those changes publicly available. However, you do not need to release your own original code that merely uses or links against the LGPL code.

This makes the LGPL very different from the more restrictive GPL and AGPL licenses.

I recommend that you keep a local the GitHub repository for Gerbil Scheme:

You will find useful tutorial examples for using Gerbil Scheme in the sub-directory gerbil/src/tutorial.

While Gerbil Scheme is another dialect in the Lisp family, it is like Racket Scheme in that it is “opinionated”, reflecting the Gerbil developers’ style and philosophy.

Gerbil is built on Gambit Scheme, a high-performance, retargetable compiler. Gerbil inherits a legacy of speed and portability while introducing a state-of-the-art module and object system inspired by Racket. The result is a language engineered for creating efficient, concurrent, and robust long-running applications, positioning it as a powerful tool for tasks ranging from web services to distributed systems.

Gerbil has a comprehensive standard library that provides a “batteries included” experience uncommon in the often-fragmented Scheme ecosystem. Rather than relying on a disparate collection of third-party packages for fundamental operations, Gerbil offers canonical, high-quality, and officially maintained implementations for core functionalities. This includes built-in libraries for HTTP clients and servers, JSON parsing and serialization, cryptographic primitives, and database drivers. This approach favors a strong, coherent core, ensuring that developers have a stable and predictable foundation for building complex systems.

Gerbil Scheme can also function as a systems language with its “Integrated FFI” (Foreign Function Interface). This FFI allows Gerbil code to interface directly and efficiently with libraries written in C. The FFI is useful for specialized applications like high-speed data parsing or hardware interaction.

Gerbil provides powerful general-purpose primitives and expects the developer to compose these primitive components to interact with specific protocols, often for systems-level software development.

Navigating any programming ecosystem begins with understanding its tooling for package management and the core libraries that form the bedrock of application development. In Gerbil, these components are designed with the same philosophy of directness and control that characterizes the language itself.

Gerbil includes a command-line package manager gxpkg, invoked as gerbil pkg or its alias gxpkg.

The essential commands for managing packages include:

• gerbil pkg install - installs a package from a specified Git repository.
• gerbil pkg update <package-name|all> - updates one or all installed packages.
• gerbil pkg list - lists all installed packages.
• gerbil pkg search - searches configured package directories for packages matching a keyword.

A key feature of gxpkg is that packages are sourced directly from public Git repositories on providers like GitHub, GitLab, or Bitbucket. For a repository to be recognized as a Gerbil package, it must contain a gerbil.pkg manifest file that declares the package’s namespace and dependencies, and a build.ss script that defines how to compile the package.

This direct-from-source model has significant security implications so I manually inspect the source code for 3rd party packages that I use. The build.ss script is not sandboxed and runs with the user’s privileges.

Package discovery in Gerbil is facilitated through package directories. These are themselves Git repositories containing a package-list file that maps package names to their repository URLs and descriptions. By default gxpkg is configured to search the “Mighty Gerbils” directory, which contains packages developed and maintained by the Gerbil Core Team.

The primary community-curated package list can be found at github.com/vyzo/gerbil-directory. Developers can add additional directories using the gerbil pkg dir -a command, allowing for the creation of private or specialized package collections.

TBD TBD: configure book examples as a community curated package…. TBD

Many examples in this book rely on :std/net/request and :std/text/json so we will give a brief overview here.

The foundation for all network communication in Gerbil is the :std/net/request library that is a comprehensive HTTP client built into the standard library. This module obviates the need for a third-party HTTP client for most use cases. Its key features are:

• Full Method Support: Dedicated procedures for all standard HTTP methods (http-get, http-post, http-put, http-delete, etc.), plus an http-any for custom methods.
• Rich Request Customization: Keyword arguments for setting custom headers, URL parameters, cookies, authentication credentials, and the request body.
• Secure Communication: Integrated SSL/TLS context management for HTTPS requests, using the system’s certificate authorities for verification by default.
• Introspective Response Objects: Requests return a request object that provides access to the status code, headers, and response body in various formats (raw bytes, text, or parsed JSON).

This library is the sole tool required to interact with any RESTful API, from the complex and we will in later chapters use it to access commercial services offered by OpenAI and Google Gemini as well as locally hosted Large Language Models (LLMs) using Ollama.

For handling the responses from web services we will use the Data Interchange Layer package :std/text/json.

Virtually all modern web APIs use JSON as their data interchange format. Gerbil provides a canonical, flexible, and efficient JSON processing library in :std/text/json. This module is the essential counterpart to :std/net/request, handling the serialization of Scheme data into JSON strings for request bodies and the parsing of JSON responses back into Scheme objects.

TBD

Assuming that you have Emacs installed withthe file ~/.emacs and the directory ~/.emacs.d/, copy the following two files into ~/.emacs.d/:

• https://github.com/gambit/gambit/blob/master/misc/gambit.el
• https://github.com/mighty-gerbils/gerbil/blob/master/etc/gerbil-mode.el

Then add the following to your ~/.emacs file:

The Google Gemini API provides developers with access to Google’s state-of-the-art family of Gemini Large Language Models (LLMs), representing a significant leap forward in multimodal artificial intelligence. Unlike earlier models that primarily processed text, the Gemini series—comprising models like the highly capable Gemini Ultra, the versatile Gemini Pro, and the efficient Gemini Nano—was designed from the ground up to seamlessly understand, operate across, and combine different types of information, including text, code, images, audio, and video. This native multimodality allows for the development of sophisticated applications that can reason about complex inputs, such as analyzing the steps in a video, interpreting charts and diagrams within a document, or generating creative text based on a visual prompt. The API offers a streamlined and powerful interface, enabling developers to integrate these advanced reasoning and generation capabilities into their own software, pushing the boundaries of what’s possible in domains ranging from data analysis and content creation to building the next generation of intelligent, context-aware user experiences.

Here you will learn how to send prompts to the Google Gemini AI models. For information on creating effective prompts please read my blog article Notes on effectively using AI.

In this section, we’ll explore a practical example of interacting with a modern web API by building a client for Google’s Gemini Large Language Model. The following program defines a function named gemini that takes a text prompt and returns the model’s generated response. This involves several common tasks in modern software development: retrieving sensitive information like an API key from environment variables, dynamically constructing a JSON payload according to the API’s specification, setting the correct HTTP headers for authentication and content type, and executing an HTTP POST request. Upon receiving a successful response, the code demonstrates how to parse the returned JSON data to extract the specific piece of information we need—the generated text from a complex, nested data structure. This serves as a simple example for making outbound web requests and handling the data interchange that is central to working with external services.

The core logic resides within the gemini function, which uses a series of let* bindings to sequentially construct the components of the API request. First, it defines the necessary HTTP headers, including the API key. Next, it builds the body-data as a nested hash table, precisely matching the structure required by the Gemini API, before serializing it into a JSON string using json-object->string. Finally, the full endpoint URL is assembled by concatenating the base URL with the specific model being used. The actual network communication is handled by the http-post function, which sends all the prepared information to the Google servers.

Once the http-post call returns, the program immediately checks the response status. If the request was successful (status code 200), it proceeds to parse the data; otherwise, nothing is returned. The parsing logic is a chain of data extraction operations on the JSON response, which is first converted into a Gerbil Scheme hash table. Using a combination of hash-ref to access values by their keys (like ’candidates and ’content) and the function car to access the first element of a list, the code navigates the nested data structure to isolate the desired text content. This sequence elegantly demonstrates how Gerbil Scheme’s standard library functions for handling lists and hash tables can be composed to efficiently process structured data from external APIs.

Change directory to source-code/gemini an run:

Notice how Gemini initially returned the JSON results in Markdown format and I modified the prompt to get the output format I wanted. Another good technique is to give LLMs an example of the output format you want in the prompt.

Ollama is a powerful and user-friendly tool designed to simplify the process of running large language models (LLMs) locally on personal hardware. In a landscape often dominated by cloud-based APIs, Ollama democratizes access to advanced AI by providing a simple command-line interface that bundles model weights, configurations, and a tailored execution environment into a single, easy-to-install package. It allows developers, researchers, and enthusiasts to download and interact with a wide range of popular open-source models, such as Llama 3, Mistral, and Phi-3, with just a single command. Beyond its interactive chat functionality, Ollama also exposes a local REST API, enabling the seamless integration of these locally-run models into custom applications without the latency, cost, or privacy concerns associated with remote services. This focus on accessibility and local deployment makes it an indispensable tool for offline development, rapid prototyping, and leveraging the power of modern LLMs while maintaining full control over data and infrastructure.

This next program in file gerbil_scheme_book/source_code/ollama/ollama.ss provides a practical demonstration of network programming and data handling in Gerbil Scheme by creating a simple client for the Ollama API. Ollama is a fantastic tool that allows you to run powerful large language models, like Llama 3, Mistral, and Gemma, directly on your own machine. Our ollama function will encapsulate the entire process of communicating with a locally running Ollama instance. It will take a text prompt as input, construct the necessary JSON payload specifying the model and prompt, send it to the Ollama server’s /api/generate endpoint via an HTTP POST request, and then carefully parse the server’s JSON response. The goal is to extract and return only the generated text, while also including basic error handling to gracefully manage any non-successful API responses, making for a robust and reusable utility.

The ollama function begins by using a let* block to define the necessary components for the API request: the server endpoint, the required HTTP headers, and the request body-data. The body is first constructed as a Gerbil hash-table, which is the natural way to represent a JSON object, and then serialized into a JSON string using json-object->string. Note that the “stream” parameter is explicitly set to #f to ensure we receive the complete response at once rather than as a series of events. The core of the function is the http-post call, which performs the actual network request.

After the request is made, the code immediately checks the status of the response. A status code of 200 indicates success, prompting the code to parse the JSON body using request-json and extract the generated text from the ’response field of the resulting hash-table. If the request fails for any reason, a descriptive error is raised, including the HTTP status and response body, which is crucial for debugging. The function’s design, with its optional model: keyword argument, makes it trivial to switch between different models you have downloaded through Ollama, providing a flexible interface for interacting with local large language models.

Linux Installation

Open your terminal and run the following command to download and execute the installation script:

macOS Installation

• Download the Ollama application from the official website: [https://ollama.com/download}(https://ollama.com/download).
• Unzip the downloaded file.
• Move the Ollama.app file to your /Applications folder.
• Run the application. An Ollama icon will appear in the menu bar.

This will also install the ollama command line program.

Pulling the Model

After installing Ollama on either Linux or macOS, open your terminal and run the following command to download the gemma3:latest model:

After this is complete, you can run the local API service using:

You need to have Ollama installed on your system and you should pull the model you want to experiment with.

A few comments: In the second example I added “Just output the bash script and nothing else.” to the end of the prompt. Without this, the model will generate a 100 lines of design notes, instructions how to make the bash script executable, etc. I didn’t want that, just the bash script.

In the third example, I used the same prompt but used displayln to print the result in a more useful format.

The OpenAI API serves as the primary gateway for developers to harness the groundbreaking capabilities of OpenAI’s suite of artificial intelligence models, most notably the influential Generative Pre-trained Transformer (GPT) series. Since the release of GPT-3, and continuing with more advanced successors like GPT-4, this API has fundamentally reshaped the landscape of software development by making sophisticated natural language understanding, generation, and reasoning accessible as a programmable service. It allows developers to integrate functionalities such as text summarization, language translation, code generation, sentiment analysis, and conversational AI into their applications through simple HTTP requests. By abstracting away the immense complexity of training and hosting these massive models, the OpenAI API has catalyzed a wave of innovation, empowering everyone from individual hobbyists to large enterprises to build intelligent applications that can write, read, and comprehend human language with unprecedented fluency and coherence.

Here you will learn how to send prompts to the OpenAI GPT AI models. For information on creating effective prompts please read my blog article Notes on effectively using AI.

This next program in file gerbil_scheme_book/source_code/openai/openai.ss provides another practical example of interfacing with a modern web API from Gerbil Scheme. We will define a function, openai, that acts as a simple client for the OpenAI Chat Completions API. This function takes a user’s prompt as its primary argument and includes optional keyword arguments to specify the AI model and a system-prompt to set the context for the conversation. Before making the request, it securely retrieves the necessary API key from an environment variable, a best practice that avoids hard-coding sensitive credentials. The core logic involves constructing a proper JSON payload containing the model and messages, setting the required HTTP headers for authorization and content type, and then sending this data via an HTTP POST request. Finally, it parses the JSON response from the OpenAI servers to extract and return the generated text content from the AI model, while also including basic error handling for failed requests.

The implementation begins by importing the necessary standard libraries for handling HTTP requests (:std/net/request) and JSON data manipulation (:std/text/json). Inside the openai function, a let* block is used to sequentially bind variables for the request. It first constructs the HTTP headers and the request body, which is a hash-table representing the JSON structure required by the OpenAI API, including the model name and a list of messages for the “system” and “user” roles. This hash-table is then serialized into a JSON string. The http-post procedure is called with the API endpoint, headers, and the serialized data to perform the web request.

Upon receiving a response, the code demonstrates robust handling of the result. It first checks if the HTTP status code is 200, indicating success. If the request was successful, it parses the JSON text from the response body back into a hash-table. It then carefully navigates the nested structure of this response data using a chain of hash-ref and car calls to drill down through the choices array and message object to finally extract the desired content string. If the HTTP request failed, the else branch is triggered, raising an error with the status code and the response body, which provides valuable debugging information to the user.

In the following example I run the Gerbil Scheme interpreter, loading the file “openai.ss”, and then entering an interactive REPL:

Notice that I repeated the second example, displaying the string response in a more readable format. As we also see in this example, Large Language Models will in general produce different output when called with the same prompt.

Sometimes we might want the output in a specific format, like JSON:

This example is not good enough! When you use LLMs in your applications it is better to one-shot prompt with the exact output format you need in your application. Here is an example prompt:

Let’s run this one-shot prompt in a Gerbil Scheme REPL:

Dear reader, are you excited about integrating LLMs into your applications but you want to miniize costs?

Groq is rapidly making a name for itself in the AI community as a cloud-based large language model (LLM) inference provider, distinguished by its revolutionary hardware and remarkably low-cost, high-speed performance. At the heart of Groq’s impressive capabilities lies its custom-designed Language Processing Unit (LPU), a departure from the conventional GPUs that have long dominated the AI hardware landscape. Unlike GPUs, which are general-purpose processors, the LPU is an application-specific integrated circuit (ASIC) meticulously engineered for the singular task of executing LLM inference. This specialization allows for a deterministic and streamlined computational process, eliminating many of the bottlenecks inherent in more versatile hardware. The LPU’s architecture prioritizes memory bandwidth and minimizes latency, enabling it to process and generate text at a blistering pace, often an order of magnitude faster than its GPU counterparts. This focus on inference, the process of using a trained model to make predictions, positions Groq as a compelling solution for real-time applications where speed is paramount.

The practical implications of Groq’s technological innovation are multifaceted, offering a potent combination of affordability, speed, and a diverse selection of open-source models. The efficiency of the LPU translates directly into a more cost-effective pricing structure for users, with a pay-as-you-go model based on the number of tokens processed. This transparent and often significantly cheaper pricing democratizes access to powerful AI, enabling developers and businesses of all sizes to leverage state-of-the-art models without prohibitive upfront costs. The platform’s raw speed is a game-changer, facilitating near-instantaneous responses that are crucial for interactive applications like chatbots, content generation tools, and real-time data analysis. Furthermore, Groq’s commitment to the open-source community is evident in its extensive library of available models, including popular choices like Meta’s Llama series, Mistral’s Mixtral, and Google’s Gemma. This wide array of options provides users with the flexibility to select the model that best suits their specific needs, all while benefiting from the unparalleled inference speeds and economic advantages offered by Groq’s unique hardware.

Here you will learn how to send prompts to the Groq LMS inference service. For information on creating effective prompts please read my blog article Notes on effectively using AI.

This project is stored in the directory gerbil_scheme_book/source_code/groq_llm_inference. There is one common utility file groq_inference.ss and currently two very short example scripts that use this utility:

• kimi2.ss - Uses Moonshot AI’s Kimi2 model (MOE 1 trillion paramters, with 32B active).
• gpt-oss-120b.ss - Uses OpenAI’s open source model gpt-oss-120b.

Both of these models are practical models that are excellent for data manipulation, coding, and general purpose use.

It’s important to note that both models leverage a Mixture of Experts (MoE) architecture. This is a significant departure from traditional “dense” transformer models where every parameter is activated for every input token. In an MoE model, a “router” network selectively activates a small subset of “expert” sub-networks for each token, allowing for a massive total parameter count while keeping the computational cost for inference relatively low. The comparison, therefore, is between two different implementations and philosophies of the MoE approach.

Here is the project Makefile:

Features:

• Architecture: A very large-scale Mixture of Experts (MoE) model.
• Parameters: It has a staggering 1 trillion total parameters. For any given token during inference, it activates approximately 32 billion of these parameters. This represents a very sparse activation (around 3.2%).
• Specialization: Kimi2 is highly optimized for agentic capabilities, meaning it excels at using tools, reasoning through multi-step problems, and advanced code synthesis.
• Training Innovation: It was trained using a novel optimizer called MuonClip, designed to ensure stability during large-scale MoE training runs, which have historically been prone to instability.
• Context Window: It supports a large context window of up to 128,000 tokens, making it suitable for tasks involving long documents or extensive codebases.
• Licensing: While the model weights are publicly available (“open-weight”), its specific licensing and training data details are proprietary to Moonshot AI.

Features:

• Architecture: Also a Mixture of Experts (MoE) model, but at a smaller scale than Kimi2.
• Parameters: It has a total of 117 billion parameters, with a much smaller active set of around 5.1 billion parameters per token. This results in a similarly sparse activation (around 4.4%).
• Efficiency and Accessibility: A primary feature is its optimization for efficient deployment. It’s designed to run on a single 80 GB GPU (like an H100), making it significantly more accessible for researchers and smaller organizations.
• Focus: Like Kimi2, it is designed for high-reasoning, agentic tasks, and general-purpose use.
• Licensing: It is a true open-source model, released under the permissive Apache 2.0 license. This allows for broad use, modification, and redistribution.
• Training: It was trained using a combination of reinforcement learning and distillation techniques from OpenAI’s more advanced proprietary models.

Here we construct a practical, reusable Gerbil Scheme function for interacting with the Groq API, a service renowned for its high-speed large language model inference. The function, named groq_inference, encapsulates the entire process of making a call to Groq’s OpenAI-compatible chat completions endpoint. It demonstrates essential real-world programming patterns, such as making authenticated HTTP POST requests, dynamically building a complex JSON payload from Scheme data structures, and securely managing credentials using environment variables. This example not only provides a useful utility for integrating AI into your applications but also serves as an excellent case study in using Gerbil’s standard libraries for networking (:std/net/request) and data interchange (:std/text/json), complete with robust error handling for both network issues and malformed API responses.

The implementation begins by defining the groq_inference function, which accepts a model and a prompt, along with an optional keyword argument for a system message. Its first action is a crucial security and configuration check: it attempts to fetch the GROQ_API_KEY from the environment variables, raising an immediate error if it’s not found. The core of the function then uses a let* block to sequentially build the components of the HTTP request. It constructs the authorization headers and then assembles the JSON body using a combination of quasiquotation and the list->hash-table procedure to create the nested structure required by the API. This body is then serialized into a JSON string, and finally, the http-post function is called with the endpoint, headers, and data to execute the network request.

Upon receiving a response, the function demonstrates robust result processing and error handling. It first checks if the HTTP status code is 200 (OK), indicating a successful request. If it is, a series of let* bindings are used to safely parse the JSON response and navigate the nested data structure to extract the final content string from response[‘choices’][0][‘message’][‘content’], with checks at each step to prevent errors on an unexpected response format. If the content is successfully extracted, it is returned as the result of the function. However, if the HTTP status is anything other than 200, the function enters its error-handling branch, raising a descriptive error that includes the failing status code and the raw text body of the response, providing valuable debugging information to the caller.

These two scripts are simple enough to just list without comment:

kimi2.ss

gpt-oss-120b.ss

Running the kimi2 example:

Note, the utility must be comiled one time: gxc groq_inference.ss. The compiled library by default will be in the directory ~/.gerbil/lib/groq/ because we set this project’s module name to groq in the file gerbil.pkg.

Running the gpt-oss-120b example:

Wikidata is a free, collaborative, and multilingual knowledge base that functions as the central structured data repository for the Wikimedia ecosystem, including projects like Wikipedia, Wikivoyage, and Wiktionary. Launched in 2012, its mission is to create a common source of open data that can be used by anyone, anywhere. Unlike Wikipedia, which contains prose articles, Wikidata stores information in a machine-readable format structured around items (representing any concept or object), which are described by properties (like “population” or “author”) and corresponding values. For example, the item for “Earth” has a property “instance of” with the value “planet”. This structured approach allows for data consistency across hundreds of language editions of Wikipedia and enables powerful, complex queries through its SPARQL endpoint. By providing a centralized, queryable, and interlinked database of facts, Wikidata not only supports Wikimedia projects but also serves as a crucial resource for researchers, developers, and applications worldwide that require reliable and openly licensed structured information.

This code in file wikidata.ss provides a client for interacting with the Wikidata Query Service, a powerful public SPARQL endpoint for accessing the vast, collaboratively edited knowledge base of Wikidata. The code encapsulates the entire process of querying this service, starting with a raw SPARQL query string. It properly formats the request by URL-encoding the query, constructs the full request URL, and sets the appropriate HTTP headers, including the Accept header for the SPARQL JSON results format and a User-Agent header, which is a requirement for responsible API usage. Upon receiving a response, the module parses the JSON data and then transforms the verbose, nested structure of the standard SPARQL results format into a more convenient and idiomatic Gerbil Scheme data structure—either a list of hash tables or a list of association lists. The file also includes several example functions that demonstrate how to query for specific facts, such as Grace Hopper’s birth date, and how to perform more complex, multi-stage queries, like first finding the unique identifiers for entities like Bill Gates and Microsoft and then discovering the relationships that connect them within the knowledge graph.

The core of this example is built around the query-wikidata and query-wikidata/alist functions. These procedures handle the low-level details of HTTP communication with the Wikidata SPARQL endpoint. They take a SPARQL query string, URI-encode it, and embed it into a GET request URL. They set a User-Agent string, a best practice that helps service operators identify the source of traffic; the functions allow this header to be overridden via an optional argument. After a successful request, the real data processing begins. The raw JSON response from a SPARQL endpoint is deeply nested, with each result variable wrapped in an object containing its type and value. The helper functions process-sparql-results and process-sparql-results-alist traverse this structure to extract the essential value of each binding, returning a clean list of results where each result is either a hash table or an association list mapping variable names (as symbols) to their values.

The included test functions, test1 and test2, serve as practical examples of this code’s capabilities. The test2 function is a straightforward lookup, retrieving the birth date and birthplace for a specific Wikidata entity (Grace Hopper, wd:Q7249). In contrast, test1 demonstrates a more powerful, dynamic pattern. It first runs a query to find the URIs for “Bill Gates” and “Microsoft” based on their English labels and types (human and business, respectively). It then uses these dynamically discovered URIs to construct a second query that finds all direct properties linking the two entities. This two-step approach is a common and robust method for interacting with linked data systems. Furthermore, the test1-ua and test2-ua variants illustrate how to provide a custom User-Agent string, for instance by reading it from an environment variable, showcasing the flexibility of the primary query functions.

We use a Makefile target to start a Gerbil Scheme REPL with the example code loaded:

Here is the Makefile:

And sample output:

Before the field of deep learning revolutionized the field of NLP, I created a commercial NLP product Knowledge Books Systems NLP that I first implemented in Common Lisp, then in Ruby, and then in Gambit Scheme. Because of the processing speed of my library, I still feel that it is a useful code because it processes text efficiently performing:

• Part of speech tagging
• Key phrase extraction
• Categorizes text

A more modern approach to writing code for Natural Language Processing involves applying computational techniques to analyze, understand, and generate human language, bridging the gap between human communication and computer interpretation. The field heavily relies on machine learning, predominantly using languages like Python due to its extensive ecosystem of specialized libraries such as NLTK (Natural Language Toolkit), spaCy, and Hugging Face’s transformers. These tools provide the building blocks for implementing a wide array of NLP tasks, from foundational steps like tokenization (splitting text into words or sentences) and part-of-speech tagging to more complex applications like sentiment analysis, named entity recognition (identifying people and places), machine translation, and text summarization. At its core, coding for NLP is about converting unstructured text into a structured data format that machine learning models can process, and then using those models to derive meaningful insights, power conversational agents, or generate new, coherent text, thereby enabling software to interact with the world in a more human-like manner.

The project directory gerbil_scheme_book/source_code/SchemeKBS contains hand written NLP utilities, the sub-directory generated-code contains classification and linguistic data as literal data embedded directly in Scheme source code. These files were auto-generated by Ruby utilities I wrote in 2005. The sub-directory data contains additional linguistic data files.

The Makefile provides a standard set of targets for common development tasks. A key feature of this setup is the use of a project-local build environment. All Gerbil build artifacts, compiled modules, and package dependencies are installed into a .gerbil directory within the project root, rather than the user’s global ~/.gerbil directory.

This is achieved by temporarily setting the HOME environment variable to the current directory for all Gerbil compiler (gxc) and interpreter (gxi) commands:

This approach ensures that the project is self-contained and builds are reproducible, without interfering with your global Gerbil installation.

The build.ss script is expected to contain the primary compilation and linking logic to produce the final executable(s).

make test

This target runs a smoke test on the application. It first ensures all modules are compiled (by depending on compile-mods), then executes testapp.ss with a predefined test data file (climate_g8.txt). The output is written to .gerbil/test-output.json, and a 300-byte preview of the output is printed to the console:

make test-fast

This target is a variation of test that also runs the interpreter-based smoke test. The primary purpose is to provide a quick feedback loop during development, bypassing any potentially slow static executable linking steps that might be part of the main build target.

make compile-mods

This is a utility target that pre-compiles all core Scheme source files (.ss) into the project-local .gerbil directory using the Gerbil compiler (gxc). The test and test-fast targets depend on this to ensure modules are up-to-date before running the test application. This separation speeds up subsequent runs, as modules are not recompiled unnecessarily.

This program serves as the main command-line interface for our NLP text analysis library. Its primary responsibility is to orchestrate the text processing workflow by parsing command-line arguments, invoking the core analysis engine, and serializing the results into a structured JSON format. The utility is designed to read the path to a source text file and a destination output file from the user. After processing the input file with the process-file function from the underlying library, it constructs a JSON object containing extracted data such as significant words, tags, key phrases, and scored categories. To accomplish this without external dependencies, the program includes a minimal, custom-built set of functions for escaping special characters and writing JSON-compliant strings and arrays, demonstrating fundamental principles of data serialization, file I/O, and application entry point logic in a functional programming context.

The program’s logic is centered in the main function, which acts as the application controller. It begins by parsing the program’s command-line arguments, using the member procedure to check for the presence of -i (input file), -o (output file), and -h (help) flags. The control flow is straightforward: if the help flag is present or if the required file arguments are missing, a help message is displayed. Otherwise, the core logic proceeds within a with-output-to-file block, which ensures that the output is correctly directed to the user-specified file. Inside this block, the external process-file function is called, and its returned data structures are placed into a hash table, which is then passed to our custom JSON writer.

A notable feature of this code is its self-contained approach to JSON serialization. Instead of relying on a third-party library, we build the JSON output manually through a series of specialized helper functions. The json-escape function handles the critical task of properly escaping special characters within strings to ensure the output is valid. Building on this, procedures like write-json-string-list and write-json-categories use a common Scheme pattern, the named let loop, to iterate over lists and recursively construct the JSON array syntax, carefully managing the placement of commas between elements. The final json-write function assembles the complete JSON object by explicitly printing the keys and calling the appropriate helper for each value, providing a clear and direct implementation of a data serialization routine.

We will not discuss the following code files:

• fasttag.ss - part of speech tagger.
• place-names.ss - identify place names in text.
• summarize.ss - summarize text.
• utils.ss - misc. utility functions.
• category.ss - classifies (or categorizes) text.
• key-phrases.ss - extracts key phrases from text.
• main.ss - main, or top level, interface functions.
• proper-names.ss - identify proper names in text.

On some systems, you might run into link compatibility probelems. I did on macOS when I brew installed Gerbil Scheme and later brew updated openssl to a newer version.

As a result of this configuration problem, the make test target runs an interpretter target, bypassing any potential link problems.

The Foreign Function Interface (FFI) in Gerbil Scheme provides a powerful bridge to leverage existing C libraries directly within Scheme programs. This allows developers to extend Scheme applications with highly optimized, domain-specific functionality written in C, while still enjoying the high-level abstractions and rapid development style of Scheme. In this chapter, we will explore an end-to-end example of integrating the Raptor RDF parsing and serialization library into Gerbil Scheme, showing how to bind C functions and expose them as Scheme procedures.

Raptor is a mature C library for parsing, serializing, and manipulating RDF (Resource Description Framework) data in a variety of syntaxes, including RDF/XML, Turtle, N-Triples, and JSON-LD. By accessing Raptor from Gerbil Scheme, we open the door to semantic web applications, linked data processing, and graph-based reasoning directly within a Scheme environment. This example illustrates the mechanics of building FFI bindings, handling C-level memory and data types, and translating them into idiomatic Scheme representations. For reference, the official Raptor API documentation is available here: Raptor2 API Reference.

We will walk through the process step by step: setting up the Gerbil Scheme FFI definitions, mapping C functions and structs into Scheme, and writing test programs that parse RDF data and extract triples. Along the way, we will highlight practical issues such as error handling, symbol exporting, and resource cleanup. By the end of this chapter, you should have both a working Gerbil Scheme binding to Raptor and a general blueprint for integrating other C libraries into your Scheme projects. For background on Gerbil Scheme’s FFI itself, consult the Gerbil Documentation: FFI.

The library is located in the file gerbil_scheme_book/source_code/RaptorRDF_FFI/ffi.ss. This code demonstrates how to use the Foreign Function Interface (FFI) to integrate with the Raptor RDF C library. It provides a Scheme-accessible procedure, raptor-parse-file->ntriples, which parses an RDF file in a specified syntax (such as Turtle or RDF/XML) and returns the results as an N-Triples–formatted string. This example highlights the practical use of FFI in Gerbil Scheme: exposing a C function to Scheme, managing memory safely across the boundary, and translating RDF data into a representation that Scheme programs can process directly.

The C portion begins by including the raptor2.h header and defining a callback function, triples_to_iostr, which takes RDF statements and writes them to a Raptor iostream in N-Triples format. This callback escapes subjects, predicates, and objects correctly and ensures triples are terminated with a period and newline, conforming to the N-Triples standard. The main work is performed in parse_file_to_ntriples, which initializes a Raptor world and parser, configures the statement handler to use the callback, and sets up an iostream that accumulates parsed triples into a string buffer. Error checks are in place at every step, ensuring resources such as the world, parser, URIs, and iostream are properly freed if initialization fails.

After setup, the parser processes the input file identified by its filename and syntax. Each RDF statement is converted into N-Triples and appended to the output string via the iostream. Once parsing is complete, the parser, URIs, iostream, and world are released, leaving a fully materialized string containing the N-Triples serialization. This string is returned to Scheme through the FFI, where Gambit copies it into a managed Scheme string. On the Scheme side, the define-c-lambda form binds this C function as the procedure raptor-parse-file->ntriples, exposing it with the expected (filename syntax-name) -> ntriples-string interface. The result is a clean abstraction: Scheme code can call raptor-parse-file->ntriples with an RDF file and syntax, receiving back normalized N-Triples ready for further processing in Gerbil Scheme.

This Gerbil Scheme test code in the file test.ss exercises the FFI binding raptor-parse-file->ntriples by creating a minimal Turtle input, invoking the parser in two modes (“turtle” and “guess”), and asserting that both produce the same canonical N-Triples output. It’s designed to be self-contained: it writes a temporary .ttl file, runs the conversion twice, compares results against an expected string, then cleans up and prints status.

This code exports main and imports the FFI wrapper. Utility helpers include write-file (persist a string to disk), read-file (characterwise file read; defined but unused here), and assert-equal, which prints PASS/FAIL labels and exits with non-zero status on mismatch. In function main a small Turtle document defines a simple triple using the ex: prefix; the corresponding expected N-Triples string is the fully expanded IRI form with a terminating period and newline.

The test proceeds in two phases: first it calls raptor-parse-file->ntriples ttl-file “turtle” and checks the result; then it repeats using “guess” to confirm the parser’s auto-detection path yields identical serialization. After both assertions pass, it deletes the temporary file and prints “All tests passed.” The result is a minimal but effective smoke test verifying the FFI, Raptor’s parsing/serialization, and the contract that both explicit syntax selection and guessing produce stable N-Triples output.

We use a Makefile to build an executable on macOS:

Alternatively, the Makefile for Linux can me run using make -f Makefile.linux:

Test output is:

The example for this chapter is a complete, self-contained bridge between Gerbil Scheme and a well-established C RDF toolchain. It demonstrates how to load RDF data into an in-memory store and execute SPARQL queries from Gerbil via a Foreign Function Interface (FFI). Under the hood it uses Serd and Sord to parse and hold triples, and Rasqal/Raptor to prepare, execute, and serialize SPARQL results. The project ships with both a small C demo CLI to test the C language part of this project and a Gerbil-based client, making it easy to explore end-to-end from raw data files to query results printed on the console.

The core is a compact C wrapper that exposes four functions: initialize the store with a data file (rdf_init), run a SPARQL query and get results as a string (rdf_query_copy), and clean up (rdf_free). Data is parsed with Serd and stored in Sord’s in-memory model; queries are executed with Rasqal, and results are serialized to TSV by Raptor with the column header removed for line-per-row output. On the Gerbil side, rdfwrap.ss defines the FFI bindings and test.ss provides a simple CLI program, illustrating a clean pattern for calling native libraries from Gerbil without excess machinery.

Using the project is straightforward. After installing the required libraries (Homebrew recipes are listed in the README for macOS and there is a separate Linux Makefile for Linux users), make produces a shared library, a C demo (DEMO_rdfwrap), and a Gerbil executable (TEST_client). You can point either binary at a small example dataset (TTL or NT representations are included) and supply a SPARQL SELECT query to print results. The Gerbil client supports sensible defaults, inline queries, and reading queries from files via the @file convention, making it convenient for quick experiments, regression checks, or embedding SPARQL query capabilities into larger Gerbil programs.

This example application aims to be an educational, practical starting point rather than a full-featured RDF database because data is stored strictly in memory. It focuses on clarity and minimal surface area: a tiny C API, a thin FFI layer, and a simple CLI that you can adapt. From here, natural extensions include loading multiple graphs, supporting additional RDF syntaxes, handling named graphs/datasets, improving error reporting, and streaming or structuring results beyond TSV. If you’re learning Gerbil FFI, integrating SPARQL into Scheme, or want a small reference for wiring C libraries into a Lisp workflow, the example in this chapter provides a concise, working template.

Here we will be using Sord which is very lightweight C library for storing RDF triples in memory.

Limitation: Sord itself is just a data store. It does not have a built-in SPARQL engine. However, it’s designed to be used with other libraries, and we will pair it with Rasqal (from the Redland RDF suite) to add SPARQL query capability.

Rasqal is a SPARQL query library. It can parse a SPARQL query and execute it against a graph datastore like Sord or Redland project’s librdf. We could have used librdf with Rasqal in our implementation but I felt that it is a better example wrapping libraries from different projects.

This project is in the directory gerbil_scheme_book/source_code/SparqlRdfStore and contains a sub-directory C-source for our implementation of a C language wrapper for the Sord and Rasqal libraries. The top level project direcory contains Gerbil Scheme source files, example RDF data files, and a file containing a test SPARQL query:

The following Makefile builds the project for macOS:

Alternatively, you can run make -f Makefile.linux to build for Linux:

Here we write a wrapper that will later be called from Gerbil Scheme code.

The following listing shows the wrapper C-source/wrapper.c. This C wrapper provides a simplified, high-level interface for handling RDF data by leveraging the combined power of the serd, sord, rasqal, and raptor2 libraries. The primary goal of this wrapper is to abstract away the complexities of library initialization, data parsing, query execution, and results serialization. It exposes a minimal API for loading an RDF graph from a Turtle file and executing SPARQL queries against it, returning the results in a straightforward, easy-to-parse string format. This makes it an ideal solution for applications that need to embed RDF query functionality without managing the intricate details of the underlying RDF processing stack.

The core functionality begins with initialization and data loading, handled by the rdf_init() function. This function sets up the necessary in-memory storage by creating a SordWorld and a SordModel, which serve as the container for the RDF graph. It then calls the internal helper function load_turtle_into_sord() that uses the serd parser to efficiently read and load the triples from a specified Turtle (.ttl) file into the sord model. This process establishes the in-memory database that all subsequent queries will be executed against. The path to the data file is stored globally for later use by the query engine.

Once the data is loaded, the rdf_query() function provides the mechanism for executing SPARQL queries. It orchestrates the rasqal query engine to parse and prepare the SPARQL query string. The function then uses the raptor2 library to create a URI reference to the original data file and associates it with the query as a data graph. After executing the query, rasqal and raptor work together to serialize the query results into a Tab-Separated Values (TSV) formatted string. As a final convenience, the code processes this string to remove the header row, ensuring that the returned buffer contains only the result data, with each row representing a solution.

Finally, the wrapper provides essential memory management and utility functions. The rdf_free() function is responsible for cleanly deallocating all resources, including the sord world and model, the serd environment, and any other globally allocated memory, preventing memory leaks. The rdf_query_copy() function is a simple convenience utility that executes a query via rdf_query() and returns a new, separately allocated copy of the result string, which can be useful for certain memory management patterns. The code also includes an optional main function, enabled by the RDF_DEMO_MAIN macro, which demonstrates the wrapper’s usage and allows it to function as a standalone command-line tool for quick testing.

The shim code is n the source file rdfwrap.ss:

This Scheme source file serves as a crucial Foreign Function Interface (FFI) shim, creating a bridge between the high-level Scheme environment and the low-level C functions compiled into the libRDFWrap.dylib shared library. Its purpose is to “wrap” the C functions, making them directly callable from Scheme code as if they were native procedures. By handling the data type conversions and function call mechanics, this shim abstracts away the complexity of interoperating between the two languages, providing a clean and idiomatic Scheme API for the underlying RDF processing engine.

The entire FFI definition is encapsulated within a (begin-ffi …) block, which sets up the necessary context for interfacing with foreign code. Inside this block, the first step is the c-declare form. This form contains a string of C code that declares the function prototypes for the C library’s public API. By providing the signatures for rdf_init, rdf_query_copy, and rdf_free, it informs the Scheme FFI about the exact names, argument types, and return types of the C functions it needs to connect with. This declaration acts as a contract, ensuring that the Scheme bindings will match the expectations of the compiled C library.

Following the declaration, the script defines the actual Scheme procedures using define-c-lambda. Each of these forms creates a direct binding from a new Scheme function to a C function. For instance, (define-c-lambda rdf-init …) creates a Scheme function named rdf-init that calls the C function of the same name. This form also specifies the marshalling of data types between the two environments, such as converting a Scheme string to a C char-string (const char*).

Notably, the Scheme function rdf-query is explicitly mapped to the C function rdf_query_copy. This is a deliberate design choice to simplify memory management. The rdf_query_copy function in C returns a distinct, newly allocated copy of the result string. This prevents the Scheme garbage collector from trying to manage memory that was allocated by the C library’s malloc, avoiding potential conflicts and memory corruption. The final binding for rdf-free provides a way to call the C cleanup function, ensuring that all resources allocated by the C library are properly released.

The main test program is in the file test.ss:

This Scheme script provides a user-friendly command-line interface (CLI) for the underlying C-based RDF wrapper. It acts as a high-level controller, responsible for parsing user input, managing file operations, and invoking the core RDF processing functions exposed by the C library. The script allows a user to specify an RDF data file and a SPARQL query directly on the command line or from a file. By handling argument parsing and I/O, it simplifies the process of interacting with the RDF engine, making it accessible and easy to use for quick queries and testing without needing to write and compile C code.

The script’s main function serves as the primary entry point and is responsible for processing command-line arguments. It intelligently determines the RDF data file path and the SPARQL query string from the arguments provided by the user. If the user omits these arguments, the script falls back to sensible defaults: “mini.nt” for the data file and a simple SELECT query to fetch all triples. Furthermore, it includes a basic help mechanism, displaying a usage message if the user provides “-h” or “—help” as an argument, guiding them on the correct command structure.

A key feature of the script is its ability to read the SPARQL query from two different sources. By default, it treats the command-line argument as the query string itself. However, if the query argument is prefixed with an “@” symbol (e.g., @myquery.sparql), the script interprets the rest of the string as a filename. It then proceeds to open and read the entire contents of that file, loading it into memory as the query to be executed. This flexibility allows users to easily run complex, multi-line queries that would be cumbersome to type directly into the terminal.

After parsing the inputs, the script interfaces directly with the C wrapper’s functions. It first calls rdf-init to load the specified RDF data file into the in-memory model. If initialization is successful, it passes the prepared SPARQL query to the rdf-query function, which executes the query and returns the results as a single string. The script then prints this result string to standard output. Finally, to ensure proper resource management and prevent memory leaks, it calls rdf-free to clean up the C library’s allocated resources before the program terminates.

Test the C library code:

Test the Gerbil Scheme client code:

Writing command line utilities is one of the most rewarding ways to use Gerbil Scheme. A small, fast executable that does one thing well can become a trusted part of your daily workflow, and Gerbil makes this style of development unusually pleasant. With a standard library that includes tools for argument parsing, clean error reporting, and multi-call interfaces, you can focus on the logic of your utility rather than boilerplate. The result is code that feels both expressive and pragmatic: terse enough to fit in a few dozen lines, yet powerful enough to interoperate with your wider toolchain. In a world of ever-growing frameworks and layers of abstraction, a lean Gerbil command line program can be a breath of fresh air.

The Gerbil ecosystem is designed around building and packaging binaries, so distribution is straightforward. A single source file can be compiled into a standalone executable, or for more ambitious projects, you can define a build script that packages multiple tools under one roof. Command line options are handled declaratively with getopt, while helper libraries provide standard exit codes, usage banners, and even multi-call support for utilities that expose multiple subcommands. This structured approach encourages writing utilities that are not only functional but also friendly to end users, programs that behave consistently and provide clear feedback. Once you have mastered this pattern, it becomes natural to treat Gerbil as both a systems language and a scripting tool.

Equally important is the mindset that comes with writing command line utilities. Each utility should feel like a sharp instrument, simple to invoke with predictable behavior and minimal dependencies. Gerbil lets you achieve this balance while still giving you access to higher-level abstractions when needed, whether you are parsing JSON, making HTTP requests, or wrapping a C library through the foreign function interface. This chapter will show how to construct utilities using both a simple structure and then using a more flexible structure by providing minimal “hello world” executable for each approach. We will explore both the mechanics of building programs, and also the design philosophy that makes small tools enduringly useful.

Here we look at two examples of ways to structure Gerbil Scheme command line applications set up in two project directories:

• command_line_utilities_first_demo_START_HERE - This is the simplest structure and the one I usually use.
• command_line_utilities - This is a more general purpose structure using a separate lib.ss and main.ss source files.

Let’s take a look at the directory structure and code, then build and run the example:

This program (file test-mytool.ss) demonstrates how to construct a complete, albeit simple, command-line utility in Gerbil Scheme. It leverages the powerful :std/cli/getopt library to handle argument parsing, a common requirement for such tools. The script is designed to accept two specific command-line options: a mandatory —name option that requires a string value, and an optional boolean flag, —verbose (or its short form -v), which toggles additional output. The program’s logic includes robust argument validation, printing a usage message and exiting if the required name is not provided. Upon successful execution, it greets the user by name, optionally printing a “verbose on” message if the corresponding flag was set, showcasing a standard pattern for creating interactive and user-friendly command-line applications.

The example file test-tool.ss: