Code for Natural Language Processing (NLP)

Before the field of deep learning revolutionized the field of NLP, I created a commercial NLP product Knowledge Books Systems NLP that I first implemented in Common Lisp, then in Ruby, and then in Gambit Scheme. Because of the processing speed of my library, I still feel that it is a useful code because it processes text efficiently performing:

• Part of speech tagging
• Key phrase extraction
• Categorizes text

A more modern approach to writing code for Natural Language Processing involves applying computational techniques to analyze, understand, and generate human language, bridging the gap between human communication and computer interpretation. The field heavily relies on machine learning, predominantly using languages like Python due to its extensive ecosystem of specialized libraries such as NLTK (Natural Language Toolkit), spaCy, and Hugging Face’s transformers. These tools provide the building blocks for implementing a wide array of NLP tasks, from foundational steps like tokenization (splitting text into words or sentences) and part-of-speech tagging to more complex applications like sentiment analysis, named entity recognition (identifying people and places), machine translation, and text summarization. At its core, coding for NLP is about converting unstructured text into a structured data format that machine learning models can process, and then using those models to derive meaningful insights, power conversational agents, or generate new, coherent text, thereby enabling software to interact with the world in a more human-like manner.

On some macOS systems, you might run into link compatibility problems. I assume that macOS and Linux using readers followed the installation instructions in the Preface. I had a few Gerbil vs. OpenSSL issues on macOS that I fixed using:

As a result of this configuration problem, the make test target or the NLP example runs an interpreter target, bypassing any potential link problems.

I recommend that you start running this example interpretively using make test before compiling this example using make.

The project directory gerbil_scheme_book/source_code/NLP contains hand written NLP utilities, the sub-directory generated-code contains classification and linguistic data as literal data embedded directly in Scheme source code. These files were auto-generated by Ruby utilities I wrote in 2005. The sub-directory data contains additional linguistic data files.

The Makefile provides a standard set of targets for common development tasks. A key feature of this setup is the use of a project-local build environment. All Gerbil build artifacts, compiled modules, and package dependencies are installed into a .gerbil directory within the project root, rather than the user’s global ~/.gerbil directory.

This is achieved by temporarily setting the HOME environment variable to the current directory for all Gerbil compiler (gxc) and interpreter (gxi) commands:

This approach ensures that the project is self-contained and builds are reproducible, without interfering with your global Gerbil installation.

The build.ss script is expected to contain the primary compilation and linking logic to produce the final executable(s).

make test

This target runs a smoke test on the application. It first ensures all modules are compiled (by depending on compile-mods), then executes nlp.ss with a predefined test data file (climate_g8.txt). The output is written to .gerbil/test-output.json, and a 300-byte preview of the output is printed to the console:

make test-fast

This target is a variation of test that also runs the interpreter-based smoke test. The primary purpose is to provide a quick feedback loop during development, bypassing any potentially slow static executable linking steps that might be part of the main build target.

make compile-mods

This is a utility target that pre-compiles all core Scheme source files (.ss) into the project-local .gerbil directory using the Gerbil compiler (gxc). The test and test-fast targets depend on this to ensure modules are up-to-date before running the test application. This separation speeds up subsequent runs, as modules are not recompiled unnecessarily.

This program serves as the main command-line interface for our NLP text analysis library. Its primary responsibility is to orchestrate the text processing workflow by parsing command-line arguments, invoking the core analysis engine, and serializing the results into a structured JSON format. The utility is designed to read the path to a source text file and a destination output file from the user. After processing the input file with the process-file function from the underlying library, it constructs a JSON object containing extracted data such as significant words, tags, key phrases, and scored categories. To accomplish this without external dependencies, the program includes a minimal, custom-built set of functions for escaping special characters and writing JSON-compliant strings and arrays, demonstrating fundamental principles of data serialization, file I/O, and application entry point logic in a functional programming context.

The program’s logic is centered in the main function, which acts as the application controller. It begins by parsing the program’s command-line arguments, using the member procedure to check for the presence of -i (input file), -o (output file), and -h (help) flags. The control flow is straightforward: if the help flag is present or if the required file arguments are missing, a help message is displayed. Otherwise, the core logic proceeds within a with-output-to-file block, which ensures that the output is correctly directed to the user-specified file. Inside this block, the external process-file function is called, and its returned data structures are placed into a hash table, which is then passed to our custom JSON writer.

A notable feature of this code is its self-contained approach to JSON serialization. Instead of relying on a third-party library, we build the JSON output manually through a series of specialized helper functions. The json-escape function handles the critical task of properly escaping special characters within strings to ensure the output is valid. Building on this, procedures like write-json-string-list and write-json-categories use a common Scheme pattern, the named let loop, to iterate over lists and recursively construct the JSON array syntax, carefully managing the placement of commas between elements. The final json-write function assembles the complete JSON object by explicitly printing the keys and calling the appropriate helper for each value, providing a clear and direct implementation of a data serialization routine.

We will not discuss the following code files:

• fasttag.ss - part of speech tagger.
• place-names.ss - identify place names in text.
• summarize.ss - summarize text.
• utils.ss - misc. utility functions.
• category.ss - classifies (or categorizes) text.
• key-phrases.ss - extracts key phrases from text.
• main.ss - main, or top level, interface functions.
• proper-names.ss - identify proper names in text.

On some systems, you might run into link compatibility probelems. I did on macOS when I brew installed Gerbil Scheme and later brew updated openssl to a newer version.

As a result of this configuration problem, the make test target runs an interpretter target, bypassing any potential link problems.