Cover Material, Copyright, and License

Copyright 2020-2024 Mark Watson. All rights reserved. This book may be shared using the Creative Commons “share and share alike, no modifications, no commercial reuse” license.

This eBook will be updated occasionally so please periodically check the leanpub.com web page for this book for updates.

If you read my eBooks free online then please consider hiring me for consulting work https://markwatson.com.

This is this edition released September 2024.

Please visit the author’s website.

Preface

While this is a book on the Hy Lisp language, we have a wider theme here. In an age where artificial intelligence (AI) is a driver of the largest corporations and government agencies, the question is how do individuals and small organizations take advantage of AI technologies given the disadvantages of small scale. The material I chose to write about here is selected to help you, dear reader, survive as a healthy small fish in a big bond.

I have been using Lisp languages professionally since 1982 and have written books covering the Common Lisp and Scheme languages. Most of my career has involved working on AI projects so tools for developing AI applications will be a major theme. In addition to covering the Hy language, you will get experience with AI tools and techniques that will help you craft your own AI platforms regardless of whether you are a consultant, work at a startup, or a corporation.

The latest version of this book (updated in May 2023) has major code changes that were required to support changes in the Hy language version 0.26.0.

The code examples can be found in my GitHub repository https://github.com/mark-watson/hy-lisp-python.

This book covers many programming topics using the Lisp language Hy that compiles to Python AST and is compatible with code, libraries, and frameworks written in Python. The main topics we will cover and write example applications for are:

• Relational and graph databases
• Web app development
• Web scraping
• Accessing semantic web and linked data sources like Wikipedia, DBpedia, and Wikidata
• Automatically constructing Knowledge Graphs from text documents, semantic web and linked data
• Deep Learning
• Natural Language Processing (NLP) using Deep Learning

The topics were chosen because of my work experience and the theme of this book is how to increase programmer productivity and happiness using a Lisp language in a bottom-up development style. This style relies heavily on the use of an interactive REPL for exploring APIs and writing new code. I chose the above topics based on my experience working as a developer and researcher. Please note: you will see the term REPL frequently in this book. REPL stands for Read Eval Print Loop.

Some of the examples are very simple (e.g., the web app examples) while some are more complex (e.g., Deep Learning and knowledge graph examples). Regardless of the simplicity or complexity of the examples I hope that you find the code interesting, useful in your projects, and fun to experiment with.

Setting Up Your Development Environment

In September 2024 I changed the way that I build and run both Hy language and Python scripts and programs. I now use venv and install dependencies in each Hy or Python project directory. I use a shell script venv_setup.sh that I keep on my PATH to create a directory venv. Please note that this method uses more disk space but keeps the dependencies of each small of large project separate.

 1 #!/bin/zsh
 2 
 3 # Check if the directory has a requirements.txt file
 4 if [ ! -f "requirements.txt" ]; then
 5     echo "No requirements.txt file found in the current directory."
 6     exit 1
 7 fi
 8 
 9 # Create a virtual environment in the .venv directory
10 python3 -m venv venv
11 
12 # Activate the virtual environment
13 source venv/bin/activate
14 
15 # Upgrade pip to the latest version
16 pip3 install --upgrade pip
17 
18 # Install dependencies from requirements.txt
19 pip3 install -r requirements.txt
20 
21 # Display installed packages
22 pip3 list
23 
24 echo "Virtual environment setup complete. Activate it with:"
25 echo "source venv/bin/activate"
26 echo ""

You will find requirements.txt files in most example subdirectories for this book.

To free disk space for the venv directories, I define a top level Makefile in the GitGub repository for the example programs https://github.com/mark-watson/hy-lisp-python:

1 clean:
2 	rm -rf */__pycache__ */venv

What is Lisp Programming Style?

I will give some examples here and also show exploratory Hy language REPL examples later in the book. How often do you search the web for documentation on how to use a library, write some code only to discover later that you didn’t use the API correctly? I reduce the amount of time that I spend writing code by having a Lisp REPL open so that I can experiment with API calls and returned results while reading the documentation.

When I am working on new code or a new algorithm I like to have a Lisp REPL open and try short snippets of code to get working code for solving low level problems, building up to more complex code. As I figure out how to do things I enter code that works and which I want to keep in a text editor and then convert this code into my own library. I then iterate on loading my new library into a REPL and stress test it, look for API improvements, etc.

I find, in general, that a “bottom-up” approach gets me to working high quality systems faster than spending too much time doing up front planning and design. The problem with spending too much up front time on design is that we change our minds as to what makes the most sense to solve a problem as we experiment with code. I try to avoid up front time spent on work that I will have to re-work or even toss out.

Hy is Python, But With a Lisp Syntax

When I need a library for a Hy project I search for Python libraries and either write a thin Hy language “wrapper” around the Python library or just call the Python APIs directly from Hy code. You will see many examples of both approaches in this book.

How This Book Reflects My Views on Artificial Intelligence and the Future of Society and Technology

Since starting work on AI in 1982 I have seen the field progress from a niche technology where even international conferences had small attendances to a field that is generally viewed as transformative. In the USA there is legitimate concern that economic adversaries like China will exceed our abilities to develop core AI technologies and integrate these technologies into commercial and military systems. As I write this in February 2020, some people in our field including myself believe that the Chinese company Baidu may have already passed Google and Microsoft in applied AI.

Even though most of my professional work in the last five years has been in Deep Learning (and before that I worked with the Knowledge Graph at Google on a knowledge representation problem and application), I believe that human level Artificial General Intelligence (AGI) will use hybrid Deep Learning, “old fashioned” symbolic AI, and techniques that we have yet to discover.

This belief that Deep Learning will not get us to AGI capabilities is a motivation for me to use the Hy language because it offers transparent access to Python Deep Learning frameworks with a bottom-up Lisp development style that I have used for decades using symbolic AI and knowledge representation.

I hope you find that Hy meets your needs as it does my own.

About the Book Cover

The official Hy Language logo is an octopus:

Usually I use photographs that I take myself for covers of my LeanPub books. Although I have SCUBA dived since I was 13 years old, sadly I have no pictures of an octopus that I have taken myself. I did find a public domain picture I liked (that is the cover of this book) on Wikimedia. Cover Credit: Thanks to Wikimedia user Pseudopanax for placing the cover image in the public domain.

A Request from the Author

I spent time writing this book to help you, dear reader. I release this book under the Creative Commons “share and share alike, no modifications, no commercial reuse” license and set the minimum purchase price to $8.00 in order to reach the most readers. You can read this book for free online or purchase it by visiting https://leanpub.com/hy-lisp-python.

If you would like to support my work please consider purchasing my books on Leanpub and star my git repositories that you find useful on GitHub. You can also interact with me on social media on Mastodon and Twitter.

I enjoy writing and your support helps me write new editions and updates for my books and to develop new book projects. Thank you!

Acknowledgements

I thank my wife Carol for editing this manuscript, finding typos, and suggesting improvements.

I would like to thank Pascal (Reddit user chuchana) for corrections and suggestions. I would like to thank Carlos Ungil for catching a typo and reporting it. I would like to thank Jud Taylor for finding several typo errors. I would like to thank Dave Smythe for finding some typos.

Introduction to the Hy Language

The Hy programming language is a Lisp language that inter-operates smoothly with Python. We start with a few interactive examples that I encourage you to experiment with as you read. Then we will look at Hy data types and commonly used built-in functions that are used in the remainder of this book.

I assume that you know at least a little Python and more importantly the Python ecosystem and general tools like pip.

Please start by installing Hy in your current Python environment:

pip install git+https://github.com/hylang/hy.git

Using Python Libraries

Using Python libraries like TensorFlow, Keras, BeautifulSoup, etc. are the reason I use the Hy language. Importing Python code and libraries and calling out to Python is simple and here we look at sufficient examples so that you will understand example code that we will look at later.

For example, in the chapter Responsible Web Scraping we will use the BeautifulSoup library. We will look at some Python code snippets and the corresponding Hy language versions of these snippets. Let’s first look at a Python example that we will then convert to Hy:

1 from bs4 import BeautifulSoup
2 
3 raw_data = '<html><body><a href="http://markwatson.com">Mark</a></body></html>'
4 soup = BeautifulSoup(raw_data)
5 a_tags = soup.find_all("a")
6 print("a tags:", a_tags)

In the following listing notice how we import other code and libraries in Hy. The special form setv is used to define variables in a local context. Since the setv statements in lines 4, 6, and 7 are used at the top level, they are global in the Python/Hy module named after the root name of the source file.

 1 $ hy
 2 hy 0.18.0 using CPython(default) 3.7.4 on Darwin
 3 => (import bs4 [BeautifulSoup])
 4 => (setv raw-data "<html><body><a href=\"http://markwatson.com\">Mark</a></body></ht\
 5 ml>")
 6 => (setv soup (BeautifulSoup raw-data "lxml"))
 7 => (setv a (.find-all soup "a"))
 8 => (print "atags:" a)
 9 atags: [<a href="http://markwatson.com">Mark</a>]
10 => (type a)
11 <class 'bs4.element.ResultSet'>
12 => (dir a)
13 ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__dict__', '\
14 __dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribut\
15 e__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__in\
16 it_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', \
17 '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__r\
18 mul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '\
19 __weakref__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop'\
20 , 'remove', 'reverse', 'sort', 'source']

Notice in lines 3 and 6 that we can have “-“ characters inside of variable and function names (raw-data and find-all in this case) in the Hy language where we might use “_” underscore characters in Python. Like Python, we can use type get get the type of a value and dir to see what symbols are available for a object.

Global vs. Local Variables

Although I don’t generally recommend it, sometimes it is convenient to export local variables defined with setv to be global variables in the context of the current module (that is defined by the current source file). As an example:

 1 Marks-MacBook:deeplearning $ hy
 2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
 3 => (defn foo []
 4 ... (global x)
 5 ... (setv x 1)
 6 ... (print x))
 7 => (foo)
 8 1
 9 => x
10 1
11 => 

Before executing function foo the global variable x is undefined (unless you coincidentally already defined somewhere else). When function foo is called, a global variable x is defined and then it equal to the value 1.

Using Python Code in Hy Programs

If there is a Python source file, named for example, test.py in the same directory as a Hy language file:

1 def factorial (n):
2   if n < 2:
3     return 1
4   return n * factorial(n - 1)

This code will be in a module named test because that is the root source code file name. We might import the Python code using the following in Python:

1 import test
2 
3 print(test.factorial(5))

and we can use the following in Hy to import the Python module test (defined in test.py):

1 (import test)
2 
3 (print (test.factorial 5))

Running this interactively in Hy:

1 $ hy
2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
3 => (import test)
4 => test
5 <module 'test' from '/Users/markw/GITHUB/hy-lisp-python/test.py'>
6 => (print (test.factorial 5))
7 120

If we only wanted to import BeautifulSoup from the Python BeautifulSoup library bs4 we can specify this in the import form:

1 (import bs4 [BeautifulSoup])

Using Hy Libraries in Python Programs

There is nothing special about importing and using Hy library code or your own Hy scripts in Python programs. The directory hy-lisp-python/use_hy_in_python in the git repository for this book https://github.com/mark-watson/hy-lisp-python contains an example Hy script get_web_page.hy that is a slightly modified version of code we will explain and use in the later chapter on web scraping and a short Python script use_hy_stuff.py that uses a function defined in Hy:

get_web_page.hy:

 1 (import argparse os)
 2 (import urllib.request [Request urlopen])
 3 
 4 (defn get-raw-data-from-web [aUri [anAgent
 5                                    {"User-Agent" "HyLangBook/1.0"}]]
 6   (setv req (Request aUri :headers anAgent))
 7   (setv httpResponse (urlopen req))
 8   (setv data (.read httpResponse))
 9   data)
10 
11 (defn main_hy []
12   (print (get-raw-data-from-web "http://markwatson.com")))

We define two functions here. Notice the optional argument anAgent defined in lines 4-5 where we provide a default value in case the calling code does not provide a value. In the next Python listing we import the file in the last listing and call the Hy function main on line 4 using the Python calling syntax.

Hy is the same as Python once it is compiled to an abstract syntax tree (AST).

hy-lisp-python/use_in_python:

1 import hy
2 from get_web_page import main_hy
3 
4 main_hy()

What I want you to understand and develop a feeling for is that Hy and Python are really the same but with a different syntax and that both languages can easily be used side by side.

Replacing the Python slice (cut) Notation with the Hy Functional Form

In Python we use a special notation for extracting sub-sequences from lists or strings:

$ python
Python 3.7.3 (default, Mar 27 2019, 16:54:48) 
>>> s = '0123456789'
>>> s[2:4]
'23'
>>> s[-4:]
'6789'
>>> s[-4:-1]
'678'
>>> 

In Hy this would be:

$ hy
hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
=> (setv s "0123456789")
=> (cut s 2 4)
'23'
=> (cut s -4)
'6789'
=> (cut s -4 -1)
'678'
=> 

It also works to use cut with setv to destructively change a list; for example:

=> (setv x [0 1 2 3 4 5 6 7 8])
=> x
[0, 1, 2, 3, 4, 5, 6, 7, 8]
=> (cut x 2 4)
[2, 3]
=> (setv (cut x 2 4) [22 33])
=> x
[0, 1, 22, 33, 4, 5, 6, 7, 8]

Iterating Through a List With Index of Each Element

We will use lfor as a form of Python list comprehension; for example:

 1 => (setv sentence "The ball rolled")
 2 => (lfor i (enumerate sentence) i)
 3 [(0, 'T'), (1, 'h'), (2, 'e'), (3, ' '), (4, 'b'), (5, 'a'), (6, 'l'), (7, 'l'), (8,\
 4  ' '), (9, 'r'), (10, 'o'), (11, 'l'), (12, 'l'), (13, 'e'), (14, 'd')]
 5 => (setv vv (lfor i (enumerate sentence) i))
 6 => vv
 7 [(0, 'T'), (1, 'h'), (2, 'e'), (3, ' '), (4, 'b'), (5, 'a'), (6, 'l'), (7, 'l'), (8,\
 8  ' '), (9, 'r'), (10, 'o'), (11, 'l'), (12, 'l'), (13, 'e'), (14, 'd')]
 9 => (for [[a b] vv]
10 ... (print a b))
11 0 T
12 1 h
13 2 e
14 3  
15 4 b
16 5 a
17 6 l
18 7 l
19 8  
20 9 r
21 10 o
22 11 l
23 12 l
24 13 e
25 14 d
26 => 

On line 2, the expression (enumerate sentence) generates one character at a time from a string. enumerate operating on a list will generate one list element at a time.

Line 9 shows an example of destructuring: the values in the list vv are tuples (tuples are like lists but are immutable, that is, once a tuple is constructed the values it holds can not be changed) with two values. The values in each tuple are copied into binding variables in the list [a b]. We could have used the following code instead but it is more verbose:

=> (for [x vv]
    (setv a (first x))
    (setv b (second x))
... (print a b))
0 T
1 h
2 e
3  
4 b
 . . .
13 e
14 d
=> 

Formatted Output

I suggest using the Python format method when you need to format output. In the following repl listing, you can see a few formatting options: insert any Hy data into a string (line 3), print values with a specific width and right justified (in line 5 the width for both values is 15 characters), print values with a specific width and left justified (in line 7), and limiting the number of characters values can be expressed as (in line 9 the object “cat” is expressed as just the first two characters and the value 3.14159 is expressed as just three numbers, the period not counting).

$ hy
hy 0.26.0 using CPython(default) 3.11.1 on Darwin
=> (.format "first: {} second: {}" "cat" 3.14159)
'first: cat second: 3.14159'
=> (.format "first: {:>15} second: {:>15}" "cat" 3.14159)
'first:             cat second:         3.14159'
=> (.format "first: {:15} second: {:15}" "cat" 3.14159)
'first: cat             second:         3.14159'
=> (.format "first: {:.2} second: {:.3}" "cat" 3.14159)
'first: ca second: 3.14'
=> 

Notice the calling .format here returns a string value rather than writing to an output stream.

Importing Libraries from Different Directories on Your Laptop

I usually write applications by first implementing simpler low-level utility libraries that are often not in the same directory path as the application that I am working on. Let’s look at a simple example of accessing the library nlp_lib.hy in the directory hy-lisp-python/nlp from the directory hy-lisp-python/webscraping:

 1 Marks-MacBook:hy-lisp-python $ pwd
 2 /Users/markw/GITHUB/hy-lisp-python
 3 Marks-MacBook:hy-lisp-python $ cd webscraping 
 4 Marks-MacBook:webscraping $ hy
 5 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
 6 => (import sys)
 7 => (sys.path.insert 1 "../nlp")
 8 => (import nlp-lib [nlp])
 9 => (nlp "President George Bush went to Mexico and he had a very good meal")
10 {'text': 'President George Bush went to Mexico and he had a very good meal', 
11   ...
12  'entities': [['George Bush', 'PERSON'], ['Mexico', 'GPE']]}
13 => (import [coref-nlp-lib [coref-nlp]])
14 => (coref-nlp "President George Bush went to Mexico and he had a very good meal")
15 {'corefs': 'President George Bush went to Mexico and President George Bush had a ver\
16 y good meal',  ...  }}}
17 => 

Here I did not install the library nlp_lib.hy using Python setuptools (which I don’t cover in this book, you can read the documentation) as a library on the system. I rely on relative paths between the library directory and the application code that uses the library.

On line 6 I am inserting the library directory into the Python system load path so the import statement on line 8 can find the nlp-lib library and on line 13 can find the coref-nlp-lib library.

Hy Looks Like Clojure: How Similar Are They?

Clojure is a dynamic general purpose Lisp language for the JVM. One of the great Clojure features is support of immutable data (read only after creation) that makes multi-threaded code easier to write and maintain.

Unfortunately, Clojure’s immutable data structures cannot be easily implemented efficiently in Python so the Hy language does not support immutable data, except for tuples. Otherwise the syntax for defining functions, using maps/hash tables/dictionaries, etc. is similar between the two languages.

The original Hy language developer Paul Tagliamonte was clearly inspired by Clojure.

The book Serious Python by Julien Danjou has an entire chapter (Chapter 9) on the Python AST (abstract syntax tree), an introduction to Hy, and an interview with Paul Tagliamonte. Recommended!

This podcast in 2015 interviews Hy developers Paul Tagliamonte, Tuukka Turto, and Morten Linderud. You can see the current Hy contributor list on github.

Plotting Data Using the Numpy and the Matplotlib Libraries

Data visualization is a common task when working with numeric data. In a later chapter on Deep Learning we will use two functions, the relu and sigmoid functions. Here we will use a few simple Hy language scripts to plot these functions.

The Numpy library supports what is called “broadcasting” in Python. In the function sigmoid that we define in the following REPL, we can pass either a single floating point number or a Numpy array as an argument. When we pass a Numpy array, then the function sigmoid is applied to each element of the Numpy array:

 1 $ hy
 2 => (import numpy :as np)
 3 => (import matplotlib.pyplot :as plt)
 4 => 
 5 => (defn sigmoid [x]
 6 ...   (/ 1.0 (+ 1.0 (np.exp (- x)))))
 7 => (sigmoid 0.2)
 8 0.549833997312478
 9 => (sigmoid 2)
10 0.8807970779778823
11 => (np.array [-5 -2 0 2 5])
12 array([-5, -2,  0,  2,  5])
13 => (sigmoid (np.array [-5 -2 0 2 5]))
14 array([0.00669285, 0.11920292, 0.5, 0.88079708, 0.99330715])
15 => 

The git repository directory hy-lisp-python/matplotlib contains two similar scripts for plotting the sigmoid and relu functions. Here is the script to plot the sigmoid function:

 1 (import numpy :as np)
 2 (import matplotlib.pyplot :as plt)
 3 
 4 (defn sigmoid [x]
 5   (/ 1.0 (+ 1.0 (np.exp (- x)))))
 6 
 7 (setv X (np.linspace -8 8 50))
 8 (plt.plot X (sigmoid X))
 9 (plt.title "Sigmoid Function")
10 (plt.ylabel "Sigmoid")
11 (plt.xlabel "X")
12 (plt.grid)
13 (plt.show)

The generated plot looks like this on macOS (Matplotlib is portable and also works on Windows and Linux):

Bonus Points: Configuration for macOS and ITerm2 for Generating Plots Inline in a Hy REPL and Shell

On the macOS ITerm2 terminal app and on most Linux terminal apps, it is possible to get inline matplotlib plots in a shell (bash, zsh, etc.), in Emacs, etc. This will take some setup work but it is well worth it especially if you work on remote servers via SSH or tmux. Here is the setup for macOS:

1   pip3 install itermplot

The add the following to your .profile, .bash_profile, or .zshrc (depending on your shell setup):

1   export MPLBACKEND="module://itermplot"

Here we run an example from the last section in a zsh shell (bash, etc. also should work):

The best part of generating inline plots is during interactive REPL-based coding sessions:

If you use a Mac laptop to SSH into a remote Linux server you need to install itermplot and set the environment variable MPLBACKEND on the remote server.

Why Lisp?

Now that we have learned the basics of the Hy Lisp language in the last chapter, I would like to move our conversation to a broader question of why we would want to use Lisp. I want to start with my personal history of why I turned to Lisp languages in the late 1970s for almost all of my creative and research oriented development and later transitioned to also using Lisp languages in production.

I Hated the Waterfall Method in the 1970s but Learned to Love a Bottom-Up Programming Style

I graduated UCSB in the mid 1970s with a degree in Physics and took a job as a scientific programmer in the 100% employee owned company SAIC. My manager had a PhD in Computer Science and our team and the organization we were in used what is known as the waterfall method where systems were designed carefully from the top down, carefully planned mostly in their entirety, and then coded up. We, and the whole industry I would guess, wasted a lot of time with early planning and design work that had to be discarded or heavily modified after some experience implementing the system.

What would be better? I grew to love bottom-up programming. When I was given a new project I would start by writing and testing small procedures for low level operations, things I was sure I would need. I then aggregated the functionality into higher levels of control logic, access to data, etc. Finally I would write the high level application.

I mostly did this for a while writing code in FORTRAN at SAIC and using Algol for weekend consulting work Salk Institute, working on hooking up lab equipment to minicomputers in Roger Guillemin’s lab (he won a Nobel Prize during that time, which was exciting). Learning Algol, a very different language than FORTRAN, helped broaden my perspectives.

I wanted a better programming language! I also wanted a more productive way to do my job both as a programmer and to make the best use of the few free hours a week that I had for my own research and learning about artificial intelligence (AI). I found my “better way” of development by adopting a bottom-up style that involves first writing low level libraries and utilities and then layering complete programs on top of well-tested low level code.

First Introduction to Lisp

In the late 1970s I discovered a Lisp implementation on my company’s DECsystem-10 timesharing computer. I had heard of Lisp when reading Bertram Raphael’s book “THE THINKING COMPUTER. Mind Inside Matter” and I learned Lisp on my own time and then, during lunch hour, taught a one day a week class to anyone at work who wanted to learn Lisp. After a few months of Lisp experience I received permission to teach an informal lunch time class to teach anyone working in my building who wanted to to learn Lisp on our DECsystem-10.

Lisp is the perfect language to support the type of bottom-up iterative programming style that I like.

Commercial Product Development and Deployment Using Lisp

My company, SAIC, identified AI as an important technology in the early 1980s. Two friends at work (Bob Beyster who founded SAIC and Joe Walkush who was our corporate treasurer and who liked Lisp from his engineering studies at MIT) arranged for the company to buy a hardware Lisp Machine, a Xerox 1108 for me. I ported Charles Forgy’s expert system development language OPS5 to run on InterLisp-D on the Xerox Lisp Machines and we successfully sold this as a product. When Coral Common Lisp was released for the Apple Macintosh in 1984, I switched my research and development to the Mac and released ExperOPS5, which also sold well, and used Common Lisp to write the first prototypes for SAIC’s ANSim neural network library. I converted my code to C++ to productize it. We also continued to use Lisp for IR&D projects and while working on the DARPA NMRD project.

Even though I proceeded to use C++ for much of my development, as well as writing C++ books for McGraw-Hill and J. Riley publishers, Lisp remained my “thinking and research” language.

Performing Bottom Up Development Inside a REPL is a Lifestyle Choice

It is my personal choice to prefer a bottom up style of coding, effectively extending the Hy (or other Lisp) language to look like something that looks custom designed and built to solve a specific problem. This is possible in Lisp languages because once a function is defined, it is for our purposes part of the Hy language. If, for example, you are writing a web application that uses a database then I believe that it makes sense to first write low level functions to perform operations that you know you will need, for example, for creating and updating customer data from the database, utility functions used in a web application (which we cover in the next chapter), etc. For the rest of your application, you use these new low level functions as if they were built into the language.

When I need to write a new low-level function, I start in a REPL and define variables (with test values) for what the function arguments will be. I then write the code for the function one line at a time using these “arguments” in expressions that will later be copied to a Hy source file. Immediately seeing results in a REPL helps me catch mistakes early, often a misunderstanding of the type or values of intermediate calculations. This style of coding works for me and I hope you like it also.

Writing Web Applications

Python has good libraries and frameworks for building web applications and here we will use the Flask library and framework “under the hood” and write two simple Hy Language web applications. We will start with a simple “Hello World” example in Python, see how to reformulate it in Hy, and then proceed with more complex examples that will show how to use HTML generating templates, sessions, and cookies to store user data for the next time they visit your web site. In a later chapter we will cover use of the SQLite and PostgreSQL databases which are commonly used to persist data for users in web applications. This pattern involves letting a user login and store a unique token for the user in a web browser cookie. In principle, you can do the same with web browser cookies but if a user visits your web site with a different browser or device then they will not have access to the data stored in cookies on a previous visit.

I like lightweight web frameworks. In Ruby I use Sinatra, in Haskell I use Spock, and when I built Java web apps I liked lightweight tools like JSP. Flask is simple but capable and using it from Hy is productive and fun. In addition to using lightweight frameworks I like to deploy web apps in the simplest way possible. We will close this chapter by discussing how to use the Heroku and Google Cloud Platform AppEngine platforms.

Getting Started With Flask: Using Python Decorators in Hy

You will need to install Flask using:

pip install flask

We will use the Hy macro with-decorator to replace Python code with annotations. Here the decorator @app.route is used to map a URI pattern with a Python callback function. In the following case we define the behavior when the index page of a web app is accessed:

1 from flask import Flask
2 
3 @app.route('/')
4   def index():
5      return "Hello World !")
6 
7 app.run()

I first used Flask with the Hy language after seeing a post of code from HN user “volent”, seen in the file flask_test.hy in the directory hy-lisp-python/webapp that is functionally equivalent to the above Python code snippet:

1 #!/usr/bin/env hy
2 
3 ;; snippet by HN user volent and modifed for
4 ;; Hy 0.26.0 with a comment from stackoverflow user plokstele:
5 
6 (import flask [Flask])
7 (setv app (Flask "Flask test"))
8 (defn [(.route app "/")] index [] "Hello World !")
9 (app.run)

I liked this example and after experimenting with the code, I then started using Hy and Flask. Please try running this example to make sure you are setup properly with Flask:

(base) Marks-MacBook:webapp $ ./flask_test.hy 
 * Serving Flask app "Flask test" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Open http://127.0.0.1:5000/ in your web browser:

Using Jinja2 Templates To Generate HTML

Jinja2 is a templating system that allows HTML markup to be supplemented with Python variable references and simple Python loops, etc. The values of application variables can be stored in a context and the HTML template has the variables values substituted with current values before returning a HTML response to the user’s web browser.

By default Jinja2 templates are stored in a subdirectory named templates. The template for this example can be found in the file hy-lisp-python/webapp/templates/template1.j2 that is shown here:

 1 <html>
 2   <head>
 3     <title>Testing Jinja2 and Flask with the Hy language</title>
 4   </head>
 5   <body>
 6      {% if name %}
 7        <h1>Hello {{name}}</h1>
 8      {% else %}
 9        <h1>Hey, please enter your name!</h1>
10      {% endif %}
11     
12     <form method="POST" action="/response">
13       Name: <input type="text" name="name" required>
14       <input type="submit" value="Submit">
15     </form>
16   </body>
17 </html>

Note that in line 6 we are using a Python if expression to check if the variable name is defined in the current app execution context.

In the context of a running Flask app, the following will render the above template with the variable name defined as None:

1 (render_template "template1.j2")

We can set values as named parameters for variables used in the template, for example:

1 (render_template "template1.j2" :name "Mark")

I am assuming that you understand the basics or HTML and also GET and POST operations in HTTP requests.

The following Flask web app defines behavior for rendering the template without the variable name set and also a HTML POST handler to pass the name entered on the HTML form back to the POST response handler:

 1 #!/usr/bin/env hy
 2 
 3 (import flask [Flask render_template request])
 4 
 5 (setv app (Flask "Flask and Jinja2 test"))
 6 
 7 (defn [(.route app "/")]
 8   index []
 9     (render_template "template1.j2"))
10 
11 (defn [(.route app "/response" :methods ["POST"])]
12   response []
13     (setv name (request.form.get "name"))
14     (print name)
15     (render_template "template1.j2" :name name))
16 
17 (app.run)

Please note that there is nothing special about the names inside the with-decorator code blocks: the functions index and response could have arbitrary names like a123 an b17. I used the function names index and response because they help describe what the functions do.

Open http://127.0.0.1:5000/ in your web browser:

Handling HTTP Sessions and Cookies

There is a special variable session that Flask maintains for each client of a Flask web app. Different people using a web app will have independent sessions. In a web app, we can set a session value by treating the session for a given user as a dictionary:

=> (setv (get session "name") "Mark")
=> session
{'name': 'Mark'}

Inside a Jinja2 template you can use a simple Python expression to place a session variable’s value into the HTML generated from a template:

{{ session['name'] }}

In a web app you can access the session using:

(get session "name")

In order to set the value of a named cookie, we can:

1 (import flask [Flask render_template request make-response])
2 
3 (defn [(.route app "/response" :methods ["POST"])]
4   response []
5     (setv name (request.form.get "name"))
6     (print name)
7     (setv a-response (make-response (render-template "template1.j2" :name name)))
8     (a-response.set-cookie "hy-cookie" name)
9     a-response)

Values of named cookies can be retrieved using:

(request.cookies.get "name")

The value for request is defined in the execution context by Flask when handling HTTP requests. Here is a complete example of handling cookies in the file cookie_test.hy:

 1 #!/usr/bin/env hy
 2 
 3 (import flask [Flask render_template request make-response])
 4 
 5 (setv app (Flask "Flask and Jinja2 test"))
 6 
 7 (defn [(.route app "/")]
 8   index []
 9     (setv cookie-data (request.cookies.get "hy-cookie"))
10     (print "cookie-data:" cookie-data)
11     (setv a-response (render_template "template1.j2" :name cookie-data))
12     a-response)
13 
14 (defn [(.route app "/response" :methods ["POST"])]
15   response []
16     (setv name (request.form.get "name"))
17     (print name)
18     (setv a-response (make-response (render-template "template1.j2" :name name)))
19     (a-response.set-cookie "hy-cookie" name)
20     a-response)
21 
22 (app.run)

I suggest that you not only try running this example as-is but also try changing the template, and generally experiment with the code. Making even simple code changes helps to understand the code better.

Deploying Hy Language Flask Apps to Google Cloud Platform AppEngine

The example for this section is in a separate github repository that you should clone or copy to a new project for a starter project if you intend to deploy to AppEngine.

This AppEngine example is very similar to that in the last section except that it also serves a static asset and has a small Python stub main program to load the Hy language library and import the Hy language code.

Here is the Python stub main program:

1 import hy
2 import flask_test
3 from flask_test import app
4 
5 if __name__ == '__main__':
6     # Used when running locally only. When deploying to Google App
7     # Engine, a webserver process such as Gunicorn will serve the app.
8     app.run(host='localhost', port=9090, debug=True)

The Hy app is slightly different than we saw in the last section. On line 6 we specify the location of static assets and we do not call the run() method on the app object.

 1 (import flask [Flask render_template request])
 2 (import os)
 3 
 4 (setv port (int (os.environ.get "PORT" 5000)))
 5 
 6 (setv app (Flask "Flask test" :static_folder "./static" :static_url_path "/static"))
 7 
 8 (defn [(.route app "/")]
 9   index []
10     (render_template "template1.j2"))
11 
12 (defn [(.route app "/response" :methods ["POST"])]
13   response []
14     (setv name (request.form.get "name"))
15     (print name)
16     (render_template "template1.j2" :name name))
17     
18 (app.run)

I assume that you have some experience with GCP and have the following:

• GCP command line tools installed.
• You have created a new project on the GCP AppEngine console named something like hy-gcp-test (if you choose a name already in use, you will get a warning).

After cloning or otherwise copying this project, you use the command line tools to deploy and test your Flask app:

gcloud auth login
gcloud config set project hy-gcp-test
gcloud app deploy
gcloud app browse

If you have problems, look at your logs:

gcloud app logs tail -s default

You can edit changes locally and test locally using:

python main.py

Any changes can be tested by deploying again:

gcloud app deploy

Please note that every time you deploy, a new instance is created. You will want to use the GCP AppEngine console to remove old instances, and remove all instances when you are done.

Going forward

You can make a copy of this example, create a GitHub repo, and follow the above directions as a first step to creating Hy language application on AppEngine. The Google Cloud Platform has many services that you can use in your app (using the Python APIs, called from your Hy program), including:

• Storage and Databases.
• Big Data.
• Machine Learning.

Wrap-up

I like to be able to implement simple things simply, without a lot of ceremony. Once you work through these examples I hope you feel that you can generate Hy and Flask based web apps quickly and with very little code required.

To return to the theme of bottom-up programming, I find that starting with short low level utility functions and placing them in a separate file makes reuse simple and makes future similar projects even easier. For each language I work with, I collect snippets of useful code and short utilities kept in separate files. When writing code I start looking in my snippets directory for the language I am using to implement low level functionality even before doing a web search. When I work in Common Lisp I keep all low level code that I have written in small libraries contained in a single Quicklisp source root directory and for Python and Hy I use Python’s setuptools library to generate libraries that are installed globally on my laptop for easy reuse. It is worth some effort to organize your work for future reuse.

Responsible Web Scraping

I put the word “Responsible” in the chapter title to remind you that just because it is easy (as we will soon see) to pull data from web sites, it is important to respect the property rights of web site owners and abide by their terms and conditions for use. This Wikipedia article on Fair Use provides a good overview of using copyright material.

The web scraping code we develop here uses the Python BeautifulSoup and URI libraries.

For my work and research, I have been most interested in using web scraping to collect text data for natural language processing but other common applications include writing AI news collection and summarization assistants, trying to predict stock prices based on comments in social media which is what we did at Webmind Corporation in 2000 and 2001, etc.

Using the Python BeautifulSoup Library in the Hy Language

There are many good libraries for parsing HTML text and extracting both structure (headings, what is in bold font, etc.) and embedded raw text. I particularly like the Python Beautiful Soup library and we will use it here.

In line 4 for the following listing of file get_web_page.hy, I am setting the default user agent to a descriptive string “HyLangBook” but for some web sites you might need to set this to appear as a Firefox or Chrome browser (iOS, Android, Windows, Linux, or macOS). The function get-raw-data gets the entire contents of a web site as a single string value.

1 (import urllib.request [Request urlopen])
2 
3 (defn get-raw-data-from-web [aUri [anAgent {"User-Agent" "HyLangBook/1.0"}]]
4   (setv req (Request aUri :headers anAgent))
5   (setv httpResponse (urlopen req))
6   (setv data (.read httpResponse))
7   data)

Let’s test this function in a REPL:

 1 $ hy
 2 => (import get-page-data [get-raw-data-from-web])
 3 => (get-raw-data-from-web "http://knowledgebooks.com")
 4 b'<!DOCTYPE html><html><head><title>KnowledgeBooks.com - research on the Knowledge M\
 5 anagement, and the Semantic Web ...'
 6 => 
 7 => (import get-page-data [get-page-html-elements])
 8 => (get-page-html-elements "http://knowledgebooks.com")
 9 {'title': [<title>KnowledgeBooks.com - research on the Knowledge Management, and the\
10  Semantic Web </title>],
11 'a': [<a class="brand" href="#">KnowledgeBooks.com  </a>,  ...
12 => 

This REPL session shows the the function get-raw-data-from-web defined in the previous listing returns a web page as a string. In line 9 we use a function get-page-html-elements to find all elements in a string containing HTML. This function is defined in the next listing and shows how to parse and process the string contents of a web pages. Note: you will need to install the lxml library for this example (using pip or pip3 depending on your Python configuration):

1 pip install lxml

The following listing of file get_page_data.hy uses the Beautiful Soup library to parse the string data for HTML text from a web site. The function get-page-html-elements returns names and associated data with each element in HTML represented as a string (the extra code on lines 20-24 is just debug example code):

 1 (import get_web_page [get-raw-data-from-web])
 2 
 3 (import bs4 [BeautifulSoup])
 4 
 5 (defn get-element-data [anElement]
 6   {"text" (.getText anElement)
 7    "name" (. anElement name)
 8    "class" (.get anElement "class")
 9    "href" (.get anElement "href")})
10 
11 (defn get-page-html-elements [aUri]
12   (setv raw-data (get-raw-data-from-web aUri))
13   (setv soup (BeautifulSoup raw-data "lxml"))
14   (setv title (.find_all soup "title"))
15   (setv a (.find_all soup "a"))
16   (setv h1 (.find_all soup "h1"))
17   (setv h2 (.find_all soup "h2"))
18   {"title" title "a" a "h1" h1 "h2" h2})
19 
20 (setv elements (get-page-html-elements "http://markwatson.com"))
21 
22 (print (get elements "a"))
23 
24 (for [ta (get elements "a")] (print (get-element-data ta)))

The function get-element-data defined in lines 5-9 accepts as an argument an HTML element object (as defined in the Beautiful soup library) and extracts data, if available, for text, name, class, and href values. The function get-page-html-elements defied in lines 11-18 accepts as an argument a string containing a URI and returns a dictionary (or map, or hash table) containing lists of all a, h1, h2, and title elements in the web page pointed to by the input URI. You can modify get-page-html-elements to add additional HTML element types, as needed.

Here is the output (with many lines removed for brevity):

 1 {'text': 'Mark Watson artificial intelligence consultant and author',
 2  'name': 'a', 'class': ['navbar-brand'], 'href': '#'}
 3 {'text': 'Home page', 'name': 'a', 'class': None, 'href': '/'}
 4 {'text': 'My Blog', 'name': 'a', 'class': None,
 5  'href': 'https://mark-watson.blogspot.com'}
 6 {'text': 'GitHub', 'name': 'a', 'class': None,
 7  'href': 'https://github.com/mark-watson'}
 8 {'text': 'Twitter', 'name': 'a', 'class': None, 'href': 'https://twitter.com/mark_l_\
 9 watson'}
10 {'text': 'WikiData', 'name': 'a', 'class': None, 'href': 'https://www.wikidata.org/w\
11 iki/Q18670263'}

Getting HTML Links from the DemocracyNow.org News Web Site

I financially support and rely on both NPR.org and DemocracyNow.org news as my main sources of news so I will use their news sites for examples here and in the next section. Web sites differ so much in format that it is often necessary to build highly customized web scrapers for individual web sites and to maintain the web scraping code as the format of the site changes in time.

Before working through this example and/or the example in the next section use the file Makefile to fetch data:

make data

This should copy the home pages for both web sites to the files:

• democracynow_home_page.html (used here)
• npr_home_page.html (used for the example in the next section)

The following listing shows democracynow_front_page.hy

 1 #!/usr/bin/env hy
 2 
 3 (import get-web-page [get-web-page-from-disk])
 4 (import bs4 [BeautifulSoup])
 5 
 6 ;; you need to run 'make data' to fetch sample HTML data for dev and testing
 7 
 8 (defn get-democracy-now-links []
 9   (setv test-html (get-web-page-from-disk "democracynow_home_page.html"))
10   (setv bs (BeautifulSoup test-html :features "lxml"))
11   (setv all-anchor-elements (.findAll bs "a"))
12   (lfor e all-anchor-elements
13           :if (> (len (.get-text e)) 0)
14           (, (.get e "href") (.get-text e))))
15 
16 (when (= __name__ "__main__")
17   (for [[uri text] (get-democracy-now-links)]
18     (print uri ":" text)))

This simply prints our URIs and text (separated with the string “:”) for each link on the home page. On line 13 we discard any anchor elements that do not contain text. On line 14 the comma character at the start of the return list indicates that we are constructing a tuple. Lines 16-18 define a main function that is used when running this file at the command line. This is similar to how main functions can be defined in Python to allow a library file to also be run as a command line tool.

A few lines of output from today’s front page is:

/2020/1/7/the_great_hack_cambridge_analytica : Meet Brittany Kaiser, Cambridge Analy\
tica Whistleblower Releasing Troves of New Files from Data Firm
/2019/11/8/remembering_orangeburg_massacre_1968_south_carolina : Remembering the 196\
8 Orangeburg Massacre When Police Shot Dead Three Unarmed Black Students
/2020/1/15/democratic_debate_higher_education_universal_programs : Democrats Debate \
Wealth Tax, Free Public College & Student Debt Relief as Part of New Economic Plan
/2020/1/14/dahlia_lithwick_impeachment : GOP Debate on Impeachment Witnesses Intensi\
fies as Pelosi Prepares to Send Articles to Senate
/2020/1/14/oakland_california_moms_4_housing : Moms 4 Housing: Meet the Oakland Moth\
ers Facing Eviction After Two Months Occupying Vacant House
/2020/1/14/luis_garden_acosta_martin_espada : “Morir Soñando”: Martín Espada Reads P\
oem About Luis Garden Acosta, Young Lord & Community Activist

The URIs are relative to the root URI https://www.democracynow.org/.

Getting Summaries of Front Page from the NPR.org News Web Site

This example is similar to the example in the last section except that text from home page links is formatted to provide a daily news summary. I am assuming that you ran the example in the last section so the web site home pages have been copied to local files.

The following listing shows npr_front_page_summary.hy

 1 #!/usr/bin/env hy
 2 
 3 (import get-web-page [get-web-page-from-disk])
 4 (import bs4 [BeautifulSoup])
 5 
 6 ;; you need to run 'make data' to fetch sample HTML data for dev and testing
 7 
 8 (defn get-npr-links []
 9   (setv test-html (get-web-page-from-disk "npr_home_page.html"))
10   (setv bs (BeautifulSoup test-html :features "lxml"))
11   (setv all-anchor-elements (.findAll bs "a"))
12   (setv filtered-a
13     (lfor e all-anchor-elements
14           :if (> (len (.get-text e)) 0)
15           #((.get e "href") (.get-text e))))
16   filtered-a)
17 
18 (defn create-npr-summary []
19   (setv links (get-npr-links))
20   (setv filtered-links (lfor [uri text] links :if (> (len (.strip text)) 40) (.strip\
21  text)))
22   (.join "\n\n" filtered-links))
23 
24 (when (= __name__ "__main__")
25   (print (create-npr-summary)))

In lines 12-15 we are filtering out (or removing) all anchor HTML elements that do not contain text. The following shows a few lines of the generated output for data collected today:

January 16, 2020  Birds change the shape of their wings far more than
planes. The complexities of bird flight have posed a major design challenge
for scientists trying to translate the way birds fly into robots.

FBI Vows To Warn More Election Officials If Discovering A Cyberattack

January 16, 2020  The bureau was faulted after the Russian attack on the
2016 election for keeping too much information from state and local
authorities. It says it'll use a new policy going forward.

Ukraine Is Investigating Whether U.S. Ambassador Yovanovitch Was Surveilled

January 16, 2020  Ukraine's Internal Affairs Ministry says it's asking the
FBI to help determine whether international laws were broken, or "whether it
is just a bravado and a fake information" from a U.S. politician.

Electric Burn: Those Who Bet Against Elon Musk And Tesla Are Paying A Big Price

January 16, 2020  For years, Elon Musk skeptics have shorted Tesla stock, confident \
the electric carmaker was on the brink of disaster. Instead, share value has skyrock\
eted, costing short sellers billions.

TSA Says It Seized A Record Number Of Firearms At U.S. Airports Last Year

The examples seen here are simple but should be sufficient to get you started gathering text data from the web.

Using the Brave Search APIs

Note: I started using the Brave search APIs in June 2024 and replaced the Microsoft Bing search chapter in previous editions with the following material.

You will need to get a free API key at https://brave.com/search/api/ to use the following code examples. You can use the search API 2000 times a month for free or pay $5/month to get 20 million API calls a month.

Setting an Environment Variable for the Access Key for Brave Search APIs

Once you get a key for https://brave.com/search/api/ set the following environment variable:

export BRAVE_SEARCH_API_KEY=BSGhQ-Nd-......

That is not my real subscription key!

Example Search Script

The following shows the file brave.hy:

It takes very little Hy code to access the Brave search APIs. Here we define a function named brave_search that takes one parameter query. We get the API subscription ket from an environment variable, define the URI for the Brave search endpoint, and set up an HTTP request to this endpoint. I encourage you, dear reader, to experiment with printing out the HTTP response to see all data returned from the Brave search API. Here we only collect the tile, URL, and description for each search result:

(import os requests)
(import pprint [pprint])

(defn brave_search [query]
  (setv subscription-key (get os.environ "BRAVE_SEARCH_API_KEY"))
  (setv endpoint "https://api.search.brave.com/res/v1/web/search")

  ;; Construct a request
  (setv params {"q" query})
  (setv headers {"X-Subscription-Token" subscription-key})

  ;; Call the API
  (setv response (requests.get endpoint :headers headers :params params))

  ;; Pull out results
  (setv results (get (get (response.json) "web") "results"))

  ;; Create a list of lists containing title, URL, and description
  (setv res (lfor result results
                 [(get result "title")
                  (get result "url")
                  (get result "description")]))

  ;; Return the results
  res)

;; Example usage:
;;(setv search-results (brave-search "site:wikidata.org Sedona Arizona"))
;;(pprint search-results)

You can use search hints like “site:wikidata.org” to only search specific web sites. In the following example I use the search query:

1 "site:wikidata.org Sedona Arizona"

The example call:

(setv search-results (brave-search "site:wikidata.org Sedona Arizona"))
(pprint search-results)

produces the output (edited for brevity):

[['Sedona - Wikidata',
  'https://m.wikidata.org/wiki/Q80041',
  'city in counties of Yavapai and Coconino, <strong>Arizona</strong>, United '
  'States'],
 ['Category:People from Sedona, Arizona - Wikidata',
  'https://www.wikidata.org/wiki/Q8748837',
  'All structured data from the main, Property, Lexeme, and EntitySchema '
  'namespaces is available under the Creative Commons CC0 License; text in the '
  'other namespaces is available under the Creative Commons '
  'Attribution-ShareAlike License; additional terms may apply.'],
 ['Category:Films set in Sedona, Arizona - Wikidata',
  'https://www.wikidata.org/wiki/Q25109087',
  'All structured data from the main, Property, Lexeme, and EntitySchema '
  'namespaces is available under the Creative Commons CC0 License; text in the '
  'other namespaces is available under the Creative Commons '
  'Attribution-ShareAlike License; additional terms may apply.'],

 ...

Wrap-up

In addition to using automated web scraping to get data for my personal research, I often use automated web search. I find the Brave search APIs are the most convenient to use and I like paying for services that I use. The search engine Duck Duck Go also provides free search APIs but even though I use Duck Duck Go for 90% of my manual web searches, when I build automated systems I prefer to rely on services that I pay for.

Deep Learning

Most of my professional career since 2014 has involved Deep Learning, mostly with TensorFlow using the Keras APIs. In the late 1980s I was on a DARPA neural network technology advisory panel for a year. I wrote the first prototype of the SAIC ANSim neural network library commercial product, and I wrote the neural network prediction code for a bomb detector my company designed and built for the FAA for deployment in airports. More recently I have used GAN (generative adversarial networks) models for synthesizing numeric spreadsheet data and LSTM (long short term memory) models to synthesize highly structured text data like nested JSON and for NLP (natural language processing). I have 55 USA and several European patents using neural network and Deep Learning technology.

The Hy language utilities and example programs we develop here all use TensorFlow and Keras “under the hood” to do the heavy lifting. Keras is a simpler to use API for TensorFlow and I usually use Keras rather than the lower level TensorFlow APIs.

There are other libraries and frameworks that might interest you in addition to TensorFlow and Keras. I particularly like the Flux library for the Julia programming language. Currently Python has the most comprehensive libraries for Deep Learning but other languages that support differential computing (more on this later) like Julia and Swift may gain popularity in the future.

Here we will learn a vocabulary for discussing Deep Learning neural network models, look at possible architectures, and show two Hy language examples that should be sufficient to get you familiar to using Keras with the Hy language. If you already have Deep Learning application development experience you might want to skip the following review material and go directly to the Hy language examples.

If you want to use Deep Learning professionally, there are two specific online resources that I recommend: Andrew Ng leads the efforts at deeplearning.ai and Jeremy Howard leads the efforts at fast.ai. Here I will show you how to use a few useful techniques. Andrew and Jeremy will teach you skills that may lead a professional level of expertise if you take their courses.

There are many Deep Learning neural architectures in current practical use; a few types that I use are:

• Multi-layer perceptron networks with many fully connected layers. An input layer contains placeholders for input data. Each element in the input layer is connected by a two-dimensional weight matrix to each element in the first hidden layer. We can use any number of fully connected hidden layers, with the last hidden layer connected to an output layer.
• Convolutional networks for image processing and text classification. Convolutions, or filters, are small windows that can process input images (filters are two-dimensional) or sequences like text (filters are one-dimensional). Each filter uses a single set of learned weights independent of where the filter is applied in an input image or input sequence.
• Autoencoders have the same number of input layer and output layer elements with one or more hidden fully connected layers. Autoencoders are trained to produce the same output as training input values using a relatively small number of hidden layer elements. Autoencoders are capable of removing noise in input data.
• LSTM (long short term memory) process elements in a sequence in order and are capable of remembering patterns that they have seen earlier in the sequence.
• GAN (generative adversarial networks) models comprise two different and competing neural models, the generator and the discriminator. GANs are often trained on input images (although in my work I have applied GANs to two-dimensional numeric spreadsheet data). The generator model takes as input a “latent input vector” (this is just a vector of specific size with random values) and generates a random output image. The weights of the generator model are trained to produce random images that are similar to how training images look. The discriminator model is trained to recognize if an arbitrary output image is original training data or an image created by the generator model. The generator and discriminator models are trained together.

The core functionality of libraries like TensorFlow are written in C++ and take advantage of special hardware like GPUs, custom ASICs, and devices like Google’s TPUs. Most people who work with Deep Learning models don’t need to even be aware of the low level optimizations used to make training and using Deep Learning models more efficient. That said, in the following section I am going to show you how simple neural networks are trained and used.

Simple Multi-layer Perceptron Neural Networks

I use the terms Multi-layer perceptron neural networks, backpropagation neural networks and delta-rule networks interchangeably. Backpropagation refers to the model training process of calculating the output errors when training inputs are passed in the forward direction from input layer, to hidden layers, and then to the output layer. There will be an error which is the difference between the calculated outputs and the training outputs. This error can be used to adjust the weights from the last hidden layer to the output layer to reduce the error. The error is then backprogated backwards through the hidden layers, updating all weights in the model. I have detailed example code in any of my older artificial intelligence books. Here I am satisfied to give you an intuition to how simple neural networks are trained.

The basic idea is that we start with a network initialized with random weights and for each training case we propagate the inputs through the network towards the output neurons, calculate the output errors, and back-up the errors from the output neurons back towards the input neurons in order to make small changes to the weights to lower the error for the current training example. We repeat this process by cycling through the training examples many times.

The following figure shows a simple backpropagation network with one hidden layer. Neurons in adjacent layers are connected by floating point connection strength weights. These weights start out as small random values that change as the network is trained. Weights are represented in the following figure by arrows; in the code the weights connecting the input to the output neurons are represented as a two-dimensional array.

Each non-input neuron has an activation value that is calculated from the activation values of connected neurons feeding into it, gated (adjusted) by the connection weights. For example, in the above figure, the value of Output 1 neuron is calculated by summing the activation of Input 1 times weight W1,1 and Input 2 activation times weight W2,1 and applying a “squashing function” like Sigmoid or Relu (see figures below) to this sum to get the final value for Output 1’s activation value. We want to flatten activation values to a relatively small range but still maintain relative values. To do this flattening we use the Sigmoid function that is seen in the next figure, along with the derivative of the Sigmoid function which we will use in the code for training a network by adjusting the weights.

Simple neural network architectures with just one or two hidden layers are easy to train using backpropagation and I have examples of from-scratch code for this several of my previous books. You can see Java and Common Lisp from-scratch implementations in two of my books that you can read online: Practical Artificial Intelligence Programming With Java and Loving Common Lisp, or the Savvy Programmer’s Secret Weapon. However, here we are using Hy to write models using the TensorFlow framework which has the huge advantage that small models you experiment with on your laptop can be scaled to more parameters (usually this means more neurons in hidden layers which increases the number of weights in a model) and run in the cloud using multiple GPUs.

Except for pedantic purposes, I now never write neural network code from scratch, instead I take advantage of the many person-years of engineering work put into the development of frameworks like TensorFlow, PyTorch, mxnet, etc. We now move on to two examples built with TensorFlow.

Deep Learning

Deep Learning models are generally understood to have many more hidden layers than simple multi-layer perceptron neural networks and often comprise multiple simple models combined together in series or in parallel. Complex architectures can be iteratively developed by manually adjusting the size of model components, changing the components, etc. Alternatively, model architecture search can be automated. At Capital One I used Google’s AdaNet project that efficiently searches for effective model architectures inside a single TensorFlow session. The model architecture used here is simple: one input layer representing the input values in a sample of University of Wisconsin cancer data, one hidden layer, and an output layer consisting of one neuron whose activation value will be interpreted as a prediction of benign or malignant.

The material in this chapter is intended to serve two purposes:

• If you are already familiar with Deep Learning and TensorFlow then the examples here will serve to show you how to call the TensorFlow APIs from Hy.
• If you have little or no exposure with Deep Learning then the short Hy language examples will provide you with concise code to experiment with and you can then decide to study further.

Once again, I recommend that you consider taking two online Deep Learning course sequences. For no cost, Jeremy Howard provides lessons at fast.ai that are very good and the later classes use PyTorch which is a framework that is similar to TensorFlow. For a modest cost Andrew Ng provides classes at deeplearning.ai that use TensorFlow. I have been working in the field of machine learning since the 1980s, but I still take Andrew’s online classes to stay up-to-date. In the last eight years I have taken his Stanford University machine learning class twice and also his complete course sequence using TensorFlow. I have also worked through much of Jeremy’s material. I recommend both course sequences without reservation.

Using Keras and TensorFlow to Model The Wisconsin Cancer Data Set

The University of Wisconsin cancer database has 646 samples. Each sample has 9 input values and one output value:

• Cl.thickness: Clump Thickness
• Cell.size: Uniformity of Cell Size
• Cell.shape: Uniformity of Cell Shape
• Marg.adhesion: Marginal Adhesion
• Epith.c.size: Single Epithelial Cell Size
• Bare.nuclei: Bare Nuclei
• Bl.cromatin: Bland Chromatin
• Normal.nucleoli: Normal Nucleoli
• Mitoses: Mitoses
• Class: Class (0 for benign, 1 for malignant)

Each row represents a sample with different measurements related to cell properties, and the final column ‘Class’ indicates whether the sample is benign (0) or malignant (1).

Let’s perform some basic analysis on this data:

• Check for missing values.
• Get the summary statistics of the dataset.
• Check the balance of the classes (benign and malignant).

Here’s the analysis:

• Missing Values: There are no missing values in the dataset. Each column has complete data.
• Summary Statistics: The mean and median (50%) values of most features are quite different, indicating that the data distribution for these features might be skewed.
• The range (min to max) for all features is from 1 to 10, indicating that the measurements are likely based on a scale or ranking system of 1 to 10.
• Class Balance: The dataset is somewhat imbalanced. Approximately 65% of the samples are benign (0) and 35% are malignant (1). This imbalance might influence the performance of machine learning models trained on this data.

Now, it would be beneficial to visualize the data to get a better understanding of the distribution of each feature and the relationship between different features:

Here are histograms of each feature, broken down by class (benign or malignant). Some observations:

Cl.thickness, Cell.size, Cell.shape, Marg.adhesion, Epith.c.size, Bare.nuclei, Bl.cromatin, Normal.nucleoli: For these features, higher values seem to be associated with the malignant class. This might suggest that these characteristics are significant in the determination of malignancy. Mitoses: This feature shows a different trend, with a majority of both benign and malignant cases having low mitoses values. However, there are more malignant cases with higher mitoses values than benign cases.

We will use separate training and test files hy-lisp-python/deeplearning/train.csv and hy-lisp-python/deeplearning/test.csv. Here are a few samples from the training file:

6,2,1,1,1,1,7,1,1,0
2,5,3,3,6,7,7,5,1,1
10,4,3,1,3,3,6,5,2,1
6,10,10,2,8,10,7,3,3,1
5,6,5,6,10,1,3,1,1,1
1,1,1,1,2,1,2,1,2,0
3,7,7,4,4,9,4,8,1,1
1,1,1,1,2,1,2,1,1,0

After you look at this data, if you did not have much experience with machine learning then it might not be obvious how to build a model to accept a sample for a patient like we see in the Wisconsin data set and then predict if the sample implies benign or cancerous outcome for the patient. Using TensorFlow with a simple neural network model, we will implement a model in about 40 lines of Hy code to implement this example.

Since there are nine input values we will need nine input neurons that will represent the input values for a sample in either training or separate test data. These nine input neurons (created in lines 9-10 in the following listing) will be completely connected to twelve neurons in a hidden layer. Here, completely connected means that each of the nine input neurons is connected via a weight to each hidden layer neuron. There are 9 * 12 = 108 weights between the input and hidden layers. There is a single output layer neuron that is connected to each hidden layer neuron.

Notice that in lines 12 and 14 in the following listing that we specify a relu activation function while the activation function connecting the hidden layer to the output layer uses the sigmoid activation function that we saw plotted earlier.

There is an example in the git example repo directory hy-lisp-python/matplotlib in the file plot_relu.hy that generated the following figure:

The following listing shows the use of the Keras TensorFlow APIs to build a model (lines 9-19) with one input layer, two hidden layers, and an output layer with just one neuron. After we build the model, we define two utility functions train (lines 21-23) to train a model given training inputs (x argument) and corresponding training outputs (y** argument), and we also define predict (lines 25-26) using a trained model to make a cancer or benign prediction given test input values (x-data argument).

Lines 28-33 show a utility function load-data that loads a University of Wisconsin cancer data set CSV file, scales the input and output values to the range [0.0, 1.0] and returns a list containing input (x-data) and target output data (y-data). You may want to load this example in a REPL and evaluate load-data on one of the CSV files.

The function main (lines 35-45) loads training and test (evaluation of model accuracy on data not used for training), trains a model, and then tests the accuracy of the model on the test (evaluation) data:

 1 #!/usr/bin/env hy
 2 
 3 (import argparse)
 4 (import os)
 5 (import tensorflow [keras])
 6 (import tensorflow.keras [layers])
 7 
 8 (import pandas [read-csv])
 9 (import pandas)
10 
11 (defn build-model []
12   (setv model (keras.models.Sequential))
13   (.add model (keras.layers.Dense 9
14                  :activation "relu"))
15   (.add model (keras.layers.Dense 12
16                  :activation "relu"))
17   (.add model (keras.layers.Dense 1
18                  :activation "sigmoid"))
19   (.compile model :loss      "binary_crossentropy"
20                   :optimizer (keras.optimizers.RMSprop))
21   model)
22 
23 (defn first [x] (get x 0))
24 
25 (defn train [batch-size model x y]
26   (for [it (range 50)]
27     (.fit model x y :batch-size batch-size :epochs 10 :verbose False)))
28 
29 (defn predict [model x-data]
30     (.predict model x-data))
31 
32 (defn load-data [file-name]
33   (setv all-data (read-csv file-name :header None))
34   (setv x-data10 (. all-data.iloc [#((slice 0 10) [0 1 2 3 4 5 6 7 8])] values))
35   (setv x-data (* 0.1 x-data10))
36   (setv y-data (. all-data.iloc [#((slice 0 10) [9])] values))
37   [x-data y-data])
38 
39 (defn main []
40   (setv xyd (load-data "train.csv"))
41   (setv model (build-model))
42   (setv xytest (load-data "test.csv"))
43   (train 10 model (. xyd [0]) (. xyd [1]))
44   (print "* predictions (calculated, expected):")
45   (setv predictions (list (map first (predict model (. xytest [0])))))
46   (setv expected (list (map first (. xytest [1]))))
47   (print
48     (list
49       (zip predictions expected))))
50 
51 (main)

The following listing shows the output:

1 * predictions (calculated, expected):
2 [(0.9998953, 1), (0.9999737, 1), (0.9172243, 1), (0.9975936, 1), (0.38985246, 0), (0\
3 .4301587, 0), (0.99999213, 1), (0.855, 0), (0.3810781, 0), (0.9999431, 1)]

Let’s look at the first test case: the “real” output from the training data is a value of 1 and the calculated predicted value (using the trained model) is 0.9759052. In making predictions, we can choose a cutoff value, 0.5 for example, and interpret any calculated prediction value less than the cutoff as a Boolean false prediction and calculated prediction value greater to or equal to the cutoff value is a Boolean true prediction.

Using a LSTM Recurrent Neural Network to Generate English Text Similar to the Philosopher Nietzsche’s Writing

We will translate a Python example program from Google’s Keras documentation (listing of LSTM.py that is included with the example Hy code) to Hy. This is a moderately long example and you can use the original Python and the translated Hy code as a general guide for converting other models implemented in Python using Keras that you want use in Hy. I have, in most cases, kept the same variable names to make it easier to compare the Python and Hy code.

Note that using the nietzsche.txt data set requires a fair amount of memory. If your computer has less than 16G of RAM, you might want to reduce the size of the training text by first running the following example until you see the printout “Create sentences and next_chars data…” then kill the program. The first time you run this program, the training data is fetched from the web and stored locally. You can manually edit the file ~/.keras/datasets/nietzsche.txt to remove 75% of the data by:

1     pushd ~/.keras/datasets/
2     mv nietzsche.txt nietzsche_large.txt
3     head -800 nietzsche_large.txt > nietzsche.txt
4     popd

The next time you run the example, the Keras example data loading utilities will notice a local copy and even though the file now is much smaller, the data loading utilities will not download a new copy.

When I start training a new Deep Learning model I like to monitor system resources using the top command line activity, watching for page faults when training on a CPU which might indicate that I am trying to train too large a model for my system memory. If you are using CUDA and a GPU then use the CUDA command line utilities for monitoring the state of the GPU utilization. It is beyond the scope of this introductory tutorial, but the tool TensorBoard is very useful for monitoring the state of model training.

There are a few things that make the following example code more complex than the example using the University of Wisconsin cancer data set. We need to convert each character in the training data to a one-hot encoding which is a vector of all 0.0 values except for a single value of 1.0. I am going to show you a short REPL session so that you understand how this works and then we will look at the complete Hy code example.

 1 $ hy
 2 => (import keras.callbacks [LambdaCallback])
 3 Using TensorFlow backend.
 4 => (import keras.models [Sequential])
 5 => (import keras.layers [Dense LSTM])
 6 => (import keras.optimizers [RMSprop])
 7 => (import keras.utils.data_utils [get_file])
 8 => (import numpy :as np) ;; note the syntax for aliasing a module name
 9 => (import random sys io)
10 => (with [f (io.open "/Users/markw/.keras/datasets/nietzsche.txt" :encoding "utf-8")]
11 ... (setv text (.read f)))
12 => (cut text 98 130)
13 'philosophers, in so far as they '
14 => (setv chars (sorted (list (set text))))
15 => chars
16 ['\n', ' ', '!', '"', "'", '(', ')', ',', '-', '.', '0', '1', '2', '3', '4', '5', '6\
17 ', '7', '8', '9', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', '\
18 K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', '_', 'a', \
19 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r',\
20  's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
21 => (setv char_indices (dict (lfor i (enumerate chars) (, (last i) (first i)))))
22 => char_indices
23 {'\n': 0, ' ': 1, '!': 2, '"': 3, "'": 4, '(': 5, ')': 6, ',': 7, '-': 8, '.': 9, '0\
24 ': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9': \
25 19, ':': 20, ';': 21, '?': 22, 'A': 23, 'B': 24, 'C': 25, 'D': 26, 'E': 27, 'F': 28,\
26  'G': 29, 'H': 30, 'I': 31, 'J': 32, 'K': 33, 'L': 34, 'M': 35, 'N': 36, 'O': 37, 'P\
27 ': 38, 'Q': 39, 'R': 40, 'S': 41, 'T': 42, 'U': 43, 'V': 44, 'W': 45, 'X': 46, 'Y': \
28 47, '_': 48, 'a': 49, 'b': 50, 'c': 51, 'd': 52, 'e': 53, 'f': 54, 'g': 55, 'h': 56,\
29  'i': 57, 'j': 58, 'k': 59, 'l': 60, 'm': 61, 'n': 62, 'o': 63, 'p': 64, 'q': 65, 'r\
30 ': 66, 's': 67, 't': 68, 'u': 69, 'v': 70, 'w': 71, 'x': 72, 'y': 73, 'z': 74}
31 => (setv indices_char (dict (lfor i (enumerate chars) i)))
32 => indices_char
33 {0: '\n', 1: ' ', 2: '!', 3: '"', 4: "'", 5: '(', 6: ')', 7: ',', 8: '-', 9: '.', 10\
34 : '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: '\
35 9', 20: ':', 21: ';', 22: '?', 23: 'A', 24: 'B', 25: 'C', 26: 'D', 27: 'E', 28: 'F',\
36  29: 'G', 30: 'H', 31: 'I', 32: 'J', 33: 'K', 34: 'L', 35: 'M', 36: 'N', 37: 'O', 38\
37 : 'P', 39: 'Q', 40: 'R', 41: 'S', 42: 'T', 43: 'U', 44: 'V', 45: 'W', 46: 'X', 47: '\
38 Y', 48: '_', 49: 'a', 50: 'b', 51: 'c', 52: 'd', 53: 'e', 54: 'f', 55: 'g', 56: 'h',\
39  57: 'i', 58: 'j', 59: 'k', 60: 'l', 61: 'm', 62: 'n', 63: 'o', 64: 'p', 65: 'q', 66\
40 : 'r', 67: 's', 68: 't', 69: 'u', 70: 'v', 71: 'w', 72: 'x', 73: 'y', 74: 'z'}
41 => (setv maxlen 40)
42 => (setv s "Oh! I saw 1 dog (yesterday)")
43 => (setv x_pred (np.zeros [1 maxlen (len chars)]))
44 => (for [[t char] (lfor j (enumerate s) j)]
45 ... (setv (get x_pred 0 t (get char_indices char)) 1))
46 => x_pred
47 array([[[0., 0., 0., ..., 0., 0., 0.],
48         [0., 0., 0., ..., 0., 0., 0.],
49         [0., 0., 1., ..., 0., 0., 0.],   // here 1. is the third character "!"
50         ...,
51         [0., 0., 0., ..., 0., 0., 0.],
52         [0., 0., 0., ..., 0., 0., 0.],
53         [0., 0., 0., ..., 0., 0., 0.]]])
54 => 

For lines 48-54, each line represents a single character one-hot encoded. Notice how the third character shown on line 50 has a value of “1.” at index 2, which corresponds to the one-hot encoding of the letter “!”.

Now that you have a feeling for how one-hot encoding works, hopefully the following example will make sense to you. We will further discuss one-hot-encoding after the next code listing. For training, we take 40 characters (the value of the variable maxlen) at a time, and using one one-hot encode a character at a time as input and the target output will be the one-hot encoding of the following character in the input sequence. We are iterating on training the model for a while and then given a few characters of text, predict a likely next character - and keep repeating this process. The generated text is then used as input to the model to generate yet more text. You can repeat this process until you have generated sufficient text.

This is a powerful technique that I used to model JSON with complex deeply nested schemas and then generate synthetic JSON in the same schema as the training data. Here, training a model to mimic the philosopher Nietzsche’s writing is much easier than learning highly structured data like JSON:

 1 #!/usr/bin/env hy
 2 
 3 ;; This example was translated from the Python example in the Keras
 4 ;; documentation at: https://keras.io/examples/lstm_text_generation/
 5 ;; The original Python file LSTM.py is included in the directory
 6 ;; hy-lisp-python/deeplearning for reference.
 7 
 8 (import keras.callbacks [LambdaCallback])
 9 (import keras.models [Sequential])
10 (import keras.layers [Dense LSTM])
11 (import keras.optimizers [RMSprop])
12 (import keras [utils])
13 ;;(import keras.utils.data_utils [get_file])
14 (import numpy :as np) ;; note the syntax for aliasing a module name
15 (import random sys io)
16 
17 (setv path
18       (utils.get_file        ;; this saves a local copy in ~/.keras/datasets
19         "nietzsche.txt"
20         :origin "https://s3.amazonaws.com/text-datasets/nietzsche.txt"))
21 
22 (with [f (io.open path :encoding "utf-8")]
23   (setv text (.read f))) ;; note: sometimes we use (.lower text) to
24                          ;;       convert text to all lower case
25 (print "corpus length:" (len text))
26 
27 (setv chars (sorted (list (set text))))
28 (print "total chars (unique characters in input text):" (len chars))
29 ;;(setv char_indices (dict (lfor i (enumerate chars) (, (last i) (first i)))))
30 (setv char_indices (dict (lfor i (enumerate chars) #((get i -1) (get i 0)))))
31 (setv indices_char (dict (lfor i (enumerate chars) i)))
32 
33 ;; cut the text in semi-redundant sequences of maxlen characters
34 (setv maxlen 40)
35 (setv step 3) ;; when we sample text, slide sampling window 3 characters
36 (setv sentences (list))
37 (setv next_chars (list))
38 
39 (print "Create sentences and next_chars data...")
40 (for [i (range 0 (- (len text) maxlen) step)]
41   (.append sentences (cut text i (+ i maxlen)))
42   (.append next_chars (get text (+ i maxlen))))
43 
44 (print "Vectorization...")
45 (setv x (np.zeros [(len sentences) maxlen (len chars)] :dtype np.bool))
46 (setv y (np.zeros [(len sentences) (len chars)] :dtype np.bool))
47 (for [[i sentence] (lfor j (enumerate sentences) j)]
48   (for [[t char] (lfor j (enumerate sentence) j)]
49     (setv (get x i t (get char_indices char)) 1))
50   (setv (get y i (get char_indices (get next_chars i))) 1))
51 (print "Done creating one-hot encoded training data.")
52 
53 (print "Building model...")
54 (setv model (Sequential))
55 (.add model (LSTM 128 :input_shape [maxlen (len chars)]))
56 (.add model (Dense (len chars) :activation "softmax"))
57 
58 (setv optimizer (RMSprop 0.01))
59 (.compile model :loss "categorical_crossentropy" :optimizer optimizer)
60 
61 (defn sample [preds &optional [temperature 1.0]]
62   (setv preds (.astype (np.array preds) "float64"))
63   (setv preds (/ (np.log preds) temperature))
64   (setv exp_preds (np.exp preds))
65   (setv preds (/ exp_preds (np.sum exp_preds)))
66   (setv probas (np.random.multinomial 1 preds 1))
67   (np.argmax probas))
68 
69 (defn on_epoch_end [epoch [not-used None]]
70   (print)
71   (print "----- Generating text after Epoch:" epoch)
72   (setv start_index (random.randint 0 (- (len text) maxlen 1)))
73   (for [diversity [0.2 0.5 1.0 1.2]]
74     (print "----- diversity:" diversity)
75     (setv generated "")
76     (setv sentence (cut text start_index (+ start_index maxlen)))
77     (setv generated (+ generated sentence))
78     (print "----- Generating with seed:" sentence)
79     (sys.stdout.write generated)
80     (for [i (range 400)]
81       (setv x_pred (np.zeros [1 maxlen (len chars)]))
82       (for [[t char] (lfor j (enumerate sentence) j)]
83         (setv (get x_pred 0 t (get char_indices char)) 1))
84 ;;      (setv preds (first (model.predict x_pred :verbose 0)))
85       (setv preds (get (model.predict x_pred :verbose 0) 0))
86       ;;;(print "** preds=" preds)
87       (setv next_index (sample preds diversity))
88       (setv next_char (get indices_char next_index))
89       (setv sentence (+ (cut sentence 1) next_char))
90       (sys.stdout.write next_char)
91       (sys.stdout.flush))
92     (print)))
93 
94 (setv print_callback (LambdaCallback :on_epoch_end on_epoch_end))
95 
96 (model.fit x y :batch_size 128 :epochs 60 :callbacks [print_callback])

In lines 52-54 we defined a model using the Keras APIs and in lines 56-57 compiled the model using a categorical crossentropy loss function with an RMSprop optimizer.

In lines 59-65 we define a function sample that takes a first required argument preds which is a one-hot predicted encoded character that might look like (maxlen or 40 values):

[2.80193929e-02 6.78635418e-01 7.85831537e-04 4.92034527e-03 . . . 6.62320468e-04 9.14627407e-03 2.31375365e-04]

Now, here the predicted one hot encoding values are not strictly 0 or 1, rather they are small floating point numbers of a single number much larger than the others. The largest number is 6.78635418e-01 at index 1 which corresponds to a one-hot encoding for a “ “ space character.

If we print out the number of characters in text and the unique list of characters (variable chars) in the training text file nietzsche.txt we see:

corpus length: 600893
['\n', ' ', '!', '"', "'", '(', ')', ',', '-', '.', '0', '1', '2', '3', '4', '5', '6\
', '7', '8', '9', ':', ';', '=', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', '\
J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', \
'[', ']', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n',\
 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'Æ', 'ä', 'æ', 'é', 'ë'\
]

A review of one-hot encoding:

Let’s review our earlier discussion of one-hot encoding with a simpler case. It is important to understand how we one-hot encode input text to inputs for the model and decode back one-hot vectors to text when we use a trained model to generate text. It will help to see the dictionaries for converting characters to indices and then reverse indices to original characters as we saw earlier, some output removed:

char_indices:
 {'\n': 0, ' ': 1, '!': 2, '"': 3, "'": 4, '(': 5, ')': 6, ',': 7, '-': 8, '.': 9, '\
0': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9':\
 19, 
   . . .
 'f': 58, 'g': 59, 'h': 60, 'i': 61, 'j': 62, 'k': 63, 'l': 64, 'm': 65, 'n': 66, 'o\
': 67, 'p': 68, 'q': 69, 'r': 70, 's': 71, 't': 72, 'u': 73, 'v': 74, 'w': 75, 'x': \
76, 'y': 77, 'z': 78, 'Æ': 79, 'ä': 80, 'æ': 81, 'é': 82, 'ë': 83}
indices_char:
 {0: '\n', 1: ' ', 2: '!', 3: '"', 4: "'", 5: '(', 6: ')', 7: ',', 8: '-', 9: '.', 1\
0: '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: \
'9', 
   . . .
 'o', 68: 'p', 69: 'q', 70: 'r', 71: 's', 72: 't', 73: 'u', 74: 'v', 75: 'w', 76: 'x\
', 77: 'y', 78: 'z', 79: 'Æ', 80: 'ä', 81: 'æ', 82: 'é', 83: 'ë'}

We prepare the input and target output data in lines 43-48 in the last code listing. Using a short string, let’s look in the next REPL session listing at how these input and output training examples are extracted for an input string:

 1 Marks-MacBook:deeplearning $ hy
 2 => (setv text "0123456789abcdefg")
 3 => (setv maxlen 4)
 4 => (setv i 3)
 5 => (cut text i (+ i maxlen))
 6 '3456'
 7 => (cut text (+ 1 maxlen))
 8 '56789abcdefg'
 9 => (setv i 4)                 ;; i is the for loop variable for
10 => (cut text i (+ i maxlen))  ;; defining sentences and next_chars
11 '4567'
12 => (cut text (+ i maxlen))
13 '89abcdefg'
14 => 

So the input training sentences are each maxlen characters long and the next-chars target outputs each start with the character after the last character in the corresponding input training sentence.

This script pauses during each training epoc to generate text given diversity values of 0.2, 0.5, 1.0, and 1.2. The smaller the diversity value the more closely the generated text matches the training text. The generated text is more realistic after many training epocs. In the following, I list a highly edited copy of running through several training epochs. I only show generated text for diversity equal to 0.2:

 1 ----- Generating text after Epoch: 0
 2 ----- diversity: 0.2
 3 ----- Generating with seed: ocity. Equally so, gratitude.--Justice r
 4 ocity. Equally so, gratitude.--Justice read in the become to the conscience the seen\
 5 er and the conception that the becess of the power to the procentical that the becau\
 6 se and the prostice of the prostice and the will to the conscience of the power of t\
 7 he perhaps the self-distance of the all the soul and the world and the soul of the s\
 8 oul of the world and the soul and an an and the profound the self-dister the all the\
 9  belief and the
10 
11 ----- Generating text after Epoch: 8
12 ----- diversity: 0.2
13 ----- Generating with seed: nations
14 laboring simultaneously under th
15 nations
16 laboring simultaneously under the subjection of the soul of the same to the subjecti\
17 on of the subjection of the same not a strong the soul of the spiritual to the same \
18 really the propers to the stree be the subjection of the spiritual that is to probab\
19 ly the stree concerning the spiritual the sublicities and the spiritual to the proce\
20 ssities the spirit to the soul of the subjection of the self-constitution and proper\
21 s to the
22 
23 ----- Generating text after Epoch: 14
24 ----- diversity: 0.2
25 ----- Generating with seed:  to which no other path could conduct us
26  to which no other path could conduct us a stronger that is the self-delight and the\
27  strange the soul of the world of the sense of the sense of the consider the such a \
28 state of the sense of the sense of the sense of such a sandine and interpretation of\
29  the process of the sense of the sense of the sense of the soul of the process of th\
30 e world in the sense of the sense of the spirit and superstetion of the world the se\
31 nse of the
32 
33 ----- Generating text after Epoch: 17
34 ----- diversity: 0.2
35 ----- Generating with seed: hemselves although they could easily hav
36 hemselves although they could easily have been moral morality and the self-in which \
37 the self-in the world to the same man in the standard to the possibility that is to \
38 the strength of the sense-in the former the sense-in the special and the same man in\
39  the consequently the soul of the superstition of the special in the end to the poss\
40 ible that it is will not be a sort of the superior of the superstition of the same m\
41 an to the same man

Here we trained on examples, translated to English, of the philosopher Nietzsche. I have used similar code to this example to train on highly structured JSON data and the resulting LSTM bsed model was usually able to generate similarly structured JSON. I have seen other examples where the training data was code in C++.

How is this example working? The model learns what combinations of characters tend to appear together and in what order.

In the next chapter we will use pre-trained Deep Learning models for natural language processing (NLP).

Natural Language Processing

I have been working in the field of Natural Language Processing (NLP) since 1985 so I ‘lived through’ the revolutionary change in NLP that has occurred since 2014: deep learning results out-classed results from previous symbolic methods.

I will not cover older symbolic methods of NLP here, rather I refer you to my previous books Practical Artificial Intelligence Programming With Java, [Loving Common Lisp, The Savvy Programmer’s Secret Weapon, and Haskell Tutorial and Cookbook for examples. We get better results using Deep Learning (DL) for NLP and the library spaCy (https://spacy.io) that we use in this chapter provides near state-of-the-art performance. The authors of spaCy frequently update it to use the latest breakthroughs in the field.

You will learn how to apply both DL and NLP by using the state-of-the-art full-feature library spaCy. This chapter concentrates on how to use spaCy in the Hy language for solutions to a few selected problems in NLP that I use in my own work. I urge you to also review the “Guides” section of the spaCy documentation where examples are in Python but after experimenting with the examples in this chapter you should have no difficulty in translating any spaCy Python examples to the Hy language.

If you have not already done so install the spaCy library and the full English language model:

pip install spacy
python -m spacy download en

You can use a smaller model (which requires loading “en_core_web_sm” instead of “en” in the following examples):

pip install spacy
python -m spacy download en_core_web_sm

Exploring the spaCy Library

We will use the Hy REPL to experiment with spaCy, Lisp style. The following REPL listings are all from the same session, split into separate listings so that I can talk you through the examples:

 1 Marks-MacBook:nlp $ hy
 2 hy 0.17.0+108.g919a77e using CPython(default) 3.7.3 on Darwin
 3 => (import spacy)
 4 => (setv nlp-model (spacy.load "en"))
 5 => (setv doc (nlp-model "President George Bush went to Mexico and he had a very good\
 6  meal"))
 7 => doc
 8 President George Bush went to Mexico and he had a very good meal
 9 => (dir doc)
10 ['_', '__bytes__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__fo\
11 rmat__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init_\
12 _', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__ne__', '__new\
13 __', '__pyx_vtable__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__\
14 setstate__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '_bulk_merg\
15 e', '_py_tokens', '_realloc', '_vector', '_vector_norm', 'cats', 'char_span', 'count\
16 _by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', 'from_disk', 'get_\
17 extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_nered', 'is_parsed'\
18 , 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'merge', 'noun_chunks', 'noun\
19 _chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', 'sentiment', 'sen\
20 ts', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws', 'to_array', 't\
21 o_bytes', 'to_disk', 'to_json', 'user_data', 'user_hooks', 'user_span_hooks', 'user_\
22 token_hooks', 'vector', 'vector_norm', 'vocab']

In lines 3-6 we import the spaCy library, load the English language model, and create a document from input text. What is a spaCy document? In line 9 we use the standard Python function dir to look at all names and functions defined for the object doc returned from applying a spaCy model to a string containing text. The value printed shows many built in “dunder” (double underscore attributes), and we can remove these:

In lines 23-26 we use the dir function again to see the attributes and methods for this class, but filter out any attributes containing the characters “__”:

23 => (lfor
24 ... x (dir doc)
25 ... :if (not (.startswith x "__"))
26 ... x)
27 ['_', '_bulk_merge', '_py_tokens', '_realloc', '_vector', '_vector_norm', 'cats', 'c\
28 har_span', 'count_by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', '\
29 from_disk', 'get_extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_ne\
30 red', 'is_parsed', 'is_sentenced', 'is_tagged', 'lang', 'lang_', 'mem', 'merge', 'no\
31 un_chunks', 'noun_chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', \
32 'sentiment', 'sents', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws\
33 ', 'to_array', 'to_bytes', 'to_disk', 'to_json', 'user_data', 'user_hooks', 'user_sp\
34 an_hooks', 'user_token_hooks', 'vector', 'vector_norm', 'vocab']
35 =>

The to_json method looks promising so we will import the Python pretty print library and look at the pretty printed result of calling the to_json method on our document stored in doc:

 36 => (import [pprint [pprint]])
 37 => (pprint (doc.to_json))
 38 {'ents': [{'end': 21, 'label': 'PERSON', 'start': 10},
 39           {'end': 36, 'label': 'GPE', 'start': 30}],
 40  'sents': [{'end': 64, 'start': 0}],
 41  'text': 'President George Bush went to Mexico and he had a very good meal',
 42  'tokens': [{'dep': 'compound',
 43              'end': 9,
 44              'head': 2,
 45              'id': 0,
 46              'pos': 'PROPN',
 47              'start': 0,
 48              'tag': 'NNP'},
 49             {'dep': 'compound',
 50              'end': 16,
 51              'head': 2,
 52              'id': 1,
 53              'pos': 'PROPN',
 54              'start': 10,
 55              'tag': 'NNP'},
 56             {'dep': 'nsubj',
 57              'end': 21,
 58              'head': 3,
 59              'id': 2,
 60              'pos': 'PROPN',
 61              'start': 17,
 62              'tag': 'NNP'},
 63             {'dep': 'ROOT',
 64              'end': 26,
 65              'head': 3,
 66              'id': 3,
 67              'pos': 'VERB',
 68              'start': 22,
 69              'tag': 'VBD'},
 70             {'dep': 'prep',
 71              'end': 29,
 72              'head': 3,
 73              'id': 4,
 74              'pos': 'ADP',
 75              'start': 27,
 76              'tag': 'IN'},
 77             {'dep': 'pobj',
 78              'end': 36,
 79              'head': 4,
 80              'id': 5,
 81              'pos': 'PROPN',
 82              'start': 30,
 83              'tag': 'NNP'},
 84             {'dep': 'cc',
 85              'end': 40,
 86              'head': 3,
 87              'id': 6,
 88              'pos': 'CCONJ',
 89              'start': 37,
 90              'tag': 'CC'},
 91             {'dep': 'nsubj',
 92              'end': 43,
 93              'head': 8,
 94              'id': 7,
 95              'pos': 'PRON',
 96              'start': 41,
 97              'tag': 'PRP'},
 98             {'dep': 'conj',
 99              'end': 47,
100              'head': 3,
101              'id': 8,
102              'pos': 'VERB',
103              'start': 44,
104              'tag': 'VBD'},
105             {'dep': 'det',
106              'end': 49,
107              'head': 12,
108              'id': 9,
109              'pos': 'DET',
110              'start': 48,
111              'tag': 'DT'},
112             {'dep': 'advmod',
113              'end': 54,
114              'head': 11,
115              'id': 10,
116              'pos': 'ADV',
117              'start': 50,
118              'tag': 'RB'},
119             {'dep': 'amod',
120              'end': 59,
121              'head': 12,
122              'id': 11,
123              'pos': 'ADJ',
124              'start': 55,
125              'tag': 'JJ'},
126             {'dep': 'dobj',
127              'end': 64,
128              'head': 8,
129              'id': 12,
130              'pos': 'NOUN',
131              'start': 60,
132              'tag': 'NN'}]}
133 =>