Deep Learning

Most of my professional career since 2014 has involved Deep Learning, mostly with TensorFlow using the Keras APIs. In the late 1980s I was on a DARPA neural network technology advisory panel for a year. I wrote the first prototype of the SAIC ANSim neural network library commercial product, and I wrote the neural network prediction code for a bomb detector my company designed and built for the FAA for deployment in airports. More recently I have used GAN (generative adversarial networks) models for synthesizing numeric spreadsheet data and LSTM (long short term memory) models to synthesize highly structured text data like nested JSON and for NLP (natural language processing). I have 55 USA and several European patents using neural network and Deep Learning technology.

The Hy language utilities and example programs we develop here all use TensorFlow and Keras “under the hood” to do the heavy lifting. Keras is a simpler to use API for TensorFlow and I usually use Keras rather than the lower level TensorFlow APIs.

There are other libraries and frameworks that might interest you in addition to TensorFlow and Keras. I particularly like the Flux library for the Julia programming language. Currently Python has the most comprehensive libraries for Deep Learning but other languages that support differential computing (more on this later) like Julia and Swift may gain popularity in the future.

Here we will learn a vocabulary for discussing Deep Learning neural network models, look at possible architectures, and show two Hy language examples that should be sufficient to get you familiar to using Keras with the Hy language. If you already have Deep Learning application development experience you might want to skip the following review material and go directly to the Hy language examples.

If you want to use Deep Learning professionally, there are two specific online resources that I recommend: Andrew Ng leads the efforts at deeplearning.ai and Jeremy Howard leads the efforts at fast.ai. Here I will show you how to use a few useful techniques. Andrew and Jeremy will teach you skills that may lead a professional level of expertise if you take their courses.

There are many Deep Learning neural architectures in current practical use; a few types that I use are:

  • Multi-layer perceptron networks with many fully connected layers. An input layer contains placeholders for input data. Each element in the input layer is connected by a two-dimensional weight matrix to each element in the first hidden layer. We can use any number of fully connected hidden layers, with the last hidden layer connected to an output layer.
  • Convolutional networks for image processing and text classification. Convolutions, or filters, are small windows that can process input images (filters are two-dimensional) or sequences like text (filters are one-dimensional). Each filter uses a single set of learned weights independent of where the filter is applied in an input image or input sequence.
  • Autoencoders have the same number of input layer and output layer elements with one or more hidden fully connected layers. Autoencoders are trained to produce the same output as training input values using a relatively small number of hidden layer elements. Autoencoders are capable of removing noise in input data.
  • LSTM (long short term memory) process elements in a sequence in order and are capable of remembering patterns that they have seen earlier in the sequence.
  • GAN (generative adversarial networks) models comprise two different and competing neural models, the generator and the discriminator. GANs are often trained on input images (although in my work I have applied GANs to two-dimensional numeric spreadsheet data). The generator model takes as input a “latent input vector” (this is just a vector of specific size with random values) and generates a random output image. The weights of the generator model are trained to produce random images that are similar to how training images look. The discriminator model is trained to recognize if an arbitrary output image is original training data or an image created by the generator model. The generator and discriminator models are trained together.
Deep learning architecture overview: TensorFlow/Keras model training pipeline

The core functionality of libraries like TensorFlow are written in C++ and take advantage of special hardware like GPUs, custom ASICs, and devices like Google’s TPUs. Most people who work with Deep Learning models don’t need to even be aware of the low level optimizations used to make training and using Deep Learning models more efficient. That said, in the following section I am going to show you how simple neural networks are trained and used.

Simple Multi-layer Perceptron Neural Networks

I use the terms Multi-layer perceptron neural networks, backpropagation neural networks and delta-rule networks interchangeably. Backpropagation refers to the model training process of calculating the output errors when training inputs are passed in the forward direction from input layer, to hidden layers, and then to the output layer. There will be an error which is the difference between the calculated outputs and the training outputs. This error can be used to adjust the weights from the last hidden layer to the output layer to reduce the error. The error is then backprogated backwards through the hidden layers, updating all weights in the model. I have detailed example code in any of my older artificial intelligence books. Here I am satisfied to give you an intuition to how simple neural networks are trained.

The basic idea is that we start with a network initialized with random weights and for each training case we propagate the inputs through the network towards the output neurons, calculate the output errors, and back-up the errors from the output neurons back towards the input neurons in order to make small changes to the weights to lower the error for the current training example. We repeat this process by cycling through the training examples many times.

The following figure shows a simple backpropagation network with one hidden layer. Neurons in adjacent layers are connected by floating point connection strength weights. These weights start out as small random values that change as the network is trained. Weights are represented in the following figure by arrows; in the code the weights connecting the input to the output neurons are represented as a two-dimensional array.

Example Backpropagation network with One Hidden Layer

Each non-input neuron has an activation value that is calculated from the activation values of connected neurons feeding into it, gated (adjusted) by the connection weights. For example, in the above figure, the value of Output 1 neuron is calculated by summing the activation of Input 1 times weight W1,1 and Input 2 activation times weight W2,1 and applying a “squashing function” like Sigmoid or Relu (see figures below) to this sum to get the final value for Output 1’s activation value. We want to flatten activation values to a relatively small range but still maintain relative values. To do this flattening we use the Sigmoid function that is seen in the next figure, along with the derivative of the Sigmoid function which we will use in the code for training a network by adjusting the weights.

Sigmoid Function and Derivative of Sigmoid Function (SigmoidP)

Simple neural network architectures with just one or two hidden layers are easy to train using backpropagation and I have examples of from-scratch code for this several of my previous books. You can see Java and Common Lisp from-scratch implementations in two of my books that you can read online: Practical Artificial Intelligence Programming With Java and Loving Common Lisp, or the Savvy Programmer’s Secret Weapon. However, here we are using Hy to write models using the TensorFlow framework which has the huge advantage that small models you experiment with on your laptop can be scaled to more parameters (usually this means more neurons in hidden layers which increases the number of weights in a model) and run in the cloud using multiple GPUs.

Except for pedantic purposes, I now never write neural network code from scratch, instead I take advantage of the many person-years of engineering work put into the development of frameworks like TensorFlow, PyTorch, mxnet, etc. We now move on to two examples built with TensorFlow.

Deep Learning

Deep Learning models are generally understood to have many more hidden layers than simple multi-layer perceptron neural networks and often comprise multiple simple models combined together in series or in parallel. Complex architectures can be iteratively developed by manually adjusting the size of model components, changing the components, etc. Alternatively, model architecture search can be automated. At Capital One I used Google’s AdaNet project that efficiently searches for effective model architectures inside a single TensorFlow session. The model architecture used here is simple: one input layer representing the input values in a sample of University of Wisconsin cancer data, one hidden layer, and an output layer consisting of one neuron whose activation value will be interpreted as a prediction of benign or malignant.

The material in this chapter is intended to serve two purposes:

  • If you are already familiar with Deep Learning and TensorFlow then the examples here will serve to show you how to call the TensorFlow APIs from Hy.
  • If you have little or no exposure with Deep Learning then the short Hy language examples will provide you with concise code to experiment with and you can then decide to study further.

Once again, I recommend that you consider taking two online Deep Learning course sequences. For no cost, Jeremy Howard provides lessons at fast.ai that are very good and the later classes use PyTorch which is a framework that is similar to TensorFlow. For a modest cost Andrew Ng provides classes at deeplearning.ai that use TensorFlow. I have been working in the field of machine learning since the 1980s, but I still take Andrew’s online classes to stay up-to-date. In the last eight years I have taken his Stanford University machine learning class twice and also his complete course sequence using TensorFlow. I have also worked through much of Jeremy’s material. I recommend both course sequences without reservation.

Using Keras and TensorFlow to Model The Wisconsin Cancer Data Set

The University of Wisconsin cancer database has 646 samples. Each sample has 9 input values and one output value:

  • Cl.thickness: Clump Thickness
  • Cell.size: Uniformity of Cell Size
  • Cell.shape: Uniformity of Cell Shape
  • Marg.adhesion: Marginal Adhesion
  • Epith.c.size: Single Epithelial Cell Size
  • Bare.nuclei: Bare Nuclei
  • Bl.cromatin: Bland Chromatin
  • Normal.nucleoli: Normal Nucleoli
  • Mitoses: Mitoses
  • Class: Class (0 for benign, 1 for malignant)

Each row represents a sample with different measurements related to cell properties, and the final column ‘Class’ indicates whether the sample is benign (0) or malignant (1).

Let’s perform some basic analysis on this data:

  • Check for missing values.
  • Get the summary statistics of the dataset.
  • Check the balance of the classes (benign and malignant).

Here’s the analysis:

  • Missing Values: There are no missing values in the dataset. Each column has complete data.
  • Summary Statistics: The mean and median (50%) values of most features are quite different, indicating that the data distribution for these features might be skewed.
  • The range (min to max) for all features is from 1 to 10, indicating that the measurements are likely based on a scale or ranking system of 1 to 10.
  • Class Balance: The dataset is somewhat imbalanced. Approximately 65% of the samples are benign (0) and 35% are malignant (1). This imbalance might influence the performance of machine learning models trained on this data.

Now, it would be beneficial to visualize the data to get a better understanding of the distribution of each feature and the relationship between different features:

Histogram Plots

Here are histograms of each feature, broken down by class (benign or malignant). Some observations:

Cl.thickness, Cell.size, Cell.shape, Marg.adhesion, Epith.c.size, Bare.nuclei, Bl.cromatin, Normal.nucleoli: For these features, higher values seem to be associated with the malignant class. This might suggest that these characteristics are significant in the determination of malignancy. Mitoses: This feature shows a different trend, with a majority of both benign and malignant cases having low mitoses values. However, there are more malignant cases with higher mitoses values than benign cases.

We will use separate training and test files source_code_for_examples/deeplearning/train.csv and source_code_for_examples/deeplearning/test.csv. Here are a few samples from the training file:

1 6,2,1,1,1,1,7,1,1,0
2 2,5,3,3,6,7,7,5,1,1
3 10,4,3,1,3,3,6,5,2,1
4 6,10,10,2,8,10,7,3,3,1
5 5,6,5,6,10,1,3,1,1,1
6 1,1,1,1,2,1,2,1,2,0
7 3,7,7,4,4,9,4,8,1,1
8 1,1,1,1,2,1,2,1,1,0

After you look at this data, if you did not have much experience with machine learning then it might not be obvious how to build a model to accept a sample for a patient like we see in the Wisconsin data set and then predict if the sample implies benign or cancerous outcome for the patient. Using TensorFlow with a simple neural network model, we will implement a model in about 40 lines of Hy code to implement this example.

Since there are nine input values we will need nine input neurons that will represent the input values for a sample in either training or separate test data. The input layer is declared on line 11 of the following listing. The input neurons are completely connected to a first hidden layer of nine neurons (line 12), which is in turn completely connected to a second hidden layer of twelve neurons (line 13). Here, completely connected means that each neuron in one layer is connected via a weight to each neuron in the next layer; for example, there are 9 * 12 = 108 weights between the two hidden layers. There is a single output layer neuron (line 14) that is connected to each neuron in the second hidden layer.

Notice that in lines 12 and 13 in the following listing that we specify a relu activation function while the activation function connecting the last hidden layer to the output layer uses the sigmoid activation function that we saw plotted earlier.

There is an example in the git example repo directory source_code_for_examples/matplotlib_examples in the file plot_relu.hy that generated the following figure:

Relu Function

The following listing shows the use of the Keras APIs to build a model (lines 9-17) with one input layer, two hidden layers, and an output layer with just one neuron. After we build the model, we define two utility functions train (lines 19-20) to train a model given training inputs (x argument) and corresponding training outputs (y argument) for 500 epochs (passes through the training data) by default, and we also define predict (lines 22-23) using a trained model to make a cancer or benign prediction given test input values (x-data argument).

Lines 25-29 show a utility function load-data that loads all rows of a University of Wisconsin cancer data set CSV file using the pandas library, selects feature columns 0-8 and the label column 9 with iloc, scales the input values to the range [0.1, 1.0] (the target output values are already 0 or 1) and returns a list containing input (x-data) and target output data (y-data). You may want to load this example in a REPL and evaluate load-data on one of the CSV files.

The function main (lines 31-41) loads training and test (evaluation of model accuracy on data not used for training), trains a model, and then tests the accuracy of the model on the test (evaluation) data. It prints the fraction of test samples that are classified correctly (rounding each prediction to 0 or 1) followed by the first ten predictions:

 1 #!/usr/bin/env hy
 2 
 3 (import keras [Input])
 4 (import keras.models [Sequential])
 5 (import keras.layers [Dense])
 6 (import keras.optimizers [RMSprop])
 7 (import pandas [read-csv])
 8 
 9 (defn build-model []
10   (setv model (Sequential))
11   (.add model (Input :shape #(9)))           ; nine input features
12   (.add model (Dense 9 :activation "relu"))  ; first hidden layer
13   (.add model (Dense 12 :activation "relu")) ; second hidden layer
14   (.add model (Dense 1 :activation "sigmoid"))
15   (.compile model :loss "binary_crossentropy"
16                   :optimizer (RMSprop))
17   model)
18 
19 (defn train [batch-size model x y [epochs 500]]
20   (.fit model x y :batch-size batch-size :epochs epochs :verbose False))
21 
22 (defn predict [model x-data]
23   (.predict model x-data :verbose 0))
24 
25 (defn load-data [file-name]
26   (setv all-data (read-csv file-name :header None))
27   (setv features (get all-data.iloc #((slice None) (slice 0 9)))) ; columns 0-8
28   (setv labels (get all-data.iloc #((slice None) [9])))             ; column 9
29   [(* 0.1 features.values) labels.values])
30 
31 (defn main []
32   (setv [x-train y-train] (load-data "train.csv"))
33   (setv [x-test y-test] (load-data "test.csv"))
34   (setv model (build-model))
35   (train 10 model x-train y-train)
36   (setv predictions (lfor p (predict model x-test) (round (float (get p 0)) 4)))
37   (setv expected (lfor e y-test (int (get e 0))))
38   (setv correct (sum (gfor [p e] (zip predictions expected) (= (round p) e))))
39   (print f"* accuracy on {(len expected)} test samples: {(/ correct (len expected)) :.3f}")
40   (print "* first 10 predictions (calculated, expected):")
41   (print (list (zip (cut predictions 10) (cut expected 10)))))
42 
43 (when (= __name__ "__main__")
44   (main))

The following listing shows the output:

1 $ uv run hy wisconsin.hy
2 * accuracy on 151 test samples: 0.960
3 * first 10 predictions (calculated, expected):
4 [(1.0, 1), (1.0, 1), (0.9593, 1), (0.9233, 1), (0.0, 0), (0.0, 0), (1.0, 1), (0.0, 0), (0.0, 0), (1.0, 1)]

Let’s look at the third test case: the “real” output from the test data is a value of 1 and the calculated predicted value (using the trained model) is 0.9593. Your numbers will differ slightly from run to run because the model’s initial weights are random. In making predictions, we can choose a cutoff value, 0.5 for example, and interpret any calculated prediction value less than the cutoff as a Boolean false prediction and calculated prediction value greater to or equal to the cutoff value is a Boolean true prediction.

Using a LSTM Recurrent Neural Network to Generate English Text Similar to the Philosopher Nietzsche’s Writing

We will translate the Python example program Character-level text generation with LSTM from the Keras documentation to Hy. This is a moderately long example and you can use the original Python and the translated Hy code as a general guide for converting other models implemented in Python using Keras that you want use in Hy. I have kept the same overall structure as the Python version, but split the script into small functions (and used Hy’s kebab-case names like char-indices instead of char_indices) so that each step can be tried in a REPL and tested without training the full model.

Note that using the full nietzsche.txt data set requires a fair amount of memory, and training for the default 60 epochs takes a long time on a CPU. The script accepts two command line options: —max-chars to only use the beginning of the corpus and —epochs to train for fewer epochs. For example, on a laptop with less than 16G of RAM, or just to see the program work quickly, try:

1     uv run hy lstm.hy --epochs 5 --max-chars 150000

The first time you run this program, the training data is fetched from the web and stored locally in ~/.keras/datasets/nietzsche.txt; later runs reuse the local copy.

When I start training a new Deep Learning model I like to monitor system resources using the top command line activity, watching for page faults when training on a CPU which might indicate that I am trying to train too large a model for my system memory. If you are using CUDA and a GPU then use the CUDA command line utilities for monitoring the state of the GPU utilization. It is beyond the scope of this introductory tutorial, but the tool TensorBoard is very useful for monitoring the state of model training.

There are a few things that make the following example code more complex than the example using the University of Wisconsin cancer data set. We need to convert each character in the training data to a one-hot encoding which is a vector of all 0.0 values except for a single value of 1.0. I am going to show you a short REPL session so that you understand how this works and then we will look at the complete Hy code example.

 1 $ uv run hy
 2 => (import numpy :as np)
 3 => (import lstm [load-text char-maps one-hot])
 4 => (setv text (load-text))
 5 => (cut text 98 130)
 6 'philosophers, in so far as they '
 7 => (setv [chars char-indices indices-char] (char-maps text))
 8 => (len chars)
 9 84
10 => (cut chars 0 12)
11 ['\n', ' ', '!', '"', "'", '(', ')', ',', '-', '.', '0', '1']
12 => (get char-indices "!")
13 2
14 => (get indices-char 2)
15 '!'
16 => (setv x-pred (one-hot "Oh! I saw 1 dog (yesterday)" char-indices))
17 => x-pred.shape
18 (1, 40, 84)
19 => (get x-pred 0)
20 array([[0., 0., 0., ..., 0., 0., 0.],
21        [0., 0., 0., ..., 0., 0., 0.],
22        [0., 0., 1., ..., 0., 0., 0.],
23        ...,
24        [0., 0., 0., ..., 0., 0., 0.],
25        [0., 0., 0., ..., 0., 0., 0.],
26        [0., 0., 0., ..., 0., 0., 0.]])
27 => (lfor t (range 4) (get indices-char (int (np.argmax (get x-pred 0 t)))))
28 ['O', 'h', '!', ' ']
29 => 

The function char-maps builds two dictionaries that map each of the 84 unique characters in the corpus to an index and back. The function one-hot returns an array with one row of 84 values for each of the 40 (maxlen) character positions. In lines 20-26, each row represents a single character one-hot encoded. Notice how the third character shown on line 22 has a value of “1.” at index 2, which corresponds to the one-hot encoding of the character “!”. Rows after the end of our 27-character string are all zeros. The last expression decodes the first four rows back into characters by finding the index of the largest value in each row.

Now that you have a feeling for how one-hot encoding works, hopefully the following example will make sense to you. We will further discuss one-hot-encoding after the next code listing. For training, we take 40 characters (the value of the variable maxlen) at a time, and one-hot encode each character as input and the target output will be the one-hot encoding of the following character in the input sequence. We are iterating on training the model for a while and then given a few characters of text, predict a likely next character - and keep repeating this process. The generated text is then used as input to the model to generate yet more text. You can repeat this process until you have generated sufficient text.

This is a powerful technique that I used to model JSON with complex deeply nested schemas and then generate synthetic JSON in the same schema as the training data. Here, training a model to mimic the philosopher Nietzsche’s writing is much easier than learning highly structured data like JSON:

  1 #!/usr/bin/env hy
  2 
  3 ;; Character-level LSTM text generation, translated from the Python example
  4 ;; "Character-level text generation with LSTM" in the Keras documentation
  5 ;; and updated for Keras 3.
  6 ;;
  7 ;; Usage: uv run hy lstm.hy [--epochs N] [--max-chars N]
  8 
  9 (import argparse random sys)
 10 (import numpy :as np)
 11 (import keras [Input])
 12 (import keras.callbacks [LambdaCallback])
 13 (import keras.models [Sequential])
 14 (import keras.layers [Dense LSTM])
 15 (import keras.optimizers [RMSprop])
 16 (import keras.utils [get-file])
 17 
 18 (setv maxlen 40) ; length of each input sequence, in characters
 19 (setv step 3)    ; slide the sampling window 3 characters at a time
 20 
 21 (defn load-text [[max-chars None]]
 22   "Download nietzsche.txt once (Keras caches it in ~/.keras/datasets) and
 23   return its text, optionally truncated to MAX-CHARS to save memory."
 24   (setv path (get-file "nietzsche.txt"
 25     :origin "https://s3.amazonaws.com/text-datasets/nietzsche.txt"))
 26   (with [f (open path :encoding "utf-8")]
 27     (setv text (.read f)))
 28   (if max-chars (cut text max-chars) text))
 29 
 30 (defn char-maps [text]
 31   "Return [chars char-indices indices-char] for the unique characters in TEXT."
 32   (setv chars (sorted (set text)))
 33   [chars
 34    (dfor [i c] (enumerate chars) c i)
 35    (dfor [i c] (enumerate chars) i c)])
 36 
 37 (defn make-sequences [text]
 38   "Cut TEXT into overlapping maxlen-character sentences and, for each one,
 39   the single character that follows it."
 40   (setv starts (range 0 (- (len text) maxlen) step))
 41   [(lfor i starts (cut text i (+ i maxlen)))
 42    (lfor i starts (get text (+ i maxlen)))])
 43 
 44 (defn one-hot [sentence char-indices]
 45   "One-hot encode SENTENCE as a (1, maxlen, number-of-chars) array."
 46   (setv x (np.zeros #(1 maxlen (len char-indices))))
 47   (for [[t char] (enumerate sentence)]
 48     (setv (get x 0 t (get char-indices char)) 1))
 49   x)
 50 
 51 (defn vectorize [sentences next-chars char-indices]
 52   "Build the one-hot training inputs x and targets y."
 53   (setv n-chars (len char-indices))
 54   (setv x (np.zeros #((len sentences) maxlen n-chars) :dtype bool))
 55   (setv y (np.zeros #((len sentences) n-chars) :dtype bool))
 56   (for [[i sentence] (enumerate sentences)]
 57     (for [[t char] (enumerate sentence)]
 58       (setv (get x i t (get char-indices char)) 1))
 59     (setv (get y i (get char-indices (get next-chars i))) 1))
 60   [x y])
 61 
 62 (defn build-model [n-chars]
 63   (setv model (Sequential))
 64   (.add model (Input :shape #(maxlen n-chars)))
 65   (.add model (LSTM 128))
 66   (.add model (Dense n-chars :activation "softmax"))
 67   (.compile model :loss "categorical_crossentropy" :optimizer (RMSprop 0.01))
 68   model)
 69 
 70 (defn sample [preds [temperature 1.0]]
 71   "Pick the index of the next character from the probability vector PREDS.
 72   Low temperatures favor the most likely character; high ones add variety."
 73   (setv preds (/ (np.log (.astype (np.array preds) "float64")) temperature))
 74   (setv exp-preds (np.exp preds))
 75   (setv preds (/ exp-preds (np.sum exp-preds)))
 76   (np.argmax (np.random.multinomial 1 preds 1)))
 77 
 78 (defn generate [model seed char-indices indices-char
 79                 [n-chars 400] [temperature 1.0]]
 80   "Generate N-CHARS characters that continue the maxlen-character SEED."
 81   (setv sentence seed
 82         generated "")
 83   (for [_ (range n-chars)]
 84     (setv preds (get (model.predict (one-hot sentence char-indices) :verbose 0) 0))
 85     (setv next-char (get indices-char (sample preds temperature)))
 86     (+= generated next-char)
 87     (setv sentence (+ (cut sentence 1 None) next-char)))
 88   generated)
 89 
 90 (defn main []
 91   (setv parser (argparse.ArgumentParser
 92                  :description "Train a character-level LSTM on Nietzsche"))
 93   (.add-argument parser "--epochs" :type int :default 60)
 94   (.add-argument parser "--max-chars" :type int :default None
 95                  :help "only use the first N characters of the corpus (saves memory)")
 96   (setv args (.parse-args parser))
 97 
 98   (setv text (load-text args.max-chars))
 99   (print "corpus length:" (len text))
100   (setv [chars char-indices indices-char] (char-maps text))
101   (print "total chars (unique characters in input text):" (len chars))
102   (setv [sentences next-chars] (make-sequences text))
103   (print "number of training sequences:" (len sentences))
104   (setv [x y] (vectorize sentences next-chars char-indices))
105   (setv model (build-model (len chars)))
106 
107   (defn on-epoch-end [epoch [logs None]]
108     (print "\n----- Generating text after Epoch:" epoch)
109     (setv start (random.randint 0 (- (len text) maxlen 1)))
110     (setv seed (cut text start (+ start maxlen)))
111     (for [diversity [0.2 0.5 1.0 1.2]]
112       (print "----- diversity:" diversity)
113       (print "----- Generating with seed:" seed)
114       (print (+ seed (generate model seed char-indices indices-char
115                                :temperature diversity)))))
116 
117   (model.fit x y :batch-size 128 :epochs args.epochs
118              :callbacks [(LambdaCallback :on-epoch-end on-epoch-end)]))
119 
120 (when (= __name__ "__main__")
121   (main))

We run this example using:

1 uv run hy lstm.hy

The function make-sequences (lines 37-42) cuts the text into overlapping 40-character “sentences”, starting a new one every 3 characters, and pairs each one with the character that follows it. The function vectorize (lines 51-60) one-hot encodes these into the training inputs x and targets y. In lines 62-68 we define a model using the Keras APIs and compile it using a categorical crossentropy loss function with an RMSprop optimizer.

In lines 70-76 we define a function sample that takes a first required argument preds which is the model’s predicted probability for each of the 84 possible next characters, and might look like:

[2.80193929e-02 6.78635418e-01 7.85831537e-04 4.92034527e-03 . . . 6.62320468e-04 9.14627407e-03 2.31375365e-04]

Now, here the predicted values are not strictly 0 or 1, rather they are small floating point numbers with a single number much larger than the others. The optional temperature argument rescales these probabilities: a low temperature makes the most likely character even more likely and a high temperature flattens the distribution so we get more varied (and more surprising) text. Instead of always taking the largest value, sample draws a random character using these rescaled probabilities. The function generate (lines 78-88) repeatedly predicts and samples a next character, appends it, and slides the 40-character input window forward by one character. The largest number is 6.78635418e-01 at index 1 which corresponds to a one-hot encoding for a “ “ space character.

If we print out the number of characters in text and the unique list of characters (variable chars) in the training text file nietzsche.txt we see:

1 corpus length: 600893
2 ['\n', ' ', '!', '"', "'", '(', ')', ',', '-', '.', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '=', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', ']', '_', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'Æ', 'ä', 'æ', 'é', 'ë']

A review of one-hot encoding:

Let’s review our earlier discussion of one-hot encoding with a simpler case. It is important to understand how we one-hot encode input text to inputs for the model and decode back one-hot vectors to text when we use a trained model to generate text. It will help to see the dictionaries for converting characters to indices and then reverse indices to original characters as we saw earlier, some output removed:

1 char_indices:
2  {'\n': 0, ' ': 1, '!': 2, '"': 3, "'": 4, '(': 5, ')': 6, ',': 7, '-': 8, '.': 9, '0': 10, '1': 11, '2': 12, '3': 13, '4': 14, '5': 15, '6': 16, '7': 17, '8': 18, '9': 19, 
3    . . .
4  'f': 58, 'g': 59, 'h': 60, 'i': 61, 'j': 62, 'k': 63, 'l': 64, 'm': 65, 'n': 66, 'o': 67, 'p': 68, 'q': 69, 'r': 70, 's': 71, 't': 72, 'u': 73, 'v': 74, 'w': 75, 'x': 76, 'y': 77, 'z': 78, 'Æ': 79, 'ä': 80, 'æ': 81, 'é': 82, 'ë': 83}
5 indices_char:
6  {0: '\n', 1: ' ', 2: '!', 3: '"', 4: "'", 5: '(', 6: ')', 7: ',', 8: '-', 9: '.', 10: '0', 11: '1', 12: '2', 13: '3', 14: '4', 15: '5', 16: '6', 17: '7', 18: '8', 19: '9', 
7    . . .
8  'o', 68: 'p', 69: 'q', 70: 'r', 71: 's', 72: 't', 73: 'u', 74: 'v', 75: 'w', 76: 'x', 77: 'y', 78: 'z', 79: 'Æ', 80: 'ä', 81: 'æ', 82: 'é', 83: 'ë'}

We prepare the input and target output data in make-sequences in the last code listing. Using a short string, let’s look in the next REPL session listing at how these input and output training examples are extracted for an input string:

 1 $ uv run hy
 2 => (setv text "0123456789abcdefg")
 3 => (setv maxlen 4)
 4 => (setv i 3)                 ;; i is the start of one training sentence
 5 => (cut text i (+ i maxlen))  ;; the input sentence
 6 '3456'
 7 => (get text (+ i maxlen))    ;; the target: the next character
 8 '7'
 9 => (setv i 6)                 ;; with step = 3 the next sentence starts at 6
10 => (cut text i (+ i maxlen))
11 '6789'
12 => (get text (+ i maxlen))
13 'a'
14 => 

So the input training sentences are each maxlen characters long and each next-chars target output is the single character after the last character in the corresponding input training sentence.

The function on-epoch-end (lines 107-115) is called by Keras after each training epoch through a LambdaCallback. It pauses training to generate text given diversity values of 0.2, 0.5, 1.0, and 1.2. The smaller the diversity value the more closely the generated text matches the training text. The generated text is more realistic after many training epocs. In the following, I list a highly edited copy of running through several training epochs. I only show generated text for diversity equal to 0.2:

 1 $ uv run hy lstm.hy
 2 ----- Generating text after Epoch: 0
 3 ----- diversity: 0.2
 4 ----- Generating with seed: ocity. Equally so, gratitude.--Justice r
 5 ocity. Equally so, gratitude.--Justice read in the become to the conscience the seener and the conception that the becess of the power to the procentical that the because and the prostice of the prostice and the will to the conscience of the power of the perhaps the self-distance of the all the soul and the world and the soul of the soul of the world and the soul and an an and the profound the self-dister the all the belief and the
 6 
 7 ----- Generating text after Epoch: 8
 8 ----- diversity: 0.2
 9 ----- Generating with seed: nations
10 laboring simultaneously under th
11 nations
12 laboring simultaneously under the subjection of the soul of the same to the subjection of the subjection of the same not a strong the soul of the spiritual to the same really the propers to the stree be the subjection of the spiritual that is to probably the stree concerning the spiritual the sublicities and the spiritual to the processities the spirit to the soul of the subjection of the self-constitution and propers to the
13 
14 ----- Generating text after Epoch: 14
15 ----- diversity: 0.2
16 ----- Generating with seed:  to which no other path could conduct us
17  to which no other path could conduct us a stronger that is the self-delight and the strange the soul of the world of the sense of the sense of the consider the such a state of the sense of the sense of the sense of such a sandine and interpretation of the process of the sense of the sense of the sense of the soul of the process of the world in the sense of the sense of the spirit and superstetion of the world the sense of the
18 
19 ----- Generating text after Epoch: 17
20 ----- diversity: 0.2
21 ----- Generating with seed: hemselves although they could easily hav
22 hemselves although they could easily have been moral morality and the self-in which the self-in the world to the same man in the standard to the possibility that is to the strength of the sense-in the former the sense-in the special and the same man in the consequently the soul of the superstition of the special in the end to the possible that it is will not be a sort of the superior of the superstition of the same man to the same man

Here we trained on examples, translated to English, of the philosopher Nietzsche. I have used similar code to this example to train on highly structured JSON data and the resulting LSTM based model was usually able to generate similarly structured JSON. I have seen other examples where the training data was code in C++.

How is this example working? The model learns what combinations of characters tend to appear together and in what order.

In the next chapter we will use pre-trained Deep Learning models for natural language processing (NLP).

Optional Practice Problems

  1. Model Tuning in wisconsin.hy The program prints the model’s accuracy on the test data. Try to improve it (or reach the same accuracy with less training) by experimenting with:

    • Changing the hidden layer sizes or adding another Dense layer.
    • Using the Adam optimizer instead of RMSprop.
    • Modifying the activation functions (e.g., changing hidden layers to tanh or elu).
  2. Hyperparameter Tuning in lstm.hy In lstm.hy, the sequence length maxlen is set to 40, and the sampling step size step is set to 3. Modify these hyperparameters:

    • Decrease maxlen to 20 or increase it to 60. How does this affect text coherence and training time?
    • Change step to 1 or 5. How does this change the total number of sentences generated for training data, and what is the impact on memory usage?
  3. Saving and Reloading the LSTM Model Because training a character-level LSTM model takes a substantial amount of time, it is impractical to retrain it every time you want to generate text. Modify lstm.hy to:

    • Save the model weights and structure to disk using (.save model "nietzsche_lstm.keras") at the end of training.
    • Write a separate Hy script (e.g., generate.hy) that loads the saved model and generates text from a given seed without retraining.