3 Neural Networks

This book is about “deep learning”, but deep learning is really just a name for the modern use of (deep) neural networks. Before we get to deep learning, we need to first look at traditional (shallow) neural networks and understand the basic concepts and issues with them. In this chapter we will cover those basics, preparing us to understand the more recent innovations that fall into the category of deep learning.

What is a neural network?

A neural network, or more properly an artificial neural network¹, is a type of machine learning model. Under the surface it’s a collection of mathematical operations that take raw input data and, based on the internal structure and parameters of the neural network, it produces an estimate or prediction of some quantity as the output. Just as with other ML models, the parameters of the neural network are learned from training data.

Unlike other machine learning models, neural networks are explicitly inspired by the structure and, to some degree, the function of animal brains. They are collections of neuron-like structures that are connected to each other as a network, similar to real neural networks in real brains. The neurons, or nodes, in artificial neural networks are much simpler than real neurons and are designed to have convenient mathematical properties, rather than to simulate or closely mimic the mechanics of real neurons. That said, the basis for exploring these types of models has been the flexibility and power of real brains. For this reason, artificial neural networks became one of the main approaches to the problem of artificial intelligence.

As we will discuss in the rest of this book, neural networks are not one single model, but due to the flexibility of how the components of neural networks can be combined, they form a very large class of machine learning models. We will first look at the simplest form of neural network and in later chapters learn about many of the variations that are used for specific tasks.

What are some tasks that neural networks can accomplish?

Because of the variety of internal structures possible with neural networks, they are able to be applied to many different kinds of tasks. In recent years they have proven to be the best solution to several tasks, including most computer vision and language related tasks.

Some examples of tasks that neural networks perform well on:

• Identifying what is in a photograph
• Recognizing faces and fingerprints
• Transcribing speech
• Translating text from one language to another
• Generating images and text from scratch or from prompts
• Transforming raw images into other styles
• Playing games like Atari and go (typically in conjunction with other algorithms)
• Web search
• Product recommendations
• Recognizing and transcribing handwritten or printed text
• Compressing data

Some of these tasks have only become tractable in the past decade due to the development of (useable) deep neural networks. On some tasks neural networks have even been able to match or surpass human performance. Other tasks have been addressed with neural networks for decades. The success and promise of neural networks across so many tasks is one of the main reasons why they are of so much interest to researchers and engineers. They are not one-trick ponies. They can be adapted to almost any machine learning task.

The building blocks of neural networks

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Activation functions

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Neural network layers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Connections, weights, and biases

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Learning via gradient descent

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Backpropagation of the gradient

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Vanishing gradients and parameter initialization

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Output layers

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

What does a neural network do?

Having looked at the mechanics of how neural networks work, it’s natural to ask “what does a neural network actually do?” in a more abstract sense and “how is a neural network different from other types of machine learning models?”.

In most traditional machine learning models, much of the effort from the person training the model is around feature engineering, i.e. creating, iterating, and selecting the best features. The goal is to find the information or ways to represent the information that best allows the model to cleanly separate classes in the data or pick out the pattern needed to predict numerical values (i.e. regression). While some of this can be automated to a certain extent, it is one of the main focuses when training traditional machine learning models.

Neural networks present a different set of capabilities and shift the focus away from feature engineering. One way to think of this is that a neural network is learning how to warp the existing feature “space” in a way that makes the classes easily separable by the final layer of the network. Similarly, when performing a regression task, the network is learning to represent the features in a way that allows it to easily perform a simple linear regression in the last layer. The flexibility within the network allows it to combine the inputted raw features in different ways until it lands on the best way to represent these features for the goal of the network.

This can be contrasted to traditional methods such as a support vector machine classifier, where the person training the model needs to select a transform kernel that will project the features into a space that allows for easy separation of the classes. Done properly, a neural network will learn the equivalent of this projection kernel on its own.

An important aspect of neural networks that makes this possible is the layer structure of the network. This lends itself to a hierarchical representation of data/features that is natural to many types of data. Thinking about image data, we can imagine the hierarchy of “features”, going from individual pixels, to basic lines, to curves, shapes, and complicated structures, such as faces. A neural network tends to naturally learn these types of hierarchical levels in data.

Generalization in a model is about finding the common patterns, while ignoring the “noise” specific to individual examples. One way to do this is by learning “compressed” representations of data. Encoding data in a smaller number of bits typically results in “loss”, meaning that it is not quite the same as the original. If you can create a compressed encoding that can then decompress in a way that has relatively high fidelity to the original (by some measure), that typically means that non-essential aspects (i.e. noise) have been removed. Neural networks are often able to create these kinds of compressed representations of data internally, preserving the fundamental patterns and helping them generalize.

As we will discuss later, a final very important aspect of neural networks is their composibility. Because of their layered, hierarchical nature, neural networks can learn to represent data for one task and then be repurposed for other tasks by modifying the later layers in the network. This access to intermediate feature representations from somewhere in the middle of the model is unlike most other types of ML models, where typically only the final result is of any value.

From basic neural networks to deep learning

This content is not available in the sample book. The book can be purchased on Leanpub at http://leanpub.com/zefsguide2dl.

Resources

Some further resources for learning the basics of neural networks and deep learning:

Courses

• The NYU Deep Learning course (Theme 1) by Yann LeCun & Alfredo Canziani.
• Fast.ai’s Practical Deep Learning for Coders, with Sylvain Gugger and Jeremy Howard.
• Andrew Ng’s Neural Networks and Deep Learning on Coursera.

Books

• “Neural Networks and Deep Learning”, by Michael Nielsen
• “Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python” by Sebastian Raschka, Yuxi (Hayden) Liu, and Vahid Mirjalili. Packt Publishing.

Other online resources

• Neural Networks series by 1Blue3Brown (Grant Sanderson).
• How Neural Networks Work by Brandon Rohrer.