Anatomy of Deep Learning Principles

Chapter 1 Programming and Math Fundamentals

• 1.1 Python quick start

  • 1.1.1 Python installation

    • Python interpreter installation
    • jupyter notebook programming environment
    • Anaconda installation tool

  • 1.1.2 Object, print() function, type conversion, comment, variable, input() function

    • 1. Objects

    • 2. Print function print()

    • 3. Type conversion

    • 4. Notes

    • 5. Variables

    • 6. input() function

      • 1.1.3 Operation

    • [subscript operator []](#subscript-operator-)

    • String formatting

  • 1.1.4 Control Statements

    • 1. if statement
    • 2. while statement
    • 3. for statement

  • 1.1.5 Python commonly used container types

    • 1. list (list)
    • index
    • slice
    • for traverse all elements
    • 2. tuple (tuple)
    • 3. set (collection)
    • 4. dict (dictionary)

  • 1.1.6 Functions

    • math math package
    • Global and local variables
    • Anonymous/Lambda Function (anonymous/lambda function)
    • Nested functions, closures
    • yield and generators

  • 1.1.7 Classes and Objects

  • 1.1.8 Getting Started with Matplotlib

    • subplot()
    • Axes objects
    • mplot3d
    • display image

• 1.2 tensor library numpy

  • 1.2.1 What is a tensor?

    • 1 vector
    • The norm of the vector
    • 2 Matrix
    • 3 dimensional tensor

  • 1.2.2 Create ndarray object

    • 1. array()
    • 2. Multidimensional array type ndarray
    • 3. asarray()
    • 4. The tolist() method of ndarray
    • 5. astype() and reshape()
    • 6. arange() and linspace()
    • 7. full(), empty(), zeros(), ones(), eye()
    • 8. Common functions for creating tensors of random values
    • 9. Add, Repeat & Lay, Merge & Split, Edge Fill, Add Axis & Swap Axis
    • Repeat repeat()
    • laying tile()
    • merge concatenate()
    • overlay stack()
    • column_stack(), hstack(), vstack()
    • split split()
    • Edge Padding
    • Add Axis
    • Swap axes

  • 1.2.3 Indexing and slicing of ndarry arrays

  • 1.2.4 Tensor calculation

    • 1. Element-by-element calculation
    • Hadamard Product
    • 2. Cumulative calculation
    • 3. Dot Product
    • 4 Broadcast Broadcasting

• 1.3 Calculus

  • 1.3.1 Functions

  • 1.3.2 Four arithmetic and compound operations

    • Arithmetic
    • Composite

  • 1.3.3 Limits, derivatives

    • 1. The limit of the sequence
    • 2. Limit and continuity of function
    • 3. Derivatives of functions

  • 1.3.4 The Four Arithmetic Operations of Derivatives and the Chain Derivation Rule

  • 1.3.5 Calculation graph, forward calculation, backpropagation derivation

  • 1.3.6 Partial derivatives and gradients of multivariable functions

  • 1.3.7 Derivative of vector-valued function and Jacobian matrix

  • 1.3.8 Integral

• 1.4 Probability Basics

  • 1.4.1 Probability

  • 1.4.2 Conditional probability, joint probability, total probability formula, Bayesian formula

  • 1.4.3 Random variables

  • 1.4.4 Probability distribution sequence of discrete random variables

  • 1.4.5 Probability Density of Continuous Random Variables

  • 1.4.6 Distribution functions of random variables

  • 1.4.7 Expectation, variance, covariance, covariance matrix

    • 1. Mean and Expectation
    • 2. Variance, standard deviation
    • 3. Covariance, covariance matrix

Chapter 2 Gradient descent method

• 2.1 Necessary conditions for function extremum

• 2.2 Gradient descent method (gradient descent)

• 2.3 Parameter optimization strategy of gradient descent method

  • 2.3.1 Momentum momentum method
  • 2.3.2 Adagrad method
  • 2.3.3 Adadelta method
  • 2.3.4 RMSprop method
  • 2.3.5 Adam method

• 2.4 Gradient verification

  • 2.4.1 Comparing numerical and analytical gradients
  • 2.4.2 Generic numerical gradients
  • 2.5 Separation gradient descent algorithm and parameter optimization strategy
  • 2.5.1 Parameter optimizer
  • 2.5.2 Gradient descent method accepting parameter optimizer

Chapter 3 Linear Regression, Logistic Regression and Softmax Regression

• 3.1 Linear regression

  • 3.1.1 Dining car profit problem

• 3.1.2 Machine Learning and Artificial Intelligence

  • 1. Machine Learning
  • 2. The relationship between machine learning and artificial intelligence
  • 3. Classification of machine learning

• 3.1.3 What is linear regression?

  • 3.1.4 Normal equations to solve linear regression problems

• 3.1.5 Gradient descent method to solve linear regression problems

• 3.1.6 Debug learning rate

• 3.1.7 Gradient verification

• 3.1.8 Prediction

• 3.1.9 Linear regression with multiple features

  • 1. Multi-feature linear regression
  • 2. Fitting plane
  • 3. Temperature and pressure problems

• 3.1.10 Normalization of data

• 3.2 Evaluation of the model

  • 3.2.1 Underfitting and overfitting
  • 3.2.2 Verification set, test set
  • 3.2.3 Learning Curve
  • 3.2.4 Forecasting the output of the dam
  • 3.2.5 Bias and variance (Bias-Variance)

• 3.3 Regularization - The loss function of adding the regular term becomes

• 3.5 Logistic regression

  • 3.5.1 Logistic regression

  • 3.5.2 numpy implementation of logistic regression

    • 1. Generate data
    • 2. Code implementation of gradient descent method
    • 3. Calculate the loss function value
    • 4. Decision curve
    • 5. Prediction accuracy
    • 6. Logistic Regression with Scikit-Learn Library

  • 3.5.3 Actual combat: numpy implementation of iris classification

• 3.6 softmax regression

  • 3.6.1 spiral data set

  • 3.6.2 softmax function

  • 3.6.3 softmax regression

    • Multi-sample form

  • 3.6.4 Multi-classification cross-entropy loss

  • 3.6.5 Calculate cross entropy loss by weighted sum

  • 3.6.6 Gradient calculation of softmax regression

    • 1. The gradient of the cross-entropy loss on the weighted sum
    • 2. The gradient of the cross-entropy loss with respect to the weight parameter

  • 3.6.7 Implementation of gradient descent method for softmax regression

  • 2.6.8 Softmax regression of spiral data set

• 3.7 Batch Gradient Descent and Stochastic Gradient Descent

  • 3.7.1 MNIST handwritten digit set

  • 3.7.2 Training logistic regression with partial training samples

  • 3.7.3 Batch Gradient Descent Method and Implementation

    • Softmax regression of Fasion MNIST training set

  • 3.7.4 Stochastic Gradient Descent

    • Summarize

Chapter 4 Neural Networks

• 4.1 Neural Network

  • 4.1.1 Perceptrons and neurons

    • 1. Perceptron
    • 2. Neurons

  • 4.1.2 Activation function

    • 1. Step function sign(x)
    • 2. Tanh function
    • 4. ReLU function

  • 4.1.3 Neural Networks and Deep Learning

  • 4.1.4 Forward calculation of multiple samples

  • 4.1.5 Output

  • 4.1.6 Loss function

    • 1. Mean square error loss
    • 2. Binary classification cross entropy loss
    • 3. Multi-classification cross-entropy loss

  • 4.1.7 Neural Network Training Based on Numerical Gradients

  • 4.1.8 Deep Learning

• 4.2 Reverse derivation

  • 4.2.1 Forward calculation and reverse derivation

  • 4.2.2 Computation graph

  • 4.2.3 The gradient of the loss function with respect to the output

    • 1. The gradient of the binary cross-entropy loss function on the output
    • 2. The gradient of the mean square error loss function on the output
    • 3. The gradient of the multi-class cross entropy loss function on the output

  • 4.2.4 Derivation of back propagation of 2-layer neural network

    • 1. Reverse derivation of single sample
    • 2. Multi-sample vectorized representation of reverse derivation
    • 3. Gradient calculation formula in column vector form

  • 4.2.5 Python implementation of 2-layer neural network

  • 4.2.6 Derivation of backpropagation of any layer neural network

• 4.3 Implement a simple deep learning framework

  • 4.3.1 Training process of neural network

  • 4.3.2 Code implementation of the network layer

  • 4.3.3 Gradient test of network layer

  • 4.3.4 Neural Network Class

  • 4.3.5 Gradient test of neural network

  • 4.3.6 MNIST data handwritten digit recognition based on deep learning framework

  • 4.3.7 Improved general neural network framework: separate weighted sum and activation function

    • Gradient Validation

  • 4.3.8 Independent parameter optimizer

  • 4.3.9 fashion-mnist classification training

  • 4.3.9 Read and write model parameters

Chapter 5 Basic Techniques for Improving Neural Network Performance

• 5.1 Data processing

  • 5.1.1 Data Augmentation

  • 5.1.2 Normalization

  • 5.1.3 Feature Engineering

    • 1. Data dimensionality reduction and principal component analysis

  • 2 Whitening

• 5.2 Parameter debugging

  • 5.2.1 Weight initialization
  • 5.2.2 Optimization parameters

• 5.3 Batch Normalization

  • 5.3.1 What is batch normalization?
  • 5.3.2 Reverse derivation of batch normalization
  • 5.3.3 Code Implementation of Batch Normalization

• 5.4 Regularization Regularization

  • 5.4.1 Weight regularization
  • 5.4.2 Dropout
  • 5.4.3 Early stopping method (Early stopping)

Chapter 6 Convolutional Neural Network CNN

• 6.1 Convolution

  • 6.1.1 What is convolution?

    • span

  • 6.1.2 Convolution of one-dimensional signal

  • 6.1.3 Two-dimensional convolution

  • span

  • 6.1.4 Multiple input channels and multiple output channels

  • 6.1.5 Pooling

• 6.2 Convolutional Neural Network

  • 6.2.1 Fully connected neurons and convolutional neurons

  • 6.2.2 Convolutional Layer and Convolutional Neural Network

  • 6.2.3 Reverse derivation and code implementation of convolutional layer and pooling layer

  • Reverse derivation of convolutional layer

    • The reverse derivation of the pooling layer

  • 6.2.4 Implementation of convolutional neural network

• 6.3 Convolution matrix multiplication

  • 6.3.1 Matrix multiplication of 1D sample convolution
  • 6.3.2 Matrix multiplication of 2D sample convolution
  • 6.3.3 Matrix multiplication for reverse derivation of 1D convolution
  • 6.3.4 Matrix multiplication for reverse derivation of 2D convolution

• 6.4 Fast convolution based on coordinate index

  • Gradient Test
  • Time comparison with non-accelerated convolution

• 6.5 Typical convolutional neural network structure

  • 6.5.1 LeNet-5
  • 6.5.2 AlexNet
  • 6.5.3 VGG
  • 6.5.4 Gradient Explosion and Vanishing Problems of Deep Neural Networks
  • 6.5.5 Residual Networks (ResNets)
  • 6.5.6 Google Inception Network
  • 6.5.7 Network in Network (NiN)

Chapter 7 Recurrent Neural Network RNN

• 7.1 Sequence problems and models

  • 7.1.1 Stock Price Prediction Problem

  • 7.1.2 Probabilistic sequence model, language model

    • 1. Probabilistic sequence model
    • 2. Language Model

  • 7.1.3 Autoregressive model

  • 7.1.4 Generate autoregressive data

  • 7.1.5 Time window method

  • 7.1.6 Time window sampling

  • 7.1.7 Time window method modeling and training

  • 7.1.8 Long-term forecast and short-term forecast

  • 7.1.9 Stock Price Prediction

  • 7.1.10 k-gram language model

• 7.2 Recurrent Neural Networks

  • 7.2.1 Acyclic neural network without memory function
  • 7.2.2 Recurrent neural network with memory function

• 7.3 Backpropagation through time

• 7.4 Implementation of single-layer recurrent neural network

  • 7.4.1 Initialize model parameters

  • 7.4.2 Forward calculation

  • 7.4.3 Loss function

  • 7.4.4 Reverse derivation

  • 7.4.5 Gradient verification

  • 7.4.6 Gradient descent training

  • 7.4.7 Sampling of sequence data

  • 7.4.8 RNN training and prediction of sequence data

    • Training on sequence data
    • predict
    • Training and prediction of stock data

• 7.5 RNN language model and text generation

  • 7.5.1 Character table

  • 7.5.2 Sampling of character sequence samples

  • 7.5.3 RNN model training and prediction

    • predict

• 7.6 Gradient explosion and gradient disappearance of RNN network

• 7.7 Long Short-Term Memory Network (LSTM)

  • 7.7.1 LSTM neuron: cell

  • 7.7.2 Reverse derivation of LSTM

  • 7.7.3 LSTM code implementation

    • Gradient Test
    • Text generation
    • predict

  • 7.7.4 Variations of LSTM

• 7.8 Gated Recurrent Unit (GRU)

  • 7.8.1 Working principle of GRU
  • 7.8.2 GRU code implementation

• 7.9 Class Representation and Implementation of Recurrent Neural Network

  • 7.9.1 Implementing Recurrent Neural Networks with Classes
  • 7.9.2 Class implementation of recurrent neural network unit
  • 7.10 Multilayer, Bidirectional Recurrent Neural Network
  • 7.10.1 Multilayer Recurrent Neural Network
  • 7.10.2 Training and prediction of multi-layer recurrent neural network
  • 7.10.3 Bidirectional Recurrent Neural Network

• 7.11 Sequence to sequence (seq2seq) model

  • machine translation

  • 7.11.1 Implementation of Seq2Seq model

  • 7.11.2 Seq2Seq for character-level machine translation

    • 1. Character word list
    • 2. Read training samples and build character vocabulary
    • 3. Training character-level Seq2Seq model

  • 7.11.3 Seq2Seq machine translation based on Word2Vec

    • 1. Word vectorization Word2Vec's skip-gram method

  • 7.11.4 Seq2Seq model based on word embedding layer

    • 1. Word embedding layer
    • 2. Seq2Seq model using word embedding layer

  • 7.11.5 Attention mechanism

Chapter 8 Generating Models

• 8.1 Generate model

• 8.2 Autoencoders

  • 8.2.1 Autoencoder
  • 8.2.2 Sparse Encoder
  • 8.2.3 Implementation of Autoencoder

• 8.3 Variational Autoencoders

  • 8.3.1 What is a variational autoencoder?
  • 8.3.2 Loss function
  • 8.3.3 Parameter resampling
  • 8.3.4 Reverse Derivation
  • 8.3.4 Implementation of Variational Autoencoder

• 8.4 Generating Adversarial Networks

  • 8.4.1 Principle of GAN

    • 1. Discriminator and Generator
    • 2. Loss function
    • 3. Training process

  • 8.4.2 Code implementation of GAN training process

• 8.5 GAN modeling example

  • 8.5.1 GAN modeling of a set of real numbers

    • 1. Real data: a set of real numbers
    • 2. Define discriminator and generator functions
    • 3. Real data iterator, noise data iterator
    • 4. Intermediate result drawing function
    • 5. Training GAN

  • 8.5.2 GAN modeling of two-dimensional coordinate points

    • 1. Real data: coordinate points sampled on the elliptic curve
    • 2. Real data iterator, noise iterator
    • 3. Define the generator and discriminator of the GAN model
    • 4. Training GAN model

  • 8.5.3 GAN modeling of MNIST dataset

    • 1. Read training data
    • 2. Define the data iterator
    • 3. Define the generator and discriminator and its optimizer
    • 4. Training model

  • 8.5.4 GAN training techniques

• 8.6 GAN loss function and its probability explanation

  • 8.6.1 The global optimal solution of the loss function of GAN
  • 8.6.2 Kullback–Leibler divergence and Jensen–Shannon divergence
  • 8.6.3 Maximum Likelihood Interpretation of GAN

• 8.7 Improved loss function: Wasserstein GAN (WGAN)

  • 8.7.1 Principle of Wasserstein GAN
  • 8.7.2 WGAN code implementation

• 8.8 Deep convolutional confrontation network DCGAN

  • 8.8.1 Transposed convolution of 1D vectors
  • 8.8.2 2D transposed convolution
  • 8.8.3 Implementation of convolutional confrontation network DCGAN