While the term deep learning was first associated with neural networks in 2000 by Igor Aizenberg and colleagues, it has only become popular in the last 5 years. Prior to this, we called this type of algorithm an artificial neural network (ANN). However, deep learning refers to something broader than ANNs and includes many other areas of connected machines. Therefore, to clarify, we will be discussing the ANN form of DL for much of the remainder of this book. However, we will also discuss some other forms of DL that can be used in games, in Chapter 5, Introducing DRL.

The first form of a multilayer perceptron (MLP) network, or what we now call an ANN, was introduced by Alexey Ivakhnenko...

The inspiration for neural networks or multilayer perceptrons is the human brain and nervous system. At the heart of our nervous system is the neuron pictured above the computer analog, which is a perceptron:

The neurons in our brain collect input, do something, and then spit out a response much like the computer analog, the perceptron. A perceptron takes a set of inputs, sums them all up, and passes them through an activation function. That activation function determines whether to send output, and at what level to send it when activated. Let's take a closer look at the perceptron, as follows:

On the left-hand side of the preceding diagram, you can see the set of inputs getting pushed in, plus a constant bias. We will get more into the bias later. Then the inputs are multiplied...

Thus far, we have been looking at a simple example of a single perceptron and how to train it. This worked well for our small dataset, but as the number of inputs increases, the complexity of our networks increases, and this cascades into the math as well. The following diagram shows a multilayer perceptron, or what we commonly refer to as an ANN:

In the diagram, we see a network with one input, one hidden, and one output layer. The inputs are now shared across an input layer of neurons. The first layer of neurons processes the inputs, and outputs the results to be processed by the hidden layer and so on, until they finally reach the output layer.

Multilayer networks can get quite complex, and the code for these models is often abstracted away by high-level interfaces such as Keras, PyTorch, and so on. These tools work...

TensorFlow (TF) is quickly becoming the technology that powers many DL applications. There are other APIs, such as Theano, but it is the one that has gathered the greatest interest and mostly applies to us. Overarching frameworks, such as Keras, offer the ability to deploy TF or Theano models, for instance. This is great for prototyping and building a quick proof of concept, but, as a game developer, you know that when it comes to games, the dominant requirements are always performance and control. TF provides better performance and more control than any higher-level framework such as Keras. In other words, to be a serious DL developer, you likely need and want to learn TF.

TF, as its name suggests, is all about tensors. A tensor is a mathematical concept that describes a set of data organized in n dimensions, where n could be 1, 2 x 2, 4 x 4 x 4, and so on...

Calculating the activation of a neuron, the forward part, or what we call feed-forward propagation, is quite straightforward to process. The complexity we encounter now is training the errors back through the network. When we train the network now, we start at the last output layer and determine the total error, just as we did with a single perceptron, but now we need to sum up all errors across the output layer. Then we need to use this value to backpropagate the error back through the network, updating each of the weights based on their contribution to the total error. Understanding the contribution of a single weight in a network with thousands or millions of weights could be quite complicated, except thankfully for the help of differentiation and the chain rule. Before we get to the complicated math, we first need to discuss the...

While we have covered a lot of important ground we will need for understanding DL, what we haven't done yet is build something that can really do anything. One of the first problems we tackle when starting with DL is to build autoencoders to encode and reform data. Working through this exercise allows us to confirm that what goes into a network can also come back out of a network and essentially reassures us that an ANN is not a complete black box. Building and working with autoencoders also allows us to tweak and test various parameters in order to understand their function. Let's get started by opening up the Chapter_1_5.py listing and following these steps:

1. We will go through the listing section by section. First, we input the base layers Input and Dense, then Model, all from the tensorflow.keras module, with the following imports...

Use these additional exercises to assist in your learning and test your knowledge further.

Answer the following questions:

1. Name three different activation functions. Remember, Google is your friend.
2. What is the purpose of a bias?
3. What would you expect to happen if you reduced the number of epochs in one of the chapter samples? Did you try it?
4. What is the purpose of backpropagation?
5. Explain the purpose of the Cost function.
6. What happens when you increase or decrease the number of encoding dimensions in the Keras autoencoder example?
7. What is the name of the layer type that we feed input into?
8. What happens when you increase or decrease the batch size?
9. What is the shape of the input Tensor for the Keras example? Hint: we already have a print statement displaying this.
10. In the last exercise, how many MNIST samples do we train and test with?

As we progress in the book,...

In this chapter, we explored the foundations of DL from the basics of the simple single perceptron to more complex multilayer perceptron models. We started with the past, present, and future of DL and, from there, we built a basic reference implementation of a single perceptron so that we could understand the raw simplicity of DL. Then we built on our knowledge by adding more perceptrons into a multiple layer implementation using TF. Using TF allowed us to see how a raw internal model is represented and trained with a much more complex dataset, MNIST. Then we took a long journey through the math, and although a lot of the complex math was abstracted away from us with Keras, we took an in-depth look at how gradient descent and backpropagation work. Finally, we finished off the chapter with another reference implementation from Keras that featured an autoencoder. Auto encoding...