TensorFlow is a powerful open source software library developed by the Google Brain team for deep neural networks (DNNs). It was first made available under the Apache 2.x License in November 2015; as of today, its GitHub repository (https://github.com/tensorflow/tensorflow) has more than 17,000 commits, with roughly 845 contributors in just two years. This by itself is a measure of both the popularity and performance of TensorFlow. The following graph shows a comparison of the popular deep learning frameworks where it is clearly visible that TensorFlow is the leader among them:

Let's first learn what exactly TensorFlow is and why is it so popular among DNN researchers and engineers. TensorFlow, the open source deep learning library allows one to deploy deep neural networks computation on one or more CPU, GPUs in a server, desktop or mobile using the single TensorFlow API. You might ask, there are so many other deep learning libraries such as Torch, Theano, Caffe, and MxNet; what makes TensorFlow special? Most of other deep learning libraries like TensorFlow have auto-differentiation, many are open source, most support CPU/GPU options, have pretrained models, and support commonly used NN architectures like recurrent neural networks (RNNs), convolutional neural networks (CNNs), and deep belief networks (DBNs). So, what more is there in TensorFlow? Let's list them for you:

• It works with all the cool languages. TensorFlow works with Python, C++, Java, R, and Go.
• TensorFlow works on multiple platforms, even mobile and distributed.
• It is supported by all cloud providers--AWS, Google, and Azure.
• Keras, a high-level neural network API, has been integrated with TensorFlow.
• It has better computational graph visualizations because it is native while the equivalent in Torch/Theano is not nearly as cool to look at.
• TensorFlow allows model deployment and ease of use in production.
• TensorFlow has very good community support.
• TensorFlow is more than a software library; it is a suite of software that includes TensorFlow, TensorBoard, and TensorServing.

The Google research blog (https://research.googleblog.com/2016/11/celebrating-tensorflows-first-year.html) lists some of the fascinating projects done by people around the world using TensorFlow:

• Google Translate is using TensorFlow and tensor processing units (TPUs)
• Project Magenta, which can produce melodies using reinforcement learning-based models, employs TensorFlow
• Australian marine biologists are using TensorFlow to find and understand sea-cows, which are on the verge of extinction
• A Japanese farmer used TensorFlow to develop an application to sort cucumbers using physical parameters like size and shape

The list is long, and the possibilities in which one can use TensorFlow are even greater. This book aims to provide you with an understanding of TensorFlow as applied to deep learning models such that you can adapt them to your dataset with ease and develop useful applications. Each chapter contains a set of recipes which deal with the technical issues, the dependencies, the actual code, and its understanding. We have built the recipes one on another such that, by the end of each chapter, you have a fully functional deep learning model.

In this recipe, you will learn how to do a fresh installation of TensorFlow 1.3 on different OSes (Linux, Mac, and Windows). We will find out about the necessary requirements to install TensorFlow. TensorFlow can be installed using native pip, Anaconda, virtualenv, and Docker on Ubuntu and macOS. For Windows OS, one can use native pip or Anaconda.

As Anaconda works on all the three OSes and provides an easy way to not only install but also to maintain different project environments on the same system, we will concentrate on installing TensorFlow using Anaconda in this book. More details about Anaconda and managing its environment can be read from https://conda.io/docs/user-guide/index.html.

The code in this book has been tested on the following platforms:

• Windows 10, Anaconda 3, Python 3.5, TensorFlow GPU, CUDA toolkit 8.0, cuDNN v5.1, NVDIA® GTX 1070
• Windows 10/ Ubuntu 14.04/ Ubuntu 16.04/macOS Sierra, Anaconda3, Python 3.5, TensorFlow (CPU)

The prerequisite for TensorFlow installation is that the system has Python 2.5 or higher installed. The recipes in this book have been designed for Python 3.5 (the Anaconda 3 distribution). To get ready for the installation of TensorFlow, first ensure that you have Anaconda installed. You can download and install Anaconda for Windows/macOS or Linux from https://www.continuum.io/downloads.

After installation, you can verify the installation using the following command in your terminal window:

conda --version

Once Anaconda is installed, we move to the next step, deciding whether to install TensorFlow CPU or GPU. While almost all computer machines support TensorFlow CPU, TensorFlow GPU can be installed only if the machine has an NVDIA® GPU card with CUDA compute capability 3.0 or higher (minimum NVDIA® GTX 650 for desktop PCs).

For TensorFlow GPU, it is imperative that CUDA toolkit 7.0 or greater is installed, proper NVDIA® drivers are installed, and cuDNN v3 or greater is installed. On Windows, additionally, certain DLL files are needed; one can either download the required DLL files or install Visual Studio C++. One more thing to remember is that cuDNN files are installed in a different directory. One needs to ensure that directory is in the system path. One can also alternatively copy the relevant files in CUDA library in the respective folders.

We proceed with the recipe as follows:

1. Create a conda environment using the following at command line (If you are using Windows it will be better to do it as Administrator in the command line):

conda create -n tensorflow python=3.5

2. Activate the conda environment:

# Windows    
activate tensorflow
#Mac OS/ Ubuntu:    
source activate tensorflow
  

3. The command should change the prompt:

# Windows
(tensorflow)C:>
# Mac OS/Ubuntu    
(tensorflow)$

4. Next, depending on the TensorFlow version you want to install inside your conda environment, enter the following command:

## Windows
# CPU Version only
(tensorflow)C:>pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.3.0cr2-cp35-cp35m-win_amd64.whl

# GPU Version  
(tensorflow)C:>pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.3.0cr2-cp35-cp35m-win_amd64.whl

## Mac OS
# CPU only Version
(tensorflow)$ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.3.0cr2-py3-none-any.whl

# GPU version
(tensorflow)$ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.3.0cr2-py3-none-any.whl

## Ubuntu
# CPU only Version
(tensorflow)$ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.3.0cr2-cp35-cp35m-linux_x86_64.whl

# GPU Version
(tensorflow)$ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.3.0cr2-cp35-cp35m-linux_x86_64.whl

5. On the command line, type python.
6. Write the following code:

import tensorflow as tf
message = tf.constant('Welcome to the exciting world of Deep Neural Networks!')
with tf.Session() as sess:
    print(sess.run(message).decode())

7. You will receive the following output:

8. Deactivate the conda environment at the command line using the command deactivate on Windows and source deactivate on MAC/Ubuntu.

TensorFlow is distributed by Google using the wheels standard. It is a ZIP format archive with the .whl extension. Python 3.6, the default Python interpreter in Anaconda 3, does not have wheels installed. At the time of writing the book, wheel support for Python 3.6 exists only for Linux/Ubuntu. Therefore, while creating the TensorFlow environment, we specify Python 3.5. This installs pip, python, and wheel along with a few other packages in the conda environment named tensorflow.

Once the conda environment is created, the environment is activated using the source activate/activate command. In the activated environment, use the pip install command with appropriate TensorFlow-API URL to install the required TensorFlow. Although there exists an Anaconda command to install TensorFlow CPU using conda forge TensorFlow documentation recommends using pip install. After installing TensorFlow in the conda environment, we can deactivate it. Now you are ready to execute your first TensorFlow program.

When the program runs, you may see a few warning (W) messages, some information (I) messages, and lastly the output of your code:

Welcome to the exciting world of Deep Neural Networks!

Congratulations for successfully installing and executing your first TensorFlow code! We will go through the code in more depth in the next recipe.

Additionally, you can also install Jupyter notebook:

1. Install ipython as follows:

conda install -c anaconda ipython
  

2. Install nb_conda_kernels:

conda install -channel=conda-forge nb_conda_kernels
  

3. Launch the Jupyter notebook:

jupyter notebook 

If you already have TensorFlow installed on your system, you can use pip install --upgrade tensorflow to upgrade it.

The first program that you learn to write in any computer language is Hello world. We maintain the convention in this book and start with the Hello world program. The code that we used in the preceding section to validate our TensorFlow installation is as follows:

import tensorflow as tf
message = tf.constant('Welcome to the exciting world of Deep Neural Networks!')
 with tf.Session() as sess:
     print(sess.run(message).decode()) 

Let's go in depth into this simple code.

1. Import tensorflow this imports the TensorFlow library and allows you to use its wonderful features.

 import tensorflow as tf 

2. Since the message we want to print is a constant string, we use tf.constant:

message = tf.constant('Welcome to the exciting world of Deep Neural Networks!')

3. To execute the graph element, we need to define the Session using with and run the session using run:

with tf.Session() as sess:
     print(sess.run(message).decode()) 

4. The output contains a series of warning messages (W), depending on your computer system and OS, claiming that code could run faster if compiled for your specific machine:

The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 
The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 

5. If you are working with TensorFlow GPU, you also get a list of informative messages (I) giving details of the devices used:

Found device 0 with properties:  
name: GeForce GTX 1070 
major: 6 minor: 1 memoryClockRate (GHz) 1.683 
pciBusID 0000:01:00.0 
Total memory: 8.00GiB 
Free memory: 6.66GiB 
DMA: 0  
0:   Y  
Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0) 

6. At the end is the message we asked to print in the session:

Welcome to the exciting world of Deep Neural Networks

The preceding code is divided into three main parts. There is the import block that contains all the libraries our code will use; in the present code, we use only TensorFlow. The import tensorflow as tf statement gives Python access to all TensorFlow's classes, methods, and symbols. The second block contains the graph definition part; here, we build our desired computational graph. In the present case, our graph consists of only one node, the tensor constant message consisting of byte string, "Welcome to the exciting world of Deep Neural Networks". The third component of our code is running the computational graph as Session; we created a Session using the with keyword. Finally , in the Session, we run the graph created above.

Let's now understand the output. The warning messages that are received tell you that TensorFlow code could run at an even higher speed, which can be achieved by installing TensorFlow from source (we will do this in a later recipe in this chapter). The information messages received inform you about the devices used for computation. On their part, both messages are quite harmless, but if you don't like seeing them, adding this two-line code will do the trick:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'   

The code is to ignore all messages till level 2. Level 1 is for information, 2 for warnings, and 3 for error messages.

The program prints the result of running the graph created the graph is run using the sess.run() statement. The result of running the graph is fed to the print function, which is further modified using the decode method. The sess.run evaluates the tensor defined in the message. The print function prints on stdout the result of the evaluation:

b'Welcome to the exciting world of Deep Neural Networks' 

This says that the result is a byte string. To remove string quotes and b (for byte), we use the method decode().

TensorFlow is very unlike other programming languages. We first need to build a blueprint of whatever neural network we want to create. This is accomplished by dividing the program into two separate parts, namely, definition of the computational graph and its execution. At first, this might appear cumbersome to the conventional programmer, but it is this separation of the execution graph from the graph definition that gives TensorFlow its strength, that is, the ability to work on multiple platforms and parallel execution.

Computational graph: A computational graph is a network of nodes and edges. In this section, all the data to be used, in other words, tensor Objects (constants, variables, and placeholders) and all the computations to be performed, namely, Operation Objects (in short referred as ops), are defined. Each node can have zero or more inputs but only one output. Nodes in the network represent Objects (tensors and Operations), and edges represent the Tensors that flow between operations. The computation graph defines the blueprint of the neural network but Tensors in it have no value associated with them yet.

To build a computation graph we define all the constants, variables, and operations that we need to perform. Constants, variables, and placeholders will be dealt with in the next recipe. Mathematical operations will be dealt in detail in the recipe for matrix manipulations. Here, we describe the structure using a simple example of defining and executing a graph to add two vectors.

Execution of the graph: The execution of the graph is performed using Session Object. The Session Object encapsulates the environment in which tensor and Operation Objects are evaluated. This is the place where actual calculations and transfer of information from one layer to another takes place. The values of different tensor Objects are initialized, accessed, and saved in Session Object only. Up to now the tensor Objects were just abstract definitions, here they come to life.

We proceed with the recipe as follows:

1. We consider a simple example of adding two vectors, we have two inputs vectors v_1 and v_2 they are to be fed as input to the Add operation. The graph we want to build is as follows:

2. The corresponding code to define the computation graph is as follows:

v_1 = tf.constant([1,2,3,4]) 
v_2 = tf.constant([2,1,5,3]) 
v_add = tf.add(v_1,v_2)  # You can also write v_1 + v_2 instead 

3. Next, we execute the graph in the session:

with tf.Session() as sess: 
    prin(sess.run(v_add)) 

The above two commands are equivalent to the following code. The advantage of using with block is that one need not close the session explicitly.

sess = tf.Session() 
print(ses.run(tv_add)) 
sess.close() 

4. This results in printing the sum of two vectors:

[3 3 8 7] 

The building of a computational graph is very simple; you go on adding the variables and operations and passing them through (flow the tensors) in the sequence you build your neural network layer by layer. TensorFlow also allows you to use specific devices (CPU/GPU) with different objects of the computation graph using with tf.device(). In our example, the computational graph consists of three nodes, v_1 and v_2 representing the two vectors, and Add is the operation to be performed on them.

Now, to bring this graph to life, we first need to define a session object using tf.Session(); we gave the name sess to our session object. Next, we run it using the run method defined in Session class as follows:

run (fetches, feed_dict=None, options=None, run_metadata) 

This evaluates the tensor in fetches; our example has tensor v_add in fetches. The run method will execute every tensor and every operation in the graph leading to v_add. If instead of v_add, you have v_1 in fetches, the result will be the value of vector v_1:

[1,2,3,4]  

Fetches can be a single tensor/operation object or more, for example, if the fetches is [v_1, v_2, v_add], the output will be the following:

[array([1, 2, 3, 4]), array([2, 1, 5, 3]), array([3, 3, 8, 7])] 

In the same program code, we can have many session objects.

You must be wondering why we have to write so many lines of code for a simple vector addition or to print a small message. Well, you could have very conveniently done this work in a one-liner:

print(tf.Session().run(tf.add(tf.constant([1,2,3,4]),tf.constant([2,1,5,3])))) 

Writing this type of code not only affects the computational graph but can be memory expensive when the same operation ( OP) is performed repeatedly in a for loop. Making a habit of explicitly defining all tensor and operation objects not only makes the code more readable but also helps you visualize the computational graph in a cleaner manner.

If you are working on Jupyter Notebook or Python shell, it is more convenient to use tf.InteractiveSession instead of tf.Session. InteractiveSession makes itself the default session so that you can directly call run the tensor Object using eval() without explicitly calling the session, as described in the following example code:

sess = tf.InteractiveSession() 
 
v_1 = tf.constant([1,2,3,4]) 
v_2 = tf.constant([2,1,5,3]) 
 
v_add = tf.add(v_1,v_2) 
 
print(v_add.eval()) 
 
sess.close() 

TensorFlow in the simplest terms provides a library to define and perform different mathematical operations with tensors. A tensor is basically an n-dimensional matrix. All types of data, that is, scalar, vectors, and matrices are special types of tensors:

TensorFlow supports three types of tensors:

• Constants
• Variables
• Placeholders

Constants: Constants are the tensors whose values cannot be changed.

Variables: We use variable tensors when the values require updating within a session. For example, in the case of neural networks, the weights need to be updated during the training session, which is achieved by declaring weights as variables. The variables need to be explicitly initialized before use. Another important thing to note is that constants are stored in the computation graph definition; they are loaded every time the graph is loaded. In other words, they are memory expensive. Variables, on the other hand, are stored separately; they can exist on the parameter server.

Placeholders: These are used to feed values into a TensorFlow graph. They are used along with feed_dict to feed the data. They are normally used to feed new training examples while training a neural network. We assign a value to a placeholder while running the graph in the session. They allow us to create our operations and build the computation graph without requiring the data. An important point to note is that placeholders do not contain any data and thus there is no need to initialize them as well.

Let's start with constants:

1. We can declare a scalar constant:

t_1 = tf.constant(4)   

2. A constant vector of shape [1,3] can be declared as follows:

t_2 = tf.constant([4, 3, 2]) 

3. To create a tensor with all elements zero, we use tf.zeros(). This statement creates a zero matrix of shape [M,N] with dtype (int32, float32, and so on):

tf.zeros([M,N],tf.dtype)   

Let's take an example:

zero_t = tf.zeros([2,3],tf.int32) 
# Results in an 2×3 array of zeros: [[0 0 0], [0 0 0]] 

4. We can also create tensor constants of the same shape as an existing Numpy array or tensor constant as follows:

tf.zeros_like(t_2) 
# Create a zero matrix of same shape as t_2  
tf.ones_like(t_2) 
# Creates a ones matrix of same shape as t_2

5. We can create a tensor with all elements set to one; here, we create a ones matrix of shape [M,N]:

tf.ones([M,N],tf.dtype) 

Let's take an example:

ones_t = tf.ones([2,3],tf.int32) 
# Results in an 2×3 array of ones:[[1 1 1], [1 1 1]] 

Let's proceed to sequences:

1. We can generate a sequence of evenly spaced vectors, starting from start to stop, within total num values:

tf.linspace(start, stop, num) 

2. The corresponding values differ by (stop-start)/(num-1).
3. Let's take an example:

range_t = tf.linspace(2.0,5.0,5) 
# We get: [ 2.    2.75  3.5   4.25  5.  ]  

4. Generate a sequence of numbers starting from the start (default=0), incremented by delta (default =1), until, but not including, the limit:

tf.range(start,limit,delta) 

Here is an example:

range_t = tf.range(10) 
# Result: [0 1 2 3 4 5 6 7 8 9] 
 

TensorFlow allows random tensors with different distributions to be created:

1. To create random values from a normal distribution of shape [M,N] with the mean (default =0.0) and standard deviation (default=1.0) with seed, we can use the following:

t_random = tf.random_normal([2,3], mean=2.0, stddev=4, seed=12) 

# Result: [[ 0.25347459  5.37990952  1.95276058], [-1.53760314  1.2588985   2.84780669]]

2. To create random values from a truncated normal distribution of shape [M,N] with the mean (default =0.0) and standard deviation (default=1.0) with seed, we can use the following:

t_random = tf.truncated_normal([1,5], stddev=2, seed=12) 
# Result: [[-0.8732627 1.68995488 -0.02361972 -1.76880157 -3.87749004]]

3. To create random values from a given gamma distribution of shape [M,N] in the range [minval (default=0), maxval] with seed, perform as follows:

t_random = tf.random_uniform([2,3], maxval=4, seed=12) 

# Result: [[ 2.54461002  3.69636583  2.70510912], [ 2.00850058  3.84459829  3.54268885]] 

4. To randomly crop a given tensor to a specified size, do as follows:

tf.random_crop(t_random, [2,5],seed=12) 

Here, t_random is an already defined tensor. This will result in a [2,5] tensor randomly cropped from tensor t_random.

Many times we need to present the training sample in random order; we can use tf.random_shuffle() to randomly shuffle a tensor along its first dimension. If t_random is the tensor we want to shuffle, then we use the following:

tf.random_shuffle(t_random) 

5. Randomly generated tensors are affected by the value of the initial seed. To obtain the same random numbers in multiple runs or sessions, the seed should be set to a constant value. When there are large numbers of random tensors in use, we can set the seed for all randomly generated tensors using tf.set_random_seed(); the following command sets the seed for random tensors for all sessions as 54:

tf.set_random_seed(54)  

Let's now turn to the variables:

1. They are created using the variable class. The definition of variables also includes the constant/random values from which they should be initialized. In the following code, we create two different tensor variables, t_a and t_b. Both will be initialized to random uniform distributions of shape [50, 50], minval=0, and maxval=10:

rand_t = tf.random_uniform([50,50], 0, 10, seed=0) 
t_a = tf.Variable(rand_t) 
t_b = tf.Variable(rand_t) 

2. In the following code, we define two variables weights and bias. The weights variable is randomly initialized using normal distribution, with mean zeros and standard deviation of two, the size of weights is 100×100. The bias consists of 100 elements each initialized to zero. Here we have also used the optional argument name to give a name to the variable defined in the computational graph.

weights = tf.Variable(tf.random_normal([100,100],stddev=2)) 
bias = tf.Variable(tf.zeros[100], name = 'biases') 

3. In all the preceding examples, the source of initialization of variables is some constant. We can also specify a variable to be initialized from another variable; the following statement will initialize weight2 from the weights defined earlier:

weight2=tf.Variable(weights.initialized_value(), name='w2') 

4. The definition of variables specify how the variable is to be initialized, but we must explicitly initialize all declared variables. In the definition of the computational graph, we do it by declaring an initialization Operation Object:

intial_op = tf.global_variables_initializer(). 

5. Each variable can also be initialized separately using tf.Variable.initializer during the running graph:

bias = tf.Variable(tf.zeros([100,100]))
 with tf.Session() as sess:
     sess.run(bias.initializer)

6. Saving variables: We can save the variables using the Saver class. To do this, we define a saver Operation Object:

saver = tf.train.Saver()  

7. After constants and variables, we come to the most important element placeholders, they are used to feed data to the graph. We can define a placeholder using the following:

tf.placeholder(dtype, shape=None, name=None) 

8. dtype specifies the data type of the placeholder and must be specified while declaring the placeholder. Here, we define a placeholder for x and calculate y = 2 * x using feed_dict for a random 4×5 matrix:

x = tf.placeholder("float")
y = 2 * x
data = tf.random_uniform([4,5],10)
with tf.Session() as sess:
     x_data = sess.run(data)
     print(sess.run(y, feed_dict = {x:x_data})) 
 

To find out the value, we need to create the session graph and explicitly use the run command with the desired tensor values as fetches:

print(sess.run(t_1))  
# Will print the value of t_1 defined in step 1 

Very often, we will need constant tensor objects with a large size; in this case, to optimize memory, it is better to declare them as variables with a trainable flag set to False:

t_large = tf.Variable(large_array, trainable = False) 

TensorFlow was designed to work impeccably with Numpy, hence all the TensorFlow data types are based on those of Numpy. Using tf.convert_to_tensor(), we can convert the given value to tensor type and use it with TensorFlow functions and operators. This function accepts Numpy arrays, Python Lists, and Python scalars and allows interoperability with tensor Objects.

The following table lists some of the common TensorFlow supported data types (taken from TensorFlow.org):

Note that unlike Python/Numpy sequences, TensorFlow sequences are not iterable. Try the following code:

for i in tf.range(10)  

You will get an error:

#TypeError("'Tensor' object is not iterable.")

Matrix operations, such as performing multiplication, addition, and subtraction, are important operations in the propagation of signals in any neural network. Often in the computation, we require random, zero, ones, or identity matrices.

This recipe will show you how to get different types of matrices and how to perform different matrix manipulation operations on them.

We proceed with the recipe as follows:

1. We start an interactive session so that the results can be evaluated easily:

import tensorflow as tf

#Start an Interactive Session
sess = tf.InteractiveSession()

#Define a 5x5 Identity matrix
I_matrix = tf.eye(5)
print(I_matrix.eval()) 
# This will print a 5x5 Identity matrix

#Define a Variable initialized to a 10x10 identity matrix
X = tf.Variable(tf.eye(10))
X.initializer.run()  # Initialize the Variable
print(X.eval()) 
# Evaluate the Variable and print the result

#Create a random 5x10 matrix
A = tf.Variable(tf.random_normal([5,10]))
A.initializer.run()

#Multiply two matrices
product = tf.matmul(A, X)
print(product.eval())

#create a random matrix of 1s and 0s, size 5x10
b = tf.Variable(tf.random_uniform([5,10], 0, 2, dtype= tf.int32))
b.initializer.run()
print(b.eval())
b_new = tf.cast(b, dtype=tf.float32)
#Cast to float32 data type

# Add the two matrices
t_sum = tf.add(product, b_new)
t_sub = product - b_new
print("A*X _b\n", t_sum.eval())
print("A*X - b\n", t_sub.eval()) 
 

2. Some other useful matrix manipulations, like element-wise multiplication, multiplication with a scalar, elementwise division, elementwise remainder of a division, can be performed as follows:

import tensorflow as tf

# Create two random matrices
a = tf.Variable(tf.random_normal([4,5], stddev=2))
b = tf.Variable(tf.random_normal([4,5], stddev=2))

#Element Wise Multiplication
A = a * b

#Multiplication with a scalar 2
B = tf.scalar_mul(2, A)

# Elementwise division, its result is
C = tf.div(a,b)

#Element Wise remainder of division
D = tf.mod(a,b)

init_op = tf.global_variables_initializer()
with tf.Session() as sess:
     sess.run(init_op)
     writer = tf.summary.FileWriter('graphs', sess.graph)
     a,b,A_R, B_R, C_R, D_R = sess.run([a , b, A, B, C, D])
     print("a\n",a,"\nb\n",b, "a*b\n", A_R, "\n2*a*b\n", B_R, "\na/b\n", C_R, "\na%b\n", D_R)

writer.close() 

All arithmetic operations of matrices like add, sub, div, multiply (elementwise multiplication), mod, and cross require that the two tensor matrices should be of the same data type. In case this is not so they will produce an error. We can use tf.cast() to convert Tensors from one data type to another.

If we are doing division between integer tensors, it is better to use tf.truediv(a,b) as it first casts the integer tensors to floating points and then performs element-wise division.

1. The first step in using TensorBoard is to identify which OPs summaries you would like to have. In the case of DNNs, it is customary to know how the loss term (objective function) varies with time. In the case of Adaptive learning rate, the learning rate itself varies with time. We can get the summary of the term we require with the help of tf.summary.scalar OPs. Suppose, the variable loss defines the error term and we want to know how it varies with time, then we can do this as follows:

loss = tf... 
tf.summary.scalar('loss', loss) 

2. You can also visualize the distribution of gradients, weights, or even output of a particular layer using tf.summary.histogram:

output_tensor  = tf.matmul(input_tensor, weights) + biases 
tf.summary.histogram('output', output_tensor)

3. The summaries will be generated during the session. Instead of executing every summary operation individually, you can define tf.merge_all_summaries OPs in the computation graph to get all summaries in a single run.
4. The generated summary then needs to be written in an event file using tf.summary.Filewriter:

writer = tf.summary.Filewriter('summary_dir', sess.graph) 

5. This writes all the summaries and the graph in the 'summary_dir' directory.
6. Now, to visualize the summaries, you need to invoke TensorBoard from the command line:

tensorboard --logdir=summary_dir 

7. Next, open your browser and type the address http://localhost:6006/ (or the link you received after running the TensorBoard command).
8. You will see something like the following, with many tabs on the top. The Graphs tab will display the graph: