The TensorFlow core is the lower-level API on which the higher-level TensorFlow modules are built. In this section, we will go over a quick overview of TensorFlow core and learn about the basic elements of TensorFlow.
Tensors are the basic components in TensorFlow. A tensor is a multidimensional collection of data elements. It is generally identified by shape, type, and rank. Rank refers to the number of dimensions of a tensor, while shape refers to the size of each dimension. You may have seen several examples of tensors before, such as in a zero-dimensional collection (also known as a scalar), a one-dimensional collection (also known as a vector), and a two-dimensional collection (also known as a matrix).
A scalar value is a tensor of rank 0 and shape []. A vector, or a one-dimensional array, is a tensor of rank 1 and shape [number_of_columns
] or [number_of_rows
]. A matrix, or a two-dimensional array, is a tensor of rank 2 and shape [number_of_rows
, number_of_columns
]. A three-dimensional array is a tensor of rank 3. In the same way, an n-dimensional array is a tensor of rank n.
A tensor can store data of one type in all of its dimensions, and the data type of a tensor is the same as the data type of its elements.
The following are the most commonly used data types in TensorFlow:
Note
Use TensorFlow data types for defining tensors instead of native data types from Python or data types from NumPy.
Tensors can be created in the following ways:
- By defining constants, operations, and variables, and passing the values to their constructor
- By defining placeholders and passing the values to
session.run()
- By converting Python objects, such as scalar values, lists, NumPy arrays, and pandas DataFrames, with the
tf.convert_to_tensor()
function
Let's explore different ways of creating Tensors.
The constant valued tensors are created using the tf.constant()
function, and has the following definition:
tf.constant(
value,
dtype=None,
shape=None,
name='const_name',
verify_shape=False
)
Let's create some constants with the following code:
const1=tf.constant(34,name='x1')
const2=tf.constant(59.0,name='y1')
const3=tf.constant(32.0,dtype=tf.float16,name='z1')
Let's take a look at the preceding code in detail:
- The first line of code defines a constant tensor,
const1
, stores a value of 34
, and names it x1
. - The second line of code defines a constant tensor,
const2
, stores a value of 59.0
, and names it y1
. - The third line of code defines the data type as
tf.float16
for const3
. Use the dtype
parameter or place the data type as the second argument to denote the data type.
Let's print the constants const1
, const2
, and const3
:
print('const1 (x): ',const1)
print('const2 (y): ',const2)
print('const3 (z): ',const3)
When we print these constants, we get the following output:
const1 (x): Tensor("x:0", shape=(), dtype=int32)
const2 (y): Tensor("y:0", shape=(), dtype=float32)
const3 (z): Tensor("z:0", shape=(), dtype=float16)
Note
Upon printing the previously defined tensors, we can see that the data types of const1
and const2
are automatically deduced by TensorFlow.
To print the values of these constants, we can execute them in a TensorFlow session with the tfs.run()
command:
print('run([const1,const2,c3]) : ',tfs.run([const1,const2,const3]))
We will see the following output:
run([const1,const2,const3]) : [34, 59.0, 32.0]
The TensorFlow library contains several built-in operations that can be applied on tensors. An operation node can be defined by passing input values and saving the output in another tensor. To understand this better, let's define two operations, op1
and op2
:
op1 = tf.add(const2, const3)
op2 = tf.multiply(const2, const3)
Let's print op1
and op2
:
print('op1 : ', op1)
print('op2 : ', op2)
The output is as follows, and shows that op1
and op2
are defined as tensors:
op1 : Tensor("Add:0", shape=(), dtype=float32)
op2 : Tensor("Mul:0", shape=(), dtype=float32)
To print the output from executing these operations, the op1
and op2
tensors have to be executed in a TensorFlow session:
print('run(op1) : ', tfs.run(op1))
print('run(op2) : ', tfs.run(op2))
The output is as follows:
run(op1) : 91.0
run(op2) : 1888.0
Some of the built-in operations of TensorFlow include arithmetic operations, math functions, and complex number operations.
While constants store the value at the time of defining the tensor, placeholders allow you to create empty tensors so that the values can be provided at runtime. The TensorFlow library provides the tf.placeholder()
function with the following signature to create placeholders:
tf.placeholder(
dtype,
shape=None,
name=None
)
As an example, let's create two placeholders and print them:
p1 = tf.placeholder(tf.float32)
p2 = tf.placeholder(tf.float32)
print('p1 : ', p1)
print('p2 : ', p2)
The following output shows that each placeholder has been created as a tensor:
p1 : Tensor("Placeholder:0", dtype=float32)
p2 : Tensor("Placeholder_1:0", dtype=float32)
Let's define an operation using these placeholders:
mult_op = p1 * p2
In TensorFlow, shorthand symbols can be used for various operations. In the preceding code, p1 * p2
is shorthand for tf.multiply(p1,p2)
:
print('run(mult_op,{p1:13.4, p2:61.7}) : ',tfs.run(mult_op,{p1:13.4, p2:61.7}))
The preceding command runs mult_op
in the TensorFlow session and feeds the values dictionary (the second argument to the run()
operation) with the values for p1
and p2
.
The output is as follows:
run(mult_op,{p1:13.4, p2:61.7}) : 826.77997
We can also specify the values dictionary by using the feed_dict
parameter in the run()
operation:
feed_dict={p1: 15.4, p2: 19.5}
print('run(mult_op,feed_dict = {p1:15.4, p2:19.5}) : ',
tfs.run(mult_op, feed_dict=feed_dict))
The output is as follows:
run(mult_op,feed_dict = {p1:15.4, p2:19.5}) : 300.3
Let's look at one final example, which is of a vector being fed to the same operation:
feed_dict={p1: [2.0, 3.0, 4.0], p2: [3.0, 4.0, 5.0]}
print('run(mult_op,feed_dict={p1:[2.0,3.0,4.0], p2:[3.0,4.0,5.0]}):',
tfs.run(mult_op, feed_dict=feed_dict))
The output is as follows:
run(mult_op,feed_dict={p1:[2.0,3.0,4.0],p2:[3.0,4.0,5.0]}):[ 6. 12. 20.]
The elements of the two input vectors are multiplied in an element-wise fashion.
Tensors from Python objects
Tensors can be created from Python objects such as lists, NumPy arrays, and pandas DataFrames. To create tensors from Python objects, use the tf.convert_to_tensor()
function with the following definition:
tf.convert_to_tensor(
value,
dtype=None,
name=None,
preferred_dtype=None
)
Let's practice doing this by creating some tensors and printing their definitions and values:
tf_t=tf.convert_to_tensor(5.0,dtype=tf.float64)
print('tf_t : ',tf_t)
print('run(tf_t) : ',tfs.run(tf_t))
The output is as follows:
tf_t : Tensor("Const_1:0", shape=(), dtype=float64)
run(tf_t) : 5.0
a1dim = np.array([1,2,3,4,5.99])
print("a1dim Shape : ",a1dim.shape)
tf_t=tf.convert_to_tensor(a1dim,dtype=tf.float64)
print('tf_t : ',tf_t)
print('tf_t[0] : ',tf_t[0])
print('tf_t[0] : ',tf_t[2])
print('run(tf_t) : \n',tfs.run(tf_t))
The output is as follows:
a1dim Shape : (5,)
tf_t : Tensor("Const_2:0", shape=(5,), dtype=float64)
tf_t[0] : Tensor("strided_slice:0", shape=(), dtype=float64)
tf_t[0] : Tensor("strided_slice_1:0", shape=(), dtype=float64)
run(tf_t) :
[ 1. 2. 3. 4. 5.99]
a2dim = np.array([(1,2,3,4,5.99),
(2,3,4,5,6.99),
(3,4,5,6,7.99)
])
print("a2dim Shape : ",a2dim.shape)
tf_t=tf.convert_to_tensor(a2dim,dtype=tf.float64)
print('tf_t : ',tf_t)
print('tf_t[0][0] : ',tf_t[0][0])
print('tf_t[1][2] : ',tf_t[1][2])
print('run(tf_t) : \n',tfs.run(tf_t))
The output is as follows:
a2dim Shape : (3, 5)
tf_t : Tensor("Const_3:0", shape=(3, 5), dtype=float64)
tf_t[0][0] : Tensor("strided_slice_3:0", shape=(), dtype=float64)
tf_t[1][2] : Tensor("strided_slice_5:0", shape=(), dtype=float64)
run(tf_t) :
[[ 1. 2. 3. 4. 5.99]
[ 2. 3. 4. 5. 6.99]
[ 3. 4. 5. 6. 7.99]]
a3dim = np.array([[[1,2],[3,4]],
[[5,6],[7,8]]
])
print("a3dim Shape : ",a3dim.shape)
tf_t=tf.convert_to_tensor(a3dim,dtype=tf.float64)
print('tf_t : ',tf_t)
print('tf_t[0][0][0] : ',tf_t[0][0][0])
print('tf_t[1][1][1] : ',tf_t[1][1][1])
print('run(tf_t) : \n',tfs.run(tf_t))
The output is as follows:
a3dim Shape : (2, 2, 2)
tf_t : Tensor("Const_4:0", shape=(2, 2, 2), dtype=float64)
tf_t[0][0][0] : Tensor("strided_slice_8:0", shape=(), dtype=float64)
tf_t[1][1][1] : Tensor("strided_slice_11:0", shape=(), dtype=float64)
run(tf_t) :
[[[ 1. 2.][ 3. 4.]]
[[ 5. 6.][ 7. 8.]]]
In the previous sections, we learned how to define tensor objects of different types, such as constants, operations, and placeholders. The values of parameters need to be held in an updatable memory location while building and training models with TensorFlow. Such updatable memory locations for tensors are known as variables in TensorFlow.
To summarize this, TensorFlow variables are tensor objects in that their values can be modified during the execution of the program.
Although tf.Variable
seems to be similar to tf.placeholder
, they have certain differences. These are listed in the following table:
In TensorFlow, a variable can be created with the API function tf.Variable()
. Let's look at an example of using placeholders and variables and create the following model in TensorFlow:
- Define the model parameters
w
and b
as variables with the initial values [.3]
and [-0.3]
:
w = tf.Variable([.3], tf.float32)
b = tf.Variable([-.3], tf.float32)
- Define the input placeholder
x
and the output operation node y
:
x = tf.placeholder(tf.float32)
y = w * x + b
- Print the variables and placeholders
w
, v
, x
, and y
:
print("w:",w)
print("x:",x)
print("b:",b)
print("y:",y)
The output depicts the type of nodes as Variable
, Placeholder
, or operation node, as follows:
w: <tf.Variable 'Variable:0' shape=(1,) dtype=float32_ref>
x: Tensor("Placeholder_2:0", dtype=float32)
b: <tf.Variable 'Variable_1:0' shape=(1,) dtype=float32_ref>
y: Tensor("add:0", dtype=float32)
The preceding output indicates that x
is a Placeholder
tensor, y
is an operation tensor, and that w
and b
are variables with a shape of (1,)
and a data type of float32
.
The variables in a TensorFlow session have to be initialized before they can be used. We can either initialize a single variable by running its initializer operation or we can initialize all or a group of variables.
For example, to initialize the w
variable, we can use the following code:
tfs.run(w.initializer)
TensorFlow provides a convenient function that can initialize all of the variables:
tfs.run(tf.global_variables_initializer())
Note
TensorFlow also provides the tf.variables_initializer()
function so that you can initialize a specific set of variables.
The global convenience function for initializing these variables can be executed in an alternative way. Instead of executing inside the run()
function of a session object, the run function of the object returned by the initializer function itself can be executed:
tf.global_variables_initializer().run()
After the variables have been initialized, execute the model to get the output for the input values of x = [1,2,3,4]
:
print('run(y,{x:[1,2,3,4]}) : ',tfs.run(y,{x:[1,2,3,4]}))
The output is as follows:
run(y,{x:[1,2,3,4]}) : [ 0. 0.30000001 0.60000002 0.90000004]
Tensors generated from library functions
TensorFlow provides various functions to generate tensors with pre-populated values. The generated values from these functions can be stored in a constant or variable tensor. Such generated values can also be provided to the tensor constructor at the time of initialization.
As an example, let's generate a 1-D tensor that's been pre-populated with 100
zeros:
a=tf.zeros((100,))
print(tfs.run(a))
Some of the TensorFlow library functions that populate these tensors with different values at the time of their definition are listed as follows:
- Populating all of the elements of a tensor with similar values:
tf.ones_like()
, tf.ones()
, tf.fill()
, tf.zeros()
, andtf.zeros_like()
- Populating tensors with sequences:
tf.range()
,and tf.lin_space()
- Populating tensors with a probability distribution:
tf.random_uniform()
, tf.random_normal()
, tf.random_gamma()
,and tf.truncated_normal()
Obtaining variables with the tf.get_variable()
If a variable is defined with a name that has already been used for another variable, then an exception is thrown by TensorFlow. Thetf.get_variable()
function makes it convenient and safe to create a variable in place of using thetf.Variable()
function. The tf.get_variable()
function returns a variable that has been defined with a given name. If the variable with the given name does not exist, then it will create the variable with the specified initializer and shape.
Consider the following example:
w = tf.get_variable(name='w',shape=[1],dtype=tf.float32,initializer=[.3])
b = tf.get_variable(name='b',shape=[1],dtype=tf.float32,initializer=[-.3])
The initializer can either be a list of values or another tensor. An initializer can also be one of the built-in initializers. Some of these are as follows:
tf.ones_initializer
tf.constant_initializer
tf.zeros_initializer
tf.truncated_normal_initializer
tf.random_normal_initializer
tf.random_uniform_initializer
tf.uniform_unit_scaling_initializer
tf.orthogonal_initializer
The tf.get_variable()
function only returns the global variables when the code is run across multiple machines in distributed TensorFlow. The local variables can be retrieved by using the tf.get_local_variable()
function.
Note
Sharing or reusing variables: Getting variables that have already been defined promotes reuse. However, an exception will be thrown if the reuse flags are not set by using tf.variable_scope.reuse_variable()
or tf.variable.scope(reuse=True)
.
Now that we have learned how to define tensors, constants, operations, placeholders, and variables, let's learn about the next level of abstraction in TensorFlow that combines these basic elements to form a basic unit of computation: the computation graph.