Basic neural networks

Our logistic regression model worked well enough, but was fundamentally linear in nature. Doubling the intensity of a pixel doubled its contribution to the score, but we might only really care if a pixel was above a certain threshold or put more weight on changes to small values. Linearity may not capture all the nuances of the problem. One way to handle this issue is to transform our input with a nonlinear function. Let's look at a simple example in TensorFlow.

First, be sure to load the required modules (tensorflow, numpy, and math) and start an interactive session:

import tensorflow as tf
import numpy as np
import math

sess = tf.InteractiveSession()

In the following example, we create three five-long vectors of normal random numbers, truncated to keep them from being too extreme, with different centers:

x1 = tf.Variable(tf.truncated_normal([5],
                 mean=3, stddev=1./math.sqrt(5)))
x2 = tf.Variable(tf.truncated_normal([5],
                 mean=-1...