Understanding saliency

We saw earlier that the intermediate layers of our ConvNet seemed to encode some pretty clear detectors of face edges. It is harder to distinguish, however, whether our network understands what a smile actually is. You will notice in our smiling faces dataset that all pictures have been taken on the same background at the same approximate angle from the camera. Moreover, you will notice that the individuals in our dataset tend to smile as they lift their head up high and clear, yet mostly tilt their head downward while frowning. That's a lot of opportunity for our network to overfit on some irrelevant pattern. Hence, how do we actually know that our network understands that a smile has more to do with the movement of a person’s lips than it has to do with the angle at which someone’s face is tilted? As we saw in our neural network fails...