Before we explain how to train the network, let's have a look at the full architecture of Faster R-CNN:
Figure 5.14: Full architecture of Faster R-CNN. Note that it can work with any input size
Because of its unique architecture, Faster R-CNN cannot be trained like a regular CNN. If each of the two parts of the network were trained separately, the feature extractors of each part would not share the same weights. In the next section, we will explain the training of each section and how to make the two sections share the convolutional weights.
