If we have access to a rich enough training set for our application, does it even make sense to use a pretrained model? This question is legitimate if the similarity between the original and target tasks is too low. Pretraining a model, or even downloading pretrained weights, can be costly. However, researchers demonstrated through various experiments that, in most cases, it is better to initialize a network with pretrained weights (even from a dissimilar use case) than with random ones.
Transfer learning makes sense when the tasks or their domains share at least some basic similarities. For instance, images and audio files can both be stored as two-dimensional tensors, and CNNs (such as ResNet ones) are commonly applied to both. However, the models are relying on completely different features for visual and audio recognition. It would typically not benefit a model for visual recognition to receive the weights from a network trained for...
