The need for specialized architectures

Inductive bias, or learning bias, refers to a set of assumptions a learning algorithm makes to generalize the function it learns on training data to unseen data. Inductive bias is not inherently a bad thing and is different from “bias” in the context of bias and variance in learning theory. We use and design inductive bias either through model architectures or through feature engineering. For instance, a Convolutional Neural Network (CNN) works better on images than a standard Feed Forward Network (FFN) on pure pixel input because the CNN has the locality and spatial bias that FFNs do not have. Although the FFN is theoretically a universal approximator, we can learn better models with the inductive bias the CNN has.

Deep learning is thought to be a completely data-driven approach where the feature engineering and final task are learned end to end, thus avoiding the inductive bias that the modelers bake in while designing the...