Creating general representations through unsupervised deep learning
The representations that are learned through unsupervised deep learning can be directly used as-is in downstream supervised tasks by predictive supervised models or consumed directly by end users. There are a handful of generally impactful unsupervised methods that utilize neural networks that are meant to be used primarily as feature extractors. Let’s take a look at a couple of unsupervised feature extractors:
- Unsupervised pre-trained word tokenizers: These are used heavily by variants of the transformers architecture and were introduced in Chapter 8, Exploring Supervised Deep Learning, in the Representing text data for supervised deep learning section.
- Unsupervised pre-trained word embeddings: These methods leverage unsupervised learning and attempt to perform language modeling, similar to masked language modeling in transformers. However, word embeddings-based methods have been overtaken by transformer...
