Unique challenges of generative models
Given the powerful applications that generative models are applied to, what are the major challenges in implementing them? As described, most of these models utilize complex data, requiring us to fit large models with sufficiently diverse inputs to capture all the nuances of their features and distribution. That complexity arises from sources including:
- Range of variation: The number of potential images generated from a set of three color channel pixels is immense, as is the vocabulary of many languages
- Heterogeneity of sources: Language models, in particular, are often developed using a mixture of data from several websites
- Size: Once data becomes large, it becomes more difficult to catch duplications, factual errors (such as mistranslations), noise (such as scrambled images), and systematic biases
- Rate of change: Many developers of LLMs struggle to keep model information current with the state of the world and thus provide relevant answers to...
