The era of multimodal interactions
Multimodal interaction in large language models (LLMs) refers to the ability of these models to understand “input prompts” and generate content as “output completions” in multiple modalities, typically combining text with other forms of data, such as images, audio, or even video. It’s the capacity to process and generate information using different sensory channels.
We already know that LLMs such as GPT-4 perform well with text input and outputs. Renowned LLMs such as GPT-4 have already demonstrated exceptional proficiency with textual inputs and outputs. The recent surge in advanced image generation models, including DALL-E 3 and Midjourney, further illustrates this progress. The next significant leap in generative AI applications is anticipated to incorporate groundbreaking capabilities, extending to text-to-video and image-to-video conversions, thus broadening the horizons of AI’s creative and functional...
