Visual prompt models

Prompt-based models have been an attractive part of artificial intelligence in many aspects. These kinds of models can take guidance in the form of a pattern and create the respective output by understanding it. The prompt can be in many forms or data formats. Textual prompt-based models or visual prompt-based models are also available. A textual prompt is a free text that indicates what the model should do or provide as output. Similarly, a visual prompt is a visual guidance that helps the model understand the task or the instruction itself.

Models such as CLIP are capable of understanding images and text at the same time and mapping them to a single vector space. In this vector space, text with similar semantic meaning to images (that visually present the same described objects or scenes in the text) are closer together. A simple approach to use more capabilities of the models is to ground them by using some external data. For example, imagine that you are...