Visual dialogue model
The visual dialogue model (VDM) enables chat based on images. VDM applies technologies from computer vision, Natural Language Processing (NLP) and chatbots. It has found major applications such as explaining to blind people about images, to doctors about medical scans, virtual companions and so on. Next, we will see the algorithm to solve this challenge.
Algorithm for VDM
The algorithm discussed here is proposed by Lu et al (https://research.fb.com/wp-content/uploads/2017/11/camera_ready_nips2017.pdf). Lu et al proposed a GAN-based VDM. The generator generates answers and the discriminator ranks those answers. The following is a schematic representation of the process:
Architecture of the VDMs based on GAN techniques [Reproduced from Lu et al.]
The history of chat, the current question and image are fed as an input to the generator. Next, we will see how the generator works.
Generator
The generator has an encoder and decoder. The encoder takes an image, question, and history...
