Multimodality

An application may be considered multimodal if it uses several input and/or output modalities. In this sense, all the apps presented in this book are multimodal, as they use voice as well as a GUI either in the output or input.

However, in the previous examples the modalities were not synchronized or used to provide alternative ways for handling the same bits of information. For example, in the MusicBrain app in Chapter 5, Form-filling Dialogs the input was oral and the output visual, and in the SimpleParrot app described in this chapter, the input and output are oral (requiring speech recognition and synthesis) and visual (selecting a language and the push to speak button in the input and toasts for output), but they correspond to different elements in the interface.

In this section, we will describe how to develop multimodal applications in which the modalities can be seamlessly combined to input or output data, so the user can select the most convenient modality to use at...