The MiniWoB benchmark

In December 2016, OpenAI published a dataset called MiniWoB that contains 80 browser-based tasks. These tasks are observed at the pixel level (strictly speaking, besides pixels, a text description of tasks is given to the agent) and are supposed to be communicated with the mouse and keyboard actions using the Virtual Network Computing (VNC) client. VNC is a standard remote desktop protocol by which a VNC server allows clients to connect to and work with a server’s GUI applications using the mouse and keyboard via the network.

The 80 tasks vary a lot in terms of complexity and the actions required from the agent. Some tasks are very simple, even for RL, like “click on the dialog’s close button,” or “push the single button,” but some require multiple steps, for example, “open collapsed groups and click on the link with some text,” or “select a specific date using the date picker tool”...