Things to try
In this chapter, we only started playing with MiniWoB++ by looking at some of the easiest environments from the full set of over 100 problems, so there is plenty of uncharted territory ahead. If you want to practice, there are several items you can experiment with:
-
Testing the robustness of demonstrations to noisy clicks.
-
The action space for the clicking approach could be improved by predicting the x and y coordinates of the place to click.
-
DOM data could be used instead of (or in addition to) screen pixels. Then, the prediction will be the element of the tree to be clicked.
-
Try other problems. There is a wide variety of them, requiring keyboard events to be generated, the sequence of actions planned, etc.
-
Very recently, the LaVague project was published (https://github.com/lavague-ai/LaVague), which uses LLMs for web automation...
