Why exploration is important

In this book, lots of environments and methods have been discussed, and in almost every chapter, exploration was mentioned. Very likely, you’ve already got ideas about why it’s important to explore the environment effectively, so I’m just going to discuss the main reasons.

Before that, it might be useful to agree on the term “effective exploration.” In theoretical RL, a strict definition of this exists, but the high-level idea is simple and intuitive. Exploration is effective when we don’t waste time in states of the environment that have already been seen by and are familiar to the agent. Rather than taking the same actions again and again, the agent needs to look for a new experience. As we’ve already discussed, exploration has to be balanced by exploitation, which is the opposite and means using our knowledge to get the best reward in the most efficient way. Let’s now quickly discuss why...