Breadth-first versus depth-first crawling

Now that you have the ability to navigate to different pages, as well as the ability to avoid getting stuck in a loop, you have one more important choice to make when crawling a website. In general, there are two main approaches to covering all pages by following links: breadth-first, and depth-first. Imagine that you are scraping a single web page that contains 20 links. Naturally, you would follow the first link on the page. On the second page, there are ten more links. Herein lies your decision: follow the first link on the second page, or go back to the second link on the first page.

Depth-first

If you choose to follow the first link on the second page, this would be considered...