Summary

In this chapter, we looked under the hood at the components that make a solid web scraping system. We used colly to scrape HTML pages that did not require JavaScript. We used chrome-protocol to drive web browsers to scrape sites that do require JavaScript. Finally, we examined dataflowkit and saw how its architecture opens the door for building distributed web crawlers. There is more to learn and do when it comes to building distributed systems in Go, but this is where the scope of this book ends. I hope you check out some other publications on building applications in Go and continue to hone your skills!