Web scraping at, its core, is collecting publicly available information from the internet for a specific purpose. It has taken form under many different names, such as following:

• Spiders
• Crawlers
• Bots

Although the name may carry a negative connotation, the practice of web scraping has been around since the beginning of the internet and has grown into various technologies and techniques. In fact, some companies have built their entire business on web scraping!

There are many different use cases where you might need to build a web scraper. All cases center around the fact that information on the internet is often disparate, but can be very valuable when collected into one single package. Often, in these cases, the person collecting the information does not have a working or business relationship with the producers of the data, meaning they cannot request the information to be packaged and delivered to them. Because of the lack of this relationship, the one who needs the data has to rely on their own means to gather the information.

One well-known use case for web scraping is indexing websites for the purpose of building a search engine. In this case, a web scraper would visit different websites and follow references to other websites in order to discover all of the content available on the internet. By collecting some of the content from the pages, you could respond to search queries by matching the terms to the contents of the pages you have collected. You could also suggest similar pages if you track how pages are linked together, and rank the most important pages by the number of connections they have to other sites.

Googlebot is the most famous example of a web scraper used to build a search engine. It is the first step in building the search engine as it downloads, indexes, and ranks each page on a website. It will also follow links to other websites, which is how it is able to index a substantial portion of the internet. According to Googlebot's documentation, the scraper attempts to reach each web page every few seconds, which requires them to reach estimates of well into billions of pages per day!

If your goal is to build a search engine, albeit on a much smaller scale, you will find enough tools in this book to collect the information you need. This book will not, however, cover indexing and ranking pages to provide relevant search results.

Another known use case is to find specific products or services sold through various websites and track their prices. You would be able to see who sells the item, who has the lowest price, or when it is most likely to be in stock. You might even be interested in similar products from different sources. Having a web scraper periodically visit websites to monitor these products and services would be easily solve this problem. This is very similar to tracking prices for flights, hotels, and rental cars as well.

Sites like camelcamelcamel (https://camelcamelcamel.com/) build their business model around such a case. According to their blog post explaining how their system works, they actively collect pricing information from multiple retailers every half hour to every few hours, covering millions of products. This allows users to view pricing differences across multiple platforms, as well as get notified if the price of an item drops.

This type of web scraper requires very careful parsing of the web pages to extract only the content that is relevant. In later chapters, you will learn how to extract information from HTML pages in order to collect this information.

Data scientists often need hundreds of thousands of data points in order to build, train, and test machine learning models. In some cases, this data is already pre-packaged and ready for consumption. Most of the time, the scientist would need to venture out on their own and build a custom dataset. This is often done by building a web scraper to collect raw data from various sources of interest, and refining it so it can be processed later on. These web scrapers also need to periodically collect fresh data to update their predictive models with the most relevant information.

A common use case that data scientists run into is determining how people feel about a specific subject, known as sentiment analysis. Through this process, a company could look for discussions surrounding one of their products, or their overall presence, and gather a general consensus. In order to do this, the model must be trained on what a positive comment and a negative comment are, which could take thousands of individual comments in order to make a well-balanced training set. Building a web scraper to collect comments from relevant forums, reviews, and social media sites would be helpful in constructing such a dataset.

These are just a few examples of web scrapers that drive large business such as Google, Mozenda, and Cheapflights.com. There are also companies that will scrape the web for whatever available data you need, for a fee. In order to run scrapers at such a large scale, you would need to use a language that is fast, scalable, and easy to maintain.

Go is a programming language created by Google employees in 2007. At the time of its creation, the goal was to build a language that was fast, safe, and simple. Go first officially released its 1.0 version in 2012 and is one of the fastest growing programming languages today. According to the Stack Overflow 2018 Developer Survey, Go is ranked in the top five of the most-loved languages and the top three in the most-wanted languages.

Go powers many large-scale web infrastructure platforms and tools such as Docker, Kubernetes, and Terraform. These platforms enable companies to build production-scale products supporting Fortune 500 companies. This is mostly the result of the design of the Go language, making it straightforward and clear to work with. Many other companies using Go for their development often tout the performance improvements over other languages.

The architecture of the Go programming language, as well as its standard libraries, make it a great choice for building web scrapers that are fast, scalable, and maintainable. Go is a statically typed, garbage-collected language with a syntax closer to C/C++. The syntax of the language will feel very familiar to developers coming from object-oriented programming languages. Go also has a few functional programming elements as well, such as higher-order functions. With all that being said, there are three main reasons why Go is a great fit for web scraping:

Speed is one of the primary objectives of the Go programming language. Many benchmarks put the speed of Go on par with that of C++, Java, and Rust, and miles ahead of languages such as Python and Ruby. Benchmark tests should always be considered with a bit of skepticism, but Go consistently stands out as a language with extremely high-performance numbers. This speed is typically coupled with a low resource footprint, as the runtime is very lightweight and does not use much RAM. One of the hidden benefits of this is being able to run Go programs on smaller machines, or to run multiple instances on the same machine, without significant overhead. This reduces the cost of operating a web scraper at larger scales.

This speed is inherently important in building web scrapers, and becomes more noticeable at larger scales. Take, for example, a web scraper that requires two minutes to scrape a page; you could theoretically process 720 pages in a day. If you were able to reduce that time to one minute per page, you would double the amount of pages per day to 1,440! Better yet, this would be done at the same cost. The speed and efficiency of Go allow you to do more with less.

One of the contributing factors to its speed is the fact that Go is statically typed. This makes the language ideal for building systems at a large scale and being confident in how your program will run in production. Also, since Go programs are built with a compiler instead of being run with an interpreter, it allows you to catch more bugs at compile time and greatly reduces the dreaded runtime errors.

This safety net is also extended to the Go garbage collector. Garbage collection means that you do not need to manually allocate and deallocate memory. This helps prevent memory leaks that might occur from mishandling objects in your code. Some may argue that garbage collection impedes the performance of your application, however, the Go garbage collector adds very little overhead in terms of interfering with your code execution. Many source report that the pauses caused by Go's garbage collector are less than one millisecond. In most cases, it's a very small price to pay to avoid chasing down memory leaks in the future. This certainly holds true for web scrapers.

As web scrapers grow in both size and complexity, it can be difficult to track all of the errors that may occur during processing. Thinking on the scale of processing thousands of web pages per day, one small bug could cause significantly affect the collection of data. At the end of the day, data missed is money lost, so preventing as many known errors as possible before the system is running is critical to your system.

Beyond the architecture of the Go programming language itself, the standard library offers all the right packages you need to make web scraping easy. Go offers a built-in HTTP client in the net/http package that is fully-featured out of the box, but also allows for a lot of customization. Making an HTTP request is as simple, as follows:

http.Get("http://example.com")

Also a part of the net/http package are utilities to structure HTTP requests, HTTP responses, and all of the HTTP status codes, which we will dive into later in this book. You will rarely need any third-party packages to handle communication with web servers. The Go standard library also has tools to help analyze HTTP requests, quickly consume HTTP response bodies, and debug the requests and responses in your web scraper. The HTTP client in the net/http package is also very configurable, letting you tune special parameters and methods to suit your specific needs. This typically will not need to be done, but the option exists if you encounter such a situation.

This simplicity will help eliminate some of the guesswork of writing code. You will not need to determine the best way to make an HTTP request; Go has already worked it out and provided you with the best tools you need to get the job done. Even when you need more than just the standard library, the Go community has built tools that follow the same culture of simplicity. This certainly makes integrating third-party libraries an easy task.

Before you get started building a web scraper, you will need the proper tools. Setting up a development environment for writing Go code is relatively simple. There are not a lot of external tools that you will need to install, and there is support for all major computing platforms. For all of the tools listed in this chapter, you will find individual instructions for Windows, Mac, and Linux systems. Also, since all of the tools we will use are open source, you will be able to access the source code and build it for your specific needs if necessary.

First and foremost, you'll need to install the Go programming language and tools on your machine. The installation process varies for different operating systems so please follow the instructions at https://golang.org/doc/install. On the installation page, you will find instructions for downloading Go for your platform, as well as the minimum operating system requirements.

This is a screenshot from the installation page from the Go website, containing all of the instructions necessary for installing Go on your computer:

You can also build the language from source if you are so inclined. By the end of the installation, you should have the all of the Go libraries, the Go command line, and a simple hello world project built to ensure that everything was installed properly.

It is very important to follow the instructions all the way through testing your installation. Go can be a little tricky sometimes with respect to $GOPATH. Once you set up your $GOPATH, you must ensure that following is done:

• You have the required src, bin, and pkg directories
• All source code is contained within the src directory
• The folder structure inside your src directory mimics what you want your package names to be

By completing the testing section, you will save yourself a lot of frustration in the future.

You will also need to install the Git version control software. This will be used to download third-party libraries onto your machine. The go get command relies on Git being installed on your system to download libraries and install them directly into your $GOPATH. You may also feel free to use Git to download the examples for each chapter. All of the examples in this book will be using open source libraries that are available on GitHub. You can install Git for your system by following the instructions at https://git-scm.com/download.

The following is a screenshot of the Git download page, containing the links for your respective operating system:

The second tool you will need is a good text editor or Integrated Development Environment (IDE). If you are not familiar with IDEs, they are basically text editors that are custom-built for writing applications for specific programming languages. One well-known IDE for Go is GoLand by JetBrains. This comes with built-in support for syntax highlighting, run and debug modes, built-in version control, and package management.

The following is a screenshot of the GoLand IDE displaying the standard Hello World program:

If you prefer to use a text editor, there are many available and they often have plugins for Go that make developing easier. Two of the best text editors available today are Visual Studio Code by Microsoft and Atom by GitHub. Both of these are general purpose editors that also have plugins for syntax highlighting, building, and running Go code. This way you can add what you need without too much overhead.

This screenshot is the same Hello World program, displayed in Visual Studio Code:

Finally, the Atom Version of the Hello World program looks like the following screenshot:

Both the Visual Studio Code and Atom are excellent choices for building Go applications due to the level of community support for the plugins, which I highly recommend installing. Alternatively, you can write Go programs in any text editor and run the code using your terminal or Command Prompt with the standard Go commands.

In this chapter, you learned a few of the use cases for building a web scraper and examples of businesses related to them. You also learned a few of the strengths of the Go programming language and created a development environment suitable for building your web scraper. These steps should help you get started on that path.

In Chapter 2, The Request/Response Cycle, we look at how to communicate with web servers in Go. We will learn the basics of how your web scraper communicates with web servers.