Before you get started building a web scraper, you will need the proper tools. Setting up a development environment for writing Go code is relatively simple. There are not a lot of external tools that you will need to install, and there is support for all major computing platforms. For all of the tools listed in this chapter, you will find individual instructions for Windows, Mac, and Linux systems. Also, since all of the tools we will use are open source, you will be able to access the source code and build it for your specific needs if necessary.
How to set up a Go development environment
Go language and tools
First and foremost, you'll need to install the Go programming language and tools on your machine. The installation process varies for different operating systems so please follow the instructions at https://golang.org/doc/install. On the installation page, you will find instructions for downloading Go for your platform, as well as the minimum operating system requirements.
It would be a good idea for you to spend some extra time browsing the Go programming language website to learn more about the language, read tutorials, and find the standard library documentation.
This is a screenshot from the installation page from the Go website, containing all of the instructions necessary for installing Go on your computer:
You can also build the language from source if you are so inclined. By the end of the installation, you should have the all of the Go libraries, the Go command line, and a simple hello world project built to ensure that everything was installed properly.
It is very important to follow the instructions all the way through testing your installation. Go can be a little tricky sometimes with respect to $GOPATH. Once you set up your $GOPATH, you must ensure that following is done:
- You have the required src, bin, and pkg directories
- All source code is contained within the src directory
- The folder structure inside your src directory mimics what you want your package names to be
By completing the testing section, you will save yourself a lot of frustration in the future.
Since the release of version 1.11, the Go team has announced support for Go modules, which allows you to develop outside of the $GOPATH. Because this feature is still considered experimental, this book will continue with the classic method for Go development.
Git
You will also need to install the Git version control software. This will be used to download third-party libraries onto your machine. The go get command relies on Git being installed on your system to download libraries and install them directly into your $GOPATH. You may also feel free to use Git to download the examples for each chapter. All of the examples in this book will be using open source libraries that are available on GitHub. You can install Git for your system by following the instructions at https://git-scm.com/download.
The Git command-line tool is a vast set of commands used for versioning, storing, and retrieving source code. These commands are the basis that power the GitHub website. It is highly recommended that you learn how to use the tool to interact with the GitHub site, rather than going through the UI.
The following is a screenshot of the Git download page, containing the links for your respective operating system:
Editor
The second tool you will need is a good text editor or Integrated Development Environment (IDE). If you are not familiar with IDEs, they are basically text editors that are custom-built for writing applications for specific programming languages. One well-known IDE for Go is GoLand by JetBrains. This comes with built-in support for syntax highlighting, run and debug modes, built-in version control, and package management.
GoLand is available as a 30-day trial, after which you must buy a license to continue using it.
The following is a screenshot of the GoLand IDE displaying the standard Hello World program:
If you prefer to use a text editor, there are many available and they often have plugins for Go that make developing easier. Two of the best text editors available today are Visual Studio Code by Microsoft and Atom by GitHub. Both of these are general purpose editors that also have plugins for syntax highlighting, building, and running Go code. This way you can add what you need without too much overhead.
This screenshot is the same Hello World program, displayed in Visual Studio Code:
Finally, the Atom Version of the Hello World program looks like the following screenshot:
Both the Visual Studio Code and Atom are excellent choices for building Go applications due to the level of community support for the plugins, which I highly recommend installing. Alternatively, you can write Go programs in any text editor and run the code using your terminal or Command Prompt with the standard Go commands.
You will need a solid internet connection. A proper internet connection will eliminate errors connecting to different websites. If you are building a web scraper that sits behind a network firewall, or if you have a weak network connection, you may encounter difficulties accessing some of the sites used as examples in this book.
