Chapter 2. HTTP and Working with the Web

The Hypertext Transfer Protocol (HTTP) is probably the most widely-used application layer protocol. It was originally developed to allow academics to share HTML documents. Nowadays, it is used as the core protocol of innumerable applications across the Internet, and it is the principle protocol of the World Wide Web.

In this chapter, we will cover the following topics:

• The HTTP protocol structure
• Using Python for talking to services through HTTP
• Downloading files
• HTTP capabilities, such as compression and cookies
• Handling errors
• URLs
• The Python standard library urllib package
• Kenneth Reitz's third-party Requests package

The urllib package is the recommended Python standard library package for HTTP tasks. The standard library also has a low-level module called http. Although this offers access to almost all aspects of the protocol, it has not been designed for everyday use. The urllib package has a simpler interface, and it deals with everything...