In this section, we'll learn about the HTTP protocol, how it works, and the security aspects of it and which methods are supported when performing a request.

This will provide you with the basic knowledge of HTTP, which is important to understand how to build tools and test for security issues in web applications.

HTTP was designed to enable communication between clients and servers.

HTTP is a TCP/IP-based communication protocol operating in the application layer. Normally, we use a web browser to interact with web applications but in this training, we will leave the browser behind and use Python to talk with web applications. This protocol is media independent.

This...

In this section, we'll take a look at the structure of a URL, the request and response headers, and an example of GET requests using Telnet to understand how it works at a low level.

I bet you have seen thousands of URLs by now. It's now time to stop and think about the URL structure. Let's see what each part means:

The first part is the protocol in web applications. The two protocols used are HTTP and HTTPS. When using HTTP, the port that will be used is 80, and when using HTTPS, the port will be 443.

The next part is the host we want to contact. Next, we can see the resource or the file location in that server. In this example, the directory is content and the resource is section. Then, we have the question mark symbol that indicates what's to come is the query string. These are the parameters that will be passed to the section...

In this section, we'll start to write Python code to perform HTTP requests using the requests library.

Requests is an Apache 2 licensed HTTP library written in Python. It was created to reduce the complexity and work needed when using urllib2 and other HTTP libraries available at the moment.

This is an example of the code needed to perform a request to api.github.com using authentication when using the urllib2 library:

import urllib2

gh_url = 'https://api.github.com'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')

auth_manager...

In this section, we will learn about the different HTTP response status codes and different classes of HTTP response codes.

Then, we'll write examples to see successful responses or errors, and finally, we'll see a redirection example.

The HTTP protocol defines five classes of response codes to indicate the status of a request:

• 1XX-Informational: The 100 range codes are used for informational purposes. It is only present in HTTP/1.1.
• 2XX-Success: The 200 range of codes are used to indicate that the action requested by the client was received, understood, accepted, and processed. The most common is 200 OK.
• 3XX-Redirection: The 300 range indicates the client that must take additional...

In this chapter, we had a brief introduction to HTTP, and we saw a basic GET request example. We also saw the different HTTP methods available that we can use to interact with web applications.

We also learned about HTTP requests. We learned how to interact with a web application using Python and the requests library. We further learned about the HTTP request anatomy and the different HTTP methods and response code.

In Chapter 3, Web Crawling with Scrapy - Mapping the Application, we'll learn how to write a Web Crawler, use Spider using Python, and how to use the Scrappy library.