Using urllib2 to download data

Before we get on to processing the data we extract from the online sources, we will first demonstrate use of the in-built urllib2 Python module for downloading data from the internet.

This will be used in all the examples later on in the chapter for parsing information downloaded from the various online sources.

In the following example, we will write a simple script that will download the text contents of a web page and print them to the terminal. This is not a practical use for this module, however it does demonstrate the use of the module for retrieving data from web resources.

We will start by importing the Python modules required for this script. We will save this script file as urllib_example.py:

import urllib2
import sys

In this line, we are taking the first argument on the command line as a URL to open and return the HTML contents of:

url = sys.argv[1]

Now, we will create a request object that represents a request to be sent to the web server. This is not...