Background briefing on tools

The organization responsible for tools and technology is affectionately known as The Puzzle Palace. They have provided some suggestions on what we'll need for the missions that we've been assigned. We'll start with an overview of the state of art in Python tools that are handed down from one of the puzzle solvers.

Some agents have already upgraded to Python 3.4. However, not all agents have done this. It's imperative that we use the latest and greatest tools.

There are four good reasons for this, as follows:

• Features: Python 3.4 adds a number of additional library features that we can use. The list of features is available at https://docs.python.org/3/whatsnew/3.4.html.
• Performance: Each new version is generally a bit faster than the previous version of Python.
• Security: While Python doesn't have any large security holes, there are new security changes in Python.
• Housecleaning: There are a number of rarely used features that were and have been removed.

Some agents may want to start looking at Python 3.5. This release is anticipated to include some optional features to provide data type hints. We'll look at this in a few specific cases as we go forward with the mission briefings. The type-analysis features can lead to improvements in the quality of the Python programming that an agent creates. The puzzle palace report is based on intelligence gathered at PyCon 2015 in Montreal, Canada. Agents are advised to follow the Python Enhancement Proposals (PEP) closely. Refer to https://www.python.org/dev/peps/.

We'll focus on Python 3.4. For any agent who hasn't upgraded to Python 3.4.3, we'll look at the best way to approach this.

If you're comfortable with working on your own, you can try to move further and download and install Python 3.5. Here, the warning is that it's very new and it may not be quite as robust as the Python version 3.4. Refer to PEP 478 (https://www.python.org/dev/peps/pep-0478/) for more information about this release.

Doing a Python upgrade

It's important to consider each major release of Python as an add-on and not a replacement. Any release of Python 2 should be left in place. Most field agents will have several side-by-side versions of Python on their computers. The following are the two common scenarios:

• The OS uses Python 2. Mac OS X and Linux computers require Python 2; this is the default version of Python that's found when we enter python at the command prompt. We have to leave this in place.
• We might also have an older Python 3, which we used for the previous missions. We don't want to remove this until we're sure that we've got everything in place in order to work with Python 3.4.

We have to distinguish between the major, minor, and micro versions of Python. Python 3.4.3 and 3.4.2 have the same minor version (3.4). We can replace the micro version 3.4.2 with 3.4.3 without a second thought; they're always compatible with each other. However, we don't treat the minor versions quite so casually. We often want to leave 3.3 in place.

Generally, we do a field upgrade as shown in the following:

1. Download the installer that is appropriate for the OS and Python version. Start at this URL: https://www.python.org/downloads/. The web server can usually identify your computer's OS and suggest the appropriate download with a big, friendly, yellow button. Mac OS X agents will notice that we now get a .pkg (package) file instead of a .dmg (disk image) containing .pkg. This is a nice simplification.
2. When installing a new minor version, make sure to install in a new directory: keep 3.3 separate from 3.4. When installing a new micro version, replace any existing installation; replace 3.4.2 with 3.4.3.
   • For Mac OS X and Linux, the installers will generally use names that include python3.4 so that the minor versions are kept separate and the micro versions replace each other.
   • For Windows, we have to make sure we use a distinct directory name based on the minor version number. For example, we want to install all new 3.4.x micro versions in C:\Python34. If we want to experiment with the Python 3.5 minor version, it would go in C:\Python35.
3. Tweak the PATH environment setting to choose the default Python.
   • This information is generally in our ~/.bash_profile file. In many cases, the Python installer will update this file in order to assure that the newest Python is at the beginning of the string of directories that are listed in the PATH setting. This file is generally used when we log in for the first time. We can either log out and log back in again, or restart the terminal tool, or we can use the source ~/.bash_profile command to force the shell to refresh its environment.
   • For Windows, we must update the advanced system settings to tweak the value of the PATH environment variable. In some cases, this value has a huge list of paths; we'll need to copy the string and paste it in a text editor to make the change. We can then copy it from the text editor and paste it back in the environment variable setting.
4. After upgrading Python, use pip3.4 (or easy_install-3.4) to add the additional packages that we need. We'll look at some specific packages in mission briefings. We'll start by adding any packages that we use frequently.

At this point, we should be able to confirm that our basic toolset works. Linux and Mac OS agents can use the following command:

MacBookPro-SLott:Code slott$ python3.4

This should confirm that we've downloaded and installed Python and made it a part of our OS settings. The greeting will show which micro version of Python 3.4 have we installed.

For Windows, the command's name is usually just python. It would look similar to the following:

C:\> python

The Mac OS X interaction should include the version; it will look similar to the following code:

MacBookPro-SLott:NavTools-1.2 slott$ python3.4
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 23 2015, 02:52:03) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0)

We've entered the python3.4 command. This shows us that things are working very nicely. We have Python 3.4.3 successfully installed.

We don't want to make a habit of using the python or python3 commands in order to run Python from the command line. These names are too generic and we could accidentally use Python 3.3 or Python 3.5, depending on what we have installed. We need to be intentional about using Python3.4.

Preliminary mission to upgrade pip

The first time that we try to use pip3.4, we may see an interaction as shown in the following:

MacBookPro-SLott:Code slott$ pip3.4 install anything
You are using pip version 6.0.8, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The version numbers may be slightly different; this is not too surprising. The packaged version of pip isn't always the latest and greatest version. Once we've installed the Python package, we can upgrade pip3.4 to the recent release. We'll use pip to upgrade itself.

It looks similar to the following code:

MacBookPro-SLott:Code slott$ pip3.4 install --upgrade pip
You are using pip version 6.0.8, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.0.3-py2.py3-none-any.whl#md5=6950e1d775fea7ea50af690f72589dbd
  Downloading pip-7.0.3-py2.py3-none-any.whl (1.1MB)
    100% |################################| 1.1MB 398kB/s 
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Successfully uninstalled pip-6.0.8

Successfully installed pip-7.0.3

We've run the pip installer to upgrade pip. We're shown some details about the files that are downloaded and new is version installed. We were able to do this with a simple pip3.4 under Mac OS X.

Some packages will require system privileges that are available via the sudo command. While it's true that a few packages don't require system privileges, it's easy to assume that privileges are always required. For Windows, of course, we don't use sudo at all.

On Mac OS X, we'll often need to use sudo -H instead of simply using sudo. This option will make sure that the proper HOME environment variable is used to manage a cache directory.

Note that your actual results may differ from this example, depending on how out-of-date your copy of pip turns out to be. This pip install --upgrade pip is a pretty frequent operation as the features advance.