Python's folder structure holds more than just the Python interpreter. Within the subfolders reside a number of important scripts, digital link libraries, and even C language modules. Not all of the scripts are used all the time, but each has a role in making the Python programming environment possible. The most important folder to know about is the site-packages folder, where most modules that will be imported in Python scripts are contained.
Python folder structure
Where modules reside
Within every Python installation is a folder called Lib, and within that folder is a folder called site-packages. On my machine, the folder sits at C:\Python27\ArcGIS10.5\Lib\site-packages.
Almost all third-party modules are copied into this folder to be imported as needed. The main exception to this rule, for our purposes, is the ArcPy module, which is stored within the ArcGIS folder in the Program Files folder (for example, C:\Program Files (x86)\ArcGIS\Desktop10.5\arcpy). To make that possible, the ArcGIS installer adjusts the Python system path (using the sys module) to make the arcpy module importable, as described next.
Installing a third-party module
To add greater functionality, thousands of third-party modules, or packages, are available for download. Online module repositories include the Python Package Index (PyPI) as well as GitHub, and others. Python 2 and Python 3 now include a module designed to make installing these packages more simple than it was in the past. This module, pip, will check for registered modules in PyPI, and install the latest version using the command install. Use pip from the command prompt by passing the command install and the name of the package to install.
If the module is not available on PyPI, pip may not be able to install it. For instance, if it’s on GitHub instead (even Python 3.7 is now hosted on https://github.com/, so GitHub is worth knowing), download the zip file of the module, and unzip it into the Lib/site-packages folder. Open a Command Prompt terminal, change directory (cd) into the newly unzipped folder, and run the script setup.py that is part of each module, using the command python setup.py install. This script will install the module, and configure the environmental variables required to make it run.
Many Python modules are only available in the GZip format, which can be unzipped using freeware such as 7Zip. Unzip the .gz file, then unzip the resulting .tar file into the Lib/site-packages folder in the Python folder.
Using Python's sys module to add a module
Python's sys module allows the user to take advantage of the system tools built into the Python interpreter. One of the most useful properties of the sys module is sys.path. It is a list of file paths which the user can modify to adjust where Python will look for a module to import, without needing administrative access.
When Python 2.7 is installed by the ArcGIS 10.5 installer, the installer takes advantage of sys.path functions to add C:\Program Files (x86)\ArcGIS\Desktop10.5\arcpy to the system path. To test this, start up the Python interpreter or an IDE, and type the following:
>>> import sys
>>> print sys.path
['', 'C:\\WINDOWS\\SYSTEM32\\python27.zip', 'C:\\Python27\\ArcGIS10.5\\Dlls', 'C:\\Python27\\ArcGIS10.5\\lib', 'C:\\Python27\\ArcGIS10.5\\lib\\plat-win', 'C:\\Python27\\ArcGIS10.5', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.5\\arcpy', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.5\\ArcToolbox\\Scripts']
The system path (stored in the sys property sys.path) includes all of the folders that ArcPy requires to automate ArcGIS. The system path incorporates all directories listed in the PYTHONPATH environment variable (if one has been created); this is separate from the Windows Path environment variable discussed earlier. The two separate path variables work together to help Python locate modules.
The sys.path.append method
The sys.path property is a mutable list, and can be appended or extended to include new file paths that will point to modules the user wants to import. To avoid the need to adjust the sys.path, copy the module into the site-packages folder; however, this is not always possible, so use the sys.path.append method as needed:
>>> sys.path.append("C:\\Projects\\Requests")
>>> sys.path
['', 'C:\\WINDOWS\\SYSTEM32\\python27.zip', 'C:\\Python27\\ArcGIS10.5\\Dlls', 'C:\\Python27\\ArcGIS10.5\\lib', 'C:\\Python27\\ArcGIS10.5\\lib\\plat-win', 'C:\\Python27\\ArcGIS10.5', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.5\\arcpy', 'C:\\Program Files (x86)\\ArcGIS\\Desktop10.5\\ArcToolbox\\Scripts','C:\\Projects\\Requests']
When the sys.path.append method is used, the adjustment is temporary. Adjust the PYTHONPATH environment variable in the Windows System Properties menu (discussed earlier in the Path environment variable section) to make a permanent change (and create the PYTHONPATH if it has not been created).
One last, valuable note: to import a module without adjusting the system path or copying the module into the site-packages folder, place the module in the folder that contains the script that is importing it. As long as the module is configured correctly, it will work normally. This is useful when there is no administrative access available to the executing machine.
