The simple buildout (mentioned in the previous section) illustrates most of Buildout's core features. Let us look at these in a bit more detail.
The bootstrap.py script installs zc.buildout itself, and gives us the bin/buildout command. This looks for build instructions in a file called buildout.cfg in the current directory. An alternative file may be specified with the -c command line option:
Note
The bin/buildout command must be re-run for any changes to the buildout configuration file to take effect.
To run a buildout in offline mode, we can use:
This is useful if we are not connected to the internet, or simply to speed up buildout execution.
Note
An offline build will fail unless all required distributions and files are available on the local machine. This usually means that you need to run the buildout at least once already.
See the output of bin/buildout --help for other options.
The buildout configuration file consists of sections, in square brackets, containing options, given as name = value pairs. Options that accept multiple values use whitespace as a delimiter. Such values may also be given on multiple, indented lines:
A value can be referenced from another value as ${section:option}. For example:
This kind of variable substitution becomes important once we start to define sections that are shared by multiple buildouts.
It is also possible to add to or remove from a multi-line option using += and -=:
Lines beginning with a # are taken as comments and ignored.
In a buildout configuration file, the [buildout] section, which controls global options, usually comes first:
The extends line is used to include other configuration files, which can be specified by relative path or remote URL. All included files are merged into a single logical buildout. If, for a particular named section, an option in an extended file is also specified in the extending file, the latter's value is used.
The extends line references a known good versions set for Plone. If you open it in a web browser, you will see something like this (truncated for brevity):
This file extends two other files, which in turn contain statements like (again truncated):
When these files are merged, Buildout will look for version pins in the [versions] section (as indicated by the ${buildout:versions} option). The Zope 2 has known good set pins numerous distributions in its [versions] block, to which the Plone known good set adds its own. We could also add or override some version pins using a [versions] section in our own top level buildout.cfg.
The parts option in the [buildout] section is used to list, in order, the steps that Buildout should follow when executing the build. Each part is defined by a section containing a recipe option.
A recipe is a named distribution which exposes the logic to perform a particular build task. It will be downloaded from PyPI as required. Most recipes accept and/or require various options, read from the relevant part section. It is possible to use a given recipe multiple times (in different parts) in a single buildout.
In the example above, we specified a single part, instance, which was defined in the following section:
This particular recipe is used to configure a Zope 2 instance, into which we install the Plone distribution, and configure an initial user called admin, with the password admin. See http://pypi.python.org/pypi/plone.recipe.zope2instance for details about other options supported by this recipe.
Several recipes, including plone.recipe.zope2instance, use the eggs option to define a working set of eggs that should be provided for any console scripts generated. When Buildout generates a script, it will embed the working set in the sys.path variable. For example, bin/instance will look something like this (truncated for brevity):
This 'path mangling' ensures that the relevant packages, at the required versions, are available at runtime, and provides for isolation between scripts and buildouts.
The working set is calculated from all distributions listed in the eggs option, in addition to their dependencies. When the relevant part is executed, Buildout will download and install distributions from PyPI as necessary. Downloaded source distributions are kept in the dist directory in the downloads cache, if one is configured. Platform-specific egg installations are kept in the eggs/ directory, either locally in buildout root or in a shared eggs cache.
Alternatively, we can specify one or more develop eggs. If no version pin says otherwise, a develop egg normally takes precedence over downloaded distributions. During development, we will usually manage all our custom code as develop eggs in the src/ directory inside the buildout. Develop eggs must be explicitly listed in the develop option in the 'buildout' section. For example:
It is also possible to use a wildcard match:
We will demonstrate a more comprehensive buildout. For more details about Buildout, see http://www.buildout.org/ and http://pypi.python.org/pypi/zc.buildout.
Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.
We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.
Inside a Plone development buildout, we may find the following files and directories:
Tip
If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.
To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).
Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.
Tip
On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.
We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:
Note
Be sure to update the paths to reflect the actual location of the .buildout directory on your system.
The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.
Packages and distributions
Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.
setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:
Here:
- The distribution name is
my.package, and the version is 1.0. - We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files
README.txt and docs/HISTORY.txt. - We ask
Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control. - We declare that
my.* is a namespace package. - We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
- We declare that this distribution depends on setuptools (to support namespace packages) and
some.package. The latter has to be version 1.0 or later.
When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.
It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:
Note
Do not do this for any Plone distributions. Use Buildout instead.
To install a develop egg
instead, we can run:
This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.
New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:
The new distribution will be placed in the dist subdirectory, which will be created if necessary.
Distributions can be uploaded to PyPI through a setup.py command:
Note
On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.
You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.
The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:
Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:
The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:
Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:
Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.
We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.
Inside a Plone development buildout, we may find the following files and directories:
Tip
If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.
To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).
Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.
Tip
On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.
We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:
Note
Be sure to update the paths to reflect the actual location of the .buildout directory on your system.
The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.
Packages and distributions
Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.
setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:
Here:
- The distribution name is
my.package, and the version is 1.0. - We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files
README.txt and docs/HISTORY.txt. - We ask
Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control. - We declare that
my.* is a namespace package. - We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
- We declare that this distribution depends on setuptools (to support namespace packages) and
some.package. The latter has to be version 1.0 or later.
When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.
It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:
Note
Do not do this for any Plone distributions. Use Buildout instead.
To install a develop egg
instead, we can run:
This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.
New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:
The new distribution will be placed in the dist subdirectory, which will be created if necessary.
Distributions can be uploaded to PyPI through a setup.py command:
Note
On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.
You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.
The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:
Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:
The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:
Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:
To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).
Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.
Tip
On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.
We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:
Note
Be sure to update the paths to reflect the actual location of the .buildout directory on your system.
The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.
Packages and distributions
Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.
setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:
Here:
- The distribution name is
my.package, and the version is 1.0. - We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files
README.txt and docs/HISTORY.txt. - We ask
Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control. - We declare that
my.* is a namespace package. - We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
- We declare that this distribution depends on setuptools (to support namespace packages) and
some.package. The latter has to be version 1.0 or later.
When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.
It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:
Note
Do not do this for any Plone distributions. Use Buildout instead.
To install a develop egg
instead, we can run:
This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.
New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:
The new distribution will be placed in the dist subdirectory, which will be created if necessary.
Distributions can be uploaded to PyPI through a setup.py command:
Note
On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.
You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.
The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:
Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:
The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:
Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:
We will now create the development buildout that will be used throughout this book. You can find this in the book's accompanying source code.
We currently have three buildout configuration files
, all found at the root of the buildout:
The first of these files, versions.cfg, is straightforward:
Note
The version pins shown here were the most appropriate versions at the time of writing. You may want to update some of these versions. See the description of z3c.checkversions mentioned in the following sections for details.
Next up, packages.cfg is a little more interesting:
This first installs two buildout extensions: mr.developer, which is used to manage develop eggs, and buildout.dumppickedversions, which will print the picked versions for any unpinned distributions at the end of the buildout run. We will describe these in more detail in the next section.
Next, we extend two known good sets—the one for our chosen Plone version, and our
own local versions.cfg, which should come last so that it can override any versions from the external sets.
We also set the unzip option to true, which means that buildout will always unzip eggs in eggs/ directory. This makes debugging easier, avoids potential problems with Zope code that does not properly set the zip_safe flag to False, and is a prerequisite for using collective.recipe.omelette. Two recipe-less sections are then defined: [sources] is used by mr.developer, and will be described in a moment. [eggs] is used to define working sets that are referenced in other buildout sections:
Finally, buildout.cfg contains the actual build instructions for our development environment:
First, we list the parts to execute, and extend packages.cfg to gain access to our working- and known good version sets. The last two options in the [buildout] section, auto-checkout and always-checkout, are used by mr.developer and described below.
Next, we define our development Zope instance. This runs on port 8080, has an administrative Zope user called admin with password admin, provides additional debug information for security violations, and is installed with a working set combining ${eggs:main} and ${eggs:devtools}.
The subsequent parts install various development tools, which are described in more detail in the subsequent sections.
Note
You may have noticed that several parts use zc.recipe.egg. This recipe is used to install one or more distributions. If a distribution declares one or more console scripts (through an entry point), these are installed to the bin/ directory refer http://pypi.python.org/pypi/zc.recipe.egg for details.
Let us now examine the various development tools that we will use throughout the book.
The first two development tools are extensions to buildout itself, which operate at build time.
We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:
Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):
Tip
Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.
With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:
Note
We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.
If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.
Note
There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.
mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.
This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.
buildout.dumppickedversions
As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.
The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:
Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.
Tip
The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.
Development Zope instance
For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Let us now examine the various development tools that we will use throughout the book.
The first two development tools are extensions to buildout itself, which operate at build time.
We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:
Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):
Tip
Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.
With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:
Note
We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.
If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.
Note
There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.
mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.
This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.
buildout.dumppickedversions
As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.
The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:
Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.
Tip
The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.
Development Zope instance
For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
The first two development tools are extensions to buildout itself, which operate at build time.
We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:
Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):
Tip
Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.
With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:
Note
We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.
If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.
Note
There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.
mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.
This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.
buildout.dumppickedversions
As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.
The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:
Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.
Tip
The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.
Development Zope instance For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:
Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):
Tip
Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.
With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:
Note
We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.
If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.
Note
There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.
mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.
This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.
buildout.dumppickedversions
As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.
The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:
Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.
Tip
The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.
Development Zope instance For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
buildout.dumppickedversions
As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.
The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:
Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.
Tip
The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.
Development Zope instance For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Development Zope instance
For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.
Note
'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.
During development, we normally run Zope in foreground mode, with:
Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.
Note
Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.
We also install various development tools into the Zope instance. These are outlined in the subsequent sections.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.
Note
plone.reload only works when Zope is running in debug mode.
The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.
Note
The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.
Tip
Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.
Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop
into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Products.PrintingMailHost
This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.
Note
In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.
This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.
Note
Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.
The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:
To run only the tests in a particular package, use the -s option:
To run only a test with a name matching a particular regular expression, use the -t option:
See the output of bin/test --help for other options.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:
Note
Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.
This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:
This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.
Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.
To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.
Note
collective.recipe.omelette works on Windows, but there are three important caveats:
- You must download
junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.
If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.
Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.
Note
Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them
with: import transaction; transaction.commit().
ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.
We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.
To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:
This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:
See the output of bin/checkversions --help for more options.
Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.
As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases.
For more information, see http://pypi.python.org/pypi/jarn.mkrelease.
The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.
Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.
Learning to help yourself
During development, there will probably be times when you are stumped. Plone is fairly well-documented, but the documentation is certainly not perfect. The mailing lists and chat room are great resources if you need help, but it is also very important to learn how to help yourself.
Documentation for Plone code can usually be found in one of the following places:
- High level and tutorial-style documentation is normally found at http://plone.org/documentation.
- Many distributions include documentation in their PyPI listings. For example, plone.testing's documentation can be found at http://pypi.python.org/pypi/plone.testing.
- Distributions often include documentation in their
README file. This is often also the source of the documentation published to PyPI. - Internal documentation in the form of comments or tests is often very valuable. Most packages use Zope interfaces (more on those in Chapter 9, Nine Core Concepts of Zope Programming) to describe their key APIs. Look for a file called
interfaces.py.
Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.
One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.
Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.
You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.
Note
It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!
Become familiar with the debugger
PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:
Tip
To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.
When this line is encountered, execution will stop, and the terminal will display:
This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.
If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.
During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.
If an exception is encountered, you will likely see a traceback. For example:
The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.
Tip
If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.
United States
Great Britain
India
Germany
France
Canada
Russia
Spain
Brazil
Australia
Singapore
Hungary
Ukraine
Luxembourg
Estonia
Lithuania
South Korea
Turkey
Switzerland
Colombia
Taiwan
Chile
Norway
Ecuador
Indonesia
New Zealand
Cyprus
Denmark
Finland
Poland
Malta
Czechia
Austria
Sweden
Italy
Egypt
Belgium
Portugal
Slovenia
Ireland
Romania
Greece
Argentina
Netherlands
Bulgaria
Latvia
South Africa
Malaysia
Japan
Slovakia
Philippines
Mexico
Thailand
Filter
4
