Using build history

When maintaining software for an embedded product, you need a way to know what has changed and how it is going to affect your product.

On a Yocto system, you may need to update a package revision (for instance, to fix a security vulnerability), and you need to make sure what the implications of this change are, for example, in terms of package dependencies and changes to the root filesystem.

Build history enables you to do just that, and we will explore it in this recipe.

How to do it...

To enable build history, add the following to your conf/local.conf file:

INHERIT += "buildhistory" 

The preceding configuration enables information gathering, including dependency graphs.

To enable the storage of build history in a local Git repository add the following line to the conf/local.conf configuration file as well:

BUILDHISTORY_COMMIT = "1"

The Git repository location can be set by the BUILDHISTORY_DIR variable, which by default is set to a buildhistory directory on your build directory.

By default, buildhistory tracks changes to packages, images, and SDKs. This is configurable using the BUILDHISTORY_FEATURES variable. For example, to track only image changes, add the following to your conf/local.conf:

BUILDHISTORY_FEATURES = "image" 

It can also track specific files and copy them to the buildhistory directory. By default, this includes only /etc/passwd and /etc/groups, but it can be used to track any important files, such as security certificates. The files need to be added with the BUILDHISTORY_IMAGE_FILES variable in your conf/local.conf file, as follows:

BUILDHISTORY_IMAGE_FILES += "/path/to/file" 

Build history will slow down the build, increase the build size, and may also grow the Git directory to an unmanageable size. The recommendation is to enable it on a build server for software releases, or in specific cases, such as when updating production software.

How it works...

When enabled, it will keep a record of the changes to each package and image in the form of a Git repository in a way that can be explored and analyzed.

Note that build history will only record changes to the build. If your project is already built, you will have to modify something or remove the tmp folder in order for build history to be generated.

The build configuration and metadata revision, as printed by Bitbake, is stored in the build-id.txt file.

For a package, build history records the following information:

• Package and recipe revision
• Dependencies
• Package size
• Files

For an image, it records the following information:

• Build configuration
• Dependency graphs
• A list of files that include ownership and permissions, as well as size and symlink information
• A list of installed packages

And for an SDK, it records the following information:

• SDK configuration
• A list of both host and target files, including ownership and permissions, as well as size and symlinks information
• Package-related information is only generated for the standard SDK, not for the extensible SDK. This includes:
  • Dependency graphs
  • A list of installed packages

For more details about using Yocto SDKs, please refer to the Preparing an SDK and Using the extensible SDK recipes in Chapter 4, Application Development.

Looking at build history

Inspecting the Git directory with build history can be done in several ways:

• Using Git tools such as gitk or git log
• Using the buildhistory-diff command-line tool, which displays the differences in a human-readable format
• Using a Django-1.8-based web interface

To install the Django web interface on a development machine, you first need to install some host dependencies:

$ sudo apt-get install python3-django 
$ sudo apt-get install python-django-registration

This will install Django 1.8 both for Python 2.7, and Python 3. The buildhistory-web interface will only currently work on Python 2.7 but the build history import script will need to run under Python 3 as that is what the Yocto 2.4 BitBake uses.

Now we can clone the web interface source and configure it:

$ cd /opt/yocto/fsl-community-bsp/sources
$ git clone git://git.yoctoproject.org/buildhistory-web
$ cd buildhistory-web/  

Edit the settings.py file to change the path to the database engine:

    DATABASES = {                                                                   
        'default': {                                                                
            'ENGINE': 'django.db.backends.sqlite3',
            'NAME': '/opt/yocto/fsl-community-bsp/sources/buildhistory-web/bhtest.db3',
            'USER': '',
            'PASSWORD': '',
            'HOST': '',
            'PORT': '',
        }                                                                           
    }  
  

You then need to set up the Django application with:

$ python manage.py migrate  

Next, import buildhistory as follows:

$ python3 warningmgr/import.py  /opt/yocto/fsl-community-bsp/sources/poky/ /opt/yocto/fsl-community-bsp/wandboard/buildhistory/  

The preceding command will need to be executed each time there is a new build.

And finally, start the web server on the localhost with:

$ python manage.py runserver  

To bind it to a different IP address and port you can do:
$ python manage.py runserver <host>:<port>
But you will need to configure your settings.py accordingly with:
ALLOWED_HOSTS = [u'<host>']

The following image shows the Buildhistory web interface home page:

Buildhistory web interface

There's more...

To maintain build history, it's important to optimize it and prevent it from growing over time. Periodic backups of build history and clean-ups of older data are important to keep the build history repository at a manageable size.

Once the buildhistory directory has been backed up, the following process will trim it and keep only the most recent history:

1. Copy your repository to a temporary RAM filesystem (tmpfs) to speed things up. Check the output of the df -h command to see which directories are tmpfs filesystems and how much space they have available, and use one. For example, in Ubuntu 16.04, the /run/ directory is available.

2. Copy build history to the /run directory as follows:

$ sudo mkdir /run/workspace
$ sudo chown ${USER} /run/workspace/
$ cp -r /opt/yocto/fsl-community-bsp/wandboard/buildhistory/ /run/workspace/
$ cd /run/workspace/buildhistory/  

3. Add a graft point for a commit 1 month ago with no parents:

$ git rev-parse "HEAD@{1 month ago}" > .git/info/grafts  

4. Make the graft point permanent:

$ git filter-branch

5. Clone a new repository to clean up the remaining Git objects:

$ git clone file://${tmpfs}/buildhistory buildhistory.new

6. Replace the old buildhistory directory with the new cleaned one:

$ rm -rf buildhistory
$ mv buildhistory.new /opt/yocto/fsl-community-bsp/wandboard/buildhistory/

7. And finally, remove the workspace:

$ rm -rf /run/workspace/