Sharing downloads

You will usually work on several projects simultaneously, probably for different hardware platforms or different target images. In such cases, it is important to optimize the build times by sharing downloads.

Getting ready

The build system runs a search for downloaded sources in a number of places:

• It tries the local downloads folder.
• It looks into the configured premirrors, which are usually local to your organization.
• It then tries to fetch from the upstream source as configured in the package recipe.
• Finally, it checks the configured mirrors. Mirrors are public alternate locations for the source.

If a package source is not found in any of the these four, the package build will fail with an error. Build warnings are also issued when upstream fetching fails and mirrors are tried, so that the upstream problem can be looked at.

The Yocto project maintains a set of mirrors to isolate the build system from problems with the upstream servers. However, when adding external layers, you could be adding support for packages that are not in the Yocto project's mirror servers, or other configured mirrors, so it is recommended that you keep a local premirror to avoid problems with source availability.

The default Poky setting for a new project is to store the downloaded package sources on the current build directory. This is the first place the build system will run a search for source downloads. This setting can be configured in your project's conf/local.conf file with the DL_DIR configuration variable.

How to do it...

To optimize the build time, it is recommended to keep a shared downloads directory between all your projects. The setup-environment script of the meta-fsl-arm layer changes the default DL_DIR to the fsl-community-bsp directory created by the repo tool. With this setup, the downloads folder will already be shared between all the projects in your host system. It is configured as:

DL_DIR ?= "${BSPDIR}/downloads/"

A more scalable setup (for instance, for teams that are remotely distributed) is to configure a premirror. For example, adding the following to your conf/local.conf file:

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "http://example.com/my-source-mirror"

A usual setup is to have a build server serve its downloads directory. The build server can be configured to prepare tarballs of the Git directories to avoid having to perform Git operations from upstream servers. This setting in your conf/local.conf file will affect the build performance, but this is usually acceptable in a build server. Add the following:

BB_GENERATE_MIRROR_TARBALLS = "1"

An advantage of this setup is that the build server's downloads folder can also be backed up to guarantee source availability for your products in the future. This is especially important in embedded products with long-term availability requirements.

In order to test this setup, you may check to see whether a build is possible just by using the premirrors with the following:

BB_FETCH_PREMIRRORONLY = "1"

This setting in your conf/local.conf file can also be distributed across the team with the TEMPLATECONF variable during the project's creation.