Configuring and building the Linux kernel

There are numerous drivers/features and build options available in the Linux kernel sources. The configuration process consists of choosing what features/drivers are going to be part of the compilation process. Depending on whether we are going to perform native compilation or cross-compilation, there are environment variables that must be defined, even before the configuration process takes place.

Specifying compilation options

The compiler that's invoked by the kernel's Makefile is $(CROSS_COMPILE)gcc. That said, CROSS_COMPILE is the prefix of the cross-compiling tools (gcc, as, ld, objcopy, and so on) and must be specified when you're invoking make or must have been exported before any make command is executed. Only gcc and its related Binutils executables will be prefixed with $(CROSS_COMPILE).

Note that various assumptions are made and options/features/flags are enabled by the Linux kernel build infrastructure based on the target architecture. To achieve that, in addition to the cross-compiler prefix, the architecture of the target must be specified as well. This can be done through the ARCH environment variable.

Thus, a typical Linux configuration or build command would look as follows:

ARCH=<XXXX> CROSS_COMPILE=<YYYY> make menuconfig

It can also look as follows:

ARCH=<XXXX> CROSS_COMPILE=<YYYY> make <make-target>

If you don't wish to specify these environment variables when you launch a command, you can export them into your current shell. The following is an example:

export CROSS_COMPILE=aarch64-linux-gnu-
export ARCH=aarch64

Remember that if these variables are not specified, the native host machine is going to be targeted; that is, if CROSS_COMPILE is omitted or not set, $(CROSS_COMPILE)gcc will result in gcc, and it will be the same for other tools that will be invoked (for example, $(CROSS_COMPILE)ld will result in ld).

In the same manner, if ARCH (the target architecture) is omitted or not set, it will default to the host where make is executed. It will default to $(uname -m).

As a result, you should leave CROSS_COMPILE and ARCH undefined to have the kernel natively compiled for the host architecture using gcc.

Understanding the kernel configuration process

The Linux kernel is a Makefile-based project that contains thousands of options and drivers. Each option that's enabled can make another one available or can pull specific code into the build. To configure the kernel, you can use make menuconfig for a ncurses-based interface or make xconfig for an X-based interface. The ncurses-based interface looks as follows:

Figure 1.1 – Kernel configuration screen

For most options, you have three choices. However, we can enumerate five types of options while configuring the Linux kernel:

• Boolean options, for which you have two choices:
  • (blank), which leaves this feature out. Once this option is highlighted in the configuration menu, you can press the <n> key to leave the feature out. It is equivalent to false. When it's disabled, the resulting configuration option is commented out in the configuration file.
  • (*), which compiles it statically in the kernel. This means it will always be there when the kernel first loads. It is equivalent to true. You can enable a feature in the configuration menu by selecting it and pressing the <y> key. The resulting option will appear as CONFIG_<OPTION>=y in the configuration file; for example, CONFIG_INPUT_EVDEV=y.
• Tristate options, which, in addition to being able to take Boolean states, can take a third state, marked as (M) in the configuration windows. This results in CONFIG_<OPTION>=m in the configuration file; for example, CONFIG_INPUT_EVDEV=m. To produce a loadable module (provided that this option allows it), you can select the feature and press the M key.
• String options, which expect string values; for example, CONFIG_CMDLINE="noinitrd console=ttymxc0,115200".
• Hex options, which expect hexadecimal values; for example, CONFIG_PAGE_OFFSET=0x80000000.
• Int options, which expect integer values; for example, CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7.

The selected options will be stored in a .config file, at the root of the source tree.

It is very difficult to know which configuration is going to work on your platform. In most cases, there will be no need to start a configuration from scratch. There are default and functional configuration files available in each arch directory that you can use as a starting point (it is important to start with a configuration that already works):

ls arch/<your_arch>/configs/

For 32-bit ARM-based CPUs, these config files can be found in arch/arm/configs/. In this architecture, there is usually one default configuration per CPU family. For instance, for i.MX6-7 processors, the default config file is arch/arm/configs/imx_v6_v7_defconfig. However, on ARM 64-bit CPUs, there is only one big default configuration to customize; it is located in arch/arm64/configs/ and is called defconfig. Similarly, for x86 processors, we can find the files in arch/x86/configs/. There will be two default configuration files here – i386_defconfig and x86_64_defconfig, for 32- and 64-bit x86 architectures, respectively.

The kernel configuration command, given a default configuration file, is as follows:

make <foo_defconfig>

This will generate a new .config file in the main (root) directory, while the old .config will be renamed .config.old. This can be useful to revert the previous configuration changes. Then, to customize the configuration, you can use the following command:

make menuconfig

Saving your changes will update your .config file. While you could share this config with your teammates, you are better off creating a default configuration file in the same minimal format as those shipped with the Linux kernel sources. To do that, you can use the following command:

make savedefconfig

This command will create a minimal (since it won't store non-default settings) configuration file. The generated default configuration file will be called defconfig and stored at the root of the source tree. You can store it in another location using the following command:

mv defconfig arch/<arch>/configs/myown_defconfig

This way, you can share a reference configuration inside the kernel sources and other developers can now get the same .config file as you by running the following command:

make myown_defconfig

Note

Note that, for cross-compilation, ARCH and CROSS_COMPILE must be set before you execute any make command, even for kernel configuration. Otherwise, you'll have unexpected changes in your configuration.

The followings are the various configuration commands you can use, depending on the target system:

• For a 64-bit x86 native compilation, it is quite straightforward (the compilation options can be omitted):

  make x86_64_defconfig
  make menuconfig

• Given a 32-bit ARM i.MX6-based board, you can execute the following command:

  ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make imx_v6_v7_defconfig
  ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make menuconfig

With the first command, you store the default options in the .config file, while with the latter, you can update (add/remove) various options, depending on your needs.

• For 64-bit ARM boards, you can execute the following commands:

  ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- make defconfig
  ARCH=aarch64 CROSS_COMPILE=aarch64-linux-gnu- make menuconfig

You may run into a Qt4 error with xconfig. In such a case, you should just use the following command to install the missing packages:

sudo apt install qt4-dev-tools qt4-qmake

Note

You may be switching from an old kernel to a new one. Given the old .config file, you can copy it into the new kernel source tree and run make oldconfig. If there are new options in the new kernel, you'll be prompted to include them or not. However, you may want to use the default values for those options. In this case, you should run make olddefconfig. Finally, to say no to every new option, you should run make oldnoconfig.

There may be a better option to find an initial configuration file, especially if your machine is already running. Debian and Ubuntu Linux distributions save the .config file in the /boot directory, so you can use the following command to copy this configuration file:

cp /boot/config-`uname -r` .config

The other distributions may not do this. So, I can recommend that you always enable the IKCONFIG and IKCONFIG_PROC kernel configuration options, which will enable access to .config through /proc/configs.gz. This is a standard method that also works with embedded distributions.

Some useful kernel configuration features

Now that we can configure the kernel, let's enumerate some useful configuration features that may be worth enabling in your kernel:

• IKCONFIG and IKCONFIG_PROC: These are the most important to me. It makes your kernel configuration available at runtime, in /proc/config.gz. It can be useful either to reuse this config on another system or simply look for the enabled state of a particular feature; for example:

  # zcat /proc/config.gz | grep CONFIG_SOUND
  CONFIG_SOUND=y
  CONFIG_SOUND_OSS_CORE=y
  CONFIG_SOUND_OSS_CORE_PRECLAIM=y
  # CONFIG_SOUNDWIRE is not set
  #  

• CMDLINE_EXTEND and CMDLINE: The first option is a Boolean that allows you to extend the kernel command line from within the configuration, while the second option is a string containing the actual command-line extension value; for example, CMDLINE="noinitrd usbcore.authorized_default=0".
• CONFIG_KALLSYMS: This is a Boolean option that makes the kernel symbol table (the mapping between symbols and their addresses) available in /proc/kallsyms. This is very useful for tracers and other tools that need to map kernel symbols to addresses. It is used while you're printing oops messages. Without this, oops listings would produce hexadecimal output, which is difficult to interpret.
• CONFIG_PRINTK_TIME: This option shows timing information while printing messages from the kernel. It may be helpful to timestamp events that occurred at runtime.
• CONFIG_INPUT_EVBUG: This allows you to debug input devices.
• CONFIG_MAGIC_SYSRQ: This allows you to have some control (such as rebooting, dumping some status information, and so on) over the system, even after a crash, by simply using some combination keys.
• DEBUG_FS: This enables support for debug filesystems, where GPIO, CLOCK, DMA, REGMAP, IRQs, and many other subsystems can be debugged from.

• FTRACE and DYNAMIC_FTRACE: These options enable the powerful ftrace tracer, which can trace the whole system. Once ftrace has been enabled, some of its enumeration options can be enabled as well:
  • FUNCTION_TRACER: This allows you to trace any non-inline function in the kernel.
  • FUNCTION_GRAPH_TRACER: This does the same thing as the previous command, but it shows a call graph (the caller and the callee functions).
  • IRQSOFF_TRACER: This allows you to track off periods of IRQs in the kernel.
  • PREEMPT_TRACER: This allows you to measure preemption off latency.
  • SCHED_TRACER: This allows you to schedule latency tracing.

Now that the kernel has been configured, it must be built to generate a runnable kernel. In the next section, we will describe the kernel building process, as well as the expected build artifacts.

Building the Linux kernel

This step requires you to be in the same shell where you were during the configuration step; otherwise, you'll have to redefine the ARCH and CROSS_COMPILE environment variables.

Linux is a Makefile-based project. Building such a project requires using the make tool and executing the make command. Regarding the Linux kernel, this command must be executed from the main kernel source directory, as a normal user.

By default, if not specified, the make target is all. In the Linux kernel sources, for x86 architectures, this target points to (or depends on) vmlinux bzImage modules targets; for ARM or aarch64 architectures, it corresponds to vmlinux zImage modules dtbs targets.

In these targets, bzImage is an x86-specific make target that produces a binary with the same name, bzImage. vmlinux is a make target that produces a Linux image called vmlinux. zImage and dtbs are both ARM- and aarch64-specific make targets. The first produces a Linux image with the same name, while the second builds the device tree sources for the target CPU variant. modules is a make target that will build all the selected modules (marked with m in the configuration).

While building, you can leverage the host's CPU performance by running multiple jobs in parallel thanks to the -j make options. The following is an example:

make -j16

Most people define their -j number as 1.5x the number of cores. In my case, I always use ncpus * 2.

You can build the Linux kernel like so:

• For a native compilation, use the following command:

  make -j16

• For a 32-bit ARM cross-compilation, use the following command:

  ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j16

Each make target can be invoked separately, like so:

ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make dtbs

You can also do the following:

ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make zImage -j16

Finally, you can also do the following:

make bzImage -j16

Note

I have used -j16 in my commands because my host has an 8-core CPU. This number of jobs must be adapted according to your host configuration.

At the end of your 32-bit ARM cross-compilation jobs, you will see something like the following:

[…]
  LZO     arch/arm/boot/compressed/piggy_data
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/hyp-stub.S
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  SHIPPED arch/arm/boot/compressed/bswapsdi2.S
  AS      arch/arm/boot/compressed/hyp-stub.o
  AS      arch/arm/boot/compressed/lib1funcs.o
  AS      arch/arm/boot/compressed/ashldi3.o
  AS      arch/arm/boot/compressed/bswapsdi2.o
  AS      arch/arm/boot/compressed/piggy.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready

By using the default targets, various binaries will result from the build process, depending on the architecture. These are as follows:

• arch/<arch>/boot/Image: An uncompressed kernel image that can be booted
• arch/<arch>/boot/*Image*: A compressed kernel image that can also be booted:

This is bzImage (which means "big zImage") for x86, zImage for ARM or aarch64, and vary for other architectures.

• arch/<arch>/boot/dts/*.dtb: This provides compiled device tree blobs for the selected CPU variant.
• vmlinux: This is a raw, uncompressed, and unstripped kernel image in ELF format. It's useful for debugging purposes but generally not used for booting purposes.

Now that we know how to (cross-)compile the Linux kernel, let's learn how to install it.

Installing the Linux kernel

The Linux kernel installation process differs in terms of native compilation or cross-compilation:

• In native installation (that is, you're installing host), you can simply run sudo make install. You must use sudo because the installation will take place in the /boot directory. If you're doing an x86 native installation, the following files are shipped:
  • /boot/vmlinuz-<version>: This is the compressed and stripped variant of vmlinux. It is the same kernel image as the one in arch/<arch>/boot.
  • /boot/System.map-<version>: This stores the kernel symbol table (the mapping between the kernel symbols and their addresses), not just for debugging, but also to allow some kernel modules to resolve their symbols and load properly. This file only contains a static kernel symbol table, while /proc/kallsyms (provided that CONFIG_KALLSYMS is enabled in the config) on the running kernel contains System.map and the loaded kernel module symbols.
  • /boot/config-<version>: This corresponds to the kernel configuration for the version that's been built.
• An embedded installation usually uses a single file kernel. Moreover, the target is not accessible, which makes manual installation preferred. Thus, embedded Linux build systems (such as Yocto or Buildroot) use internal scripts to make the kernel image available in the target root filesystem. While embedded installation may be straightforward thanks to the use of build systems, native installation (especially x86 native installation) can require running additional bootloader-related commands (such as update-grub2) to make the new kernel visible to the system.

Now that we are familiar with the kernel configuration, including the build and installation processes, let's look at kernel modules, which allow you to extend the kernel at runtime.