I assume that you have gone through Online Chapter, Kernel Workspace Setup, and have appropriately prepared an x86_64 guest VM running Ubuntu 22.04 LTS (or equivalent) and installed all the required packages. If not, I highly recommend you do this first.

To get the most out of this book, I also strongly recommend you clone this book’s GitHub repository (https://github.com/PacktPublishing/Linux-Kernel-Programming_2E) for the code and work on it in a hands-on fashion.

It’s important to understand a few things right from the outset that will help you as we proceed on our journey of building and working with a Linux kernel. Firstly, the Linux kernel and its sister projects are completely decentralized – it’s a virtual, online open-source community! The closest we come to an “office” for Linux is this: stewardship of the Linux kernel (as well as several dozen related projects) is in the capable hands of the Linux Foundation (https://linuxfoundation.org/); further, it manages the Linux Kernel Organization, a private foundation that distributes the Linux kernel to the public free of charge (https://www.kernel.org/nonprofit.html).

Some of the key points we’ll discuss in this section include the following:

• The kernel release, or version number nomenclature
• The typical kernel development workflow
• The existence of different types of kernel source trees within the repository

With this information in place, you will be better armed to move through the kernel build procedure. All right let’s go over each of the preceding points.

Understanding the Linux kernel release nomenclature

To see the kernel version number, simply run uname -r on your shell. How do you precisely interpret the output of uname -r? On our x86_64 Ubuntu distribution version 22.04 LTS guest VM, we run uname, passing the -r option switch to display just the current kernel release or version:

$ uname -r
5.19.0-40-generic

The modern Linux kernel release number nomenclature is as follows:

major#.minor#[.patchlevel][-EXTRAVERSION]

This is also often written or described as w.x[.y][-z].

The square brackets around the patchlevel and EXTRAVERSION (or the y and -z) components indicate that they are optional. The following table summarizes the meaning of the components of the release number:

Table 2.1: Linux kernel release nomenclature

So, we can now interpret our Ubuntu 22.04 LTS distribution’s kernel release number, 5.19.0-40-generic:

• Major # (or w): 5
• Minor # (or x): 19
• [patchlevel] (or y): 0
• [-EXTRAVERSION] (or -z): -40-generic

As part of an interesting exercise configuring the kernel, we will later change the localversion (aka the -EXTRAVERSION) component of the kernel we build.

Fingers-and-toes releases

Next, it’s important to understand a simple fact: with modern Linux kernels, when the kernel major and/or minor number changes, it does not imply that some tremendous or key new design, architecture, or feature has come about; no, it is simply, in the words of Linus, organic evolution.

The currently used kernel version nomenclature is a loosely time-based one, not feature-based. Thus, a new major number will pop up every so often. How often exactly? Linus likes to call it the “fingers and toes” model; when he runs out of fingers and toes to count the minor number (the x component of the w.x.y release), he updates the major number from w to w+1. Hence, after iterating over 20 minor numbers – from 0 to 19 – we end up with a new major number.

This has practically been the case since the 3.0 kernel; thus, we have the following:

• 3.0 to 3.19 (20 minor releases)
• 4.0 to 4.19 (20 minor releases)
• 5.0 to 5.19 (20 minor releases)
• 6.0 to … (it’s still moving along; you get the idea!)

Take a peek at Figure 2.1 to see this. Each minor-to-next-minor release takes approximately between 6 to 10 weeks.

Kernel development workflow – understanding the basics

Here, we provide a brief overview of the typical kernel development workflow. Anyone like you who is interested in kernel development should at least minimally understand the process.

A common misconception, especially in its baby years, was that the Linux kernel is developed in an ad hoc fashion. This is not true at all! The kernel development process has evolved to become a mostly well-oiled system with a thoroughly documented process and expectations of what a kernel contributor should know to use it well. I refer you to the preceding link for the complete details.

In order for us to take a peek into a typical development cycle, let’s assume we’ve cloned the latest mainline Linux Git kernel tree on to our system.

As mentioned earlier, at the time of writing, the 6.1 kernel is a Long-Term Stable (LTS) version with the furthest projected EOL date from now (December 2026), so we shall use it in the materials that follow.

So, how did it come to be? Obviously, it has evolved from earlier release candidate (rc) kernels and the previous stable kernel release that preceded it, which in this case would be the v6.1-rc’n’ kernels and the stable v6.0 one before it. Let’s view this evolution in two ways: via the command line and graphically via the kernel’s GitHub page.

Viewing the kernel’s Git log via the command line

We use the git log command as follows to get a human-readable log of the tags in the kernel Git tree ordered by date. Here, as we’re primarily interested in the release of the 6.1 LTS kernel, we’ve deliberately truncated the following output to highlight that portion:

$ git log --date-order --tags --simplify-by-decoration \
--pretty=format:'%ai %h %d'
2023-04-23 12:02:52 -0700 457391b03803  (tag: v6.3)
2023-04-16 15:23:53 -0700 6a8f57ae2eb0  (tag: v6.3-rc7)
2023-04-09 11:15:57 -0700 09a9639e56c0  (tag: v6.3-rc6)
2023-04-02 14:29:29 -0700 7e364e56293b  (tag: v6.3-rc5)
[ … ]
2023-03-05 14:52:03 -0800 fe15c26ee26e  (tag: v6.3-rc1)
2023-02-19 14:24:22 -0800 c9c3395d5e3d  (tag: v6.2)
2023-02-12 14:10:17 -0800 ceaa837f96ad  (tag: v6.2-rc8)
[ … ]
2022-12-25 13:41:39 -0800 1b929c02afd3  (tag: v6.2-rc1)
2022-12-11 14:15:18 -0800 830b3c68c1fb  (tag: v6.1)
2022-12-04 14:48:12 -0800 76dcd734eca2  (tag: v6.1-rc8)
2022-11-27 13:31:48 -0800 b7b275e60bcd  (tag: v6.1-rc7)
2022-11-20 16:02:16 -0800 eb7081409f94  (tag: v6.1-rc6)
2022-11-13 13:12:55 -0800 094226ad94f4  (tag: v6.1-rc5)
2022-11-06 15:07:11 -0800 f0c4d9fc9cc9  (tag: v6.1-rc4)
2022-10-30 15:19:28 -0700 30a0b95b1335  (tag: v6.1-rc3)
2022-10-23 15:27:33 -0700 247f34f7b803  (tag: v6.1-rc2)
2022-10-16 15:36:24 -0700 9abf2313adc1  (tag: v6.1-rc1)
2022-10-02 14:09:07 -0700 4fe89d07dcc2  (tag: v6.0)
2022-09-25 14:01:02 -0700 f76349cf4145  (tag: v6.0-rc7)
[ … ]
2022-08-14 15:50:18 -0700 568035b01cfb  (tag: v6.0-rc1)
2022-07-31 14:03:01 -0700 3d7cb6b04c3f  (tag: v5.19)
2022-07-24 13:26:27 -0700 e0dccc3b76fb  (tag: v5.19-rc8)
[ … ]

In the preceding output block, you can first see that, at the time I ran this git log command (late April 2023), the 6.3 kernel was just released! You can also see that seven rc kernels led up to this release, numbered as 6.3-rc1, 6.3-rc2, …, 6.3-rc7.

Delving further, we find what we’re after – you can clearly see that the stable 6.1 (LTS) kernel initial release date was 11 December 2022, and its predecessor, the 6.0 tree, was released on 2 October 2022. You can also verify these dates by looking up other useful kernel resources, such as https://kernelnewbies.org/LinuxVersions.

For the development series that ultimately led to the 6.1 kernel, this latter date (2 October 2022) marks the start of what is called the merge window for the next stable kernel for a period of approximately two weeks. In this period, developers are allowed to submit new code to the kernel tree. In reality, the actual work would have been going on from a lot earlier; the fruit of this work is now merged into mainline at this time, typically by subsystem maintainers.

We attempt to diagram a timeline of this work in Figure 2.1; you can see how the earlier kernels (from 3.0 onward) had 20 minor releases. More detail is shown for our target kernel: 6.1 LTS.

Figure 2.1: A rough timeline of modern Linux kernels, highlighting the one we’ll primarily work with (6.1 LTS)

Two weeks from the start of the merge window (2 October 2022) for the 6.1 kernel, on 16 October 2022, the merge window was closed and the rc kernel work started, with 6.1-rc1 being the first of the rc versions, of course. The -rc (also known as prepatch) trees work primarily on merging patches and fixing (regression and other) bugs, ultimately leading to what is determined by the chief maintainers (Linus Torvalds and Andrew Morton) to be a “stable” kernel tree.

The number of rc kernels or prepatches varies; typically, though, this “bugfix” window takes anywhere between 6 to 10 weeks, after which the new stable kernel is released. In the preceding output block, we can see that eight release candidate kernels (from 6.1-rc1 to 6.1-rc8) finally resulted in the stable release of the v6.1 tree on 11 December 2022, taking a total of 70 days, or 10 weeks. See the 6.x section on https://kernelnewbies.org/LinuxVersions to confirm this.

Viewing the kernel’s Git log via its GitHub page

The kernel log can be seen more visually via the releases/tags page at Linus’s GitHub tree here – https://github.com/torvalds/linux/tags:

Figure 2.2: See the v6.1 tag with the -rc’n’ release candidates that ultimately result in the v6.1 kernel seen above it (read it bottom-up)

Figure 2.2 shows us the v6.1 kernel tag in this truncated screenshot. How did we get there? Clicking on the Next button (not shown here) several times leads to the remaining pages where the v6.1-rc’n’ release candidate kernels can be spotted. Alternatively, simply navigate to https://github.com/torvalds/linux/tags?after=v6.2-rc4.

The preceding screenshot is a partial one showing how two of the various v6.1-rc’n’ release candidate kernels, 6.1-rc7 and 6.1-rc8, ultimately resulted in the release of the LTS 6.1 tree on 12 December 2022.

The work never really stops: as can be seen, by early January 2023, the v6.2-rc3 release candidate went out, ultimately resulting in the v6.2 kernel on 19 February 2023. Then, the 6.3-rc1 kernel came out on 6 March 2023, and six more followed, ultimately resulting in the release of the stable 6.3 kernel on 23 April 2023. And so it continues…

Again, by the time you’re reading this, the kernel will be well ahead. But that’s okay – the 6.1 LTS kernel will be maintained for a relatively long while (recall, the projected EOL is December 2026) and it’s thus a very significant release to products and projects!

The kernel dev workflow in a nutshell

Generically, taking the 6.x kernel series as an example, the kernel development workflow is as follows. You can simultaneously refer to Figure 2.3, where we diagram how the 6.0 kernel evolves into 6.1 LTS.

1. A 6.x stable release is made (for our purposes, consider x to be 0). Thus, the 2-week merge window for the 6.x+1 mainline kernel is opened.
2. The merge window remains open for about two weeks and new patches are merged into the mainline kernel by the various subsystem maintainers, who have been carefully accepting patches from contributors and updating their trees for a long while.
3. When around 2 weeks have elapsed, the merge window is closed.
4. Now, the “bugfix” period ensues; the rc (or mainline, prepatch) kernels start. They evolve as follows: 6.x+1-rc1, 6.x+1-rc2, ..., 6.x+1-rcn are released. This process can take anywhere between 6 to 8 weeks.
5. A “finalization” period ensues, typically about a week long. The stable release arrives and the new 6.x+1 stable kernel is released.
6. The release is handed off to the “stable team”:
   • Significant bug or security fixes result in the release of 6.x+1.y : 6.x+1.1, 6.x+1.2, ... , 6.x+1.n. We’re going to primarily work upon the 6.1.25 kernel, making the y value 25.
   • The release is maintained until the next stable release or End Of Life (EOL) date is reached, which for 6.1 LTS is projected as December 2026. Bug and security fixes will be applied right until then.

...and the whole process repeats.

Figure 2.3 : How a 6.x becomes a 6.x+1 kernel (example here particularly for how 6.0 evolves into 6.1)

So, let’s say you’re trying to submit a patch but miss the merge window. Well, there’s no help for it: you (or more likely the subsystem maintainer who’s pushing your patch series as a part of several others) will just have wait until the next merge window arrives in about 2.5 to 3 months. Hey, that’s the way it is; we aren’t in that great a rush.

Exercise

Go through what we did – following the kernel’s evolution – for the current latest stable kernel. (Quick tip: one more way to see the history of kernel releases: https://en.wikipedia.org/wiki/Linux_kernel_version_history).

So, when you now see Linux kernel releases, the names and the process involved will make sense. Let’s now move on to looking at the different types of kernel source trees out there.

Exploring the types of kernel source trees

There are several types of Linux kernel source trees. A key one is the Long Term Support (LTS) kernel. It’s simply a “special” release in the sense that the kernel maintainers will continue to backport important bugs and security fixes upon it until a given EOL date. By convention, the next kernel to be “marked” as an LTS release is the last one released each year, typically in December.

The “life” of an LTS kernel will usually be a minimum of 2 years, and it can be extended to go for several more. The 6.1.y LTS kernel that we will use throughout this book is the 23^rd LTS kernel and has a projected lifespan of 4 years – from December 2022 to December 2026.

This image snippet from the Wikipedia page on Linux kernel version history (https://en.wikipedia.org/wiki/Linux_kernel_version_history) says it all:

Figure 2.4: A look at the dev, supported (stable), and stable LTS kernels and their EOL dates; 6.1 LTS is what we work with (image credit: Wikipedia)

Interestingly, the 5.4 LTS kernel will be maintained until December 2025, and 5.10 LTS will be maintained for the same period as 6.1 LTS, up to December 2026.

Note, though, that there are several types of release kernels in the repository. Here, we mention an incomplete list, ordered from least to most stable (thus, their lifespan is from the shortest to longest time):

• -next trees: This is indeed the bleeding edge (the tip of the arrow!), subsystem trees with new patches collected here for testing and review. This is what an upstream kernel contributor will work on. If you intend to upstream your patches to the kernel (contribute), you must work with the latest -next tree.
• Prepatches, also known as -rc or mainline: These are release candidate kernels that get generated prior to a release.
• Stable kernels: As the name implies, this is the business end. These kernels are typically picked up by distributions and other projects (at least to begin with). They are also known as vanilla kernels.
• Distribution and LTS kernels: Distribution kernels are (obviously) the kernels provided by the distributions. They typically begin with a base vanilla/stable kernel. LTS kernels are the specially-maintained-for-a-longer-while kernels, making them especially useful for industry/production projects and products. Especially with regard to enterprise class distros, you’ll often find that many of them seem to be using “old” kernels. This can even be the case with some Android vendors. Now, even if uname -r shows that the kernel version is, say, 4.x based, it does not necessarily imply it’s old. No, the distro/vendor/OEM typically has a kernel engineering team (or outsources to one) that periodically updates the old kernel with new and relevant patches, especially critical security and bug fixes. So, though the version might appear outdated, it isn’t necessarily the case!

  Note, though, that this entails a huge workload on the kernel engineering team’s part, and it still is very difficult to keep up with the latest stable kernel; thus, it’s really best to work with vanilla kernels.

Two more types of kernel source trees require a mention here: Super LTS (SLTS) and chip (SoC) vendor kernels. SLTS kernels are maintained for even longer durations than the LTS kernels, by the Civil Infrastructure Platform (https://www.cip-project.org/), a Linux Foundation project. Quoting from their site:

”... The CIP project is focused on establishing an open source “base layer” of industrial grade software to enable the use and implementation of software building blocks in civil infrastructure projects. Currently, civil infrastructure systems are built from the ground up, with little re-use of existing software building blocks. The CIP project intends to create reusable building blocks that meet the safety, reliability, and other requirements of industrial and civil infrastructure.”

In fact, as of this writing, the latest CIP kernels – SLTS v6.1 and SLTS v6.1-rt – are based on the 6.1 LTS kernel and their projected EOL date is August 2033 (10 years)! As well, the SLTS 5.10 and SLTS 5.10-rt kernels have a projected EOL of January 2031. See the Wiki site here for the latest info: https://wiki.linuxfoundation.org/civilinfrastructureplatform/start#kernel_maintainership.

LTS kernels – the new mandate

A quick and important update! In September 2023 at the Open Source Summit in Bilbao, Spain, Jonathan Corbet, in his famous “kernel report” talk, made an important announcement: LTS kernels will from now on be maintained for a period of only two years.

This might come as something of a surprise to many. Why only two years? Briefly, two reasons were given:

• First, why maintain a kernel series for many years when people aren’t really using them? Many enterprise kernel vendors, as well as SoC ones, maintain their own kernels.
• Next, an unfortunate and serious issue: maintainer fatigue. It’s hard for the kernel community to keep all these LTS kernels – there are 7 major LTS versions as of now (4.14, 4.19, 5.4, 5.10, 5.15, 6.1, and 6.6) – continuously maintained! Furthermore, the burden only increases with time. For example, the 4.14 LTS series has had about 300 updates and close to 28,000 commits. Besides, old bugs eventually surface, and a lot of work ensues to fix them in modern LTS kernels. Not just that, but new bugs that surface in later kernels must be fixed and then back-ported to the older still-maintained LTS kernels. It all adds up.

As of this time, it appears that the kernel we work with here – 6.1 LTS – will be maintained for the usual time, until December 2026.

Moving along, let’s now briefly discuss the aforementioned second type of kernel source tree: silicon/chipset SoC vendor kernels tend to maintain their own kernels for various boards/silicon that they support. They typically base their kernel on an existing vanilla LTS kernel (not necessarily the latest one!) and then build upon it, adding their vendor-specific patches, Board Support Package (BSP) stuff, drivers, and so on. Of course, as time goes by, the differences – between their kernel and the latest stable one – can become quite significant, leading to difficult maintenance issues, like the need to constantly backport critical security/bugfix patches to them.

When on a project using such silicon, the perhaps-best approach is to base your work on existing industry-strength solutions like the Yocto Project (https://www.yoctoproject.org/), which does a great job in keeping recent LTS kernels maintained with vendor layers applied in sync with key security/bugfix patches. For example, as of this writing, the latest stable Yocto release – Nanbield 4.3 – supports both the 6.1 LTS as well as the more recent 6.5 non-LTS kernel; of course, the particular version can vary with the architecture (processor family).

So, the types of kernel source trees out there are aplenty. Nevertheless, I refer you to kernel.org’s Releases page to obtain details on the type of release kernels: https://www.kernel.org/releases.html. Again, for even more detail, visit How the development process works (https://www.kernel.org/doc/html/latest/process/2.Process.html#how-the-development-process-works).

Querying the repository, https://www.kernel.org/, in a non-interactive scriptable fashion can be done using curl. The following output is the state of Linux as of 06 December 2023:

$ curl -L https://www.kernel.org/finger_banner
The latest stable version of the Linux kernel is:             6.6.4
The latest mainline version of the Linux kernel is:           6.7-rc4
The latest stable 6.6 version of the Linux kernel is:         6.6.4
The latest stable 6.5 version of the Linux kernel is:         6.5.13 (EOL)
The latest longterm 6.1 version of the Linux kernel is:       6.1.65
The latest longterm 5.15 version of the Linux kernel is:      5.15.141
The latest longterm 5.10 version of the Linux kernel is:      5.10.202
The latest longterm 5.4 version of the Linux kernel is:       5.4.262
The latest longterm 4.19 version of the Linux kernel is:      4.19.300
The latest longterm 4.14 version of the Linux kernel is:      4.14.331
The latest linux-next version of the Linux kernel is:         next-20231206
$ 

By the time you read this, it’s extremely likely – certain, in fact – that the kernel has evolved further, and later versions show up. For a book such as this, the best I can do is pick close to the latest stable LTS kernel with the longest projected EOL date at the time of writing: 6.1.x LTS.

To help demonstrate this point, note that the first edition of this book used the 5.4.0 kernel, as 5.4 was the LTS kernel series with the longest lifetime at the time of its writing. Today, as I pen this, 5.4 LTS is still maintained with a projected EOL in December 2025 and the latest stable version is 5.4.268.

Which kernel should I run?

So, with all the types and types of kernels we’ve seen, it really does beg the question, which kernel should I run? The answer is nuanced, since it depends on the environment. Is it for embedded, desktop or server usage? Is it for a new project or a legacy one? What’s the intended maintenance period? Is it an SoC kernel that is maintained by a vendor? What’s the security requirement? Still, the “right answer,” straight from the mouths of senior kernel maintainers, is this:

Run the latest stable update. That is the most stable, the most secure, the best kernel we know how to create at this time. That’s the very best we can do. You should run that.

– Jon Corbet, September 2023

You have to take all of the stable/LTS releases in order to have a secure and stable system. If you attempt to cherry-pick random patches you will not fix all of the known, and unknown, problems, but rather you will end up with a potentially more insecure system, and one that contains known bugs.

– Greg Kroah-Hartman

Greg Kroah-Hartman’s practical blog article What Stable Kernel Should I Use, August 2018 (http://kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/), echoes these thoughts.

Right, now that we’re armed with the knowledge of kernel version nomenclature and types of kernel source trees, it’s definitely time to begin our journey of building our kernel.

As a convenient and quick reference, the following are the main, key steps required to build a Linux kernel from source. As the explanation for each of them is pretty detailed, you can refer back to this summary to see the big picture. The steps are as follows:

1. Obtain a Linux kernel source tree through either of the following options:
   • Downloading a specific kernel source tree as a compressed file
   • Cloning a (kernel) Git tree
2. Extract the kernel source tree into some location in your home directory (skip this step if you obtained a kernel by cloning a Git tree).
3. Configure: Get a starting point for your kernel config (the approach varies). Then edit it, selecting the kernel support options as required for the new kernel. The recommended way of doing this is with make menuconfig.
4. Build the kernel image, the loadable modules, and any required Device Tree Blobs (DTBs) with make [-j'n'] all. This builds the compressed kernel image (arch/<arch>/boot/[b|z|u]{Ii}mage), the uncompressed kernel image – vmlinux, the System.map file, the kernel module objects, and any configured DTB files.
5. Install the just-built kernel modules (on x86) with sudo make [INSTALL_MOD_PATH=<prefix-dir>] modules_install. This step installs kernel modules by default under /lib/modules/$(uname -r)/ (the INSTALL_MOD_PATH environment variable can be leveraged to change this).
6. Bootloader (x86): Set up the GRUB bootloader and the initramfs, earlier called initrd, image:sudo make [INSTALL_PATH=</new/boot/dir>] install
   • This creates and installs the initramfs or initrd image under /boot (the INSTALL_PATH environment variable can be leveraged to change this).
   • It updates the bootloader configuration file to boot the new kernel (first entry).
7. Customize the GRUB bootloader menu (optional).

This chapter, being the first of two on this kernel build topic, will cover steps 1 to 3, with a lot of required background material thrown in as well. The next chapter will cover the remaining steps, 4 to 7. So, let’s begin with step 1.