Preliminaries for the kernel build

It’s important to understand a few things right from the outset that will help you as we proceed on our journey of building and working with a Linux kernel. Firstly, the Linux kernel and its sister projects are completely decentralized – it’s a virtual, online open-source community! The closest we come to an “office” for Linux is this: stewardship of the Linux kernel (as well as several dozen related projects) is in the capable hands of the Linux Foundation (https://linuxfoundation.org/); further, it manages the Linux Kernel Organization, a private foundation that distributes the Linux kernel to the public free of charge (https://www.kernel.org/nonprofit.html).

Did you know? The terms of the GNU GPLv2 license – under which the Linux kernel’s released and will continue to be held for the foreseeable future – do not in any manner prevent original developers from charging for their work! It’s just that Linus Torvalds, and now the Linux Foundation, makes the Linux kernel software available to everybody for free. This doesn’t prevent commercial organizations from adding value to the kernel (and their products bundled with it) and charging for it, whether via an initial upfront charge or today’s typical SaaS/IaaS subscription model.

Thus, open source is definitely viable for business, as has been, and continues to be, proved daily. Customers whose core business lies elsewhere simply want value for money; businesses built around Linux can provide that by providing the customer with expert-level support, detailed SLAs, and upgrades.

Some of the key points we’ll discuss in this section include the following:

• The kernel release, or version number nomenclature
• The typical kernel development workflow
• The existence of different types of kernel source trees within the repository

With this information in place, you will be better armed to move through the kernel build procedure. All right let’s go over each of the preceding points.

Understanding the Linux kernel release nomenclature

To see the kernel version number, simply run uname -r on your shell. How do you precisely interpret the output of uname -r? On our x86_64 Ubuntu distribution version 22.04 LTS guest VM, we run uname, passing the -r option switch to display just the current kernel release or version:

$ uname -r
5.19.0-40-generic

Of course, by the time you read this, the Ubuntu 22.04 LTS kernel will very likely have been upgraded to a later release; that’s perfectly normal. The 5.19.0-40-generic kernel was the one I encountered with the Ubuntu 22.04.2 LTS at the time of writing this chapter.

The modern Linux kernel release number nomenclature is as follows:

major#.minor#[.patchlevel][-EXTRAVERSION]

This is also often written or described as w.x[.y][-z].

The square brackets around the patchlevel and EXTRAVERSION (or the y and -z) components indicate that they are optional. The following table summarizes the meaning of the components of the release number:

Release # component	Meaning	Example numbers
Major # (or w)	Main or major number; currently, we are on the 6.x kernel series, thus the major number is 6.	2, 3, 4, 5, 6
Minor # (or x)	The minor number, hierarchically under the major number.	0 onward
[patchlevel] (or y)	Hierarchically under the minor number – also called the ABI or revision – applied on occasion to the stable kernel when significant bug/security fixes are required.	0 onward
[-EXTRAVERSION] (or -z)	Also called localversion; typically used by distribution kernels and vendors to track their internal changes.	Varies; Ubuntu uses w.x.y-<z>-generic

Table 2.1: Linux kernel release nomenclature

So, we can now interpret our Ubuntu 22.04 LTS distribution’s kernel release number, 5.19.0-40-generic:

• Major # (or w): 5
• Minor # (or x): 19
• [patchlevel] (or y): 0
• [-EXTRAVERSION] (or -z): -40-generic

Note that distribution kernels may not precisely follow these conventions; it’s up to them. The regular or vanilla kernels released on https://www.kernel.org/ do follow these conventions (at least until Linus decides to change them).

Historically, in kernels before 2.6 (IOW, ancient stuff now), the minor number held a special meaning; if it was an even number, it indicated a stable kernel release, and if odd, an unstable or beta release. This is no longer the case.

As part of an interesting exercise configuring the kernel, we will later change the localversion (aka the -EXTRAVERSION) component of the kernel we build.

Fingers-and-toes releases

Next, it’s important to understand a simple fact: with modern Linux kernels, when the kernel major and/or minor number changes, it does not imply that some tremendous or key new design, architecture, or feature has come about; no, it is simply, in the words of Linus, organic evolution.

The currently used kernel version nomenclature is a loosely time-based one, not feature-based. Thus, a new major number will pop up every so often. How often exactly? Linus likes to call it the “fingers and toes” model; when he runs out of fingers and toes to count the minor number (the x component of the w.x.y release), he updates the major number from w to w+1. Hence, after iterating over 20 minor numbers – from 0 to 19 – we end up with a new major number.

This has practically been the case since the 3.0 kernel; thus, we have the following:

• 3.0 to 3.19 (20 minor releases)
• 4.0 to 4.19 (20 minor releases)
• 5.0 to 5.19 (20 minor releases)
• 6.0 to … (it’s still moving along; you get the idea!)

Take a peek at Figure 2.1 to see this. Each minor-to-next-minor release takes approximately between 6 to 10 weeks.

Kernel development workflow – understanding the basics

Here, we provide a brief overview of the typical kernel development workflow. Anyone like you who is interested in kernel development should at least minimally understand the process.

A detailed description can be found within the official kernel documentation here: https://www.kernel.org/doc/html/latest/process/2.Process.html#how-the-development-process-works.

A common misconception, especially in its baby years, was that the Linux kernel is developed in an ad hoc fashion. This is not true at all! The kernel development process has evolved to become a mostly well-oiled system with a thoroughly documented process and expectations of what a kernel contributor should know to use it well. I refer you to the preceding link for the complete details.

In order for us to take a peek into a typical development cycle, let’s assume we’ve cloned the latest mainline Linux Git kernel tree on to our system.

The details regarding the use of the powerful Git Source Code Management (SCM) tool lie beyond the scope of this book. Please see the Further reading section for useful links on learning how to use Git. Obviously, I highly recommend gaining at least basic familiarity with using Git.

As mentioned earlier, at the time of writing, the 6.1 kernel is a Long-Term Stable (LTS) version with the furthest projected EOL date from now (December 2026), so we shall use it in the materials that follow.

So, how did it come to be? Obviously, it has evolved from earlier release candidate (rc) kernels and the previous stable kernel release that preceded it, which in this case would be the v6.1-rc’n’ kernels and the stable v6.0 one before it. Let’s view this evolution in two ways: via the command line and graphically via the kernel’s GitHub page.

Viewing the kernel’s Git log via the command line

We use the git log command as follows to get a human-readable log of the tags in the kernel Git tree ordered by date. Here, as we’re primarily interested in the release of the 6.1 LTS kernel, we’ve deliberately truncated the following output to highlight that portion:

The git log command (that we use in the following code block, and in fact any other git sub-commands) will only work on a Git tree. We use the following one purely to demonstrate the evolution of the kernel. A bit later, we will show how you can clone a Git kernel source tree.

$ git log --date-order --tags --simplify-by-decoration \
--pretty=format:'%ai %h %d'
2023-04-23 12:02:52 -0700 457391b03803  (tag: v6.3)
2023-04-16 15:23:53 -0700 6a8f57ae2eb0  (tag: v6.3-rc7)
2023-04-09 11:15:57 -0700 09a9639e56c0  (tag: v6.3-rc6)
2023-04-02 14:29:29 -0700 7e364e56293b  (tag: v6.3-rc5)
[ … ]
2023-03-05 14:52:03 -0800 fe15c26ee26e  (tag: v6.3-rc1)
2023-02-19 14:24:22 -0800 c9c3395d5e3d  (tag: v6.2)
2023-02-12 14:10:17 -0800 ceaa837f96ad  (tag: v6.2-rc8)
[ … ]
2022-12-25 13:41:39 -0800 1b929c02afd3  (tag: v6.2-rc1)
2022-12-11 14:15:18 -0800 830b3c68c1fb  (tag: v6.1)
2022-12-04 14:48:12 -0800 76dcd734eca2  (tag: v6.1-rc8)
2022-11-27 13:31:48 -0800 b7b275e60bcd  (tag: v6.1-rc7)
2022-11-20 16:02:16 -0800 eb7081409f94  (tag: v6.1-rc6)
2022-11-13 13:12:55 -0800 094226ad94f4  (tag: v6.1-rc5)
2022-11-06 15:07:11 -0800 f0c4d9fc9cc9  (tag: v6.1-rc4)
2022-10-30 15:19:28 -0700 30a0b95b1335  (tag: v6.1-rc3)
2022-10-23 15:27:33 -0700 247f34f7b803  (tag: v6.1-rc2)
2022-10-16 15:36:24 -0700 9abf2313adc1  (tag: v6.1-rc1)
2022-10-02 14:09:07 -0700 4fe89d07dcc2  (tag: v6.0)
2022-09-25 14:01:02 -0700 f76349cf4145  (tag: v6.0-rc7)
[ … ]
2022-08-14 15:50:18 -0700 568035b01cfb  (tag: v6.0-rc1)
2022-07-31 14:03:01 -0700 3d7cb6b04c3f  (tag: v5.19)
2022-07-24 13:26:27 -0700 e0dccc3b76fb  (tag: v5.19-rc8)
[ … ]

In the preceding output block, you can first see that, at the time I ran this git log command (late April 2023), the 6.3 kernel was just released! You can also see that seven rc kernels led up to this release, numbered as 6.3-rc1, 6.3-rc2, …, 6.3-rc7.

Delving further, we find what we’re after – you can clearly see that the stable 6.1 (LTS) kernel initial release date was 11 December 2022, and its predecessor, the 6.0 tree, was released on 2 October 2022. You can also verify these dates by looking up other useful kernel resources, such as https://kernelnewbies.org/LinuxVersions.

For the development series that ultimately led to the 6.1 kernel, this latter date (2 October 2022) marks the start of what is called the merge window for the next stable kernel for a period of approximately two weeks. In this period, developers are allowed to submit new code to the kernel tree. In reality, the actual work would have been going on from a lot earlier; the fruit of this work is now merged into mainline at this time, typically by subsystem maintainers.

We attempt to diagram a timeline of this work in Figure 2.1; you can see how the earlier kernels (from 3.0 onward) had 20 minor releases. More detail is shown for our target kernel: 6.1 LTS.

Figure 2.1: A rough timeline of modern Linux kernels, highlighting the one we’ll primarily work with (6.1 LTS)

Two weeks from the start of the merge window (2 October 2022) for the 6.1 kernel, on 16 October 2022, the merge window was closed and the rc kernel work started, with 6.1-rc1 being the first of the rc versions, of course. The -rc (also known as prepatch) trees work primarily on merging patches and fixing (regression and other) bugs, ultimately leading to what is determined by the chief maintainers (Linus Torvalds and Andrew Morton) to be a “stable” kernel tree.

The number of rc kernels or prepatches varies; typically, though, this “bugfix” window takes anywhere between 6 to 10 weeks, after which the new stable kernel is released. In the preceding output block, we can see that eight release candidate kernels (from 6.1-rc1 to 6.1-rc8) finally resulted in the stable release of the v6.1 tree on 11 December 2022, taking a total of 70 days, or 10 weeks. See the 6.x section on https://kernelnewbies.org/LinuxVersions to confirm this.

Why does Figure 2.1 begin at the 2.6.12 kernel? The answer is simple: this is the first version from which Git kernel history is maintained.

Another question may be, why show the 5.4 (LTS) kernel in the figure? Because it too is an LTS kernel (projected EOL is December 2025), and it was the kernel we used in the first edition of this book!

Viewing the kernel’s Git log via its GitHub page

The kernel log can be seen more visually via the releases/tags page at Linus’s GitHub tree here – https://github.com/torvalds/linux/tags:

Figure 2.2: See the v6.1 tag with the -rc’n’ release candidates that ultimately result in the v6.1 kernel seen above it (read it bottom-up)

Figure 2.2 shows us the v6.1 kernel tag in this truncated screenshot. How did we get there? Clicking on the Next button (not shown here) several times leads to the remaining pages where the v6.1-rc’n’ release candidate kernels can be spotted. Alternatively, simply navigate to https://github.com/torvalds/linux/tags?after=v6.2-rc4.

The preceding screenshot is a partial one showing how two of the various v6.1-rc’n’ release candidate kernels, 6.1-rc7 and 6.1-rc8, ultimately resulted in the release of the LTS 6.1 tree on 12 December 2022.

The work never really stops: as can be seen, by early January 2023, the v6.2-rc3 release candidate went out, ultimately resulting in the v6.2 kernel on 19 February 2023. Then, the 6.3-rc1 kernel came out on 6 March 2023, and six more followed, ultimately resulting in the release of the stable 6.3 kernel on 23 April 2023. And so it continues…

Again, by the time you’re reading this, the kernel will be well ahead. But that’s okay – the 6.1 LTS kernel will be maintained for a relatively long while (recall, the projected EOL is December 2026) and it’s thus a very significant release to products and projects!

The kernel dev workflow in a nutshell

Generically, taking the 6.x kernel series as an example, the kernel development workflow is as follows. You can simultaneously refer to Figure 2.3, where we diagram how the 6.0 kernel evolves into 6.1 LTS.

1. A 6.x stable release is made (for our purposes, consider x to be 0). Thus, the 2-week merge window for the 6.x+1 mainline kernel is opened.
2. The merge window remains open for about two weeks and new patches are merged into the mainline kernel by the various subsystem maintainers, who have been carefully accepting patches from contributors and updating their trees for a long while.
3. When around 2 weeks have elapsed, the merge window is closed.
4. Now, the “bugfix” period ensues; the rc (or mainline, prepatch) kernels start. They evolve as follows: 6.x+1-rc1, 6.x+1-rc2, ..., 6.x+1-rcn are released. This process can take anywhere between 6 to 8 weeks.
5. A “finalization” period ensues, typically about a week long. The stable release arrives and the new 6.x+1 stable kernel is released.
6. The release is handed off to the “stable team”:
   • Significant bug or security fixes result in the release of 6.x+1.y : 6.x+1.1, 6.x+1.2, ... , 6.x+1.n. We’re going to primarily work upon the 6.1.25 kernel, making the y value 25.
   • The release is maintained until the next stable release or End Of Life (EOL) date is reached, which for 6.1 LTS is projected as December 2026. Bug and security fixes will be applied right until then.

...and the whole process repeats.

Figure 2.3 : How a 6.x becomes a 6.x+1 kernel (example here particularly for how 6.0 evolves into 6.1)

So, let’s say you’re trying to submit a patch but miss the merge window. Well, there’s no help for it: you (or more likely the subsystem maintainer who’s pushing your patch series as a part of several others) will just have wait until the next merge window arrives in about 2.5 to 3 months. Hey, that’s the way it is; we aren’t in that great a rush.

Exercise

Go through what we did – following the kernel’s evolution – for the current latest stable kernel. (Quick tip: one more way to see the history of kernel releases: https://en.wikipedia.org/wiki/Linux_kernel_version_history).

So, when you now see Linux kernel releases, the names and the process involved will make sense. Let’s now move on to looking at the different types of kernel source trees out there.

Exploring the types of kernel source trees

There are several types of Linux kernel source trees. A key one is the Long Term Support (LTS) kernel. It’s simply a “special” release in the sense that the kernel maintainers will continue to backport important bugs and security fixes upon it until a given EOL date. By convention, the next kernel to be “marked” as an LTS release is the last one released each year, typically in December.

The “life” of an LTS kernel will usually be a minimum of 2 years, and it can be extended to go for several more. The 6.1.y LTS kernel that we will use throughout this book is the 23^rd LTS kernel and has a projected lifespan of 4 years – from December 2022 to December 2026.

This image snippet from the Wikipedia page on Linux kernel version history (https://en.wikipedia.org/wiki/Linux_kernel_version_history) says it all:

Figure 2.4: A look at the dev, supported (stable), and stable LTS kernels and their EOL dates; 6.1 LTS is what we work with (image credit: Wikipedia)

Interestingly, the 5.4 LTS kernel will be maintained until December 2025, and 5.10 LTS will be maintained for the same period as 6.1 LTS, up to December 2026.

Note, though, that there are several types of release kernels in the repository. Here, we mention an incomplete list, ordered from least to most stable (thus, their lifespan is from the shortest to longest time):

• -next trees: This is indeed the bleeding edge (the tip of the arrow!), subsystem trees with new patches collected here for testing and review. This is what an upstream kernel contributor will work on. If you intend to upstream your patches to the kernel (contribute), you must work with the latest -next tree.
• Prepatches, also known as -rc or mainline: These are release candidate kernels that get generated prior to a release.
• Stable kernels: As the name implies, this is the business end. These kernels are typically picked up by distributions and other projects (at least to begin with). They are also known as vanilla kernels.
• Distribution and LTS kernels: Distribution kernels are (obviously) the kernels provided by the distributions. They typically begin with a base vanilla/stable kernel. LTS kernels are the specially-maintained-for-a-longer-while kernels, making them especially useful for industry/production projects and products. Especially with regard to enterprise class distros, you’ll often find that many of them seem to be using “old” kernels. This can even be the case with some Android vendors. Now, even if uname -r shows that the kernel version is, say, 4.x based, it does not necessarily imply it’s old. No, the distro/vendor/OEM typically has a kernel engineering team (or outsources to one) that periodically updates the old kernel with new and relevant patches, especially critical security and bug fixes. So, though the version might appear outdated, it isn’t necessarily the case!

  Note, though, that this entails a huge workload on the kernel engineering team’s part, and it still is very difficult to keep up with the latest stable kernel; thus, it’s really best to work with vanilla kernels.

In this book, we will work throughout with one of the latest LTS kernels with the longest lifetime.

As of the time of writing, it’s the 6.1.x LTS kernel, with a projected EOL date of December 2026, thus keeping this book’s content current and valid for years to come!

Two more types of kernel source trees require a mention here: Super LTS (SLTS) and chip (SoC) vendor kernels. SLTS kernels are maintained for even longer durations than the LTS kernels, by the Civil Infrastructure Platform (https://www.cip-project.org/), a Linux Foundation project. Quoting from their site:

”... The CIP project is focused on establishing an open source “base layer” of industrial grade software to enable the use and implementation of software building blocks in civil infrastructure projects. Currently, civil infrastructure systems are built from the ground up, with little re-use of existing software building blocks. The CIP project intends to create reusable building blocks that meet the safety, reliability, and other requirements of industrial and civil infrastructure.”

In fact, as of this writing, the latest CIP kernels – SLTS v6.1 and SLTS v6.1-rt – are based on the 6.1 LTS kernel and their projected EOL date is August 2033 (10 years)! As well, the SLTS 5.10 and SLTS 5.10-rt kernels have a projected EOL of January 2031. See the Wiki site here for the latest info: https://wiki.linuxfoundation.org/civilinfrastructureplatform/start#kernel_maintainership.

LTS kernels – the new mandate

A quick and important update! In September 2023 at the Open Source Summit in Bilbao, Spain, Jonathan Corbet, in his famous “kernel report” talk, made an important announcement: LTS kernels will from now on be maintained for a period of only two years.

Here’s some resources, in case you’d like to look into it for yourself:

• The Kernel Report, Jon Corbet, September 2023, OSS EU, Spain; YouTube: https://osseu2023.sched.com/event/4a09976cdb50e939601160ecdb45bc05#video_stream; Slides (PDF): https://lwn.net/talks/2023/kr-osseu.pdf
• Long-term support for Linux kernels is about to get a lot shorter, Liam Proven, The Register, Sept 2023: https://www.theregister.com/2023/09/26/linux_kernel_report_2023/

This might come as something of a surprise to many. Why only two years? Briefly, two reasons were given:

• First, why maintain a kernel series for many years when people aren’t really using them? Many enterprise kernel vendors, as well as SoC ones, maintain their own kernels.
• Next, an unfortunate and serious issue: maintainer fatigue. It’s hard for the kernel community to keep all these LTS kernels – there are 7 major LTS versions as of now (4.14, 4.19, 5.4, 5.10, 5.15, 6.1, and 6.6) – continuously maintained! Furthermore, the burden only increases with time. For example, the 4.14 LTS series has had about 300 updates and close to 28,000 commits. Besides, old bugs eventually surface, and a lot of work ensues to fix them in modern LTS kernels. Not just that, but new bugs that surface in later kernels must be fixed and then back-ported to the older still-maintained LTS kernels. It all adds up.

As of this time, it appears that the kernel we work with here – 6.1 LTS – will be maintained for the usual time, until December 2026.

Moving along, let’s now briefly discuss the aforementioned second type of kernel source tree: silicon/chipset SoC vendor kernels tend to maintain their own kernels for various boards/silicon that they support. They typically base their kernel on an existing vanilla LTS kernel (not necessarily the latest one!) and then build upon it, adding their vendor-specific patches, Board Support Package (BSP) stuff, drivers, and so on. Of course, as time goes by, the differences – between their kernel and the latest stable one – can become quite significant, leading to difficult maintenance issues, like the need to constantly backport critical security/bugfix patches to them.

When on a project using such silicon, the perhaps-best approach is to base your work on existing industry-strength solutions like the Yocto Project (https://www.yoctoproject.org/), which does a great job in keeping recent LTS kernels maintained with vendor layers applied in sync with key security/bugfix patches. For example, as of this writing, the latest stable Yocto release – Nanbield 4.3 – supports both the 6.1 LTS as well as the more recent 6.5 non-LTS kernel; of course, the particular version can vary with the architecture (processor family).

So, the types of kernel source trees out there are aplenty. Nevertheless, I refer you to kernel.org’s Releases page to obtain details on the type of release kernels: https://www.kernel.org/releases.html. Again, for even more detail, visit How the development process works (https://www.kernel.org/doc/html/latest/process/2.Process.html#how-the-development-process-works).

Querying the repository, https://www.kernel.org/, in a non-interactive scriptable fashion can be done using curl. The following output is the state of Linux as of 06 December 2023:

$ curl -L https://www.kernel.org/finger_banner
The latest stable version of the Linux kernel is:             6.6.4
The latest mainline version of the Linux kernel is:           6.7-rc4
The latest stable 6.6 version of the Linux kernel is:         6.6.4
The latest stable 6.5 version of the Linux kernel is:         6.5.13 (EOL)
The latest longterm 6.1 version of the Linux kernel is:       6.1.65
The latest longterm 5.15 version of the Linux kernel is:      5.15.141
The latest longterm 5.10 version of the Linux kernel is:      5.10.202
The latest longterm 5.4 version of the Linux kernel is:       5.4.262
The latest longterm 4.19 version of the Linux kernel is:      4.19.300
The latest longterm 4.14 version of the Linux kernel is:      4.14.331
The latest linux-next version of the Linux kernel is:         next-20231206
$ 

By the time you read this, it’s extremely likely – certain, in fact – that the kernel has evolved further, and later versions show up. For a book such as this, the best I can do is pick close to the latest stable LTS kernel with the longest projected EOL date at the time of writing: 6.1.x LTS.

Of course, it’s happened already! The 6.6 kernel was released on 29 October 2023 and, as of the time of writing (just before going to print), it has in fact been marked as an LTS kernel, with a projected EOL date of December 2026 (the same as that of 6.1 LTS).

To help demonstrate this point, note that the first edition of this book used the 5.4.0 kernel, as 5.4 was the LTS kernel series with the longest lifetime at the time of its writing. Today, as I pen this, 5.4 LTS is still maintained with a projected EOL in December 2025 and the latest stable version is 5.4.268.

Which kernel should I run?

So, with all the types and types of kernels we’ve seen, it really does beg the question, which kernel should I run? The answer is nuanced, since it depends on the environment. Is it for embedded, desktop or server usage? Is it for a new project or a legacy one? What’s the intended maintenance period? Is it an SoC kernel that is maintained by a vendor? What’s the security requirement? Still, the “right answer,” straight from the mouths of senior kernel maintainers, is this:

Run the latest stable update. That is the most stable, the most secure, the best kernel we know how to create at this time. That’s the very best we can do. You should run that.

– Jon Corbet, September 2023

Tip: Point your browser to https://kernel.org/; do you see the big yellow button with the kernel release number inside it? That’s the latest stable kernel as of today.

You have to take all of the stable/LTS releases in order to have a secure and stable system. If you attempt to cherry-pick random patches you will not fix all of the known, and unknown, problems, but rather you will end up with a potentially more insecure system, and one that contains known bugs.

– Greg Kroah-Hartman

Greg Kroah-Hartman’s practical blog article What Stable Kernel Should I Use, August 2018 (http://kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/), echoes these thoughts.

Right, now that we’re armed with the knowledge of kernel version nomenclature and types of kernel source trees, it’s definitely time to begin our journey of building our kernel.