You will need a modern desktop PC or laptop. Ubuntu Desktop specifies the following as "recommended system requirements" for the installation and usage of the distribution:

• A 2 GHz dual core processor or better.
• RAM:
  • Running on physical host: 2 GB or more system memory (more will certainly help).
  • Running as a guest VM: The host system should have at least 4 GB RAM (the more the better and the smoother the experience).
• 25 GB of free hard drive space (I suggest more, at least double this).
• Either a DVD drive or a USB port for the installer media (not required when setting up Ubuntu as a guest VM).
• Internet access is definitely helpful and required at times.

As performing tasks such as building a Linux kernel from source is a very memory- and CPU-intensive process, I highly recommend that you try it out on a powerful Linux system with plenty of RAM and disk space to spare as well. It should be pretty obvious – the more RAM and CPU power the host system has, the better!

Like any seasoned kernel contributor, I would say that working on a native Linux system is best. However, for the purposes of this book, we cannot assume that you will always have a dedicated native Linux box available to you. So, we will assume that you are working on a Linux guest. Working within a guest VM also adds an additional layer of isolation and thus safety. 

Cloning our code repository: The complete source code for this book is freely available on GitHub at https://github.com/PacktPublishing/Linux-Kernel-Programming. You can clone and work on it by cloning the git tree, like so:

git clone https://github.com/PacktPublishing/Linux-Kernel-Programming.git

The source code is organized chapter-wise. Each chapter is represented as a directory – for example, ch1/ has the source code for this chapter. The root of the source tree has some code that is common to all chapters, such as the source files convenient.h, klib_llkd.c, as well as others.

For efficient code browsing, I would strongly recommend that you always index the code base with ctags(1) and/or cscope(1). For example, to set up the ctags index, just cd to the root of the source tree and type ctags -R .

As discussed previously, a practical and convenient alternative to using a native Linux system is to install and use the Linux distribution as a guest OS on a VM. It's key that you install a recent Linux distribution, preferably as a VM to be safe and avoid unpleasant data loss or other surprises. The fact is when working at the level of the kernel, abruptly crashing the system (and the data loss risks that arise thereof) is actually a commonplace occurrence. I recommend using Oracle VirtualBox 6.x (or the latest stable version) or other virtualization software, such as VMware Workstation.

The host system should be either MS Windows 10 or later (of course, even Windows 7 will work), a recent Linux distribution (for example, Ubuntu or Fedora), or macOS. So, let's get started by installing our Linux guest.

Here, I won't delve into the minutiae of installing Linux as a guest on Oracle VirtualBox, the reason being that this installation is not directly related to Linux kernel development. There are many ways to set up a Linux VM; we really don't want to get into the details and the pros and cons of each of them here.

But if you are not familiar with this, don't worry. For your convenience, here are some excellent resources that will help you out:

• A very clearly written tutorial entitled Install Linux Inside Windows Using VirtualBox by Abhishek Prakash (It's FOSS!, August 2019): https://itsfoss.com/install-linux-in-virtualbox/.

• An alternate, similarly excellent resource is Install Ubuntu on Oracle VirtualBox: https://brb.nci.nih.gov/seqtools/installUbuntu.html.
  

Also, you can look up useful resources for installing a Linux guest on VirtualBox in the Further reading section at the end of this chapter. 

While you install the Linux VM, keep the following things in mind.

Installing a 64-bit Linux guest requires that CPU virtualization extension support (Intel VT-x or AMD-SV) be turned on within the host system's basic input/output system (BIOS) settings. Let's see how to do this:

1. Our first step is to ensure that our CPU supports virtualization:
   1. There are two broad ways to check this while on a Windows host:
      • One, run the Task Manager app and switch to the Performance tab. Below the CPU graph, you will see, among several other things, Virtualization, with Enabled or Disabled following it.
      • A second way to check on Windows systems is to open a Command window (cmd). In Command Prompt, type systeminfo and press Enter. Among the output seen will be the Virtualization Enabled in firmware line. It will be followed by either Yes or No.
   2. To check this while on a Linux host, from Terminal, issue the following commands (processor virtualization extension support: vmx is the check for Intel processors, smv is the check for AMD processors):

egrep --color "vmx|svm" /proc/cpuinfo

For Intel CPUs, the vmx flag will show up (in color) if virtualization is supported. In the case of AMD CPUs, svm will show up (in color). With this, we know that our CPU supports virtualization. But in order to use it, we need to enable it in the computer BIOS.

2. Enter the BIOS by pressing Del or F12 while booting (the precise key to press varies with the BIOS). Please refer to your system's manual to see which key to use. Search for terms such as Virtualization or Virtualization Technology (VT-x). Here is an example for Award BIOS:

3. Now, choose to use hardware virtualization in VirtualBox's Settings menu for your VM. To do this, click on System and then Acceleration. After that, check the boxes, as shown in the following screenshot:

This is how we enable our host processor's hardware virtualization features for optimal performance.

For most desktop/laptop systems, allocating a gigabyte of RAM and two CPUs to the guest VM should be sufficient.

However, when allocating space for the guest's disk, please be generous. Instead of the usual/default 8 GB suggested, I strongly recommend you make it 50 GB or even more. Of course, this implies that the host system has more disk space than this available! Also, you can specify this amount to be dynamically allocated or allocated on-demand. The hypervisor will "grow" the virtual disk optimally, not giving it the entire space to begin with.

For best performance, it's important to install the Oracle VirtualBox Guest Additions as well within the guest VM. These are essentially para-virtualization accelerator software, which greatly helps with optimal performance. Let's see how to do this on an Ubuntu guest session:

1. First, update your Ubuntu guest OS's software packages. You can do so using the following command:

sudo apt update

sudo apt upgrade

2. On completion, reboot your Ubuntu guest OS and then install the required packages using the following command:

sudo apt install build-essential dkms linux-headers-$(uname -r)

3. Now, from the VM menu bar, go to Devices | Insert Guest Additions CD image.... This will mount the Guest Additions ISO file inside your VM. The following screenshot shows what it looks like doing this: 

4. Now, a dialog window will pop up that will prompt you to run the installer in order to launch it. Select Run.
5. The Guest Additions installation will now take place in a Terminal window that shows up. Once complete, hit the Enter key to close the window. Then, power off your Ubuntu guest OS in order to change some settings from the VirtualBox manager, as explained next.

6. Now, to enable Shared Clipboard and Drag'n'Drop functionalities between the guest and host machines, go to General | Advanced and enable the two options (Shared Clipboard and Drag'n'Drop) as you wish with the dropdowns:

7. Then, click OK to save the settings. Now boot into your guest system, log in, and test that everything is working fine.

Remember to update the guest VM regularly and when prompted. This is an essential security requirement. You can do so manually by using the following: 

sudo /usr/bin/update-manager

Finally, to be safe, please do not keep any important data on the guest VM. We will be working on kernel development. Crashing the guest kernel is actually a commonplace occurrence. While this usually does not cause data loss, you can never tell! To be safe, always back up any important data. This applies to Fedora as well. To learn how to install Fedora as a VirtualBox guest, visit https://fedoramagazine.org/install-fedora-virtualbox-guest/.

The Raspberry Pi is a popular credit card-sized Single-Board Computer (SBC), much like a small-factor PC that has USB ports, a microSD card, HDMI, audio, Ethernet, GPIO, and more. The System on Chip (SoC) that powers it is from Broadcom, and in it is an ARM core or cluster of cores. Though not mandatory, of course, in this book, we strive to also test and run our code on a Raspberry Pi 3 Model B+ target. Running your code on different target architectures is always a good eye-opener to possible defects and helps with testing. I encourage you to do the same:

You can work on the Raspberry Pi target either using a digital monitor/TV via HDMI as the output device and a traditional keyboard/mouse over its USB ports or, more commonly for developers, over a remote shell via ssh(1). However, the SSH approach does not cut it in all circumstances. Having a serial console on the Raspberry Pi helps, especially when doing kernel debugging.

To set up your Raspberry Pi, please refer to the official documentation: https://www.raspberrypi.org/documentation/. Our Raspberry Pi system runs the "official" Raspbian (Debian for Raspberry Pi) Linux OS with a recent (as of the time of writing) 4.14 Linux kernel. On the console of the Raspberry Pi, we run the following commands:

rpi $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.6 (stretch)
Release:        9.6
Codename:       stretch
rpi $ uname -a
Linux raspberrypi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
rpi $ 

What if you don't have a Raspberry Pi, or it's not handy? Well, there's always a way – emulation! Though not as good as having the real thing, emulating the Raspberry Pi with the powerful Free and Open Source Software (FOSS) emulator called QEMU or Quick Emulator is a nice way to get started, at least.

Also, of course, you do not have to confine yourself to the Raspberry Pi family; there are several other excellent prototyping boards available. One that springs to mind is the popular BeagleBone Black (BBB) board.

By now, I expect that you have set up Linux as a guest machine (or are using a native "test" Linux box) and have cloned the book's GitHub code repository. So far, we have covered some information regarding setting up Linux as a guest VM (as well as optionally using boards such as the Raspberry Pi or the BeagleBone). Let's now move on to a key step: actually installing software components on our Linux guest system so that we can learn and write Linux kernel code on the system!

It is recommended to use one of the following or later stable version Linux distributions. As mentioned in the previous section, they can always be installed as a guest OS on a Windows or Linux host system, with the clear first choice being Ubuntu Linux 18.04 LTS Desktop. The following screenshot shows you the recommended version and the user interface:

The preceding version – Ubuntu 18.04 LTS Desktop – is the version of choice for this book, at least.  The two primary reasons for this are straightforward:

• Ubuntu Linux is one of the, if not the, most popular Linux (kernel) development workstation environments in industry use today.
• We cannot always, for lack of space and clarity, show the code/build output of multiple environments in this book. Hence, we have chosen to show the output as seen on Ubuntu 18.04 LTS Desktop.

Some other Linux distributions that can also be considered include the following:

• CentOS 8 Linux (not CentOS Stream): CentOS Linux is a distribution that's essentially a clone of the popular enterprise server distribution from RedHat (RHEL 8, in our case). You can download it from here: https://www.centos.org/download/.
• Fedora Workstation: Fedora is a very well-known FOSS Linux distribution as well. You can think of it as being a kind of test-bed for projects and code that will eventually land up within RedHat's enterprise products. Download it from https://getfedora.org/ (download the Fedora Workstation image).
• Raspberry Pi as a target: It's really best to refer to the official documentation to set up your Raspberry Pi (Raspberry Pi documentation: https://www.raspberrypi.org/documentation/). It's perhaps worth noting that Raspberry Pi "kits" are widely available that come completely pre-installed and with some hardware accessories as well. 

• BeagleBone Black as a target: The BBB is, like the Raspberry Pi, an extremely popular embedded ARM SBC for hobbyists and pros. You can get started here: https://beagleboard.org/black. The System Reference Manual for the BBB can be found here: https://cdn.sparkfun.com/datasheets/Dev/Beagle/BBB_SRM_C.pdf. Though we don't present examples running on the BBB, nevertheless, it's a valid embedded Linux system that, once properly set up, you can run this book's code on.

Before we conclude our discussion on selecting our software distribution for the book, here are a few more points to note:

• These distributions are, in their default form, FOSS and non-proprietary, and free to use as an end user.
• Though our aim is to be Linux distribution-neutral, the code has only been tested on Ubuntu 18.04 LTS and "lightly" tested on CentOS 8, and a Raspberry Pi 3 Model B+ running the Raspbian GNU/Linux 9.9 (stretch) Debian-based Linux OS.
• We will, as far as is possible, use the very latest (as of the time of writing) stable LTS
  Linux kernel version 5.4 for our kernel build and code runs. Being an LTS kernel, the 5.4 kernel is an excellent choice to run on and learn with.

• For this book, we'll log in as the user account named llkd.

Now that we have chosen our Linux distribution and/or hardware boards and VMs, it's time we install essential software packages.

The packages that are installed by default when you use a typical Linux desktop distribution, such as any recent Ubuntu, CentOS, or Fedora Linux system, will include the minimal set required by a systems programmer: the native toolchain, which includes the gcc compiler along with headers, and the make utility/packages.

In this book, though, we are going to learn how to write kernel-space code using a VM and/or a target system running on a foreign processor (ARM or AArch64 being the typical cases). To effectively develop kernel code on these systems, we will need to install some software packages. Read on.

Make sure you have installed the guest VM (as explained previously). Then, follow along:

1. Log in to your Linux guest VM and first run the following commands within a Terminal window (on a shell):

sudo apt update
sudo apt install gcc make perl

2. Install the Oracle VirtualBox Guest Additions now. Refer to How to Install VirtualBox Guest Additions in Ubuntu: https://www.tecmint.com/install-virtualbox-guest-additions-in-ubuntu/.
   

To install the packages, take the following steps:

1. Within the Ubuntu VM, first do the following:

sudo apt update

2. Now, run the following command in a single line:

sudo apt install git fakeroot build-essential tar ncurses-dev tar xz-utils libssl-dev bc stress python3-distutils libelf-dev linux-headers-$(uname -r) bison flex libncurses5-dev util-linux net-tools linux-tools-$(uname -r) exuberant-ctags cscope sysfsutils gnome-system-monitor curl perf-tools-unstable gnuplot rt-tests indent tree pstree smem libnuma-dev numactl hwloc bpfcc-tools sparse flawfinder cppcheck tuna hexdump openjdk-14-jre trace-cmd virt-what

The command installing gcc, make, and perl is done first so that the Oracle VirtualBox Guest Additions can be properly installed straight after. These (Guest Additions) are essentially para-virtualization accelerator software. It's important to install them for optimal performance.

One way to try things on an ARM machine is to actually do so on a physical ARM-based SBC; for example, the Raspberry Pi is a very popular choice. In this case, the typical development workflow is to first build the ARM code on your x86-64 host system. But to do so, we need to install a cross toolchain – a set of tools allowing you to build software on one host CPU designed to execute on a different target CPU. An x86-64 host building programs for an ARM target is a very common case, and indeed is our use case here. Details on installing the cross compiler follow shortly.

Often, an alternate way to just trying things out is to have an ARM/Linux system emulated – this alleviates the need for hardware! To do so, we recommend using the superb QEMU project (https://www.qemu.org/).

To install the required QEMU packages, do the following:

• For installation on Ubuntu, use the following:

sudo apt install qemu-system-arm

• For installation on Fedora, use the following:

sudo dnf install qemu-system-arm-<version#>

CentOS 8 does not seem to have a simple means to install the QEMU package we require. (You could always install a cross toolchain via the source, but that's challenging; or, obtain an appropriate binary package.) Due to these difficulties, we will skip showing cross-compilation on CentOS.

If you intend to write a C program that is compiled on a certain host system but must execute on another target system, then you need to compile it with what's known as a cross compiler or cross toolchain. For example, in our use case, we want to work on an x86-64 host machine. It could even be an x86-64 guest VM, no issues, but run our code on an ARM-32 target:

• On Ubuntu, you can install the cross toolchain with the following:

sudo apt install crossbuild-essential-armhf

The preceding command installs an x86_64-to-ARM-32 toolchain appropriate for ARM-32 "hard float" (armhf) systems (such as the Raspberry Pi); this is usually just fine. It results in the arm-linux-gnueabihf-<foo> set of tools being installed; where <foo> represents cross tools such as addr2line, as, g++, gcc, gcov, gprof, ld, nm, objcopy, objdump, readelf, size, strip, and so on. (The cross compiler prefix in this case is arm-linux-gnueabihf-). In addition, though not mandatory, you can install the arm-linux-gnueabi-<foo> cross toolset like this:

sudo apt install gcc-arm-linux-gnueabi binutils-arm-linux-gnueabi

• On Fedora, you can install the cross toolchain with the following:

sudo dnf install arm-none-eabi-binutils-cs-<ver#> arm-none-eabi-gcc-cs-<ver#>

Installing and using a cross toolchain might require some reading up for newbie users. You can visit the Further reading section where I have placed a few useful links that will surely be of great help.

We will now mention a few remaining points, most of them pertaining to software installation or other issues when working on particular distributions:

• On CentOS 8, you can install Python with the following command:

sudo dnf install python3

However, this does not actually create the (required) symbolic link (symlink), /usr/bin/python; why not? Check out this link for details: https://developers.redhat.com/blog/2019/05/07/what-no-python-in-red-hat-enterprise-linux-8/.

To manually create the symlink to, for example, python3, do the following:

sudo alternatives --set python /usr/bin/python3

• The kernel build might fail if the OpenSSL header files aren't installed. Fix this on CentOS 8 with the following:

sudo dnf install openssl-devel

• On CentOS 8, the lsb_release utility can be installed with the following:

sudo dnf install redhat-lsb-core

• On Fedora, do the following:
  • Install these two packages, ensuring the dependencies are met when building a kernel on Fedora systems:
    sudo dnf install openssl-devel-1:1.1.1d-2.fc31 elfutils-libelf-devel
    (the preceding openssl-devel package is suffixed with the relevant Fedora version number (.fc31 here; adjust it as required for your system).
  • In order to use the lsb_release command, you must install the redhat-lsb-core package.

Congratulations! This completes the software setup, and your kernel journey begins! Now, let's check out a few additional and useful projects to complete this chapter. It's certainly recommended that you read through these as well.

This section brings you details of some additional miscellaneous projects that you might find very useful indeed. In a few appropriate places in this book, we refer to or directly make use of some of them, thus making them important to understand. 

Let's get started with the well-known and important Linux man pages project.

You must have noticed the convention followed in most Linux/Unix literature:

• The suffixing of user commands with (1) – for example, gcc(1) or gcc.1
• System calls with (2) – for example, fork(2) or fork().2
• Library APIs with (3) – for example, pthread_create(3) or pthread_create().3

As you are no doubt aware, the number in parentheses (or after the period) denotes the section of the manual (the man pages) that the command/API in question belongs to. A quick check with man(1), via the man man command (that's why we love Unix/Linux!) reveals the sections of the Unix/Linux manual:

$ man man
[...]
A section, if provided, will direct man to look only in that section of
the manual. [...]

       The table below shows the section numbers of the manual followed by the types of pages they contain.

       1   Executable programs or shell commands
       2   System calls (functions provided by the kernel)
       3   Library calls (functions within program libraries)
       4   Special files (usually found in /dev)
       5   File formats and conventions eg /etc/passwd
       6   Games
       7   Miscellaneous (including macro packages and conventions), e.g. 
           man(7), groff(7)
       8   System administration commands (usually only for root)
       9   Kernel routines [Non standard]
[...]

So, for example, to look up the man page on the stat(2) system call, you would use the following:

man 2 stat # (or: man stat.2)

At times (quite often, in fact), the man pages are simply too detailed to warrant reading through when a quick answer is all that's required. Enter the tldr project – read on!

While we're discussing man pages, a common annoyance is that the man page on a command is, at times, too large. Take the ps(1) utility as an example. It has a large man page as, of course, it has a huge number of option switches. Wouldn't it be nice, though, to have a simplified and summarized "common usage" page? This is precisely what the tldr pages project aims to do.

In their own words, they provide "simplified and community-driven man pages." So, once installed, tldr ps provides a neat brief summary on the most commonly used ps command option switches to do something useful:

It's indeed worth checking out. If you're interested in knowing more, visit https://tldr.sh/.

Earlier, recall that we said that userspace system calls fall under section 2 of the man pages, library subroutines under section 3, and kernel APIs under section 9. Given this, then, in this book, why don't we specify the, say, printk kernel function (or API) as printk(9) – as man man shows us that section 9 of the manual is Kernel routines? Well, it's fiction, really (at least on today's Linux): no man pages actually exist for kernel APIs! So, how do you get documentation on the kernel APIs and so on? That's just what we will briefly delve into in the following section.

The community has developed and evolved the Linux kernel documentation into a good state over many years of effort. The latest version of the kernel documentation, presented in a nice and modern "web" style, can always be accessed online here: https://www.kernel.org/doc/html/latest/.

As just one example of the online kernel documentation, see the following partial screenshot of the page on Core Kernel Documentation/Basic C Library Functions (https://www.kernel.org/doc/html/latest/core-api/kernel-api.html#basic-c-library-functions):

As can be gleaned from the screenshot, the modern documentation is pretty comprehensive.

You can literally generate the full Linux kernel documentation from within the kernel source tree in various popular formats (including PDF, HTML, LaTeX, EPUB, or XML), in a Javadoc or Doxygen-like style. The modern documentation system used internally by the kernel is called Sphinx. Using make help within the kernel source tree will reveal several documentation targets, among them htmldocs, pdfdocs, and more. So, you can, for example, cd to the kernel source tree and run make pdfdocs to build the complete Linux kernel documentation as PDF documents (the PDFs, as well as some other meta-docs, will be placed in Documentation/output/latex). The first time, at least, you will likely be prompted to install several packages and utilities (we don't show this explicitly).

Static analyzers are tools that, by examining the source code, attempt to identify potential errors within it. They can be tremendously useful to you as the developer, though you must learn how to "tame" them – in the sense that they can result in false positives.

Several useful static analysis tools exist. Among them, the ones that are more relevant for Linux kernel code analysis include the following:

• Sparse: https://sparse.wiki.kernel.org/index.php/Main_Page
• Coccinelle: http://coccinelle.lip6.fr/ (requires the ocaml package installed)
• Smatch: http://smatch.sourceforge.net/, http://repo.or.cz/w/smatch.git
• Flawfinder: https://dwheeler.com/flawfinder/
• Cppcheck: https://github.com/danmar/cppcheck

For example, to install and try Sparse, do the following:

sudo apt install sparse
cd <kernel-src-tree>
make C=1 CHECK="/usr/bin/sparse"

There are also several high-quality commercial static analysis tools available. Among them are the following:

• SonarQube: https://www.sonarqube.org/​ (a free and open source community edition is available)
• Coverity Scan: https://scan.coverity.com/
• Klocwork: https://www.meteonic.com/klocwork

Static analysis tools can save the day. Time spent learning to use them effectively is time well spent!

A superb tool for tracing and profiling is the powerful Linux Tracing Toolkit next generation (LTTng) toolset, a Linux Foundation project. LTTng allows you to trace both userspace (applications) and/or the kernel code paths in minute detail. This can tremendously aid you in understanding where performance bottlenecks occur, as well as aiding you in understanding the overall code flow and thus in learning about how the code actually performs its tasks.

In order to learn how to install and use it, I refer you to its very good documentation here: https://lttng.org/docs​ (try https://lttng.org/download/ for installation for common Linux distributions). It is also highly recommended that you install the Trace Compass GUI: https://www.eclipse.org/tracecompass/. It provides an excellent GUI for examining and interpreting LTTng's output.

As an example (I can't resist!), here's a screenshot of a capture by LTTng being "visualized" by the superb Trace Compass GUI. Here, I show a couple of hardware interrupts (IRQ lines 1 and 130, the interrupt lines for the i8042 and Wi-Fi chipset, respectively, on my native x86_64 system.):

The pink color in the upper part of the preceding screenshot represents the occurrence of a hardware interrupt. Underneath that, in the IRQ vs Time tab (it's only partially visible), the interrupt distribution is seen. (In the distribution graph, the y axis is the time taken; interestingly, the network interrupt handler – in red – seems to take very little time, the i8042 keyboard/mouse controller chip's handler – in blue – takes more time, even exceeding 200 microseconds!)

Visualizing the complete memory map of the kernel Virtual Address Space (VAS) as well as any given process's user VAS is what the procmap utility is designed to do.

The description on its GitHub page sums it up:

It outputs a simple visualization of the complete memory map of a given process in a vertically-tiled format ordered by descending virtual address. The script has the intelligence to show kernel and userspace mappings as well as calculate and show the sparse memory regions that will be present. Also, each segment or mapping is scaled by relative size (and color-coded for readability). On 64-bit systems, it also shows the so-called non-canonical sparse region or 'hole' (typically close to 16,384 PB on the x86_64).

The utility includes options to see only kernel space or userspace, verbose and debug modes, the ability to export its output in convenient CSV format to a specified file, as well as other options. It has a kernel component as well and currently works (and auto-detects) on x86_64, AArch32, and Aarch64 CPUs.

Download/clone it from https://github.com/kaiwan/procmap:

We make good use of this utility in Chapter 7, Memory Management Internals - Essentials.

SEALS or Simple Embedded ARM Linux System is a very simple "skeleton" Linux base system running on an emulated ARM machine. It provides a primary Bash script that asks the end user what functionality they want via a menu, then accordingly proceeds to cross-compile a Linux kernel for ARM, then creates and initializes a simple root filesystem. It can then call upon QEMU ( qemu-system-arm) to emulate and run an ARM platform (the Versatile Express CA-9 is the default board emulated). The useful thing is, the script builds the target kernel, the root filesystem, and the root filesystem image file, and sets things up for boot. It even has a simple GUI (or console) frontend, to make usage a bit simpler for the end user. The project's GitHub page is here: https://github.com/kaiwan/seals/. Clone it and give it a try... we definitely recommend you have a look at its wiki section pages at https://github.com/kaiwan/seals/wiki for help.

An extension of the well-known Berkeley Packet Filter or BPF, eBPF is the extended BPF. (FYI, modern usage of the term is simply to refer to it as BPF, dropping the 'e' prefix). Very briefly, BPF used to provide the supporting infrastructure within the kernel to effectively trace network packets. BPF is a very recent kernel innovation – available only from the Linux 4.0 kernel onward. It extends the BPF notion, allowing you to trace much more than just the network stack. Also, it works for tracing both kernel space as well as userspace apps. In effect, BPF and its frontends are the modern approach to tracing and performance analysis on a Linux system.

To use BPF, you will need a system with the following:

• Linux kernel 4.0 or later
• Kernel support for BPF (https://github.com/iovisor/bcc/blob/master/INSTALL.md#kernel-configuration)
• The BCC or bpftrace frontends installed (link to install them on popular Linux distributions: https://github.com/iovisor/bcc/blob/master/INSTALL.md#installing-bcc)
• Root access on the target system

Using the BPF kernel feature directly is very hard, so there are several easier front ends to use. Among them, BCC and bpftrace are regarded as useful. Check out the following link to a picture that opens your eyes to just how many powerful BCC tools are available to help trace different Linux subsystems and hardware: https://github.com/iovisor/bcc/blob/master/images/bcc_tracing_tools_2019.png.

The main site for BCC can be found at https://github.com/iovisor/bcc.

The Russian Linux Verification Center, founded in 2005, is an opensource project; it has specialists in, and thus specializes in, automated testing of complex software projects. This includes comprehensive test suites, frameworks, and detailed analyses (both static and dynamic) being performed on the core Linux kernel as well as on the primarily device drivers within the kernel. This project puts a great deal of focus on the testing and verification of kernel modules as well, which many similar projects tend to skim.

Of particular interest to us here is the Online Linux Driver Verification Service page (http://linuxtesting.org/ldv/online?action=rules); it contains a list of a few verified Rules (Figure 1.11):

By glancing through these rules, we'll be able to not only see the rule but also instances of actual cases where these rules were violated by driver/kernel code within the mainline kernel, thus introducing bugs. The LDV project has successfully discovered and fixed (by sending in patches in the usual manner) several driver/kernel bugs. In a few of the upcoming chapters, we shall mention instances of these LDV rule violations (for example, memory leakage, Use After Free (UAF) bugs, and locking violations) having been uncovered, and (probably) even fixed.

Here are some useful links on the LDV website:

• The Linux Verification Center home page; http://linuxtesting.org/
• Linux Kernel Space Verification; http://linuxtesting.org/kernel
• Online Linux Driver Verification Service page with verified Rules : http://linuxtesting.org/ldv/online?action=rules
• Problems in Linux Kernel page; lists over 400 issues found in existing drivers (mostly fixed as well); http://linuxtesting.org/results/ldv

In this chapter, we covered in detail the hardware and software requirements to set up an appropriate development environment for beginning to work on Linux kernel development. In addition, we mentioned the basics and provided links, wherever appropriate, for setting up a Raspberry Pi device, installing powerful tools such as QEMU and a cross toolchain, and so on. We also threw some light on other "miscellaneous" tools and projects that you, as a budding kernel and/or device driver developer, might find useful, as well as information on how to begin looking up kernel documentation.

In this book, we definitely recommend and expect you to try out and work on kernel code in a hands-on fashion. To do so, you must have a proper kernel workspace environment set up, which we have successfully done in this chapter.

Now that our environment is ready, let's move on and explore the brave world of Linux kernel development! The next two chapters will teach you how to download, extract, configure, and build a Linux kernel from source.

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material: https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/questions. You will find some of the questions answered in the book's GitHub repo: https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/solutions_to_assgn.

To help you delve deeper into the subject with useful materials, we provide a rather detailed list of online references and links (and at times, even books) in a Further reading document in this book's GitHub repository. The Further reading document is available here: https://github.com/PacktPublishing/Linux-Kernel-Programming/blob/master/Further_Reading.md.