Using C# and .NET in systems programming

We already run into a problem. You most likely are a C# developer. Maybe you are a VB.Net developer. But no matter what language, you are a .NET developer. After all, that is what this book is about.

Traditionally, Systems programming is done in Assembly, C, and C++. Systems programming has always been the realm of hardcore developers who know the systems they are working on inside out. In the early 50s of the last century, people wrote systems software using switches. A switch in the up position meant a 1, and a switch in the down position meant a 0. These early computers had 8, 16, or even more switches that pointed to the memory address to read or write. Then, 8 switches represented all the bits in a byte for that memory address. Above these switches, there were little lights (no, not LEDS: that invention happened later). Those little lights, if illuminated, meant a 1 in that byte (and a 0 if not illuminated). That way, you could read the contents of that memory address.

Do not worry; that kind of low-level programming is not the topic of this book. If you are interested, there are good remakes of the original Altair 8800 that started a company called Microsoft. You can program that computer in this way: use the switches and lights on the front panel to enter your software. That is how Bill Gates and Paul Allen wrote their first software. But we have other tools at our disposal.

Since systems software relies on efficient, fast, and memory-aware code, people often use programming languages close to the metal. That usually means using language such as machine code – such as the switches I mentioned earlier. Assembly language is another language used, especially in the seventies and eighties of the last century. C and later C++ are other examples of languages that can take advantage of the specifics of the hardware. Most parts of Windows, for instance, are written in C.

However, systems developers do not restrict themselves to low-level languages only. Let me give you an example.

Higher-level languages for systems programming

In 1965, IBM published a manual called PL/I Language Specifications. C28-6571. This relatively obscure title is a fascinating read: it outlines the specifications of the PL/I programming language. PL/I, a sort of abbreviation for Programming Language One, is a higher-level programming language. It contains block structures to allow for recursion, many different datatypes, exception handling, and many other features we take for granted today. It truly was a high-level language. However, they used it to write parts of the early OSs inside IBM. Remember, this was in the sixties when every microsecond counted. Machines were extremely slow compared to modern systems, so they had to utilize every trick in the book to make things work. Yet, a high-level language was considered appropriate. That means there is no reason not to use a high-level language today, especially considering memory profilers’ compiler techniques and advantages.

Kernel mode and user mode

OSs and drivers are usually not built using .NET. The reason for this is that drivers and most parts of the OS run in kernel mode.

The CPU in your computer can run in two modes: kernel or system mode and user mode. User mode is where most of the applications run. The CPU shields the applications from using other memory or process spaces. The CPU protects the applications by placing them in a sandbox. That is precisely what you would want: it would be very undesirable for a program to snoop around in another application’s memory. The processor handles this level of security.

Kernel mode, however, does not have those limitations. Software running in kernel mode is less restricted, controlled, and trusted. That makes sense: parts of an OS should be able to run in all parts of the system, including in the space of other applications.

However, to run in kernel, the compiled code needs to have certain flags set, and the layout of the binaries should be very specific. That is the problem we face. Our C# code relies heavily on the .NET Runtime, and that runtime is not built to be used in Kernel mode. So, even if we could compile our code so that the OS would accept it, it still would not work due to the app not loading the runtime.

There are ways around this. There are ways to pre-compile and include the runtime classes in your binary. Then, you can modify that binary to run in kernel mode. However, the results may vary, and the whole thing would be unreliable. Unreliable code is the exact opposite of what a device driver or OS part should be, so we will not get into this in this book. It’s a hack, not a standard way of working.

Although this book does not deal with kernel-mode apps, I want to give you some insight. Especially since systems programming is usually programming “close to the metal,” so to speak, we are interacting with systems that are running in kernel mode.

Kernel mode is a mode in the CPU. A system can request the CPU to turn on kernel mode. If the code requesting it has the proper privileges, the CPU will do so, thus unlocking parts of the memory previously unavailable. The code does what it needs to do, and then the CPU returns to user mode. Since the code is still in memory doing all sorts of things, it is quite wrong to say an app is a kernel or user-mode app. Some apps can switch the CPU into that state, but the app is almost always running in mixed mode: most of the time, it is in user mode, sometimes kernel mode. Oh, and when I say CPU, I mean logical CPU. This toggling happens on that level, not on the chip itself (but it can also do that).

I have Adobe Creative Cloud installed on my machine. We all know Photoshop, Illustrator, and Premiere, but these apps are meant to be accessed through the Creative Cloud app. This app monitors the system and launches any app you need when you need it. It also updates the background and keeps track of your fonts, files, colors, and other things like that.

Whenever you read something like “runs in the background,” you might expect some systems programming going on, and indeed, there is.

For example, I get this image if I start Performance Monitor on my system and add the % Privileged Time and % User Time counters for the Adobe Desktop Service process.

Figure 0.5: Performance Monitor showing kernel and user times

The red line in Figure 0.5 shows how much time the Adobe Desktop Service spends in user time. The green line, however, shows how long the service is running in privileged time, and privileged time is just a fancy term for kernel time.

As you can see, this app is doing much work in the kernel time. Although I have to admit, I have no clue what it is doing there, but I am sure it is all for a good reason.

We will encounter kernel mode later in other chapters but we will not build apps that run in it.

Why use .NET?

So, we established that we cannot build an OS or a device driver in .NET. That might lead to the question: “Can we use .NET for systems programming?” The answer is a big yes. Otherwise, this would have been a very thin and short book.

Shall we have a look at our recently discovered definition of systems programming? “Writing software used by other software, as a part of a bigger system that works together to achieve a certain goal.” I have shortened the definition, but it is all about this.

Looking at it this way, we can use .NET to write that software. Better yet: I bet .NET is one of the best choices to do so.

.NET offers many advantages over plain C or even C++ (not the managed kind of C++, that is still .NET.)

Back in the day, when we used .NET-Framework-based applications, it would have been a bad idea to use that for systems programming. However, with the introduction of the latest versions of .NET, many disadvantages have been taken care of. With many disadvantages out of the way, .NET-based systems are a viable choice for these kinds of systems.

C and C++ still are excellent languages for low-level systems code. However, C# and .NET Core have their advantages as well.

This table lays out the differences.

Topic	C# and .net core	C/C++
Performance	.NET Core has improved performance compared to .NET Framework, but there may still be overhead due to its runtime. This won’t be an issue for most applications, but it could matter for highly performance-critical systems.	C/C++ provides direct control over hardware and, with careful optimization, can yield superior performance in performance-critical systems.
Memory management	.NET Core still provides automatic garbage collection, reducing the chance of memory leaks, but it gives less control to the developer. This is more suitable for application-level programming.	C/C++ gives developers direct control over memory allocation and deallocation, making it more suitable for systems programming that requires fine-grained memory management.
System-level programming	Some system-level programming tasks may still be more difficult in .NET Core due to its higher-level abstractions and safety features.	C/C++ is often used for system-level programming because it allows for direct hardware access and low-level system calls, which are essential for kernel development, device drivers, and so on.
Portability	.NET Core applications can run on multiple platforms without recompilation, but you must install the .NET Runtime on the target machine. This is an improvement over .NET Framework.	C and C++ code can be compiled and run on virtually any system but often requires careful management of platform-specific differences.
Runtime requirement	.NET Core applications still require the .NET Core Runtime to be installed on the target machine. This can limit its use on systems with limited resources.	C and C++ applications compile down to machine code and don’t require a separate runtime. This can be beneficial for system-level applications or when working with resource-constrained systems.
Direct control	C# and .NET Core still provide many abstractions that can increase productivity, but these abstractions can limit direct control over the system and how code runs.	C/C++ provides more direct control over the system, allowing for finely tuned optimizations and precise control over how your code runs.
Community and support	.NET Core and C# have a growing community and plenty of support resources, including for cross-platform development.	C/C++ has a large, established community, many open-source projects, and a vast amount of existing system-level code.

Table 0.1: Comparison of C# and C/C++

As you can see, both options have advantages and disadvantages. However, most of the disadvantages of .NET Core can be removed using clever tricks and smart programming. Those are the topics of the rest of this book.

C# is a very mature and well-designed language. The capabilities far exceed what developers had when they used C to build, for example, the Unix OS.

What is .NET anyway?

.NET Core is the next version of the over two decades old framework that was meant to help developers get their work done quickly.

It all started with .NET Framework 1 back in 2002. Microsoft presented it as the end-all solution to many issues developers were facing. Fun fact: the project had the internal code name Project 42. You get bonus points if you know why they chose that name.

In the years following the introduction, we have seen many different functions of .NET Framework. Microsoft released the last version of .NET Framework on April 18, 2019.

Before that, Microsoft realized they needed to support other platforms as well. They wanted .NET to be available everywhere, including Linux, Macintosh, and most mobile devices. That meant they had to make fundamental changes to the runtime and the framework. Instead of having different runtime versions for each platform, they decided to have a unified version. That became .NET Core. Microsoft released this in June 2016.

.NET Standard was a set of specifications. The specifications told all developers which features of the runtime were available in which version of the runtime. Most developers did not understand the purpose of .NET Standard and assumed it was yet another version of the runtime. But once they got the idea behind this, it made a lot of sense. If you need a specific API, look it up in the documentation, see what version of .NET Standard it was supported, and then check whether your desired runtime supported that version of .NET Standard.

An example might be helpful here. Let’s say you build an app that does some fancy drawing on the screen. You have worked with System.Drawing.Bitmap before, so you want to use that again. However, your new app should be running on .NET Core. Can you reuse your code? If you look up the documentation of the System.Drawing.Bitmap class, you see the following:

Product	Versions
.NET framework	1.1, 2.0, 3.0, 3.5, 4.0, 4.5, 4.5.1, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8, 4.8.1
. NET platform extensions	2.1, 2.2, 3.0, 3.1, 5, 6, 7, 8
Windows desktop	3.0, 3.1, 5, 6, 7, 8

Table 0.2: Support for System.Drawing.Bitmap

Darn. This class is not part of .NET Standard. It is not available in all runtimes out there. You need to find another way to draw your images.

Your app also communicates with the outside world. It uses the HttpClient class, found in the System.Net.Http namespace. Can you move that to other platforms? Again, we need to look up the documentation of that class. There, we see this table:

Product	Versions
.NET	Core 1.0, core 1.1, core 2.0, core 2.1, core 2.2, core 3.0, core 3.1, 5, 6, 7, 8
.NET framework	4.5, 4.5.1, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2, 4.8, 4.8.1
.NET standard	1.1, 1.2, 1.3, 1.4, 1.6, 2.0, 2.1
Uwp	10.0
Xamarin.ios	10.8
Xamarin.mac	3.0

Table 0.3: Support for Sstem.Net.Http.HttpClient

Now, that is more like it. HttpClient is part of the .NET Standard specification, which means that all runtimes that support the mentioned versions of .NET Standard implement this class. You are good to go!

.NET, .NET Framework, .NET Standard – what is all this?

Table 0.3 shows .NET Framework, .NET Standard, and .NET but not .NET Core. We do see .NET, though. What is this all about?

.NET Core was introduced to sit next to .NET Framework. Microsoft intended for .NET Framework to support Windows devices. However, as I explained, Microsoft later decided to support other devices, OSs, and other hardware architectures; hence the introduction of .NET Core. Then, they realized that this complicated things a lot. People lost track of what they could use and where they could use it. The solution to this was the introduction of the .NET Standard specifications, but that only worsened things – even the people who were not confused initially lost track of what was going on.

The version numbering was an issue as well. We have .NET Framework version 4.8.1 that matched .NET Standard 2.1. .NET Core 3.1 also supported .NET Standard 2.1. Many people had no idea what was happening. They could not understand why a .NET (Core) version of 3.0 was newer than .NET 4.5.

Microsoft saw this problem as well. They also had internal issues: they had to backport a lot of the code in the libraries so it would be available everywhere. To eliminate this mess once and for all, they announced that .NET Framework 4.8 would be the last version. .NET Core 3.1 would be the last version. From now on, it was all unified in something called .NET. Then, to prevent issues with the numbering, .NET started with the number 5.

They also made it easier to track when new versions would come out. Every single year, there will be a new version of .NET. So far, the odd numbers are under Long Term Support (LTS); the even numbers are under Standard Term Support (STS). STS is 18 months, and LTS is 3 years.

.NET 5 was an STS version, and since it was released in November 2020, the support ended in May 2022. .NET 6 was an LTS version. Released in November 2021, support ends November 2024. .NET 7 is again an STS, released in November 2022, with an end of life in May 2024.

By the time of writing this book, the preview versions of .NET 8 are out, and that will be an LTS version.

This is what I use in this book.

Now, the versioning is clear. The release cycle is understood. We can finally let that go. We can focus on building cool stuff instead of worrying about versions.

Programming languages – a choice to make

We are not done yet. We have figured out which version of the runtime we need. But the runtime is just that: a runtime. A set of libraries that we can use. Those libraries have a lot of tools and pre-built classes available, so we do not have to write that. That is awesome. However, we still have to write some code ourselves. We do that in a programming language, and then link to the libraries, compile the code, and have a binary we can deploy and run.

What language should we use?

Microsoft offers us three choices. Others have made their own .NET-compatible languages, but we ignore them. These days, the main languages to write .NET code are C#, F#, and Visual Basic.

F# is a language used for functional programming. This is a different approach to programming than most people are used to, but the financial domain and data-intensive systems use it a lot.

Visual Basic is an excellent language for people just getting started in development. Back in the nineties, at the end of the last century, it was one of the few options people had to build GUI systems rapidly. When .NET came along, Microsoft quickly ported Visual Basic to support this framework, so developers did not have as steep a learning curve. However, usage of Visual Basic is dwindling now that Microsoft stopped co-evolving it with C#.

C# is the language we use in this book.

Although not coupled with the available runtime, Microsoft seems to release a new version of the language around the same time they release a new version of .NET. Version 11 of the language came out in November 2022. Version 12 of C# is now in preview when writing this book.

Each new version of the language has improvements, but many are syntactic. That means that if you cannot use the latest language version, you can still use all the features in the runtime. They are officially decoupled. Sometimes, it is just a bit more typing work.

The .NET Runtime is an excellent foundation for building all sorts of systems. The ecosystem surrounding .NET is very extensive. Next, a huge group of people contributes to the framework daily. It is hard to think of a task that cannot be performed with .NET or one of the thousands of NuGet packages available.

Again, real Kernel mode systems, such as device drivers, are best built with non-managed languages. However, for all other purposes, .NET and C# are an excellent choice.