The software industry is very young. In the last 20-30 years, we’ve seen a dramatic increase in its usage.

Today, software is engraved into every aspect of our lives – our work, our transportation, and in our homes, down to the smart light bulbs many of us have installed.

With the breadth of applications and users using the software we make, there are expectations from the software. Users today expect far more from software than they did 20 years ago. And since software is so engraved in our lives, we are far more vulnerable, which makes it a risk factor.

In this section, we’ll discuss why you, as a developer, should care about metaprogramming. We’ll go through the developer concerns, how we can do some nifty automation, and cover some of the basics around metaprogramming.

Developer concerns

For us developers, we have a lot of different aspects to cover to guarantee the success of our software.

End users have high expectations of great user experiences, and they expect to be put in the pit of success. We also need to be empathetic to the different types of users our system will have and make sure it is accessible to everyone.

Our systems need to maintain the integrity of their data and help the end users do the right thing. To do this, we need to validate all the input according to what we expect or what our data model requires. In addition, we need to have business rules that enforce the integrity of the system.

The data is also something we want to protect, so security plays an important role. Users need to be authenticated and we also want to make sure the user has the correct authorization to perform the different tasks of the system.

We must also make sure that our systems don’t have security flaws that would allow hackers to breach the system. The input from users also needs to be sanitized, to prevent malicious attacks through things such as SQL injection.

For our software to be available to our users, we need to have it running somewhere, on an on-premises server or servers with a hosting partner, or in the cloud, be it physical or virtual. This means we need to think about how we package the software and then how we get it onto the running environment.

Once it is running, we have to make sure it runs all the time and doesn’t have any downtime; our users rely on it. For this, we want to consider having more than one instance of the system running so that it can fail over to a second instance if the primary one goes down.

We also need to make sure that the environment it is running in can handle the number of users it is targeting.

Instead of just having a fail-over instance, we can scale out horizontally and have a load balancing mechanism spreading the users across the different instances we have. This makes our system a distributed system.

Those are a lot of different concerns. Ideally, you want to have different people doing different aspects of the job, but this is not always the case (depending on the size of the organization, as well as its culture). Today, you’ll often see in job ads that companies are looking for full stack developers. In many cases, this could mean the expectations are that you need to work with all of the following aspects:

• User experience: This is all about the interaction, flows, and how it all feels
• Accessibility: This involves creating empathetic software that is accessible to those with disabilities
• Frontend code: This is the layout, styling, and necessary logic to make the user experience come to life
• Backend code: This is for creating the glue that represents the domain we’re working in
• Data modeling: This is how we store the data and model it for the usage we need
• Authentication and authorization: This is for making sure users are authenticated and the proper authorization policies are applied to the different features
• Security: This makes the application robust from any attacks and protects the data
• DevOps: This involves delivering features to production in a timely fashion without any ceremony

Automation

Being humans, we make mistakes and we forget things. Sometimes, this has some really bad outcomes, such as systems going down, or worse, such as being breached by hackers.

Luckily, computers are good at doing what they’re told and repeating it endlessly. They never complain and don’t make mistakes. This means there are great opportunities for streamlining a lot of our work. As the industry has matured over the years, we have seen improved workflows and tooling that can help us achieve what we aim to achieve, often taking away tedious and time-consuming tasks.

A great example of automation is what has happened in the last decade in cloud computing. Before this, we had to set up physical hardware and often had manual routines for getting our software onto that hardware. This has completely changed into us being able to spin up anything our hearts desire with a few clicks, connect it to some continuous deployment software that will build our software, and automatically get it onto the running environment. All this can be achieved in minutes, rather than hours or days.

Metaprogramming

Where am I going with all this? Wasn’t this book supposed to be about something called metaprogramming?

Metaprogramming is all about additional information surrounding your code. This information is sometimes implicit – that is, it’s already there. Sometimes, however, it needs to be added explicitly or deliberately by you, as a developer.

The computer that runs the software only understands the machine language instructions laid out in memory for the CPU to execute. For us humans, this is less than intuitive. Early on, we came up with languages that would help us write something more friendly and we could reason about more easily. This started with the assembly language and, later, higher-level languages that would compile down to assembly language.

With this tooling in place, we gained the ability to not just translate from one language into another, but also to reason about what was going on with our code. In 1978, Stephen C. Johnson, from Bell Labs, came up with what he called lint – a static code analysis tool that could be used to reason about C code and detect potential issues with it. Today, this is common with most programming languages. For web development in JavaScript or TypeScript, we could typically add tools such as ESLint to our build pipelines to do this. With C#, we have this built into the compiler, and with the Roslyn compiler, it is completely extensible with our own custom rules, something we will cover in Chapter 17, Static Code Analysis.

For programming languages such as C/C++ that compile down to something that runs natively on the CPU, we’re limited to what we can reason about at the compile level. However, with programming languages such as Java or C#, often referred to as managed languages, we’re now running code in a managed environment. The code we write compiles down to an intermediate language that will be translated on the fly while running. These languages then carry information with them about the code we wrote – this is known as metadata. This lets us treat our programs or others as data at runtime and allows us to reason about the code; we can even discover code at runtime.

With C#, from version 1, we could add additional information and more metadata. Through the use of C# attributes, we could adorn things such as types, properties on types, and methods with additional information. This information would carry through to the running program and is something we can use to reason about our software.

For instance, with attributes, we can now add additional information that we can reason about both at compile time and runtime. We can do things such as marking properties on an object with validation information, such as [Required]:

public class RegisterPerson
{
    [Required]
    public string FirstName { get; set; }
    [Required]
    public string LastName { get; set; }
    [Required]
    public string SocialSecurityNumber { get; set; }
}

This code represents what is needed to register a person. All the properties that we required have the [Required] attribute as metadata.

Now that we have added metadata to the code, we can take concrete actions based on it.

Adding explicit metadata is great for visibility and makes it very explicit in the code for the type of metadata that has been added. However, this metadata is not actionable on its own. This means that there is nothing that will inherently deal with it – for instance, a property is required, as we’ve seen.

This metadata gives us the power to not only reason about the metadata surrounding our code but put it into action. We can build our systems in a way that leverages this information to make decisions for us, or we could automate tedious tasks.

One of the most common things I’ve seen throughout my career is what I call recipe-driven development. Code bases tend to settle on a certain structure and a certain set of things developers need to do when creating features in it. These recipes are then often written down as a part of the documentation for the code base and something everyone has to read and make sure they follow. This is not necessarily a bad thing, and I think all code bases have this to some degree.

Taking a step back, there might be some potential to improve our productivity and have to write less code. The recipes and patterns could be formalized and automated. The main reason for doing so is that following recipes can be error-prone. We can forget to do something or do it wrong or maybe even mix up the ordering of steps.

Imagine that you have an API and that for every action, you have the following recipe:

• Check if the user is authorized
• Check if all the input is valid
• Check for malicious input (for example, SQL injection)
• Check if the action is allowed by the domain logic, typically business-specific rules
• Perform the business logic
• Return the correct HTTP status code and result, depending on whether or not we’re successful

Those are a lot of concerns mixed into one place. At this point, you’re probably thinking that this is not how we do things in modern ASP.NET API development. And that is correct – they are typically split into concerns and things such as the SQL injection handled by the pipeline.

Important note

We’ll revisit how ASP.NET leverages metaprogramming to make the developer experience it offers in Chapter 3, Demystifying through Existing Real-World Examples.

Even though these things might not be in the same method and spread out, they are still concerns we have to be aware of, and a recipe would then still state that these things would need to be done. Often, they are repetitive and could potentially be optimized for an improved developer experience and also reduce the risk of fatal errors in the system.

Maintaining software

Another aspect of this type of repetitive code is that all code we add to our system is code we need to maintain. Building out a feature might not take that long, but chances are the code needs to be maintained for years to come. It might not be maintained by you, but by others on the team or a successor to you. So, we should be optimizing our code bases for maintenance first. Getting features out the door in a timely fashion is expected of us, but if we don’t consider the maintenance of the code, we, as the owners of the code, will suffer when it needs to be maintained.

Maintenance is not just about keeping the code working and delivering on its promise. It’s also about its ability to change and adapt to new requirements, whether business or technical. The very beginning of a project is when you know the least about it.

So, planning for this is super hard and would require us to be able to predict the future. But we can write our code in a way that would make it more adaptable to change.

Instead of repeating all this code all over the place, we could put metadata into our code that we could leverage. This is typically what ASP.NET supports – for instance, for authorization with the [Authorize] attribute for controllers. It would require a specific policy to be fulfilled, such as the user having to be in a role. If our system has a deliberate structure for our features, you might find natural groupings of features belonging to specific roles. We could then reason about this structure by looking at the namespace metadata on the type and putting in the correct authorization rules. For developers, you replaced the need for an explicit piece of information and made it implicit through the structure. This may seem like a small thing, but throughout the lifetime of the code base, this type of mindset can have a huge impact on productivity and maintainability.

Generating code

With C#, we can go even further than just reasoning about code and making decisions based on what we find – we can generate code. Code generation can take place at compile time if we have all the information we need or are pushed to the runtime level. This opens up a lot more flexibility and gives us a vast amount of power.

As an example, if you’ve ever worked with XAML-based frontend technology such as Windows Presentation Foundation (WPF) or Universal Windows Platform (UWP) and have used data binding, you have probably come across the INotifyPropertyChanged interface. Its purpose is to enable the view controls so that you’re notified when the value of a property has changed on an instance of an object in the view to which it is bound.

Let’s say you have an object representing a person:

public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

Now, let’s say we want to make this notification appear whenever one of the properties changes. Using the INotifyPropertyChanged interface for binding purposes, the object would need to expand into the following:

public class Person : INotifyPropertyChanged
{
    private string _firstName;
    public string FirstName
    {
        get { return _firstName; }
        set
        {
            _name = value;
            RaisePropertyChanged("FirstName");
        }
    }
    public string LastName { get; set; }
    public event PropertyChangedEventHandler
      PropertyChanged;
    protected void RaisePropertyChanged(string
      propertyName)
    {
        if (PropertyChanged != null)
        {
            PropertyChanged(this, new
              PropertyChangedEventArgs(propertyName));
        }
    }
}

As you can see, creating a property is now very tedious. Imagine having all the properties of an object do this. This easily becomes code that is hard to read, and there's more code to maintain, and it’s not adding any business value to your code.

This can be improved upon thanks to the latest version of the C# compiler.

Microsoft rewrote the C# compiler a few years back. The compiler was given the name Roslyn. There were a couple of reasons they rewrote the compiler, with one being that they wanted to have the compiler itself be written in C# – a proof of the maturity of the language and the .NET runtime Also, as part of the move from Microsoft to open source, having a rewrite and doing it in the open and leaving the old license model behind made more sense. But the most important reason in my opinion was to make it more extensible, and not just for Microsoft themselves, but everyone.

Part of this extensibility is what is called Roslyn code generation. With it, we could go and make this code very close to the original. Let’s imagine we introduce some metadata in the form of an [Bindable] attribute and we create a compiler extension that makes all private fields into properties that are needed for InotifyPropertyChanged. Here, our object would look like this:

[Bindable]
public class Person
{
    private string _firstName;
    private string _lastName;
}

We could also do this at runtime. However, at runtime, we are limited to what has been compiled and can’t change the type. So, the approach would be slightly different. Instead of changing the existing type, we would need to create a new type that inherits from the original type and then extend it. This would require us to make the original properties virtual for us to override them in a generated type:

[Bindable]
public class Person
{
    public virtual string FirstName { get; set; }
    public virtual string LastName { get; set; }
}

For this to work, we would need a factory that knows how to create these objects. We would call this when we needed an instance of it.

With great power also comes great responsibility, and it needs to be a very deliberate choice to go down this path. We’ll cover this in Chapter 18, Caveats and Final Words.

We will cover the Roslyn extensibility in more depth in Chapter 15, Roslyn Compiler Extensions.

Compile time safety

There are also times when we must add certain metadata for the system to work. This is a candidate for writing a code analyzer for the Roslyn compiler. The analyzer would figure out what’s missing and let the developer know as soon as possible, providing a tight feedback loop rather than the developer having to discover the problem at runtime.

An example of this is in a platform I work on called Cratis (https://cratis.io), an event sourcing platform. For all the events being persisted, we require a unique identifier that represents the type of event. This is added as an attribute for the event:

[EventType("66f58b90-c027-41b3-aa2c-2cfd18e7db69")]
public record PersonRegistered(string FirstName, string LastName);

When calling the Append() method on the event log, the type has to be associated with the unique identifier. If there is no association between an event type and the .NET type, the Append() method will throw an exception. This is a great opportunity to perform a compile-time check of anything being sent to the Append() method and to check whether or not the type of the object has the [EventType] attribute.

We will revisit all this in Chapter 17, Static Code Analysis.

Hopefully, you now know of the great potential of metaprogramming. It is very powerful. This comes with great responsibility – the balance of code you don’t see that magically gets added either at compile time or runtime versus the explicitness in every code file is a hard one. From my experience, new developers coming into a code base with a lot of implicit automation can run into trouble and might end up not trusting the magic.

But after a while, once they get used to it, they tend to want more magic. The benefits are clear once you have experience with them, but it might be a bit scary at first. To remedy this, you should communicate the automation you have. That will at least make it adhere more to the principle of least surprise.

In the next chapter, we will dive into more concrete concepts of metaprogramming and look at what’s behind the concepts. We’ll become familiar with how the .NET runtime sees code and the metadata it produces and how this can be leveraged in a running application. Finally, we’ll learn how to extend this metadata.