Virtualization is a concept that creates virtualized resources and maps them to physical resources. This process can be done using specific hardware functionality (partitioning, via some kind of partition controller) or software functionality (hypervisor). So, as an example, if you have a physical PC-based server with 16 cores running a hypervisor, you can easily create one or more virtual machines with two cores each and start them up. Limits regarding how many virtual machines you can start is something that's vendor-based. For example, if you're running Red Hat Enterprise Virtualization v4.x (a KVM-based bare-metal hypervisor), you can use up to 768 logical CPU cores or threads (you can read more information about this at https://access.redhat.com/articles/906543). In any case, hypervisor is going to be the go-to guy that's going to try to manage that as efficiently as possible so that all of the virtual machine workloads get as much time on the CPU as possible.

I vividly remember writing my first article about virtualization in 2004. AMD just came out with its first consumer 64-bit CPUs in 2003 (Athlon 64, Opteron) and it just threw me for a loop a bit. Intel was still a bit hesitant to introduce a 64-bit CPU – a lack of a 64-bit Microsoft Windows OS might have had something to do with that as well. Linux was already out with 64-bit support, but it was a dawn of many new things to come to the PC-based market. Virtualization as such wasn't something revolutionary as an idea since other companies already had non-x86 products that could do virtualization for decades (for example, IBM CP-40 and its S/360-40, from 1967). But it sure was a new idea for a PC market, which was in a weird phase with many things happening at the same time. Switching to 64-bit CPUs with multi-core CPUs appearing on the market, then switching from DDR1 to DDR2, and then from PCI/ISA/AGP to PCI Express, as you might imagine, was a challenging time.

Specifically, I remember thinking about the possibilities – how cool it would be to run an OS, and then another couple of OSes on top of that. Working in the publishing industry, you might imagine how many advantages that would offer to anyone's workflow, and I remember really getting excited about it.

15 or so years of development later, we now have a competitive market in terms of virtualization solutions – Red Hat with KVM, Microsoft with Hyper-V, VMware with ESXi, Oracle with Oracle VM, and Google and other key players duking it out for users and market dominance. This led to the development of various cloud solutions such as EC2, AWS, Office 365, Azure, vCloud Director, and vRealize Automation for various types of cloud services. All in all, it was a very productive 15 years for IT, wouldn't you say?

But, going back to October 2003, with all of the changes that were happening in the IT industry, there was one that was really important for this book and virtualization for Linux in general: the introduction of the first open source Hypervisor for x86 architecture, called Xen. It supports various CPU architectures (Itanium, x86, x86_64, and ARM), and it can run various OSes – Windows, Linux, Solaris, and some flavors of BSD – and it's still alive and kicking as a virtualization solution of choice for some vendors, such as Citrix (XenServer) and Oracle (Oracle VM). We'll get into more technical details about Xen a little bit later in this chapter.

The biggest corporate player in the open source market, Red Hat, included Xen virtualization in initial releases of its Red Hat Enterprise Linux 5, which was released in 2007. But Xen and Red Hat weren't exactly a match made in heaven and although Red Hat shipped Xen with its Red Hat Enterprise Linux 5 distribution, Red Hat switched to KVM in Red Hat Enterprise Linux 6 in 2010, which was – at the time – a very risky move. Actually, the whole process of migrating from Xen to KVM began in the previous version, with 5.3/5.4 releases, both of which came out in 2009. To put things into context, KVM was a pretty young project back then, just a couple of years old. But there were more than a few valid reasons why that happened, varying from Xen is not in the mainline kernel, KVM is, to political reasons (Red Hat wanted more influence over Xen development, and that influence was fading with time).

Technically speaking, KVM uses a different, modular approach that transforms Linux kernels into fully functional hypervisors for supported CPU architectures. When we say supported CPU architectures, we're talking about the basic requirement for KVM virtualization – CPUs need to support hardware virtualization extensions, known as AMD-V or Intel VT. To make things a bit easier, let's just say that you're really going to have to try very hard to find a modern CPU that doesn't support these extensions. For example, if you're using an Intel CPU on your server or desktop PC, the first CPUs that supported hardware virtualization extensions date all the way back to 2006 (Xeon LV) and 2008 (Core i7 920). Again, we'll get into more technical details about KVM and provide a comparison between KVM and Xen a little bit later in this chapter and in the next.

There are various types of virtualization solutions, all of which are aimed at different use cases and are dependent on the fact that we're virtualizing a different piece of the hardware or software stack, that is, what you're virtualizing. It's also worth noting that there are different types of virtualization in terms of how you're virtualizing – by partitioning, full virtualization, paravirtualization, hybrid virtualization, or container-based virtualization.

So, let's first cover the five different types of virtualization in today's IT based on what you're virtualizing:

• Desktop virtualization (Virtual Desktop Infrastructuree (VDI)): This is used by a lot of enterprise companies and offers huge advantages for a lot of scenarios because of the fact that users aren't dependent on a specific device that they're using to access their desktop system. They can connect from a mobile phone, tablet, or a computer, and they can usually connect to their virtualized desktop from anywhere as if they're sitting at their workplace and using a hardware computer. Benefits include easier, centralized management and monitoring, much more simplified update workflows (you can update the base image for hundreds of virtual machines in a VDI solution and re-link that to hundreds of virtual machines during maintenance hours), simplified deployment processes (no more physical installations on desktops, workstations, or laptops, as well as the possibility of centralized application management), and easier management of compliance and security-related options.
• Server virtualization: This is used by a vast majority of IT companies today. It offers good consolidation of server virtual machines versus physical servers, while offering many other operational advantages over regular, physical servers – easier to backup, more energy efficient, more freedom in terms of moving workloads from server to server, and more.
• Application virtualization: This is usually implemented using some kind of streaming/remote protocol technology such as Microsoft App-V, or some solution that can package applications into volumes that can be mounted to the virtual machine and profiled for consistent settings and delivery options, such as VMware App Volumes.
• Network virtualization (and a more broader, cloud-based concept called Software-Defined Networking (SDN)): This is a technology that creates virtual networks that are independent of the physical networking devices, such as switches. On a much bigger scale, SDN is an extension of the network virtualization idea that can span across multiple sites, locations, or data centers. In terms of the concept of SDN, entire network configuration is done in software, without you necessarily needing a specific physical networking configuration. The biggest advantage of network virtualization is how easy it is for you to manage complex networks that span multiple locations without having to do massive, physical network reconfiguration for all the physical devices on the network data path. This concept will be explained in Chapter 4, libvirt Networking, and Chapter 12, Scaling Out KVM with OpenStack.
• Storage virtualization (and a newer concept Software-Defined Storage (SDS)): This is a technology that creates virtual storage devices out of pooled, physical storage devices that we can centrally manage as a single storage device. This means that we're creating some sort of abstraction layer that's going to isolate the internal functionality of storage devices from computers, applications, and other types of resources. SDS, as an extension of that, decouples the storage software stack from the hardware it's running on by abstracting control and management planes from the underlying hardware, as well as offering different types of storage resources to virtual machines and applications (block, file, and object-based resources).

If you take a look at these virtualization solutions and scale them up massively (hint: the cloud), that's when you realize that you're going to need various tools and solutions to effectively manage the ever-growing infrastructure, hence the development of various automatization and orchestration tools. Some of these tools will be covered later in this book, such as Ansible in Chapter 11, Ansible for Orchestration and Automation. For the time being, let's just say that you just can't manage an environment that contains thousands of virtual machines by relying on standard utilities only (scripts, commands, and even GUI tools). You're definitely going to need a more programmatic, API-driven approach that's tightly integrated with the virtualization solution, hence the development of OpenStack, OpenShift, Ansible, and the Elasticsearch, Logstash, Kibana (ELK) stack, which we'll cover in Chapter 14, Monitoring the KVM Virtualization Platform Using the ELK Stack.

If we're talking about how we're virtualizing a virtual machine as an object, there are different types of virtualization:

• Partitioning: This is a type of virtualization in which a CPU is divided into different parts, and each part works as an individual system. This type of virtualization solution isolates a server into partitions, each of which can run a separate OS (for example, IBM Logical Partitions (LPARs)).
• Full virtualization: In full virtualization, a virtual machine is used to simulate regular hardware while not being aware of the fact that it's virtualized. This is done for compatibility reasons – we don't have to modify the guest OS that we're going to run in a virtual machine. We can use a software- and hardware-based approach for this.

  Software-based: Uses binary translation to virtualize the execution of sensitive instruction sets while emulating hardware using software, which increases overhead and impacts scalability.

  Hardware-based: Removes binary translation from the equation while interfacing with a CPU's virtualization features (AMD-V, Intel VT), which, in turn, means that instruction sets are being executed directly on the host CPU. This is what KVM does (as well as other popular hypervisors, such as ESXi, Hyper-V, and Xen).

• Paravirtualization: This is a type of virtualization in which the guest OS understands the fact that it's being virtualized and needs to be modified, along with its drivers, so that it can run on top of the virtualization solution. At the same time, it doesn't need CPU virtualization extensions to be able to run a virtual machine. For example, Xen can work as a paravirtualized solution.
• Hybrid virtualization: This is a type of virtualization that uses full virtualization and paravirtualization's biggest virtues – the fact that the guest OS can be run unmodified (full), and the fact that we can insert additional paravirtualized drivers into the virtual machine to work with some specific aspects of virtual machine work (most often, I/O-intensive memory workloads). Xen and ESXi can also work in hybrid virtualization mode.
• Container-based virtualization: This is a type of application virtualization that uses containers. A container is an object that packages an application and all its dependencies so that the application can be scaled out and rapidly deployed without needing a virtual machine or a hypervisor. Keep in mind that there are technologies that can operate as both a hypervisor and a container host at the same time. Some examples of this type of technology include Docker and Podman (a replacement for Docker in Red Hat Enterprise Linux 8).

Next, we're going to learn how to use hypervisors.