Off-box virtualization

If you look at any traditional cloud provider, they are all made up of Virtual Machines (VMs) running on top of a hypervisor. A hypervisor's job is to isolate these VMs by sharing the same CPUs, and then capture I/O from each VM to ensure that they are abstracted from the hardware. The VM is, therefore, secure and portable, as it sees only a software-defined network interface card. The hypervisor can inspect all of the packets between the VMs and enable features such as IP whitelists and access control lists. This is depicted in the following diagram:

Figure 1.6 – In-kernel network virtualization

In in-kernel virtualization, whenever there is a need to inspect packets to and from a VM, you can put pressure on the host hypervisor by taking away its CPU cycles. This is mainly because this type of hypervisor performs packet switching, encapsulation, and enforces stateful firewall rules. However, this is not the only risk. There is another risk of having noisy neighbors. A noisy neighbor VM monopolizes bandwidth, disk I/O, and CPUs at the expense of its neighbors.

The fundamental purpose of off-box virtualization is that it no longer commits I/O virtualization into the hypervisor but to the network outside of the physical box. You can't reach the control plane that runs the virtual network from the public internet. However, you can create an explicit tunnel to reach the virtual network, which can be monitored, audited, and, in the case of an emergency, switched off as well. This is shown in the following diagram, where you can clearly see that the network I/O virtualization is not being done at the host hypervisor level:

Figure 1.7 – Off-box network virtualization

When you move network virtualization from in-kernel virtualization to off-box virtualization, it results in a dramatic change in performance and improved security posture. This is because you are no longer getting any performance overhead associated with the hypervisor. Another benefit of off-box virtualization is that you retain the flexibility to plug anything into the virtual network. You can perhaps add another bare metal host, an Non-Volatile Memory Express (NVMe) storage system, a VM, a container, or even an engineered system, such as Exadata. All of them can run on this virtual network and reach each other within two hops. Take a look at the following diagram:

Figure 1.8 – OCI's holistic architecture

OCI's unique offering of bare metal servers doesn't come with a pre-installed operating system or any software. This increases the level of security over traditional virtualization. You can choose any hypervisor that you want to install on top of the OCI bare metal instance and then deploy VMs and install applications on top of that. OCI doesn't offer any access to the memory space of these bare metal instances, allowing complete physical isolation.

As you can see, these bare metal instances are running without any adjacent co-tenants; therefore, they boost both IOPS and bandwidth.

The security benefits of off-box virtualization

Traditional server virtualization comes with an abstraction layer. This layer abstracts the application that is running on the virtualized server from compute resources, storage, and networking hardware. You can deploy this virtual infrastructure without any disruption as it has nothing to do with user experience. You have to virtualize the CPU, main memory, network access, and I/O if you want to take advantage of partitioning, isolation, and hardware independence, which all result from the virtualization process.

Traditional server virtualization in first-generation public cloud infrastructures might come at a cost to the performance overhead and lead to weaker security. The performance overhead is mainly because the hypervisor needs to manage network traffic and I/O for all of the VMs that are running on a host, causing noisy neighbor problems. Additionally, the level of security is inherently weaker because, in traditional virtualization architectures, the hypervisor has complete trust and makes access decisions on behalf of the VM. This means a hostile actor that compromises the hypervisor can easily spread beyond the single hypervisor to other systems in the same cloud. More specifically, the traditional model implies that the attacker of a compromised host/hypervisor can access any VCN because the host/hypervisor is trusted to do so. You can see an example of in-kernel network virtualization in the following diagram:

Figure 1.9 – In-kernel virtualization

The security isolation between one customer's compute resources from other customers' compute resources is critical. Fundamentally, OCI designed its security architecture with the assumption that customer-controlled compute resources can be hostile. It has a multi-pronged defense in-depth security architecture, which has been designed to minimize the security risks of traditional virtualization.

Oracle's Gen 2 Cloud infrastructure uses a unique approach that eliminates some of the disadvantages of traditional server virtualization. OCI uses off-box network virtualization, which takes network virtualization out of the software stack (hypervisor) and places it in the infrastructure. Oracle uses off-box network virtualization technology in both bare metal instances (for example, physical servers dedicated to a single customer) and VM instances.

Isolated network virtualization limits the attacker surface to only a VCN connected to the hypervisor by the control plane. OCI moves the trust from the hypervisor to the isolated network virtualization, which is implemented outside of the hypervisor. In the following diagram, you can see how this is done:

Figure 1.10 – Off-box network virtualization

Bare metal is unique as no hypervisor is needed to run resource virtualization for the network and I/O. Instead, OCI moves network virtualization into the infrastructure, resulting in dramatic performance and security gains. This is because the performance overhead associated with traditional virtualization (in the hypervisor) is eliminated.

OCI effectively uses a coprocessor in the infrastructure, which performs network virtualization, and thus allows the hypervisor to focus on other virtualization tasks. Offloading network virtualization eliminates a number of traditional hypervisor security risks by creating a security boundary between the hypervisor running on one host and the virtual networks of other VMs running on other hosts.

OCI engineered this solution in an attempt to reduce the attack surface and potential damage that an attacker might cause in the case of a compromised hypervisor.

OCI has also implemented additional layers of network isolation, which prevent malicious actors from sending unauthorized network traffic even in the extremely unlikely case of an attacker breaking through the first line of defense that is provided by off-box network virtualization.

For bare metal instances, off-box network virtualization provides a security boundary for the virtual network. This boundary prevents an attacker on a bare metal instance from gaining access to the virtual networks of other bare metal instances and VMs running on other hypervisors.