vSphere High Performance Cookbook: A cookbook is the ideal way to learn a tool as complex as vSphere. Through experiencing the real-world recipes in this tutorial you'll gain deep insight into vSphere's unique attributes and reach a high level of proficiency.

Although VMware vSphere uses various mechanisms to efficiently allocate memory, you might still encounter a situation where virtual machines are allocated insufficient physical memory.

You should know how to monitor memory usage of both the host and virtual machines. You should also know how to troubleshoot common memory performance problems, such as those involving an excessive demand for memory.

Virtual machines perform memory allocation in the same way, as an operating system handles memory allocation and deallocation. The guest operating system frees itself from a piece of physical memory by adding the memory page numbers to the guest free list.

The guest operating system free list is not accessible to the hypervisor and thus it is difficult for the hypervisor to know when to free host physical memory and when the guest physical memory is freed. The hypervisor is completely unaware of which pages are free or allocated in the guest operating system, and because of this the hypervisor cannot reclaim host physical memory when the guest operating system frees guest physical memory.

So VMware hypervisor relies on memory reclamation techniques to reclaim the host physical memory that is freed by the guest operating system. The memory reclamation techniques are:

Excessive memory demand can cause severe performance problems for one or more virtual machines on an ESXi host. When ESXi is actively swapping from the memory of a virtual machine to disk, the performance of that virtual machine will degrade. The overhead of swapping a virtual machine's memory to disk can also degrade the performance of other virtual machines, because the virtual machine expects to be writing to RAM (speeds measured in nanoseconds) but it is unknowingly writing to disk (speeds measured in milliseconds).

The metrics in the vSphere Client for monitoring swapping activity are the following:

Ballooning is a part of normal operations when memory is overcommitted. The fact that ballooning occurrence is not necessarily an indication of a performance problem. The use of the balloon driver enables the guest to give up physical memory pages that are not being used. In fact, ballooning can be a sign that you're getting extra value out of the memory you have in the host.

However, if ballooning causes the guest to give up memory that it actually needs, performance problems can occur due to guest operating system paging.

Note, however, that this is fairly uncommon because the guest operating system will always assign already-free memory to the balloon driver whenever possible, thereby avoiding any guest operating system swapping.

In the vSphere Client, use the Memory Balloon metric to monitor a host's ballooning activity. This metric represents the total amount of memory claimed by the balloon drivers of the virtual machines on the host. The memory claimed...

The amount of memory the VMkernel will try to keep free can be achieved through the Mem.MemMinFreePct parameter. MemMinFreePct determines the amount of memory that the VMkernel should keep free. vSphere 4.1 introduced a dynamic threshold of the Soft, Hard, and Low state to set appropriate thresholds and prevent virtual machine performance issues, while protecting VMkernel. The different states, based on %pRAM which is still free, determines what type of memory reclamation techniques are being used.

For MemMinFreePct, using a default value of 6 percent can be inefficient when 256 gigabyte or 512 gigabyte systems are becoming more and more mainstream. A 6 percent threshold on a 512 gigabyte results in 30 gigabyte idling most of the time. However, not all customers use large systems; some prefer to scale out rather than to scale up. In this scenario, a 6 percent MemMinFreePct might be suitable. To have the best of both worlds, VMkernel uses a sliding scale to...

To troubleshoot memory performance in a VMware vSphere environment, you should monitor the memory performance very carefully. In this aspect you should monitor the following metrics:

A lot of the time, we assume that some very popular metric would be better to monitor memory performance. However, many a times, it leads to something else. This means that these are not an indication of a memory performance issue. If this metric is combined with something else then it may indicate performance degradation.

In this aspect you should not use two of the most popular metrics just to understand whether memory is under pressure or not:

Let me show you what they essentially indicate.

Mem.consumed is the amount of memory consumed by one or all virtual machines. This is calculated as memory granted minus memory saved by sharing. Now the question is why we should not use this. The reason is that memory allocation will vary dynamically based on the VM's entitlement. It is important that a VM should get whatever it actually demands.

Similarly, Mem.vmmemctl is the amount of ballooned memory. This does not...

Both your host memory and VM memory can indicate that it is under pressure. But the main challenge to a VMware admin is how to determine that there is a memory performance issue.

There are a few things which a VMware admin should understand is that there could be a memory performance issue and those are:

Now, if you wonder what is Active memory here in relation to Consumed memory, let me tell you that Active Memory is the amount of memory that is actively used, as estimated by VMkernel based on recently touched memory pages. For a VM this is referred to the amount of guest "physical" memory actively used.

An ESXi host calculates Active memory by using the sum of all active metrics for all powered-on virtual machines plus vSphere services on the host.

There could be another side to it...

Often you need to monitor virtual machine and host memory usage; the good part about this is that VMware vSphere Client exposes two memory statistics in the Summary tab of a virtual machine. These are consumed host memory and active guest memory.

Consumed host memory is the amount of host physical memory that is allocated to the virtual machine. Please note that this value includes the virtualization overhead also.

Also note that many VMs have oversized vRAM, and the guest operating system is likely to opportunistically fill up its vRAM with unnecessary things (for example, caching everything read from disk, no matter how unlikely it will be requested again). Consumed memory only means that the VM used this memory at some point, not that it's likely to use it again.

Active guest memory is defined as the amount of guest physical memory that is currently being used by the guest operating system and its applications.

These two statistics are quite useful for analyzing...

Virtualization causes an increase in the amount of physical memory required, due to the extra memory needed by ESXi for its own code and data structures, and you need to know what are the best practice standards you have.

There are four basic principles, which you should keep in mind: