In my current project work, I have been tasked to configure our VDAs (Windows Server 2016) to claim all memory in VMware, and disable all dynamic memory mechanism on the ESXi 6.7 hosts with 512 GB of memory each. I’m sharing my notes in this blog post for future reference.
The memory sizing in VMware include the hypervisor system memory, guest memory, memory overhead and reserved memory for test machines and gold images.
System Memory (VMware host):
ESXi 6.7 requires a minimum of 4 GB of physical RAM. (ESXi Hardware Requirements (vmware.com)).
Reserved Memory for Test Machines and Gold Images
In the project environment, there is a total of 4 Test virtual machines and 2 Gold Images virtual machines on each site.
Each test virtual machine has a memory set to 30 GB.
For the Gold Images, there is a total of 2 virtual machines using 15 GB of memory each.
With 5 ESXi hosts on each site, the reserved memory on each host will be equal to 32 GB, so at least 1 test machine can be booted on each host or 2 Gold Images on a single host.
Memory overhead
Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures, such as shadow page tables. Overhead memory depends on the number of virtual CPUs and the configured memory for the guest operating system. This is because each core will address its own memory range.
VMware provides a table sample for memory overhead:
To expand the memory overhead usage, we will use the 8 VCPUs model. The target is 90 GB of RAM per VM. According to this model, 90 GB of RAM equals to 720 MB of Memory overhead. 1.5 GB will be used for safety.
Memory Sizing Formula
The formula, including all the elements above, on a 90 GB per VM target, 5 VMs per host, 512 GB of RAM on each host, is as follow:
(Total Host Memory – System Memory – (((Guest Memory + Overhead)* 5) + Reserved Memory)
(512 – 4 – (((90 + 1.5) * 5) + 32) = 18.5 (free memory left)
This would leave 18.5 GB of Memory as spare memory on each host, or a 96% memory reclamation.
Memory Reservation
Memory over-commitment will allow the virtual machines to start on a physical host even when exceeding the available physical memory. When this happen, virtual memory swap and compression will be triggered for sharing the available physical memory. This will impact performances and should not be allowed in the Citrix environment.
Each VDA virtual machine in VMWare should be configured with “Reserve All Guest Memory (All Locked)” under the memory options.
Memory compression cache
In case of a memory over-commitment, the memory page file to be swapped out will be compressed beforehand to accelerate this operation when it occurs. If memory requests comes in to access a compressed page, the page is decompressed and pushed back to the guest memory. The page is then removed from the compression cache.
In a Citrix VDA configuration with full memory reservation for each guest, the memory compression cache operation is not needed.
To disable the compression, on each host, in Advanced System Settings, set:
Mem.MemZipEnable = 0.
Large Memory Page Size
Windows Server allows software to use 4KB, 2MB and 1GB pages. We refer to 4KB pages as small pages while 2MB and 1GB pages are referred to as large pages. Large pages relieve translation lookaside buffer (TLB) pressure for virtualized memory, which results in improved workload performance.
While the biggest performance impact is achieved if large pages are used by the guest and the hypervisor, in most cases a performance impact can be observed even if large pages are used only at the hypervisor level.
To enable large pages in VMware, on each ESXi host, in Advanced System Settings, set:
Mem.AllocGuestLargePage = 1.
Backing Guest vRAM with 1GB Pages
ESXi 6.7 provides a limited support for backing guest vRAM with 1GB pages, referred as Huge Page Files in the VMware documentation. Same as for Large Memory Page Sizes, this will allow to reduce the pressure on the TLB mechanism and improve performance.
A VM with 1GB pages enabled must have full memory reservation. Otherwise, the VM will not be able to power on. All of the vRAM for VMs with 1GB pages enabled is pre-allocated on power-on.
In order to use 1GB pages for backing guest memory you must apply the option at the VM level: sched.mem.lpage.enable1GPage = “TRUE”.
Ballooning
The ballooning driver included with the VMware Tools will trigger memory pages swap to disk at the OS level when the target memory is reached. This feature is not needed with full memory commitment and the driver will be turned off.
Add the following parameter at the virtual machine level (VM Options):
sched.mem.maxmemctl = 0.
Transparent Page Sharing
This feature is the equivalent of memory deduplication. The memory pages will be scanned regularly to be de-duplicated amongst the guest on a given host. Page scanning in a hardware assisted memory virtualization can have a limited performance cost, but in a setup for Citrix with full memory allocation this feature can be intentionally be turned off.
In the guest advanced options, enter the following option:
sched.mem.pshare.enable = FALSE
On the host, to disable TPS, and avoid CPU cycles to be used for this purpose,
change the value at the host level:
Mem.ShareScanGHz = 0.
Configuration Summary
| Virtual Machine | ESXi Host |
| sched.mem.lpage.enable1GPage = TRUE | Mem.ShareScanGHz = 0 |
| sched.mem.maxmemctl = 0 | Mem.AllocGuestLargePage = 1 |
| sched.mem.pshare.enable = FALSE | Mem.MemZipEnable = 0 |
| Memory = 90 GB. Allocate all memory (locked) |
Memory Utilization alerts
When reclaiming all memory, alerts with 100% “Memory Active” can show up, and depending on the ESXi version, this is by design but is still an issue for Virtual Machines in the specified configuration.
ESXi’s active memory metric, despite being called “Memory Utilization” or “Memory Usage” in different parts of the UI, is in no way related to the in-guest memory metrics. It doesn’t show how much guest OS memory is available nor how much guest memory is in an “active” working set or “resident”. It is only used for making memory reclamation decisions in addition to other resource controls like shares, limits and reservation
“Memory Active” is a heuristic utilizing a weighted moving average based on reads and writes to a small, moving subset of pages of the Virtual Machine’s memory. The sampling of this subset of 100 random pages over one minute also incurs a minimal overhead and pre-allocating memory is designed to remove most possibilities of jitter. The Virtual Machine configurations mentioned in the Symptoms section force a full memory reservation and are not subjected to reclamation techniques, hence memory sampling is disabled and active memory will default to a display value of 100%.
Resolution:
This is a display artefact only and has no negative performance impact on the virtual machine or the host it is running on. To limit false positives, the memory activity for VMs with pre-allocated memory has been reduced to 75% in ESXi 7.0 U2.
Workaround:
Disable the Virtual machine memory usage alarm to avoid false positives at the vCenter level.
The following rules will be disabled at the vCenter level (impacts all clusters):
– VKernel VM Memory Utilization
– VKernel Host Memory Utilization
– Host memory usage
– Virtual machine memory usage
The rules will be re-added for each cluster that are not for VDAs.
VM/Host Rules
The new memory configuration for the VDA creates a de facto “static assignment” of VMs to Host in the Citrix VDA cluster. Virtual machines are now “too big to be moved”, and vMotion will not be possible.
VM/Host rules have been created at each cluster level. Virtual machines will not be allowed to start on a host not matching the VM to Host assignment rule.
WEM Memory Optimization
WEM Memory Management / Optimization was enabled in the BFI Citrix environment, when the Appstore servers were limited to 32 GB each. WEM was periodically reclaiming the committed memory of idle processes. This technique would increase the load on the storage, by flushing paged memory in RAM but idle to the page file. With the new memory setting, the optimization is turned off in all environments.
Windows Server page file size
The page file size on the VDA servers (Windows Server 2016), will be changed to a fixed 16 GB file on the system drive (C drive). This is to support the large memory size and committed memory.
In the Tasks Manager, Performances, under the memory tab the total committed memory is now equal to 90 GB (Memory) + 16 GB (Page File):
The setting is defined at the Gold Image level.
Clear Page File at Shutdown
The “Shutdown: Clear virtual memory page file” setting was enabled in the Environmental group policy.
This option is a security setting but with Citrix MCS the page file is not kept and erased at shutdown and restart. Due to the new page file size, and to eliminate potential slow down or restart failures for the week-end and scripted restarts, this option has been disabled.