Introduction
Virtual machine monitors (VMM) emulate most guest access to interrupts and the advanced programmable interrupt controller (APIC) in a virtual environment. They also virtualize all guest interrupts. These activities require the exit and reentry of the virtual machines (VM), but they are time consuming and are a major source of overhead. in order to minimize that effect, Intel(R) new processors, Xeon(R) Et v2, emulate those activities in the hardware. This new feature is called APIC virtualization (APICv). More information can be found at [6] and in chapter 29 of [7].
Figure 1 - VM-VMM interaction with and without APICv
Figure 1 shows that all virtualized activities relating interrupts and APIC to and from the guest OS have to go through VMM in systems without APICv. They are executed in the hardware for systems with APICv, not in the VMM. This way all activities can stay inside the VM, thus eliminating the need to issue the "VM exit" command resulting in reducing overhead and increasing throughput.
In his blog we will test this new feature to see if it improves the throughput and how it affects the CPU utilization.
Terminology
Advanced Programmable Interrupt Controller (APIC) is programmable interrupt controller (PIC) that can handle interrupts from multiple processors. More information about PIC can be found here[2].
Fread is reading a file using the function fread(). More information about this can be found here[8] in the download documentation link.
Fwrite is writing a file using the function fwrite(). More information about this can be found here[8] in the download documentation link.
Guest Operating System (OS) is an OS that runs in a VM. More information about the guest OS can be found here[5].
Pread is reading from a file at a given offset.
Pwrite is writing to a file at a given offset.
Random Read is reading from a file with access being made to random locations within the file. More information about it can be found here8] in the download documentation link.
Random Write is writing to a file with access being made to random locations with the file. More information about it can be found here[8] in the download documentation link.
Virtualization is a way to run multiple independent virtual operating systems on a single physical computer. More information about it can be found here[1].
Virtual Machine (VM) is a piece of computer software, firmware or hardware that creates and run VMs. More information about it can be found here[4].
The Test
Iozone v3.42 was chosen to test the file I/O performance of a system. The beta version of the 64-bit Enterprise edition of Redhat* 7 was used with kvm* installed on a system with two pre-production Intel(R) Xeon(R) E5-2697 v2 processors and 64GB of RAM. We created twelve VMs, and each VM had the same OS on it: beta version of 64-bit Enterprise edition of Redhat 7.
On each VM, Iozone will perform the test on a 100MB file with record size of 64KB.
The following steps were used to collect file I/O activities and CPU utilization:
1) Enable APICv feature in kvm
2) Turn on only one VM
3) Ssh to execute Iozone an all running VMs
4) At the main console, run the command sar (system activity report) to collect the percentage CPU utilization.
5) Increase the number of running VMs by one and repeat the above steps starting from step 3
6) Repeat until all VMs created are running
7) Disable APICv feature in kvm
8) Repeat the above steps starting from step 3 through step 7
Note that the results can vary from system to system depending on many factors like the record size, file size, type of data disk and so on.
Results
Figure 2 - Percentage Percentage CPU utilization with and without APICv
Figure 2 shows the CPU utilization increase ranging from 0.26% to 4.98% when compared to systems with and without APICv features. This makes sense in systems with APICv. They have requests activities served within VMs without going out to VMM. The result is less overhead. This translates into more requests being ready to execute within the same time frame, compared to those of systems without APICv resulting in more CPU activities. We want to figure out which components in the CPU utilization that causes it to increase in systems with APICv. Further analyzing the components of the CPU utilization revealed the values of the iowait component in systems with APICv are lower than those in system without APICv as can be seen in figure 2a. Iowait is the portion of time the CPU is idle during which the system has an outstanding disk I/O request. Since more requests were being executed with systems equipped with APICvFigure 8.
~~Figure 2a - Percentage of iowait with and without APICv
~~Figure 3 – Throughput of random read with and without APICv
With APICv the random-read throughput increase ranging from 0.29% - 2.73%
~~Figure 4 – Throughput of random write with and without APICv
With APICv the random-write throughput increase ranging from 0% - 8.51%
~~Figure 5 – Throughput of pread with and without APICv
With APICv the pread throughput increase ranging from 0.52% - 9.3%
~~Figure 6 – Throughput of pwrite with and without APICv
With APICv the pwrite throughput increase ranging from 0.8% - 3.44%
~~Figure 7 – Throughput of fread with and without APICv
With APICv the fread throughput increase ranging from 0.08% - 1.39%
~~Figure 8 – Throughput of fwrite with and without APICv
With APICv the fwrite throughput increase ranging from 0.73% - 4.18%
Conclusion
APICv will help improve the throughput due to less overhead since activities relating to APIC are done in the hardware, not in VMM. APICv also increases CPU utilization resulting in less idle time for servers.
References
[1] http://en.wikipedia.org/wiki/Virtualization
[2] http://en.wikipedia.org/wiki/Programmable_Interrupt_Controller
[3] http://en.wikipedia.org/wiki/Virtual_machine
[4] http://en.wikipedia.org/wiki/Hypervisor
[5] http://searchservervirtualization.techtarget.com/definition/guest-OS
[6] http://www.linuxplumbersconf.org/2012/wp-content/uploads/2012/09/2012-lpc-virt-intel-vt-feat-nakajima.pdf
[7] http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-system-programming-manual-325384.pdf
[8] http://www.iozone.org/