PatentTips - Virtualizing performance counters
BACKGROUND
Generally, the concept of virtualization in information processing systems allows multiple instances of one or more operating systems (each, an "OS") to run on a single information processing system, even though each OS is designed to have complete, direct control over the system and its resources. Virtualization is typically implemented by using software (e.g., a virtual machine monitor, or a "VMM") to present to each OS a "virtual machine" ("VM") having virtual resources, including one or more virtual processors, that the OS may completely and directly control, while the VMM maintains a system environment for implementing virtualization policies such as sharing and/or allocating the physical resources among the VMs (the "virtualization environment"). Each OS, and any other software, that runs on a VM is referred to as a "guest" or as "guest software," while a "host" or "host software" is software, such as a VMM, that runs outside of the virtualization environment.
A physical processor in an information processing system may support virtualization, for example, by supporting an instruction to enter a virtualization environment to run a guest on a virtual processor (i.e., a physical processor under constraints imposed by a VMM) in a VM. In the virtualization environment, certain events, operations, and situations, such as external interrupts or attempts to access privileged registers or resources, may be intercepted, i.e., cause the processor to exit the virtualization environment so that a VMM may operate, for example, to implement virtualization policies. A physical processor may also support other instructions for maintaining a virtualization environment, and may include memory or register bits that indicate or control virtualization capabilities of the physical processor.
A physical processor supporting a virtualization environment may include performance counters for logging performance monitoring information. Typically, each performance counter would be controlled by one or more control or configuration registers, or portions of a one or more control or configuration registers, associated with the counter. To virtualize these performance counters, each transition between a guest and a host or between two guests would typically require saving the state of all of the counters and their associated control registers and loading a new state for all of the counters and their associated control registers.
DETAILED DESCRIPTION
The performance of a virtualization environment may be improved by reducing the amount of state information that must be saved and loaded on transitions between a host and a guest and between guests. Embodiments of the present invention may be used to virtualize performance counters or other counters without requiring the saving and the loading of the contents of the counters and their associated control registers. Therefore, performance may be improved over a virtualization environment in which the contents of the counters and their associated control registers are saved on transitions. Embodiments of the present invention provide for performance counters to be efficiently enabled or disabled for any number of hosts and guests, for performance counters to be assigned for the exclusive use of a host or a guest, and for performance counters to be shared between any number of hosts and/or guests.
FIG. 1 illustrates an embodiment of the present invention in virtualization architecture 100. In FIG. 1, bare platform hardware 110 may be any data processing apparatus capable of executing any OS or VMM software. For example, bare platform hardware may be that of a personal computer, mainframe computer, portable computer, handheld device, set-top box, server, or any other computing system. Bare platform hardware 110 includes processor 120 and memory 130.
Processor 120 may be any type of processor, including a general purpose microprocessor, such as a processor in the Intel® Pentium® Processor Family, Itanium® Processor Family, or other processor family from Intel Corporation, or another processor from another company, or a digital signal processor or microcontroller. Although FIG. 1 shows only one such processor 120, bare platform hardware 110 may include any number of processors, including any number of multicore processors, each with any number of execution cores, and any number of multithreaded processors, each with any number of threads.
Memory 130 may be static or dynamic random access memory, semiconductor-based read-only or flash memory, magnetic or optical disk memory, any other type of medium readable by processor 120, or any combination of such mediums. Processor 120 and memory 130 may be coupled to or communicate with each other according to any known approach, such as directly or indirectly through one or more buses, point-to-point, or other wired or wireless connections. Bare platform hardware 110 may also include any number of additional devices or connections.
In addition to bare platform hardware 100, FIG. 1 illustrates VMM 140, VMs 150 and 160, guest operating systems 152 and 162, and applications 154,155, 164, and 165.
VMM 140 may be any software, firmware, or hardware host installed on or accessible to bare platform hardware 110, to present VMs, i.e., abstractions of bare platform hardware 110, to guests, or to otherwise create VMs, manage VMs, and implement virtualization policies. In other embodiments, a host may be any VMM, hypervisor, OS, or other software, firmware, or hardware capable of controlling bare platform hardware 110. A guest may be any OS, any VMM, including another instance of VMM 140, any hypervisor, or any application or other software.
Each guest expects to access physical resources, such as processor and platform registers, memory, and input/output devices, of bare platform hardware 110, according to the architecture of the processor and the platform presented in the VM. FIG. 1 shows two VMs, 150 and 160, with guest OS152 and guest applications 154 and 155 installed on VM 150 and guest OS 162 and guest applications 164 and 165 installed on VM 160. Although FIG. 1 shows only two VMs and two applications per VM, any number of VMs may be created, and any number of applications may run on each VM within the scope of the present invention.
A resource that can be accessed by a guest may either be classified as a "privileged" or a "non-privileged" resource. For a privileged resource, VMM140 facilitates the functionality desired by the guest while retaining ultimate control over the resource. Non-privileged resources do not need to be controlled by VMM 140 and may be accessed directly by a guest.
Furthermore, each guest OS expects to handle various events such as exceptions (e.g., page faults, and general protection faults), interrupts (e.g., hardware interrupts and software interrupts), and platform events (e.g., initialization and system management interrupts). These exceptions, interrupts, and platform events are referred to collectively and individually as "virtualization events" herein. Some of these virtualization events are referred to as "privileged events" because they must be handled by VMM 140 to ensure proper operation of VMs 150 and 160, protection of VMM 140 from guests, and protection of guests from each other.
At any given time, processor 120 may be executing instructions from VMM 140 or any guest, thus VMM 140 or the guest may be running on, or in control of, processor 120. When a privileged event occurs or a guest attempts to access a privileged resource, control may be transferred from the guest to VMM 140. The transfer of control from a guest to VMM 140 is referred to as a "VM exit" herein. After handling the event or facilitating the access to the resource appropriately, VMM 140 may return control to a guest. The transfer of control from VMM 140 to a guest is referred to as a "VM entry" herein.
In the embodiment of FIG. 1, processor 120 controls the operation of VMs 150 and 160 according to data stored in virtual machine control structure ("VMCS") 132. VMCS 132 is a structure that may contain state of a guest or guests, state of VMM 140, execution control information indicating how VMM 140 is to control operation of a guest or guests, information regarding VM exits and VM entries, and any other such information. Processor 120reads information from VMCS 132 to determine the execution environment of a VM and constrain its behavior. In this embodiment, VMCS 132 is stored in memory 130. In some embodiments, multiple VMCSs are used to support multiple VMs. Although FIG. 1 shows VMCS 132 stored in memory 130, storing a VMCS in a memory is not required by the present invention.
Processor 120 includes counters 171, 173, 175, and 177. In this embodiment, counters 171, 173, 175, and 177 are performance counters that may be programmed by software running on processor 120 to log performance monitoring information; however, other embodiments may include any number of counters and/or any type or size of counter. For example, any of performance counters 171, 173, 175, or 177 may be programmed to increment for each occurrence of a selected event, or to increment for each clock cycle during a selected event. The events may include any of a variety of events related to execution of program code on processor 120, such as branch mispredictions, cache hits, cache misses, translation lookaside buffer hits, translation lookaside buffer misses, etc. Therefore, performance counters 171, 173, 175, and 177 may be used for tuning or profiling program code to yield the best possible performance on processor 120.
Processor 120 also includes counter control storage locations 172, 174, 176, and 178, corresponding to counters 171, 173, 175, and 177, respectively. Counter control storage locations 172, 174, 176, and 178 may be registers or any other structures of any size, or portions of one or more registers of other structures, for storing information to control or configure counters 171, 173, 175, and 177, respectively. Counter control storage locations 172, 174, 176, and 178 may be programmed to store information to control or configure counters 171, 173, 175, and 177, respectively, such as information to enable the counter, to select the event to be counted, to select the method of counting (e.g., number of occurrences or duration of event), to select conditions for counting (e.g., based on privilege level of software executing when event is detected) and to set any other control, configuration, or other variables.
Processor 120 also includes global counter enable storage location 121, which may be one or more bits in a control register, configuration register, model specific register, or any other storage location to store a global counter enable indicator. In one embodiment, global counter enable storage location 121 may be a single bit of a programmable register, where the bit may be set to a logical one to set the global counter enable indicator to an "enable" value or a logical zero to set the global counter enable indicator to a "disable" value.
In another embodiment, a global counter enable storage location may include one bit per performance counter, where each bit may be set to a logical one to set an individual counter enable indicator for a corresponding performance counter to an "enable" value. In other embodiments, a single bit or field of a global counter enable storage location may correspond to a group of any number of performance counters. In an of these embodiments, a global enable storage location ma include an number of bits or fields that correspond to any number of performance counters, and may further include or be associated with one or more additional enable indicators that may be used to control any number of bits or fields that more directly control performance counters. For example, a storage location may include a field to store a "counter enable vector," which includes one bit per counter, and an additional bit within this storage location or in a storage location or data structure elsewhere in processor 120 or virtualization architecture 100 may be used to enable or disable the counter enable vector. In this way, individual counters may be individually controlled by guests according to a counter enable vector stored in a VMCS, but the default value of the additional bit may be set such that the individual counter control is automatically disabled when executing a VMM or other host software designed for a processor that does not support a counter enable vector model.
The value of the global counter enable indicator is used, along with any other relevant information stored in counter control storage locations 172, 174,176, and 178, by counter enable logic 170 to control the operation of counters 171, 173, 175, and 177, respectively. If the global counter enable indicator is set to an "enable" value, then each counter operates according to its individual control and configuration information. For example, counter171 may increment on a cache hit because the contents of counter control storage location 172 include an individual counter enable bit set to an "enable" value and an event select field set to count cache hits, while at the same time counter 173 may not increment because the contents of counter control storage location 174 include an individual counter enable bit set to a "disable" value. However, if the global counter enable indicator is set to a "disable" value, all counters 171, 173, 175, and 177 are disabled and do not operate according to their individual control and configuration information. For example, counter 171 would not increment on a cache hit even if the contents of counter control storage location 172 include an individual counter enable bit set to an "enable" value and an event select field set to count cache hits.
Additionally, processor 120 includes control logic 180 to support virtualization, including the virtualization of counters 171, 173, 175, and 177. Control logic 180 may be microcode, programmable logic, hard-coded logic, or any other form of control logic within processor 120. In other embodiments, control logic 180 may be implemented in any form of hardware, software, or firmware, such as a processor abstraction layer, within a processor or within any component accessible or medium readable by a processor, such as memory 130.
Control logic 180 causes processor 120 to execute method embodiments of the present invention, such as the method embodiments described below with reference to FIG. 2, for example, by causing processor 120 to include the execution of one or more micro-operations, e.g., to support virtualization, in its response to virtualization instructions or other instructions from a host or guest.
Control logic 180 includes VM entry logic 181 to transfer control of processor 120 from a host to a guest (i.e., a VM entry) and VM exit logic 182 to transfer control of processor 120 from a guest to a host (i.e., a VM exit). In some embodiments, control may also be transferred from a guest to a guest or from a host to a host. For example, in an embodiment supporting layered virtualization, software running on a VM on processor 120 may be both a guest and a host (e.g., a VMM running on a VM is a guest to the VMM that controls that VM and a host to a guest running on a VM that it controls).
Control logic 180 also includes guest state load logic 183, host state load logic 184, and guest state store logic 185. Guest state load logic 183 is to load guest state from VMCS 132 to processor 120 on a VM entry. Host state load logic 184 is to load host state from VMCS 132 to processor 120 on a VM exit. Guest state store logic 185 is to store guest state from processor 120 to VMCS 132 on a VM exit. In some embodiments control logic 180 also includes host state store logic 186 to store host state from processor 120 to VMCS 132 on a VM entry prior to guest state load logic 183 loading guest state into processor 120. In some embodiments, control logic 180 may load and store state residing in other system components, e.g., input-output devices, memory controllers.
VMCS 132 may include fields, control bits, or other data structures to support virtualization. These data structures may be checked or otherwise referred to by control logic 180 to determine how to manage a VM environment. For example, guest state load enable indicator 133 may be set to cause guest state load logic 183 to load a guest value from guest state storage location 137 into global counter enable storage location 121 in connection with a VM entry, host state load enable indicator 134 may be set to cause host state load logic 184 to load a host value from host state storage location138 into global counter enable storage location 121 in connection with a VM exit, guest state store enable indicator 135 may be set to cause guest state store logic 185 to store the contents of global enable storage location 121 in guest state storage location 137, and host state store enable indicator 136 may be set to cause host state store logic 186 to store the contents of global enable storage location 121 host state storage location138, all as further described below. In this description of this embodiment, these indicators are control bits that are set to enable or cause a desired effect, where set means writing a logical one to the bit, but any logic convention or nomenclature may be used within the scope of the present invention.
Using the mechanism described herein, a VMM may enable counting of events that occur only while a guest executes, only while the VMM executes or while both the VMM and a guest execute. Additionally, a VMM may, on a guest by guest basis, enable or disable performance counters as part of the transition (i.e., the VM entry or VM exit) between the guest and the VMM. This mechanism allows the VMM to "hide" its effect on the performance counters from a guest. Alternatively, this mechanism allows counting of events in the VMM for use by a guest, or in a guest for use by the VMM.
FIG. 2 illustrates an embodiment of the present invention in method 200, a method for virtualizing performance counters. Although method embodiments are not limited in this respect, reference is made to virtualization architecture 100 of FIG. 1 to describe the method embodiment of FIG. 2.
In box 210 of FIG. 2, a first performance counter is configured to count occurrences of a first event. For example, counter 171 may be configured, by programming counter control storage location 172, to count cache misses. In box 212, a second performance counter is configured to count occurrences of a second event. For example, counter 173 may be configured, by programming counter control storage location 174, to count cycles not halted. In other embodiments, the performance counters may be configured to count occurrences of any other events, e.g., translation lookaside buffer misses, branch mispredictions, etc.
In box 220 of FIG. 2, VMM 140 of FIG. 1 creates a VMCS (e.g., VMCS 132) for a VM. In boxes 222 to 226, VMM 140 configures VMCS 132 to implement support for virtualizing counters 171, 173, 175, and 177. In box 222, guest state load enable indicator 133 is set to cause guest state load logic 183 to load a guest value from guest state storage location 137 into global counter enable storage location 121 in connection with a VM entry. In box 224, host state load enable indicator 134 is set to cause host state load logic 184 to load a host value from host state storage location 138 into global counter enable storage location 121 in connection with a VM exit. In box 226, guest state store enable indicator 135 is set to cause guest state store logic 185 to store the contents of global enable storage location 121 in guest state storage location 136.
In box 230, a transfer of control of processor 120 from the host (i.e., VMM 140) to a guest is initiated. For example, the VMM may initiate a VM entry. The VM entry may include VM entry logic 181 causing processor 120 to execute operations or micro-operations to save the host state and load the guest state. In box 232, VM entry logic determines whether to load a guest value from guest state storage location 137 into global counter enable storage location 121 based on guest state load enable indicator 133. If guest state load enable indicator 133 is set, then, in box 234, guest state load logic 183 causes the guest value from guest state storage location 137 to be loaded into global counter enable storage location 121; otherwise, box234 is not performed. In box 236, the VM entry is completed and control is transferred to the guest. In box 238, the guest begins or continues to execute.
In box 239, counter enable logic 170 determines whether the global counter enable indicator is set. If not, method 200 continues at box 258. If so, method 200 continues at box 240. In box 240, it is determined whether the first event has occurred. If so, then, in box 244, the first performance counter operates as configured, i.e., in this embodiment, where it has been configured in box 210 to count cache misses, the first performance counter increments.
In box 250, it is determined whether the second event has occurred. If so, then, in box 254, the second performance counter operates as configured, i.e., in this embodiment, where it has been configured in box 212 to count occurrences of the second event, the second performance counter increments.
In box 258, it is determined whether a virtualization event has occurred. If not, method 200 continues at box 238. If so, then, in box 260, a transfer of control of processor 120 from the guest to the host is initiated. The VM exit may include VM exit logic 182 causing processor 120 to execute operations or micro-operations to save the guest state and load the host state.
In box 262, VM exit logic determines whether to store the contents of global counter enable storage location 121 in guest state storage location 137based on guest state store enable indicator 135. If guest state store enable indicator 135 is set, then, in box 264, guest state store logic 185 stores the contents of global counter enable storage location 121 in guest state storage location 137; otherwise, box 264 is not performed. In some embodiments, there is no guest state store enable indicator 135. In some embodiments, guest state store logic 185 always stores the contents of global counter enable storage location 121 in guest state storage location 137. In other embodiments, guest state store logic 185 never stores the contents of global counter enable storage location 121 in guest state storage location 137 as part of VM exit processing. In some embodiments, attempts by guest software to access the global counter enable storage location 121 may cause VM exits, either by software convention (e.g., the VMM is required to appropriately set controls in the VMCS to cause VM exits on access to global counter enable storage location 121) or by forcing such a VM exit by the processor.
In box 266, VM exit logic determines whether to load a host value from host state storage location 138 into global counter enable storage location 121based on host state load enable indicator 134. If host state store enable indicator 134 is set, then, in box 268, host state load logic 184 causes the host value from host state storage location 138 to be loaded into global counter enable storage location 121; otherwise, box 268 is not performed.
In box 270, the VM exit is completed and control is transferred to the host. In box 272, the host begins or continues to execute.
In box 275, counter enable logic 170 determines whether the global counter enable indicator is set. If not, method 200 continues at box 298. If so, method 200 continues at box 280. In box 280, it is determined whether the first event has occurred. If so, then, in box 284, the first performance counter operates as configured, i.e., in this embodiment, where it has been configured in box 210 to count cache misses, the first performance counter increments.
In box 290, it is determined whether the second event has occurred. If so, then, in box 294, the second performance counter operates as configured, i.e., in this embodiment, where it has been configured in box 212 to count occurrences of the second event, the second performance counter increments.
In box 298, it is determined whether a VM entry is to occur. If so, method 200 continues at box 230. If not, method 200 continues at box 299. In box299, it is determined whether a halt or other such instruction is to be executed. If not, method 200 continues at box 272. If so, method 200 ends.
FIG. 3 illustrates another embodiment of the invention in bare platform hardware 310, which may be used in virtualization architecture 100 or another virtualization architecture. Except as otherwise described, the description of bare platform hardware and its elements also applies to bare platform hardware 310, and its corresponding elements. Bare platform hardware 310 includes processor 320 and memory 330.
Processor 320 includes counters 371, 373, 375, and 377. Processor 320 also includes counter control storage locations 372, 374, 376, and 378, corresponding to counters 371, 373, 375, and 377, respectively, which each contain a counter valid storage location to store a counter valid indicator to enable or disable the corresponding counter, along with other control or configuration information. Additionally, processor 320 includes counter identifier storage locations 391, 393, 395, and 397, corresponding to counters 371, 373, 375, and 377, respectively. Counter identifier storage locations 391, 393, 395, and 397 may be registers or any other structures of any size, or portions of one or more registers of other structures, for storing information to identify or label counters 371, 373, 375, and 377, respectively. Counter identifier storage locations 391, 393, 395, and 397 may be programmed to store an identifier to uniquely or redundantly (i.e., two counter identifier storage locations could be programmed with the same value) identify counters 371, 373, 375, and 377, respectively.
VMCS 332 includes counter identifier field 333, which in this embodiment is the same number of bits as each of counter identifier storage location 391,393, 395, and 397, but in other embodiments may be any number of bits. Other embodiments may include additional counter identifier fields. Counter identifier field 333 may be programmed, by the VMM that creates VMCS 332 or by any other software, to store a value that may or may not match one or more of the values stored in counter identifier storage locations 391, 393, 395, and 397.
Processor 320 also includes comparison logic 322, which may be implemented according to any known approach, to generate a counter match signal for each counter based on a comparison of the contents of a corresponding counter identifier storage location and the contents of counter identifier field 333. In this embodiment, the counter match signal for a counter is asserted if the contents of the corresponding, counter identifier storage location match the contents of counter identifier field 333. In this embodiment, the counter match signal for a counter is also asserted if the contents of the corresponding counter identifier storage location equals a value of zero, regardless of the contents of counter identifier field 333, to provide for software that does not use the counter identifier feature of the present invention to operate as expected on hardware that does include the counter identifier feature. In another embodiment, the counter match logic may determine a "match" based at least in part on other factors such as the contents of other fields within the VMCS. For example, the value of various control registers such as the CR0, CR3 or CR4 registers in the Intel Architecture may be consulted. Alternatively, the counter match logic may include a determination of whether the software currently executing is the VMM or a guest, and enable the counters only if guest software is currently executing, based on one or more control bits in the VMCS.
Embodiments may include one or more "counter match enable" control bits or fields to enable or disable the counter match feature, by disabling the comparison logic or by any other desired approach. For example, if a counter match enable bit is set, then the comparison happens as described above. However, if the counter match enable bit is cleared, the counters are always enabled and no comparison is made, so that a "zero" value in a counter identifier storage location does not require the special treatment described above.
The counter match signal for each counter is used, along with any other relevant information stored in counter control storage locations 372, 374, 376, and 378, by counter enable logic 370 to control the operation of counters 371, 373, 375, and 377, respectively. If the counter match signal for a counter is asserted, then the counter operates according to its individual control and configuration information. For example, counter 371 may increment on a cache hit if the contents of counter control storage location 372 include an individual counter valid bit set to an "enable" value and an event select field set to count cache hits. However, if the counter match signal for a counter is not asserted, the counter is disabled and does not operate according to its individual control and configuration information. For example, counter 371 would not increment on a cache hit even if the contents of counter control storage location 372 include an individual counter valid bit set to an "enable" value and an event select field set to count cache hits.
Additionally, processor 320 includes control logic 380 to support virtualization, including the virtualization of counters 371, 373, 375, and 377. Control logic 380 causes processor 320 to execute method embodiments of the present invention, such as the method embodiments described below with reference to FIG. 4, for example, by causing processor 320 to include the execution of one or more micro-operations, e.g., to support virtualization, in its response to virtualization instructions or other instructions from a host or guest.
FIG. 4 illustrates an embodiment of the present invention in method 400, another method for virtualizing performance counters. Although method embodiments are not limited in this respect, reference is made to FIGS. 1 and 3 to describe the method embodiment of FIG. 4.
In box 410 of FIG. 4, a performance counter is configured to count occurrences of an event. For example, counter 371 may be configured, by programming counter control storage location 372, to count cache misses. In box 412, counter identifier storage location 391 is programmed with a desired counter identification value.
In box 420, a host creates a VMCS (e.g., VMCS 332) for a VM. In box 422, the host configures VMCS 332 to implement support for virtualizing counters371, 373, 375, and 377, including programming counter identifier field 333 with a value equal to the counter identification value used on box 412. In this embodiment, any number of other counter identifier fields in VMCS 332 or any other VMCSes, corresponding to any number of other guests or hosts, may also be programmed with the same value to allow any number of guests or hosts to share one or more counters.
In box 430, control of processor 320 is transferred from the host to a guest. In box 432, the guest begins or continues to execute. In box 434, a counter match signal is generated based on a comparison of counter identifier storage location 391 and counter identifier field 333. In this embodiment, the counter match signal is asserted if the contents of the counter identifier storage location match the contents of the counter identifier field.
In box 440, an occurrence of the event, i.e., in this embodiment, a cache miss, is recognized. In box 442, counter enable logic 370 determines whether to increment the performance counter based on the counter match signal and the contents of the counter valid storage location in the corresponding counter control storage locations (e.g., counter control storage location 372).
If the counter match signal is asserted and the counter's counter valid indicator is set, then, in box 444, the performance counter operates as configured, i.e., in this embodiment, where it has been configured in box 410 to count cache misses, the performance counter increments. If the counter match signal is not asserted or the counter's valid indicator is not set, box 444 is not performed.
Within the scope of the present invention, the methods illustrated in FIGS. 2 and 4 may be performed in a different order, with illustrated boxes omitted, with additional boxes added, or with a combination of reordered, omitted, or additional boxes. For example, a VMCS may configure a VMCS in any order, e.g., boxes 222 to 226 may be rearranged in any order.
Processor 120, or any other component or portion of a component designed according to an embodiment of the present invention, may be designed in various stages, from creation to simulation to fabrication. Data representing a design may represent the design in a number of manners. First, as is useful in simulations, the hardware may be represented using a hardware description language or another functional description language. Additionally or alternatively, a circuit level model with logic and/or transistor gates may be produced at some stages of the design process. Furthermore, most designs, at some stage, reach a level where they may be modeled with data representing the physical placement of various devices. In the case where conventional semiconductor fabrication techniques are used, the data representing the device placement model may be the data specifying the presence or absence of various features on different mask layers for masks used to produce an integrated circuit.