Manually setting the Uncore frequency on Intel CPUs

After receiving several inquiries about how to manually set to Uncore frequency I wrote this article on the subject so it can be used as a refence by others. In it I briefly discuss what the Uncore frequency is, why you should care about it, and suggest two ways to set it. A more detailed account on the Uncore and its impact can be found in our latest ISC paper [1] (try the pre-print version of the article if you have trouble accessing the conference proceedings for free from within your network).

Uncore frequency domain

If you are unfamiliar with the nomenclature, let’s bring you up to speed: Each component of an Intel processor belongs either to the “core” or the “Uncore” part of the chip. The core part comprises the individual computational cores and their private L1 and L2 caches; the Uncore part everything else, such as, e.g., the shared last-level cache, memory controllers, PCIe interfaces, etc. Before Intel’s Haswell microarchitecture, the core and Uncore parts of the chip shared the same frequency domain. This meant setting your CPU cores to run at a clock frequency of 3 GHz resulted in the Uncore running at the same frequency.

Starting with the Haswell microarchitecture things changed (see [1] for rationale). Processors now feature separate frequency domains for the cores (in fact, each core’s frequency can now be set individually) and the Uncore. This means you can run your cores and the Uncore at different clock speeds. Together with this change, Intel introduced a feature called Uncore frequency scaling (UFS), which, when enabled, allows the processor to dynamically change the Uncore frequency based on the current workload. When UFS is disabled the Uncore is clocked at its maximum clock frequency.

Implications

The Uncore frequency can impact a number of key characteristics of your chip. For example, the Uncore clock has a direct impact on sustained main memory bandwidth. This is shown in Figure 1 which shows full-chip sustained main memory bandwidth for the Schönauer vector triad for different Uncore frequencies measured on Haswell-EP (Xeon E5-2695 v3) and a Broadwell-EP (Xeon E5-2697 v4). Increasing the Uncore clock increases bandwidth between main memory and the cores up to the point where the interconnect is fast enough to deliver the full main memory bandwidth; at this point, main memory becomes the bottleneck and increasing the Uncore clock further will no longer result in higher bandwidth.

From a performance perspective it makes sense to make the Uncore run at least at the frequency that is required to attain the maximum main memory bandwidth (i.e., 2.1 GHz or higher for the Haswell-EP processor respectively 1.9 GHz or higher for the Broadwell-EP processor). From an energy perspective, however, it makes sense to set the Uncore frequency exactly to the frequency at which the maximum main memory bandwidth is achieved; increasing the Uncore clock beyond this point does no longer increase performance but will increase power consumption. Unfortunately, we found that UFS will run the Uncore at its maximum frequency at the slightest hint at memory starvation. While this is guaranteed to get the maximum performance, it will do so at the cost of energy-efficiency, because this performance could also have been reached at a much lower Uncore frequency. Disabling UFS will not help either: Without UFS the Uncore will always run at its maximum frequency. Optimizing energy-efficiency for memory-bound codes thus requires manually setting the Uncore frequency.

So far not manually setting the Uncore frequency only had implications on energy-efficiency. We have, however, observed that using the chip in any of the possible default settings (i.e., running the chip with either UFS enabled or disabled) can also be detrimental to performance! Although the LINPACK benchmark is notoriously compute-bound, it has a non-negligible bandwidth requirement; as a result UFS will clock the chip to its maximum frequency (this is of course also the case when UFS is disabled). LINPACK typically pushes the power consumption of a chip to its TDP limit, which means that both the cores as well as the Uncore are competing for the limited power budget. Because the Uncore is clocked higher than necessary its share of power from the budget is higher than necessary as well; this in turn leaves less of the budget for the computational cores which limits their attainable frequencies. Manually adjusting the Uncore clock allowed us to to locate the Uncore frequency sweet spot for LINPACK that leaves the maximum amount of the power budget for the CPU cores while providing sufficient memory bandwidth not to starve the cores of data (again, see [1] for details).

Setting the Uncore frequency

The Uncore frequency is adjusted via the lower 16 bits of MSR 0x620. The register contents represent upper and lower bounds for the Uncore frequency: Bits 15 through 8 encode the minimum and bits 7 to 0 the maximum frequency. To derive a frequency from each of the two eight bit groups the integer encoded in the bits has to be multiplied with 100 MHz.

To access the MSR I recommend using Intel’s msr-tools (note that the msr kernel module needs to be loaded!). You use rdmsr for reading and wrmsr for writing MSRs. Now that we have all of the theory covered, let’s have a look at some examples.

A good way to start is to find out the supported Uncore frequency range of your processor. Enabling UFS in the BIOS will set minimum and maximum allowed frequencies in MSR 0x620 to the minimum and maximum supported Uncore frequencies of your processor. Enabling UFS in the BIOS of a system containing two Xeon E5-2697 v4 processors and reading the MSR’s contents for one of the processors in the system (the -p flag is used to specify the processor number in a multi-socket system) yields the following:

broadep2:~ $ rdmsr -p 0 0x620

Note that rdmsr outputs the register’s contents in hex (unfortunately it does not prefix the hex value with 0x as it should to avoid confusion between hex and decimal!) and does not print leading zeroes. Taking the liberty of fixing rdmsr’s shortcomings, the MSR’s contents in hex are 0x0c1c. The first eight bits, i.e., 0x0c, correspond to a decimal value of 12; the second eight bits, i.e., 0x1c, to a decimal value of 28. Multiplying these values with 100 MHz yields minimum and maximum Uncore frequencies of 1.2 GHz and 2.8 GHz, respectively.

Setting minimum and maximum frequencies is done via wrmsr. To set new a new minimum frequency of 1.6 GHz and a new maximum frequency of 2 GHz the frequencies are first divided by 100 MHz, yielding decimal values of 16 and 20. These are then converted to hexadecimal (0x10 for 16 and 0x14 for 20) and concatenated: 0x1016. The command line to make the frequency adjustments would therefore be: wrmsr -p 0 0x620 0x1016.

If you’d like to fix the Uncore to a specific frequency, make the minimum and maximum frequency identical; e.g., if you want the Uncore fixed at 2 GHz you would use wrmsr -p 0 0x620 0x1616.

Note that specifying a frequency outside the supported range will not have any effect: Specifing a frequency below or above the supported range will result in the frequency being capped at the lowest or highest supported frequency, respectively.

If you’re looking for an easier way to set the Uncore frequency then the likwid tool suite might be for you. Make sure to use at least version 4.3.0 as support for setting the Uncore frequency is pretty new. Using likwid-perfctr in combination with its --umin and --umax parameters provides a more pleasant interface than accessing the MSR directly. You can find more information about setting the Uncore frequency with likwid here.

Measuring the Uncore frequency to check whether your changes had any effect is done most easily via likwid-perfctr (which is also part of the likwid tool suite). Figure 2 shows Uncore frequency and package power consumption based on the UNCORE_CLOCK and PWR_PKG_ENERGY hardware events for an idle Xeon E5-2697 v4 chip. The minimum and maximum Uncore frequencies were initially set to 1.2 GHz using wrmsr and increased by 100 MHz every five minutes. We find that changes made via wsmsr to MSR 0x620 were successfully realized as refleced by the measured Uncore frequency; in addition, the graph shows the Uncore frequencies impact on energy consumption: Going from 1.2 GHz to 2.9 GHz doubles the dissipated power—even when the chip is idle!