1 Basics

2 Terminology

• CAT Cache Allocation Technology
• CBM Capacity BitMasks
• CDP Code and Data Prioritization
• COS/CLOS Class of Service
• HW Hardware
• MBA Memory Bandwidth Allocation
• MSRs Machine Specific Registers
• PSR Intel Platform Shared Resource
• THRTL Throttle value or delay value

3 Overview

The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory bandwidth available per-core. This feature provides OS/ hypervisor the ability to slow misbehaving apps/domains by using a credit-based throttling mechanism.

4 User details

• Feature Enabling:

  Add “psr=mba” to boot line parameter to enable MBA feature.

• xl interfaces:

  1. psr-mba-show [domain-id|domain-name]:

     Show memory bandwidth throttling for domain. Under different modes, it shows different type of data.

     There are two modes: Linear mode: the input precision is defined as 100-(MBA_MAX). For instance, if the MBA_MAX value is 90, the input precision is 10%. Values not an even multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10% delay applied) by HW automatically. The response of throttling value is linear.

     Non-linear mode: input delay values are powers-of-two from zero to the MBA_MAX value from CPUID. In this case any values not a power of two will be rounded down the next nearest power of two by HW automatically. The response of throttling value is non-linear.

     For linear mode, it shows the decimal value. For non-linear mode, it shows hexadecimal value.

  2. psr-mba-set [OPTIONS] <domain-id|domain-name> <throttling>:

     Set memory bandwidth throttling for domain.

     Options: ‘-s’: Specify the socket to process, otherwise all sockets are processed.

     Throttling value set in register implies the approximate amount of delaying the traffic between core and memory. Higher throttling value result in lower bandwidth. The max throttling value (MBA_MAX) supported can be obtained through CPUID inside hypervisor. Users can fetch the MBA_MAX value using the psr-hwinfo xl command.

5 Technical details

MBA is a member of Intel PSR features, it shares the base PSR infrastructure in Xen.

5.1 Hardware perspective

MBA defines a range of MSRs to support specifying a delay value (Thrtl) per COS, with details below.

 +----------------------------+----------------+
 | MSR (per socket)           |    Address     |
 +----------------------------+----------------+
 | IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
 +----------------------------+----------------+
 | ...                        |  ...           |
 +----------------------------+----------------+
 | IA32_L2_QOS_Ext_BW_Thrtl_n |     0xD50+n    |
 +----------------------------+----------------+

When context switch happens, the COS ID of domain is written to per-hyper- thread MSR IA32_PQR_ASSOC, and then hardware enforces bandwidth allocation according to the throttling value stored in the Thrtl MSR register.

5.2 The relationship between MBA and CAT/CDP

Generally speaking, MBA is completely independent of CAT/CDP, and any combination may be applied at any time, e.g. enabling MBA with CAT disabled.

But it needs to be noticed that MBA shares COS infrastructure with CAT, although MBA is enumerated by different CPUID leaf from CAT (which indicates that the max COS of MBA may be different from CAT). In some cases, a domain is permitted to have a COS that is beyond one (or more) of PSR features but within the others. For instance, let’s assume the max COS of MBA is 8 but the max COS of L3 CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for MBA, the HW works as default value is set since COS 9 is beyond the max COS (8) of MBA.

5.3 Design Overview

• Core COS/Thrtl association

  When enforcing Memory Bandwidth Allocation, all cores of domains have the same default Thrtl MSR (COS0) which stores the same Thrtl (0). The default Thrtl MSR is used only in hypervisor and is transparent to tool stack and user.

  System administrators can change PSR allocation policy at runtime by using the tool stack. Since MBA shares COS ID with CAT/CDP, a COS ID corresponds to a 2-tuple, like [CBM, Thrtl] with only-CAT enabled, when CDP is enabled, the COS ID corresponds to a 3-tuple, like [Code_CBM, Data_CBM, Thrtl]. If neither CAT nor CDP is enabled, things are easier, since one COS ID corresponds to one Thrtl.

• VCPU schedule

  This part reuses CAT COS infrastructure.

• Multi-sockets

  Different sockets may have different MBA capabilities (like max COS) although it is consistent on the same socket. So the capability of per-socket MBA is specified.

  This part reuses CAT COS infrastructure.

5.4 Implementation Description

• Hypervisor interfaces:

  1. Boot line param: “psr=mba” to enable the feature.

  2. SYSCTL: - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.

  3. DOMCTL: - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get throttling for a domain. - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set throttling for a domain.

• xl interfaces:

  1. psr-mba-show [domain-id] Show system/domain runtime MBA throttling value. For linear mode, it shows the decimal value. For non-linear mode, it shows hexadecimal value. => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL

  2. psr-mba-set [OPTIONS] Set bandwidth throttling for a domain. => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL

  3. psr-hwinfo Show PSR HW information, including L3 CAT/CDP/L2 CAT/MBA. => XEN_SYSCTL_PSR_MBA_get_info

• Key data structure:

  1. Feature HW info

     ``` struct { unsigned int thrtl_max; bool linear; } mba;

     • Member thrtl_max

       thrtl_max is the max throttling value to be set, i.e. MBA_MAX.

     • Member linear

       linear means the response of delay value is linear or not.

     As mentioned above, MBA is a member of Intel PSR features, it shares the base PSR infrastructure in Xen. For example, the ‘cos_max’ is a common HW property for all features. So, for other data structure details, please refer to ‘intel_psr_cat_cdp.pandoc’.

6 Limitations

MBA can only work on HW which supports it (check CPUID).

7 Testing

We can execute these commands to verify MBA on different HWs supporting them.

For example: 1. User can get the MBA hardware info through ‘psr-hwinfo’ command. From result, user can know if this hardware works under linear mode or non- linear mode, the max throttling value (MBA_MAX) and so on.

root@:~$ xl psr-hwinfo --mba
Memory Bandwidth Allocation (MBA):
Socket ID       : 0
Linear Mode     : Enabled
Maximum COS     : 7
Maximum Throttling Value: 90
Default Throttling Value: 0

2. Then, user can set a throttling value to a domain. For example, set ‘10’, i.e 10% delay.

root@:~$ xl psr-mba-set 1 10

3. User can check the current configuration of the domain through ‘psr-mab-show’. For linear mode, the decimal value is shown.

root@:~$ xl psr-mba-show 1
Socket ID       : 0
Default THRTL   : 0
   ID                     NAME            THRTL
    1                 ubuntu14             10

8 Areas for improvement

N/A

9 Known issues

N/A

10 References

“INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES” Intel 64 and IA-32 Architectures Software Developer Manuals, vol3

11 History