Intel Memory Bandwidth Allocation (MBA) Feature

Revision 1.8

1 Basics

Status: Tech Preview
Architecture(s): Intel x86
Component(s): Hypervisor, toolstack
Hardware: MBA is supported on Skylake Server and beyond

2 Terminology

3 Overview

The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory bandwidth available per-core. This feature provides OS/ hypervisor the ability to slow misbehaving apps/domains by using a credit-based throttling mechanism.

4 User details

5 Technical details

MBA is a member of Intel PSR features, it shares the base PSR infrastructure in Xen.

5.1 Hardware perspective

MBA defines a range of MSRs to support specifying a delay value (Thrtl) per COS, with details below.

 +----------------------------+----------------+
 | MSR (per socket)           |    Address     |
 +----------------------------+----------------+
 | IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
 +----------------------------+----------------+
 | ...                        |  ...           |
 +----------------------------+----------------+
 | IA32_L2_QOS_Ext_BW_Thrtl_n |     0xD50+n    |
 +----------------------------+----------------+

When context switch happens, the COS ID of domain is written to per-hyper- thread MSR IA32_PQR_ASSOC, and then hardware enforces bandwidth allocation according to the throttling value stored in the Thrtl MSR register.

5.2 The relationship between MBA and CAT/CDP

Generally speaking, MBA is completely independent of CAT/CDP, and any combination may be applied at any time, e.g. enabling MBA with CAT disabled.

But it needs to be noticed that MBA shares COS infrastructure with CAT, although MBA is enumerated by different CPUID leaf from CAT (which indicates that the max COS of MBA may be different from CAT). In some cases, a domain is permitted to have a COS that is beyond one (or more) of PSR features but within the others. For instance, let’s assume the max COS of MBA is 8 but the max COS of L3 CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for MBA, the HW works as default value is set since COS 9 is beyond the max COS (8) of MBA.

5.3 Design Overview

5.4 Implementation Description

6 Limitations

MBA can only work on HW which supports it (check CPUID).

7 Testing

We can execute these commands to verify MBA on different HWs supporting them.

For example: 1. User can get the MBA hardware info through ‘psr-hwinfo’ command. From result, user can know if this hardware works under linear mode or non- linear mode, the max throttling value (MBA_MAX) and so on.

root@:~$ xl psr-hwinfo --mba
Memory Bandwidth Allocation (MBA):
Socket ID       : 0
Linear Mode     : Enabled
Maximum COS     : 7
Maximum Throttling Value: 90
Default Throttling Value: 0
  1. Then, user can set a throttling value to a domain. For example, set ‘10’, i.e 10% delay.
root@:~$ xl psr-mba-set 1 10
  1. User can check the current configuration of the domain through ‘psr-mab-show’. For linear mode, the decimal value is shown.
root@:~$ xl psr-mba-show 1
Socket ID       : 0
Default THRTL   : 0
   ID                     NAME            THRTL
    1                 ubuntu14             10

8 Areas for improvement

N/A

9 Known issues

N/A

10 References

“INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES” Intel 64 and IA-32 Architectures Software Developer Manuals, vol3

11 History

Date Revision Version Notes

2017-01-10 1.0 Xen 4.9 Design document written 2017-07-10 1.1 Xen 4.10 Changes: 1. Modify data structure according to latest codes; 2. Add content for ‘Areas for improvement’; 3. Other minor changes. 2017-08-09 1.2 Xen 4.10 Changes: 1. Remove a special character to avoid error when building pandoc. 2017-08-15 1.3 Xen 4.10 Changes: 1. Add terminology ‘HW’. 2. Change ‘COS ID of VCPU’ to ‘COS ID of domain’. 3. Change ‘COS register’ to ‘Thrtl MSR’. 4. Explain the value shown for ‘psr-mba-show’ under different modes. 5. Remove content in ‘Areas for improvement’. 2017-08-16 1.4 Xen 4.10 Changes: 1. Add ‘<>’ for mandatory argument. 2017-08-30 1.5 Xen 4.10 Changes: 1. Modify words in ‘Overview’ to make it easier to understand. 2. Explain ‘linear/non-linear’ modes before mention them. 3. Explain throttling value more accurate. 4. Explain ‘MBA_MAX’. 5. Correct some words in ‘Design Overview’. 6. Change ‘mba_info’ to ‘mba’ according to code changes. Also, modify contents of it. 7. Add context in ‘Testing’ part to make things more clear. 8. Remove ‘n<64’ to avoid out-of-sync. 2017-09-21 1.6 Xen 4.10 Changes: 1. Add ‘domain-name’ as parameter of ‘psr-mba-show/ psr-mba-set’. 2. Fix some wordings. 3. Explain how user can know the MBA_MAX. 4. Move the description of ‘Linear mode/Non-linear mode’ into section of ‘psr-mba-show’. 5. Change ‘per-thread’ to ‘per-hyper-thread’. 2017-09-29 1.7 Xen 4.10 Changes: 1. Correct some words. 2. Change ‘xl psr-mba-set 1 0xa’ to ‘xl psr-mba-set 1 10’ 2017-10-08 1.8 Xen 4.10 Changes: 1. Correct some words. ———- ——– ——– ——————————————-