-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Xen Security Advisory CVE-2015-7972 / XSA-153 version 4 x86: populate-on-demand balloon size inaccuracy can crash guests UPDATES IN VERSION 4 ==================== Normalize version tags ISSUE DESCRIPTION ================= The design of the memory populate-on-demand (PoD) system requires that a guest's memory ballooning driver reach its memory reduction target. The target is not entirely well-defined in terms of the information visible to the appropriate parts of the system, so some unknown set of guests (but probably most guests) will fail this criterion. If the guest memory balloon driver does not free sufficient memory to reach its target, the guest will proceed to run with a nonzero number of outstanding PoD pages. When the guest or management toolstack touches such a page, the hypervisor would search the guest memory for a page containing only zeroes. If no such page is found, the guest crashes. Prior to the patch for XSA-150, the search might lock up the relevant physical cpu for a while. After the patch to XSA-150, it might crash the guest even if a suitable zero page is available. This means that in the current arrangements toolstack software must apply an adjustment to a guest's PoD target as supplied to Xen. Neither xend nor libxl do this. IMPACT ====== Guests configured with PoD might be unstable, especially under load. In an affected guest, an unprivileged guest user might be able to cause a guest crash, perhaps simply by applying load so as to cause heavy memory pressure within the guest. This problem also allows an unprivileged guest user to exercise the separate vulnerability described in XSA-150: an unprivileged guest user might be able to cause a denial of service affecting the host. VULNERABLE SYSTEMS ================== The vulnerability is restricted to HVM guests which have been constructed in Populate-on-Demand mode (ie, with memory < maxmem). ARM is not vulnerable. x86 PV VMs are not vulnerable. x86 HVM domains without PoD (ie started with memory==maxmem, or without mentioning "maxmem" in the guest config file) are not vulnerable. Systems using libxl (whether via xl, or libvirt, or another higher layer) or xend (whether via xm, or libvirt, or another higher layer) are vulnerable. If the system has been stress-tested (by imposing memory load on the guest) and found to be stable, it is less likely that the guest is vulnerable. Combinations of Xen, guest, guest balloon driver, and toolstack software, which have an empirical adjustment as described in the Description, and which have been formally stress-tested in PoD mode, are less likely to be vulnerable. Migration is not capable of creating a guest with outstanding PoD. So migrating a guest which is vulnerable might crash it. However, if a guest has been migrated successfully since it booted, it is no longer vulnerable. Xen versions back to 3.4.x are affected. Vulnerability of a particular guest can be tested by the host administrator using the utility `xsa153-check.c', attached to this advisory. MITIGATION ========== Reducing the guest's memory target, after guest startup, can cause the guest's ballon driver to eliminate the PoD discrepancy. If the guest successfully balloons down, it will no longer be vulnerable. On systems using libxl this can be done with `xl mem-set', during or after each guest boot: # ./xsa153-check `xl domid name-of-guest` checked domain 621 for XSA-153: VULNERABLE (1 more outstanding pages) try using xl mem-set to reduce its memory by 1 (Mby) or perhaps reduce /local/domain/621/memory/target by 4 # xl list name-of-guest Name ID Mem VCPUs State Time(s) name-of-guest 621 512 2 r----- 156.9 # xl mem-set name-of-guest 511 # [ wait for guest to give up memory ] # ./xsa153-check `xl domid name-of-guest` checked domain 621 for XSA-153: NOT vulnerable # Alternatively, no matter the toolstack, it is possible for a host administrator to bypass the toolstack code and give ballooning instructions directly to the guest: [ suppose guest domid is 616, eg from xl domid name-of-guest ] # ./xsa153-check 616 checked domain 616 for XSA-153: VULNERABLE (1 more outstanding pages) try using xl mem-set to reduce its memory by 1 (Mby) or perhaps reduce /local/domain/616/memory/target by 4 # xenstore-read /local/domain/616/memory/target 520188 # xenstore-write /local/domain/616/memory/target 520184 # [ wait for guest to give up memory ] # ./xsa153-check `xl domid name-of-guest` checked domain 616 for XSA-153: NOT vulnerable # The memory/target value is in decimal, and is a number of kilobytes; it must be a multiple of 4, since a page is 4 Kb on affected systems. The value to write should be some amount less than the value read. It is not currently known whether use of the VM memory event inspection facilities (in-tree, this means the xc_monitor utility) might invalidate the workaround. Note that guests may become unstable if given too little memory, so large reductions of the memory target should be applied with caution, if at all. The expected offset related to XSA-153 is small (tens of pages, perhaps). If a large reduction is required, it is more likely that either the guest is still booting up (and still working to reduce the PoD memory), or that the guest's balloon driver is not functioning: # ./xsa153-check `xl domid name-of-guest` checked domain 623 for XSA-153: VULNERABLE (65536 more outstanding pages) difference is >1Mby ballon driver not running or guest still booting? # A guest without a working balloon driver will be unstable in PoD mode, especially under memory pressure; this is an inherent feature of the design of PoD. RESOLUTION ========== The attached patch fixes the problem for systems using libxl (via xl, or via libvirt, or another higher layer). At the time of writing there is no patch for xend-based systems. xsa153-libxl.patch xen-unstable, Xen 4.5, Xen 4.6 xsa153-libxl.patch Xen 4.1 - 4.4 (using libxl) (Xend was removed in Xen 4.5; so the libxl-only patch is always sufficient for Xen 4.5 and later.) $ sha256sum xsa153* 633df5d970af49476c2d279e604150c444834bb906f6568070f0c2e0ceaa3af4 xsa153-check.c f5cbc98cba758e10da0a01d9379012ec56b98a85a92bfeb0c6b8132d4b91ce77 xsa153-libxl.patch $ DEPLOYMENT DURING EMBARGO ========================= Deployment of the patches and/or mitigations described above (or others which are substantially similar) is permitted during the embargo, even on public-facing systems with untrusted guest users and administrators. But: Distribution of updated software is prohibited (except to other members of the predisclosure list). Predisclosure list members who wish to deploy significantly different patches and/or mitigations, please contact the Xen Project Security Team. (Note: this during-embargo deployment notice is retained in post-embargo publicly released Xen Project advisories, even though it is then no longer applicable. This is to enable the community to have oversight of the Xen Project Security Team's decisionmaking.) For more information about permissible uses of embargoed information, consult the Xen Project community's agreed Security Policy: http://www.xenproject.org/security-policy.html NOTE REGARDING SHORT EMBARGO ============================ This issue was quickly encountered by the Security Team during our investigations of the scope and impact of XSA-150; this issue was originally discussed in the `Incomplete Information' section of XSA-150 v1. Accordingly XSA-153 is embargoed and the embargo will end at the same time as that of XSA-150. -----BEGIN PGP SIGNATURE----- iQFABAEBCAAqFiEEI+MiLBRfRHX6gGCng/4UyVfoK9kFAmV8b/AMHHBncEB4ZW4u b3JnAAoJEIP+FMlX6CvZSHYIAMHlr9TvZ/rVo0Mnv+d9zMKGaMXn5mgW9BLORTdA zt+RUKgqdaIvEfJoPx9syxoOJccQ0LAhkTEChE1lWw3bS5vxny0D3MdYJeE0I40m krgGw5tTF2dzH1p1tpWtKNB7c29oiaJ0vIR0xF+thLtklMQk62Bro6lQgZC1IMlr KdMolXYpzxlKd3u5zGEqsODgepqN+xIFezxfxRHfi4pSkn+qK417WDrPoGNJ0Tvz DDzHdCgYcgrW5ki+hZcb5Swr5T0yHyOdj1tQB3D/v/1ZDsc9zek1Gpy1QajhqTCj T3Gwpet+suwzLvFZQnvS1aeNvYvGz4VHquaSgG8JW+UNsjI= =yNOd -----END PGP SIGNATURE-----