Dom0 kernel and ramdisk modules ================================ Xen is passed the dom0 kernel and initrd via a reference in the /chosen node of the device tree. Each node contains the following properties: - compatible Must always include at least the generic compatiblity string: "multiboot,module" Optionally a more specific compatible string may be used in addition to the above. One of: - "multiboot,kernel" -- the dom0 kernel - "multiboot,ramdisk" -- the dom0 ramdisk - "xen,xsm-policy" -- XSM policy blob It is normally recommended to include a more specific compatible string (if one applies) in addition to the generic string (which must always be present). Xen will assume that the first module which lacks a more specific compatible string is a "multiboot,kernel". Xen will examine each module, starting from the second module that lacks a specific compatible string. Xen will check each such module for the XSM Magic number: - For a module which has the XSM Magic number: it will be treated by Xen as if its compatible string was "xen,xsm-policy"; - For a module which does not have the XSM Magic: the second module lacking a compatible string will be treated by Xen as if its compatible string was "multiboot,ramdisk"; for the third and subsequent modules which lack a specific compatible string, Xen will not apply any special treatment. This means if the ramdisk module is present and does not have the compatible string "multiboot,ramdisk", then it must always be the second module. Note: This XSM Magic detection behavior was introduced by Xen 4.7. Xen 4.6 (and downwards) still requires the XSM module to have the compatible string "xen,xsm-policy". Xen 4.4 supported a different set of legacy compatible strings which remain supported such that systems supporting both 4.4 and later can use a single DTB. However when booting Xen using UEFI, the legacy compatible strings are not supported. - "xen,multiboot-module" equivalent to "multiboot,module" - "xen,linux-zimage" equivalent to "multiboot,kernel" - "xen,linux-initrd" equivalent to "multiboot,ramdisk" For compatibility with Xen 4.4 the more specific "xen,linux-*" names are non-optional and must be included. - reg Specifies the physical address of the module in RAM and the length of the module. - bootargs (optional) Command line associated with this module. See below for the priority of this field vs. other mechanisms of specifying the bootargs for the kernel. - xen,uefi-binary (UEFI boot only) String property that specifies the file name to be loaded by the UEFI boot for this module. If this is specified, there is no need to specify the reg property because it will be created by the UEFI stub on boot. This option is needed only when UEFI boot is used, the node needs to be compatible with multiboot,kernel or multiboot,ramdisk. Examples ======== A boot module of unspecified type: module@0xc0000000 { compatible = "multiboot,module"; reg = <0xc0000000 0x1234>; bootargs = "..."; }; A boot module containing a ramdisk: module@0xd0000000 { compatible = "multiboot,ramdisk", "multiboot,module"; reg = <0xd0000000 0x5678>; }; The previous examples are compatible with Xen 4.5+ only. To be compatible with Xen 4.4 as well use the legacy names: module@0xd0000000 { compatible = "xen,linux-initrd", "xen,multiboot-module"; reg = <0xd0000000 0x5678>; }; Command lines ============= Xen also checks for properties directly under /chosen to find suitable command lines for Xen and Dom0. The logic is the following: - If xen,xen-bootargs is present, it will be used for Xen. - If xen,dom0-bootargs is present, it will be used for Dom0. - If xen,xen-bootargs is _not_ present, but xen,dom0-bootargs is, bootargs will be used for Xen. - If a kernel boot module is present and has a bootargs property then the top-level bootargs will used for Xen. - If no Xen specific properties are present, bootargs is for Dom0. - If xen,xen-bootargs is present, but xen,dom0-bootargs is missing, bootargs will be used for Dom0. Most of these cases is to make booting with Xen-unaware bootloaders easier. For those you would hardcode the Xen commandline in the DTB under /chosen/xen,xen-bootargs and would let the bootloader set the Dom0 command line by writing bootargs (as for native Linux). A Xen-aware bootloader would set xen,xen-bootargs for Xen, xen,dom0-bootargs for Dom0 and bootargs for native Linux. UEFI boot and DT ================ When Xen is booted using UEFI, it doesn't read the configuration file if any multiboot module is specified. To force Xen to load the configuration file, the boolean property xen,uefi-cfg-load must be declared in the /chosen node. Creating Multiple Domains directly from Xen =========================================== It is possible to have Xen create other domains, in addition to dom0, out of the information provided via device tree. A kernel and initrd (optional) need to be specified for each guest. For each domain to be created there needs to be one node under /chosen with the following properties: - compatible For domUs: "xen,domain" - memory A 64-bit integer specifying the amount of kilobytes of RAM to allocate to the guest. - cpus An integer specifying the number of vcpus to allocate to the guest. - vpl011 An empty property to enable/disable a virtual pl011 for the guest to use. The virtual pl011 uses SPI number 0 (see GUEST_VPL011_SPI). Please note that the SPI used for the virtual pl011 could clash with the physical SPI of a physical device assigned to the guest. - nr_spis Optional. A 32-bit integer specifying the number of SPIs (Shared Peripheral Interrupts) to allocate for the domain. If nr_spis is missing, the max number of SPIs supported by the physical GIC is used, or GUEST_VPL011_SPI+1 if vpl011 is enabled, whichever is greater. - #address-cells and #size-cells Both #address-cells and #size-cells need to be specified because both sub-nodes (described shortly) have reg properties. - direct-map Only available when statically allocated memory is used for the domain. An empty property to request the memory of the domain to be direct-map (guest physical address == physical address). - domain-cpupool Optional. Handle to a xen,cpupool device tree node that identifies the cpupool where the guest will be started at boot. - sve Optional. The `sve` property enables Arm SVE usage for the domain and sets the maximum SVE vector length, the option is applicable only to Arm64 guests. A value equal to 0 disables the feature, this is the default value. Specifying this property with no value, means that the SVE vector length will be set equal to the maximum vector length supported by the platform. Values above 0 explicitly set the maximum SVE vector length for the domain, allowed values are from 128 to maximum 2048, being multiple of 128. Please note that when the user explicitly specifies the value, if that value is above the hardware supported maximum SVE vector length, the domain creation will fail and the system will stop, the same will occur if the option is provided with a non zero value, but the platform doesn't support SVE. - xen,enhanced A string property. Possible property values are: - "enabled" (or missing property value) Xen PV interfaces, including grant-table and xenstore, will be enabled for the VM. - "disabled" Xen PV interfaces are disabled. - "no-xenstore" All default Xen PV interfaces, including grant-table will be enabled but xenstore will be disabled for the VM. If the xen,enhanced property is present with no value, it defaults to "enabled". If the xen,enhanced property is not present, PV interfaces are disabled. In the future other possible property values might be added to enable only selected interfaces. - xen,domain-p2m-mem-mb Optional. A 32-bit integer specifying the amount of megabytes of RAM used for the domain P2M pool. This is in-sync with the shadow_memory option in xl.cfg. Leaving this field empty in device tree will lead to the default size of domain P2M pool, i.e. 1MB per guest vCPU plus 4KB per MB of guest RAM plus 512KB for guest extended regions. - max_grant_version Optional. A 32-bit integer specifying the maximum grant table version the domain is allowed to use (valid values are 1 or 2). If this property is missing, the value specified by Xen command line parameter gnttab=max-ver (or its default value if unspecified, i.e. 1) is used. - max_grant_frames Optional. A 32-bit integer specifying the maximum number of grant frames the domain is allowed to have. If this property is missing, the value specified by Xen command line parameter gnttab_max_frames (or its default value if unspecified, i.e. 64) is used. - max_maptrack_frames Optional. A 32-bit integer specifying the maximum number of grant maptrack frames the domain is allowed to have. If this property is missing, the value specified by Xen command line parameter gnttab_max_maptrack_frames (or its default value if unspecified, i.e. 1024) is used. Under the "xen,domain" compatible node, one or more sub-nodes are present for the DomU kernel and ramdisk. The kernel sub-node has the following properties: - compatible "multiboot,kernel", "multiboot,module" - reg Specifies the physical address of the kernel in RAM and its length. - bootargs (optional) Command line parameters for the guest kernel. - xen,uefi-binary (UEFI boot only) String property that specifies the file name to be loaded by the UEFI boot for this module. If this is specified, there is no need to specify the reg property because it will be created by the UEFI stub on boot. This option is needed only when UEFI boot is used. The ramdisk sub-node has the following properties: - compatible "multiboot,ramdisk", "multiboot,module" - reg Specifies the physical address of the ramdisk in RAM and its length. - xen,uefi-binary (UEFI boot only) String property that specifies the file name to be loaded by the UEFI boot for this module. If this is specified, there is no need to specify the reg property because it will be created by the UEFI stub on boot. This option is needed only when UEFI boot is used. Example ======= chosen { domU1 { compatible = "xen,domain"; #address-cells = <0x2>; #size-cells = <0x1>; memory = <0 131072>; cpus = <2>; vpl011; module@0x4a000000 { compatible = "multiboot,kernel", "multiboot,module"; reg = <0x0 0x4a000000 0xffffff>; bootargs = "console=ttyAMA0 init=/bin/sh"; }; module@0x4b000000 { compatible = "multiboot,ramdisk", "multiboot,module"; reg = <0x0 0x4b000000 0xffffff>; }; }; domU2 { compatible = "xen,domain"; #address-cells = <0x2>; #size-cells = <0x1>; memory = <0 65536>; cpus = <1>; module@0x4c000000 { compatible = "multiboot,kernel", "multiboot,module"; reg = <0x0 0x4c000000 0xffffff>; bootargs = "console=ttyAMA0 init=/bin/sh"; }; module@0x4d000000 { compatible = "multiboot,ramdisk", "multiboot,module"; reg = <0x0 0x4d000000 0xffffff>; }; }; }; Device Assignment ================= Device Assignment (Passthrough) is supported by adding another module, alongside the kernel and ramdisk, with the device tree fragment corresponding to the device node to assign to the guest. The dtb sub-node should have the following properties: - compatible "multiboot,device-tree" and "multiboot,module" - reg Specifies the physical address of the device tree binary fragment RAM and its length. - xen,uefi-binary (UEFI boot only) String property that specifies the file name to be loaded by the UEFI boot for this module. If this is specified, there is no need to specify the reg property because it will be created by the UEFI stub on boot. This option is needed only when UEFI boot is used. As an example: module@0xc000000 { compatible = "multiboot,device-tree", "multiboot,module"; reg = <0x0 0xc000000 0xffffff>; }; The DTB fragment is loaded at 0xc000000 in the example above. It should follow the convention explained in docs/misc/arm/passthrough.txt. The DTB fragment will be added to the guest device tree, so that the guest kernel will be able to discover the device. Static Allocation ============= Static Allocation refers to system or sub-system(domains) for which memory areas are pre-defined by configuration using physical address ranges. Memory can be statically allocated to a domain using the property "xen,static- mem" defined in the domain configuration. The number of cells for the address and the size must be defined respectively by the parent node properties "#address-cells" and "#size-cells". The property 'memory' is still needed and should match the amount of memory given to the guest. Currently, it either comes from static memory or lets Xen allocate from heap. *Mixing* is not supported. The static memory will be mapped in the guest at the usual guest memory addresses (GUEST_RAM0_BASE, GUEST_RAM1_BASE) defined by xen/include/public/arch-arm.h. Below is an example on how to specify the static memory region in the device-tree: / { chosen { #address-cells = <0x1>; #size-cells = <0x1>; ... domU1 { compatible = "xen,domain"; cpus = <2>; memory = <0x0 0x80000>; xen,static-mem = <0x30000000 0x20000000>; ... }; }; }; This will reserve a 512MB region starting at the host physical address 0x30000000 to be exclusively used by DomU1. Static Event Channel ==================== The event channel communication will be established statically between two domains (dom0 and domU also). Event channel connection information between domains will be passed to Xen via the device tree node. The event channel will be created and established in Xen before the domain started. The domain does not need to do any operation to establish a connection. Domain only needs hypercall EVTCHNOP_send(local port) to send notifications to the remote guest. There is no need to describe the static event channel info in the domU device tree. Static event channels are only useful in fully static configurations, and in those configurations, the domU device tree dynamically generated by Xen is not needed. To enable the event-channel interface for domU guests include the xen,enhanced = "no-xenstore" property in the domU Xen device tree node. Under the "xen,domain" compatible node for domU, there needs to be sub-nodes with compatible "xen,evtchn" that describe the event channel connection between two domUs. For dom0, there needs to be sub-nodes with compatible "xen,evtchn" under the chosen node. The static event channel node has the following properties: - compatible "xen,evtchn" - xen,evtchn The property is tuples of two numbers (local-evtchn link-to-foreign-evtchn) where: local-evtchn is an integer value that will be used to allocate local port for a domain to send and receive event notifications to/from the remote domain. Maximum supported value is 2^17 for FIFO ABI and 4096 for 2L ABI. It is recommended to use low event channel IDs. link-to-foreign-evtchn is a single phandle to a remote evtchn to which local-evtchn will be connected. Example ======= chosen { /* One sub-node per local event channel. This sub-node is for Dom0. */ ec1: evtchn@1 { compatible = "xen,evtchn-v1"; /* local-evtchn link-to-foreign-evtchn */ xen,evtchn = <0xa &ec2>; }; domU1 { compatible = "xen,domain"; #address-cells = <0x2>; #size-cells = <0x1>; xen,enhanced = "no-xenstore"; /* One sub-node per local event channel */ ec2: evtchn@2 { compatible = "xen,evtchn-v1"; /* local-evtchn link-to-foreign-evtchn */ xen,evtchn = <0xa &ec1>; }; ec3: evtchn@3 { compatible = "xen,evtchn-v1"; xen,evtchn = <0xb &ec5>; }; ec4: evtchn@4 { compatible = "xen,evtchn-v1"; xen,evtchn = <0xc &ec6>; }; }; domU2 { compatible = "xen,domain"; #address-cells = <0x2>; #size-cells = <0x1>; xen,enhanced = "no-xenstore"; /* One sub-node per local event channel */ ec5: evtchn@5 { compatible = "xen,evtchn-v1"; /* local-evtchn link-to-foreign-evtchn */ xen,evtchn = <0xb &ec3>; }; ec6: evtchn@6 { compatible = "xen,evtchn-v1"; xen,evtchn = <0xd &ec4>; }; }; }; Static Heap Memory ================== The static heap memory refers to parts of RAM reserved in the beginning of boot time for heap. The memory is reserved by configuration in the device tree using physical address ranges. The static heap memory declared in the device tree defines the memory areas that will be reserved to be used exclusively as heap. - For Arm32, since there are separated heaps, the static heap will be used for both domheap and xenheap. The admin should make sure that the static heap region should contain enough memory below 4GB to cater 32-bit DMA. - For Arm64, since there is a single heap, the defined static heap areas shall always go to the heap allocator. The static heap memory is an optional feature and can be enabled by adding below device tree property. - xen,static-heap Property under the top-level "chosen" node. It specifies the address and size of Xen static heap memory. Number of address and size cells for the "xen,static-heap" property is determined by the root node "#address-cells" and "#size-cells". Note that at least a 64KB alignment is required. Below is an example on how to specify the static heap in device tree: / { #address-cells = <0x2>; #size-cells = <0x2>; ... chosen { xen,static-heap = <0x0 0x30000000 0x0 0x40000000>; ... }; }; RAM starting from the host physical address 0x30000000 of 1GB size will be reserved as static heap. Static Shared Memory ==================== The static shared memory device tree nodes allow users to statically set up shared memory on dom0less system, enabling domains to do shm-based communication. - compatible "xen,domain-shared-memory-v1" - xen,shm-id An arbitrary string that represents the unique identifier of the shared memory region, with a strict limit on the number of characters(\0 included), `MAX_SHM_ID_LENGTH(16)`. e.g. "xen,shm-id = "my-shared-mem-1"". - xen,shared-mem An array takes a physical address, which is the base address of the shared memory region in host physical address space, a size, and a guest physical address, as the target address of the mapping. e.g. xen,shared-mem = < [host physical address] [guest address] [size] > It shall also meet the following criteria: 1) If the SHM ID matches with an existing region, the address range of the region shall also exactly match. 2) If the SHM ID does not match with any other existing region, it should also not overlap with any other regions. The number of cells for the host address (and size) is the same as the guest pseudo-physical address and they are inherited from the parent node. Host physical address is optional, when missing Xen decides the location (currently unimplemented). - role (Optional) A string property specifying the ownership of a shared memory region, the value must be one of the following: "owner", or "borrower" A shared memory region could be explicitly backed by one domain, which is called "owner domain", and all the other domains who are also sharing this region are called "borrower domain". If not specified, the default value is "borrower" and owner is DOMID_IO, a system domain. As an example: chosen { #address-cells = <0x1>; #size-cells = <0x1>; xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0"; ...... /* this is for Dom0 */ dom0-shared-mem@10000000 { compatible = "xen,domain-shared-memory-v1"; role = "owner"; xen,shm-id = "my-shared-mem-0"; xen,shared-mem = <0x10000000 0x10000000 0x10000000>; } domU1 { compatible = "xen,domain"; #address-cells = <0x1>; #size-cells = <0x1>; memory = <0 131072>; cpus = <2>; vpl011; /* * shared memory region identified as 0x0(xen,shm-id = <0x0>) * is shared between Dom0 and DomU1. */ domU1-shared-mem@10000000 { compatible = "xen,domain-shared-memory-v1"; role = "borrower"; xen,shm-id = "my-shared-mem-0"; xen,shared-mem = <0x10000000 0x50000000 0x10000000>; } /* * shared memory region identified as 0x1(xen,shm-id = <0x1>) * is shared between DomU1 and DomU2. */ domU1-shared-mem@50000000 { compatible = "xen,domain-shared-memory-v1"; xen,shm-id = "my-shared-mem-1"; xen,shared-mem = <0x50000000 0x60000000 0x20000000>; } ...... }; domU2 { compatible = "xen,domain"; #address-cells = <0x1>; #size-cells = <0x1>; memory = <0 65536>; cpus = <1>; /* * shared memory region identified as 0x1(xen,shm-id = <0x1>) * is shared between domU1 and domU2. */ domU2-shared-mem@50000000 { compatible = "xen,domain-shared-memory-v1"; xen,shm-id = "my-shared-mem-1"; xen,shared-mem = <0x50000000 0x70000000 0x20000000>; } ...... }; }; This is an example with two static shared memory regions. For the static shared memory region identified as "my-shared-mem-0", host physical address starting at 0x10000000 of 256MB will be reserved to be shared between Dom0 and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is explicitly defined as the owner domain, and DomU1 is the borrower domain. For the static shared memory region identified as "my-shared-mem-1", host physical address starting at 0x50000000 of 512MB will be reserved to be shared between DomU1 and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address space, and at 0x70000000 in DomU2 guest physical address space. DomU1 and DomU2 are both the borrower domain, the owner domain is the default owner domain DOMID_IO.