1 Introduction

1.1 Purpose

The domain save image is the context of a running domain used for snapshots of a domain or for transferring domains between hosts during migration.

There are a number of problems with the format of the domain save image used in Xen 4.4 and earlier (the legacy format).

• Dependant on toolstack word size. A number of fields within the image are native types such as unsigned long which have different sizes between 32-bit and 64-bit toolstacks. This prevents domains from being migrated between hosts running 32-bit and 64-bit toolstacks.

• There is no header identifying the image.

• The image has no version information.

A new format that addresses the above is required.

ARM does not yet have have a domain save image format specified and the format described in this specification should be suitable.

1.2 Not Yet Included

The following features are not yet fully specified and will be included in a future draft.

• Page data compression.

• ARM

2 Overview

The image format consists of two main sections:

• Headers
• Records

2.1 Headers

There are two headers: the image header, and the domain header. The image header describes the format of the image (version etc.). The domain header contains general information about the domain (architecture, type etc.).

2.2 Records

The main part of the format is a sequence of different records. Each record type contains information about the domain context. At a minimum there is a END record marking the end of the records section.

2.3 Fields

All the fields within the headers and records have a fixed width.

Fields are always aligned to their size.

Padding and reserved fields are set to zero on save and must be ignored during restore.

Integer (numeric) fields in the image header are always in big-endian byte order.

Integer fields in the domain header and in the records are in the endianness described in the image header (which will typically be the native ordering).

3 Headers

3.1 Image Header

The image header identifies an image as a Xen domain save image. It includes the version of this specification that the image complies with.

Tools supporting version V of the specification shall always save images using version V. Tools shall support restoring from version V. If the previous Xen release produced version V - 1 images, tools shall supported restoring from these. Tools may additionally support restoring from earlier versions.

The marker field can be used to distinguish between legacy images and those corresponding to this specification. Legacy images will have at one or more zero bits within the first 8 octets of the image.

Fields within the image header are always in big-endian byte order, regardless of the setting of the endianness bit.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| marker                                          |
+-----------------------+-------------------------+
| id                    | version                 |
+-----------+-----------+-------------------------+
| options   | (reserved)                          |
+-----------+-------------------------------------+

The endianness shall be 0 (little-endian) for images generated on an i386, x86_64, or arm host.

3.2 Domain Header

The domain header includes general properties of the domain.

 0      1     2     3     4     5     6     7 octet
+-----------------------+-----------+-------------+
| type                  | page_shift| (reserved)  |
+-----------------------+-----------+-------------+
| xen_major             | xen_minor               |
+-----------------------+-------------------------+

The legacy stream conversion tool writes a xen_major version of 0, and sets xen_minor to the version of itself.

4 Records

A record has a record header, type specific data and a trailing footer. If body_length is not a multiple of 8, the body is padded with zeroes to align the end of the record on an 8 octet boundary.

 0     1     2     3     4     5     6     7 octet
+-----------------------+-------------------------+
| type                  | body_length             |
+-----------+-----------+-------------------------+
| body...                                         |
...
|           | padding (0 to 7 octets)             |
+-----------+-------------------------------------+

Records may be mandatory or optional. Optional records have bit 31 set in their type. Restoring an image that has unrecognised or unsupported mandatory record must fail. The contents of optional records may be ignored during a restore.

The following sub-sections specify the record body format for each of the record types.

4.1 END

An end record marks the end of the image, and shall be the final record in the stream.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+

The end record contains no fields; its body_length is 0.

4.2 PAGE_DATA

The bulk of an image consists of many PAGE_DATA records containing the memory contents.

 0     1     2     3     4     5     6     7 octet
+-----------------------+-------------------------+
| count (C)             | (reserved)              |
+-----------------------+-------------------------+
| pfn[0]                                          |
+-------------------------------------------------+
...
+-------------------------------------------------+
| pfn[C-1]                                        |
+-------------------------------------------------+
| page_data[0]...                                 |
...
+-------------------------------------------------+
| page_data[N-1]...                               |
...
+-------------------------------------------------+

Note: Count is strictly > 0. N is strictly <= C and it is possible for there to be no page_data in the record if all pfns are of invalid types.

PFNs with type BROKEN, XALLOC, or XTAB do not have any corresponding page_data.

The saver uses the XTAB type for PFNs that become invalid in the guest’s P2M table during a live migration¹.

Restoring an image with unrecognised page types shall fail.

4.3 X86_PV_INFO

 0     1     2     3     4     5     6     7 octet
+-----+-----+-----------+-------------------------+
| w   | ptl | (reserved)                          |
+-----+-----+-----------+-------------------------+

4.4 X86_PV_P2M_FRAMES

 0     1     2     3     4     5     6     7 octet
+-----+-----+-----+-----+-------------------------+
| p2m_start_pfn (S)     | p2m_end_pfn (E)         |
+-----+-----+-----+-----+-------------------------+
| p2m_pfn[p2m frame containing pfn S]             |
+-------------------------------------------------+
...
+-------------------------------------------------+
| p2m_pfn[p2m frame containing pfn E]             |
+-------------------------------------------------+

4.5 X86_PV_VCPU_BASIC, EXTENDED, XSAVE, MSRS

The format of these records are identical. They are all binary blobs of data which are accessed using specific pairs of domctl hypercalls.

 0     1     2     3     4     5     6     7 octet
+-----------------------+-------------------------+
| vcpu_id               | (reserved)              |
+-----------------------+-------------------------+
| context...                                      |
...
+-------------------------------------------------+

4.6 SHARED_INFO

The content of the Shared Info page.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| shared_info                                     |
...
+-------------------------------------------------+

4.7 X86_TSC_INFO

Domain TSC information, as accessed by the XEN_DOMCTL_{get,set}tscinfo hypercall sub-ops.

 0     1     2     3     4     5     6     7 octet
+------------------------+------------------------+
| mode                   | khz                    |
+------------------------+------------------------+
| nsec                                            |
+------------------------+------------------------+
| incarnation            | (reserved)             |
+------------------------+------------------------+

4.8 HVM_CONTEXT

HVM Domain context, as accessed by the XEN_DOMCTL_{get,set}hvmcontext hypercall sub-ops.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| hvm_ctx                                         |
...
+-------------------------------------------------+

4.9 HVM_PARAMS

HVM Domain parameters, as accessed by the HVMOP_{get,set}_param hypercall sub-ops.

 0     1     2     3     4     5     6     7 octet
+------------------------+------------------------+
| count (C)              | (reserved)             |
+------------------------+------------------------+
| param[0].index                                  |
+-------------------------------------------------+
| param[0].value                                  |
+-------------------------------------------------+
...
+-------------------------------------------------+
| param[C-1].index                                |
+-------------------------------------------------+
| param[C-1].value                                |
+-------------------------------------------------+

4.10 TOOLSTACK (deprecated)

This record was only present for transitionary purposes during development. It is should not be used.

An opaque blob provided by and supplied to the higher layers of the toolstack (e.g., libxl) during save and restore.

 0     1     2     3     4     5     6     7 octet
+------------------------+------------------------+
| data                                            |
...
+-------------------------------------------------+

4.11 VERIFY

A verify record indicates that, while all memory has now been sent, the sender shall send further memory records for debugging purposes.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+

The verify record contains no fields; its body_length is 0.

4.12 CHECKPOINT

A checkpoint record indicates that all the preceding records in the stream represent a consistent view of VM state.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+

The checkpoint record contains no fields; its body_length is 0

If the stream is embedded in a higher level toolstack stream, the CHECKPOINT record marks the end of the libxc portion of the stream and the stream is handed back to the higher level for further processing.

The higher level stream may then hand the stream back to libxc to process another set of records for the next consistent VM state snapshot. This next set of records may be terminated by another CHECKPOINT record or an END record.

4.13 CHECKPOINT_DIRTY_PFN_LIST

A checkpoint dirty pfn list record is used to convey information about dirty memory in the VM. It is an unordered list of PFNs. Currently only applicable in the backchannel of a checkpointed stream. It is only used by COLO, more detail please reference README.colo.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| pfn[0]                                          |
+-------------------------------------------------+
...
+-------------------------------------------------+
| pfn[C-1]                                        |
+-------------------------------------------------+

The count of pfns is: record->length/sizeof(uint64_t).

4.14 STATIC_DATA_END

A static data end record marks the end of the static state. I.e. state which is invariant of guest execution.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+

The end record contains no fields; its body_length is 0.

4.15 X86_CPUID_POLICY

CPUID policy content, as accessed by the XEN_DOMCTL_{get,set}_cpu_policy hypercall sub-ops.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| CPUID_policy                                    |
...
+-------------------------------------------------+

4.16 X86_MSR_POLICY

MSR policy content, as accessed by the XEN_DOMCTL_{get,set}_cpu_policy hypercall sub-ops.

 0     1     2     3     4     5     6     7 octet
+-------------------------------------------------+
| MSR_policy                                      |
...
+-------------------------------------------------+

5 Layout

The set of valid records depends on the guest architecture and type. No assumptions should be made about the ordering or interleaving of independent records. Record dependencies are noted below.

Some records are used for signalling, and explicitly have zero length. All other records contain data relevant to the migration. Data records with no content should be elided on the source side, as their presence serves no purpose, but results in extra work for the restore side.

5.1 x86 PV Guest

A typical save record for an x86 PV guest image would look like:

• Image header
• Domain header
• Static data records:
  • X86_PV_INFO record
  • X86_{CPUID,MSR}_POLICY
  • STATIC_DATA_END
• X86_PV_P2M_FRAMES record
• Many PAGE_DATA records
• X86_TSC_INFO
• SHARED_INFO record
• VCPU context records for each online VCPU
  • X86_PV_VCPU_BASIC record
  • X86_PV_VCPU_EXTENDED record
  • X86_PV_VCPU_XSAVE record
  • X86_PV_VCPU_MSRS record
• END record

There are some strict ordering requirements. The following records must be present in the following order as each of them depends on information present in the preceding ones.

• X86_PV_INFO record
• X86_PV_P2M_FRAMES record
• PAGE_DATA records
• VCPU records

5.2 x86 HVM Guest

A typical save record for an x86 HVM guest image would look like:

• Image header
• Domain header
• Static data records:
  • X86_{CPUID,MSR}_POLICY
  • STATIC_DATA_END
• Many PAGE_DATA records
• X86_TSC_INFO
• HVM_PARAMS
• HVM_CONTEXT

HVM_PARAMS must precede HVM_CONTEXT, as certain parameters can affect the validity of architectural state in the context.

6 Compatibility with older versions

6.1 v3 compat with v2

A v3 stream is compatible with a v2 stream, but mandates the presense of a STATIC_DATA_END record ahead of any memory/register content. This is to ease the introduction of new static configuration records over time.

A v3-compatible reciever interpreting a v2 stream should infer the position of STATIC_DATA_END based on finding the first X86_PV_P2M_FRAMES record (for PV guests), or PAGE_DATA record (for HVM guests) and behave as if STATIC_DATA_END had been sent.

6.2 Legacy Images (x86 only)

Restoring legacy images from older tools shall be handled by translating the legacy format image into this new format.

It shall not be possible to save in the legacy format.

There are two different legacy images depending on whether they were generated by a 32-bit or a 64-bit toolstack. These shall be distinguished by inspecting octets 4-7 in the image. If these are zero then it is a 64-bit image.

This assumes the presence of the extended-info chunk which was introduced in Xen 3.0.

7 Future Extensions

All changes to this specification should bump the revision number in the title block.

All changes to the image or domain headers require the image version to be increased.

The format may be extended by adding additional record types.

Extending an existing record type must be done by adding a new record type. This allows old images with the old record to still be restored.

The image header may only be extended by appending additional fields. In particular, the marker, id and version fields must never change size or location.

8 Errata

1. For compatibility with older code, the receving side of a stream should tolerate and ignore variable sized records with zero content. Xen releases between 4.6 and 4.8 could end up generating valid HVM_PARAMS or X86_PV_VCPU_{EXTENDED,XSAVE,MSRS} records with zero-length content.