openhcl: preserve UEFI firmware across hibernation via VMGS snapshot

[Mike Ebersol (mebersol)]

Summary

Adds a VMGS firmware-image snapshot/restore path so a hibernation-enabled,
non-isolated UEFI guest sees an identical firmware binary after resume even when
the host cannot overload the firmware itself.

This is the scoped-down firmware load/store portion of #3771. The other pieces
of that WIP already landed separately:

• hibernate token lifecycle — openhcl: write hibernate token to VMGS on power transitions #4235
• host-side GET LoadFirmware overload — openhcl: add GET LoadFirmware request for hibernation firmware overload #4262

What it does

• Cold boot (store): snapshots the pristine UEFI firmware image from VTL0
  guest memory into a new VMGS file (HIBERNATION_FIRMWARE, file id 20) before
  write_uefi_config layers the dynamic config on top. The snapshot runs on any
  non-restore boot after any overload/restore has occurred, so the image the
  guest will actually run is preserved for the next hibernation. Only stored when
  the VMGS backing store is large enough (32 MB overall gate); insufficient space
  is a warning, not an error. Servicing restore is skipped (its firmware already
  has dynamic config applied and is no longer pristine).
• Resume (load): on a firmware-version-mismatch resume, prefers restoring
  the snapshotted image from VMGS (needs no host support), falling back to the
  host LoadFirmware overload, then to resuming on the current firmware.
• x86_64 RIP: the restored image's SEC entry point may differ from the
  cold-boot image's, so VTL0's RIP is rebased to it (computed via the
  now-public loader::uefi::get_sec_entry_point_offset); load_firmware applies
  the updated measured UEFI context. Computed before writing guest memory so a
  malformed image is a clean fallback.

Changes

• vmgs_format: add HIBERNATION_FIRMWARE (file id 20) and
  VMGS_HIBERNATION_FIRMWARE_MIN_SIZE.
• vmgs / vmgs_broker: add device_size() on Vmgs, the broker, and the client.
• vmgstool: parse the HIBERNATION_FIRMWARE file id.
• loader: expose uefi::get_sec_entry_point_offset.
• underhill_core/worker: store/restore helpers and resume wiring.

Scope / limitations

• Targets non-isolated VMs. CVM/isolated support is deferred.

Testing

• cargo clippy and cargo doc --no-deps pass for the modified crates
  (underhill_core + loader checked for x86_64-unknown-linux-gnu).
• cargo xtask fmt passes.

[Copilot]

Pull request overview

This PR adds a VMGS-backed snapshot/restore path for the UEFI firmware image to preserve a bit-for-bit identical firmware binary across hibernation/resume for non-isolated guests. It extends the VMGS format and broker APIs to support querying backing-store size and storing a new HIBERNATION_FIRMWARE file, and wires restore/store behavior into underhill_core.

Changes:

• Add VMGS FileId::HIBERNATION_FIRMWARE and a minimum backing-store size gate constant for storing firmware snapshots.
• Add a device_size() API to Vmgs, the VMGS broker RPC surface, and the broker client.
• Expose loader::uefi::get_sec_entry_point_offset publicly and use it during VMGS firmware restore to rebase VTL0 RIP on x86_64.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
vm/vmgs/vmgstool/src/main.rs	Accept HIBERNATION_FIRMWARE as a named VMGS file id in the CLI tool.
vm/vmgs/vmgs/src/vmgs_impl.rs	Add Vmgs::device_size() to report backing-store size.
vm/vmgs/vmgs_format/src/lib.rs	Define HIBERNATION_FIRMWARE file id and VMGS_HIBERNATION_FIRMWARE_MIN_SIZE gate.
vm/vmgs/vmgs_broker/src/client.rs	Add VmgsClient::device_size() RPC call.
vm/vmgs/vmgs_broker/src/broker.rs	Add DeviceSize broker RPC and handler.
vm/loader/src/uefi/mod.rs	Make SEC entry-point offset helper public for callers swapping firmware at runtime.
openhcl/underhill_core/src/worker.rs	Wire firmware snapshot-to-VMGS and restore-from-VMGS into hibernation resume flow, including x86_64 RIP rebase.

Suppressed comments (1)

vm/loader/src/uefi/mod.rs:228

• get_sec_entry_point_offset is now pub and is used by hibernation restore on a firmware image read from VMGS. The implementation uses unchecked slicing/indexing like &image[image_offset as usize..] while advancing offsets based on on-image fields, which can panic on malformed/corrupt input. Across the OpenHCL trust boundary this should be made non-panicking (use image.get(..)/checked arithmetic and return None on any bounds issue).

pub fn get_sec_entry_point_offset(image: &[u8]) -> Option<u64> {
    // Skip to SEC volume start.
    let mut image_offset = SEC_FIRMWARE_VOLUME_OFFSET;

    // Expect a firmware volume header for SEC volume.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

vm/vmgs/vmgs/src/vmgs_impl.rs:647

• device_size() computes bytes via sector_count * sector_size without overflow handling. Storage::capacity() in storage.rs clamps and avoids returning nonsensical sizes; here an overflow would silently wrap and could make callers think the VMGS is tiny/huge incorrectly.

    /// Returns the total size, in bytes, of the underlying VMGS backing store.
    pub fn device_size(&self) -> u64 {
        self.storage.sector_count() * self.storage.sector_size() as u64
    }

openhcl/underhill_core/src/worker.rs:4363

• On x86_64, new_rip is computed with unchecked start() + offset. The offset comes from parsing a stored firmware image (VMGS can be tampered/corrupted), so this should be validated (no overflow, and offset must land within the firmware region) similar to the host-provided offset validation in overload_vtl0_firmware.

    #[cfg(guest_arch = "x86_64")]
    let new_rip = match loader::uefi::get_sec_entry_point_offset(&firmware) {
        Some(offset) => firmware_memory.start() + offset,
        None => {

openhcl/underhill_core/src/worker.rs:3845

• PR description says an insufficient-VMGS-size snapshot attempt should be logged as a warning (normal fallback). This currently logs at info level, which makes it easy to miss when diagnosing why firmware wasn’t preserved across hibernation.

                    tracing::info!(

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Suppressed comments (5)

openhcl/underhill_core/src/worker.rs:4484

• This branch validates the host-provided offset only after LoadFirmware has already overwritten VTL0 memory. If the offset is out of range, the later return false makes the caller fall back as though the cold-boot image were still present, but the old RIP now points into the host-loaded image. Validate before replacing memory, or make this protocol violation fatal/restore the original bytes.

            let Some(new_rip) = base.checked_add(offset).filter(|_| offset < len) else {
                tracing::warn!(
                    CVM_ALLOWED,
                    offset,
                    firmware_len = len,

openhcl/underhill_core/src/worker.rs:3747

• This new VMGS restore path is not gated by isolation.is_isolated(), so an isolated UEFI VM with hibernation enabled will also overwrite VTL0 firmware from VMGS. The PR explicitly defers CVM/isolated support, and this path does not address the isolated measurement/attestation semantics; skip this restore for isolated VMs and apply the same gate to the snapshot condition below.

                if restore_vtl0_firmware_from_vmgs(gm.vtl0(), &mut measured_vtl0_info, vmgs_client)
                    .await

openhcl/underhill_core/src/worker.rs:4292

• If this write fails while replacing an existing snapshot, the old HIBERNATION_FIRMWARE file remains allocated. A subsequent clean boot can therefore leave a snapshot from an older firmware version, and the resume path will restore that stale image on a token mismatch because the file is not tied to the token. Invalidate the old snapshot on refresh failure or persist an image/token association so stale data is never selected.

    vmgs_client
        .write_file(vmgs::FileId::HIBERNATION_FIRMWARE, firmware)
        .await
        .context("failed to write UEFI firmware snapshot to VMGS")?;

openhcl/underhill_core/src/worker.rs:3845

• The PR contract says that an undersized VMGS is a warning-level fallback, but this branch emits info!. At normal log filtering operators will not get the intended warning that firmware preservation was skipped; use warn! for this outcome.

                    tracing::info!(

vm/loader/src/uefi/mod.rs:301

• The new malformed-image checks still allow the scan to finish without finding a PE32 section: for example, a SEC file whose size is only its file header skips this loop, and the Some(image_offset) below treats the post-header address as a valid entry point. restore_vtl0_firmware_from_vmgs relies on this function to reject corrupt VMGS data, so return None when the loop ends without a PE section (or return directly from the PE branch).

        // A zero-size section would not advance the scan; treat as malformed.
        let section_size = expand_3byte_integer(sh.size);
        if section_size == 0 {
            return None;
        }
        image_offset = image_offset.checked_add(section_size)?;
        file_offset = file_offset.checked_add(section_size)?;

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

openhcl/underhill_core/src/worker.rs:3747

• This new VMGS restore path is not gated on isolation.is_isolated(), so hibernation-enabled SNP/TDX guests will restore a VMGS firmware image even though this PR explicitly defers isolated/CVM support and its measurement/attestation handling. Gate the snapshot restore to non-isolated VMs (or implement the missing isolated path) before using it here.

                // logic (host overload, then the current firmware).
                if restore_vtl0_firmware_from_vmgs(gm.vtl0(), &mut measured_vtl0_info, vmgs_client)

openhcl/underhill_core/src/worker.rs:3832

• The new snapshot store is likewise enabled for isolated VMs because this condition only checks hibernation and servicing state. That causes the deferred VMGS firmware-preservation path to read the isolated VTL0 firmware and persist it without the required CVM design; add the same non-isolated gate here.

    // VMGS so a future hibernation resume can restore it bit-for-bit on a host

vm/loader/src/uefi/mod.rs:288

• This parser still returns Some(image_offset) when the SEC CORE file contains no EFI_SECTION_PE32 section (including a zero-length SEC file), because the loop falls through to the unconditional success return. The VMGS restore path treats that offset as a valid entry point and writes the malformed image into VTL0 instead of taking its documented clean fallback. Track whether a PE32 section was found and return None when it was not, with the entry point bounded by the containing file/section.

        if sh.typ == EFI_SECTION_PE32 {
            let section_header_size = size_of::<EFI_COMMON_SECTION_HEADER>() as u64;
            let pe_data_offset = image_offset.checked_add(section_header_size)?;
            let pe_offset = pe_get_entry_point_offset(image.get(pe_data_offset as usize..)?)?;
            image_offset = pe_data_offset.checked_add(pe_offset as u64)?;

vm/vmgs/vmgs_broker/src/broker.rs:53

• MeshPayload encodes enum variants by their positional numbers, so inserting DeviceSize here renumbers ReadFile and every following RPC. A client and broker built against different revisions can decode an existing request as the wrong operation. Append this variant after DeleteFile (or assign a stable explicit protocol number) to preserve the existing RPC tags.

    DeviceSize(Rpc<(), u64>),

openhcl/underhill_core/src/worker.rs:3857

• The no-space paths do not match the documented warning-only behavior: the explicit InsufficientSpace result above uses info!, while allocator exhaustion from write_file reaches this Err arm and uses error!. Distinguish allocation exhaustion from unexpected VMGS failures and log the former as a warning rather than reporting a normal capacity fallback as an error.

                    tracing::info!(
                        CVM_ALLOWED,
                        firmware_size,
                        device_size,
                        minimum_size = VMGS_HIBERNATION_FIRMWARE_MIN_SIZE,

openhcl/underhill_core/src/worker.rs:3845

• The documented insufficient-space outcome is emitted with info!, so operators cannot readily distinguish a VM that will not preserve firmware from a successful hibernation setup. This branch should use warning-level logging, consistent with the stated fallback behavior.

            vmgs_client.as_ref(),

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (6)

openhcl/underhill_core/src/worker.rs:4296

• Comparing the image length with total device_size does not guarantee that VMGS can allocate the write because allocation uses currently free blocks. If this overwrite fails, the previous snapshot remains allocated; if the subsequent hibernate-token write succeeds, resume will accept that stale image and boot the wrong firmware. Invalidate/delete the old snapshot whenever capture is skipped or fails, or make snapshot validity atomic with the token.

    if device_size < VMGS_HIBERNATION_FIRMWARE_MIN_SIZE || len > device_size {
        return Ok(StoreFirmwareOutcome::InsufficientSpace {
            firmware_size: len,
            device_size,
        });

openhcl/underhill_core/src/worker.rs:3840

• This condition is not gated on the active firmware type. MeasuredVtl0Info can expose UEFI alongside PCAT/Linux, so a PCAT or direct-Linux boot with hibernation enabled will snapshot the unused UEFI region; because current_hibernate_token is None for those modes, the halt path never deletes it. Restrict this block to FirmwareType::Uefi.

    if dps.general.hibernation_enabled && !is_restoring && !firmware_loaded_from_vmgs {

vm/vmgs/vmgs_broker/src/broker.rs:53

• Please append this new RPC variant after the existing variants instead of inserting it here. MeshPayload assigns enum field numbers from declaration order, and the derive documentation warns that updating an enum can break existing binaries (support/mesh/mesh_derive/src/lib.rs:51-55); inserting DeviceSize shifts ReadFile, WriteFile, Save, and DeleteFile, so a mixed-version broker/client can dispatch those calls incorrectly.

    DeviceSize(Rpc<(), u64>),

openhcl/underhill_core/src/worker.rs:3853

• The PR's fallback contract says an undersized VMGS is a warning rather than an error, but this path logs at info. That makes a functional loss of firmware preservation easy to miss in production diagnostics; emit a warning for InsufficientSpace while keeping actual VMGS failures as errors.

                    tracing::info!(

openhcl/underhill_core/src/worker.rs:3748

• The Other arm below explicitly treats an unrecognized token as corrupt and uses the current firmware, but this restore attempt runs before that match. If an old HIBERNATION_FIRMWARE file remains after cleanup failed, a corrupt token can therefore make a cold boot restore stale firmware instead of following the documented corrupt-token fallback. Only attempt the VMGS restore for Token::Hibernated values.

                if restore_vtl0_firmware_from_vmgs(gm.vtl0(), &mut measured_vtl0_info, vmgs_client)
                    .await

vm/loader/src/uefi/mod.rs:297

• This loop can finish without ever finding an EFI_SECTION_PE32 (for example, a malformed SEC file whose sections are non-PE or whose size ends before a section header). Because this helper is now used to validate VMGS data, the fall-through Some(image_offset) treats that image as valid, so restore overwrites VTL0 firmware and rebases RIP instead of taking the documented fallback. Return None when no PE section was found (return the computed offset directly from the PE branch).

        // A zero-size section would not advance the scan; treat as malformed.
        let section_size = expand_3byte_integer(sh.size);
        if section_size == 0 {
            return None;
        }
        image_offset = image_offset.checked_add(section_size)?;
        file_offset = file_offset.checked_add(section_size)?;

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Suppressed comments (5)

openhcl/underhill_core/src/hibernate.rs:216

• A failed replacement (or the InsufficientSpace return above) leaves any previously allocated HIBERNATION_FIRMWARE file in VMGS. The boot path still writes a hibernation token, so the next resume can read that stale image and restore the wrong firmware instead of taking the documented no-snapshot fallback. Invalidate the old snapshot when a non-restore store cannot complete, or use an atomic replacement that cannot leave stale data valid.

    vmgs_client
        .write_file(vmgs::FileId::HIBERNATION_FIRMWARE, firmware)
        .await
        .context("failed to write UEFI firmware snapshot to VMGS")?;

openhcl/underhill_core/src/hibernate.rs:257

• Power-off/reset requests can be generated repeatedly by VTL0, so a persistent VMGS deletion failure can emit an unbounded error! for every request. Other guest-triggered error paths in this worker use tracelimit::error_ratelimited! (for example, lines 4534-4548); use the rate-limited macro here as well.

            tracing::error!(

openhcl/underhill_core/src/worker.rs:3853

• When storage returns InsufficientSpace or the snapshot write fails, the previous HIBERNATION_FIRMWARE file is left intact, but the hibernate token is still retained for the next halt. After a prior snapshot this makes the next resume read stale firmware and report a successful restore, defeating the fallback to the current/host firmware. Invalidate/delete the snapshot on every non-Stored outcome, or prevent writing the hibernate token.

                Ok(hibernate::StoreFirmwareOutcome::Stored) => {}
                Ok(hibernate::StoreFirmwareOutcome::InsufficientSpace {
                    firmware_size,
                    device_size,
                }) => {

openhcl/underhill_core/src/worker.rs:3840

• The new snapshot path is guarded only by hibernation_enabled and UEFI mode; it never checks !isolation.is_isolated(). Thus an isolated UEFI guest with hibernation enabled enters this non-isolated VMGS firmware path even though this PR explicitly defers isolated/CVM support. Gate the token/restore/store flow on non-isolated VMs (or reject that configuration) before applying these assumptions to isolated firmware memory.

    if dps.general.hibernation_enabled && !is_restoring && !firmware_loaded_from_vmgs {

openhcl/underhill_core/src/worker.rs:3858

• InsufficientSpace is the documented fallback when firmware preservation is unavailable, so this should be visible as a warning rather than an info event. With the current level, filtered logs can silently miss that resumes may use the current firmware. Change this call to tracing::warn!.

                    tracing::info!(
                        CVM_ALLOWED,
                        firmware_size,
                        device_size,
                        minimum_size = VMGS_HIBERNATION_FIRMWARE_MIN_SIZE,

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

openhcl/underhill_core/src/worker.rs:3747

• Restoration is attempted before the token is classified. For a corrupt/out-of-range Token::Other, this can accept a leftover VMGS snapshot and return Some(token), so the corrupt raw value is then used as HaltState.current_token and written on the next hibernate. Restrict the VMGS restore attempt to Token::Hibernated { .. } and leave Other on the existing current-firmware path.

                if restore_vtl0_firmware_from_vmgs(gm.vtl0(), &mut measured_vtl0_info, vmgs_client)
                    .await

openhcl/underhill_core/src/worker.rs:3839

• The snapshot block is not gated on the selected load kind being UEFI. A forced PCAT/Linux boot can therefore overwrite HIBERNATION_FIRMWARE while current_hibernate_token is None and the existing hibernation token is left untouched; a later UEFI boot may consume that token and restore the image from the wrong boot. Restrict this block to LoadKind::Uefi.

    if dps.general.hibernation_enabled && !is_restoring && !firmware_loaded_from_vmgs {

openhcl/underhill_core/src/worker.rs:4420

• The host has already handled the successful LoadFirmware request by writing its image into VTL0 RAM before this returned offset is validated. If the offset is out of range, returning false does not restore the cold-boot image; the caller can then treat the overload as a fallback and this PR's snapshot path may persist the host-written bytes while keeping the old RIP. Reload/reject the image as a unit instead of continuing with the overwritten memory.

            let Some(new_rip) = base.checked_add(offset).filter(|_| offset < len) else {
                tracing::warn!(
                    CVM_ALLOWED,
                    offset,
                    firmware_len = len,

vm/loader/src/uefi/mod.rs:283

• This scan can finish without finding an EFI_SECTION_PE32, but the function still returns Some(image_offset) at its final return. The new VMGS restore path treats that as a valid entry point and only checks that it is inside the region, so a truncated/corrupt image with an FV/SEC file can be written and VTL0 RIP rebased to a non-entry address instead of falling back. Track whether the PE section was found and return None when it was not.

        let sh = EFI_COMMON_SECTION_HEADER::read_from_prefix(image.get(image_offset as usize..)?)
            .ok()?
            .0; // TODO: zerocopy: use-rest-of-range, option-to-error (https://github.com/microsoft/openvmm/issues/759)

vm/vmgs/vmgs_broker/src/broker.rs:53

• VmgsBrokerRpc derives MeshPayload without explicit variant numbers, so inserting DeviceSize here renumbers every existing variant after it (ReadFile, WriteFile, Save, and DeleteFile). The mesh derive documentation notes that enum updates can break existing binaries, so a mixed-version client/broker can decode an existing request as the wrong RPC. Append this variant after the existing ones or assign stable explicit numbers.

    DeviceSize(Rpc<(), u64>),

openhcl/underhill_core/src/worker.rs:3853

• The PR contract says that an undersized VMGS backing store is a warning while remaining non-fatal, but this branch emits info!. With normal log filtering, operators may not see that firmware preservation was skipped; use tracing::warn! here.

                    tracing::info!(

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Suppressed comments (4)

openhcl/underhill_core/src/worker.rs:3853

• The insufficient-space path is documented as a warning, but this uses tracing::info!; normal log filtering can hide that firmware preservation was skipped. Emit a warning here while retaining error! for actual VMGS failures.

                    tracing::info!(

openhcl/underhill_core/src/worker.rs:3839

• The snapshot write is likewise enabled for isolated guests, contrary to the stated non-isolated-only scope. This can copy isolated VTL0 firmware into VMGS and makes the deferred CVM behavior active; add the isolation check to this condition before calling store_firmware.

    if dps.general.hibernation_enabled && !is_restoring && !firmware_loaded_from_vmgs {

openhcl/underhill_core/src/worker.rs:3755

• When a valid snapshot restores for Token::Other, this arm returns the corrupt token unchanged. current_hibernate_token then remains Other(raw) and the next hibernate writes that same unrecognized value, bypassing the Other fallback to CURRENT below and propagating the corrupt token indefinitely. Normalize Other to CURRENT in the restore-success arm while preserving recognized hibernation tokens.

                    Some(token)

openhcl/underhill_core/src/worker.rs:3839

• supports_uefi is not equivalent to firmware_type == Uefi: the image builder can advertise both PCAT and UEFI for an x86 image, and finalize_load clears the UEFI region for LoadKind::Pcat. A PCAT cold boot therefore enters this block, snapshots an unused UEFI region, and leaves a HIBERNATION_FIRMWARE file even though no UEFI hibernate token is tracked. Gate this snapshot on matches!(firmware_type, FirmwareType::Uefi) as well.

This issue also appears on line 3853 of the same file.

    if dps.general.hibernation_enabled && !is_restoring && !firmware_loaded_from_vmgs {