|
| 1 | +# MSR state across restore |
| 2 | + |
| 3 | +## Requirement |
| 4 | + |
| 5 | +A snapshot represents the state of a VM at a point in time. This includes MSR |
| 6 | +state that can affect later execution. |
| 7 | + |
| 8 | +After `MultiUseSandbox::restore`, the destination sandbox's MSR state must match |
| 9 | +the supplied snapshot, regardless of prior execution in the sandbox. |
| 10 | + |
| 11 | +## How snapshot and restore work |
| 12 | + |
| 13 | +A snapshot saves the value of two groups of MSRs: |
| 14 | + |
| 15 | +* The MSRs you list with `SandboxConfiguration::guest_msrs`. List the ones your |
| 16 | + guest reads or writes. |
| 17 | +* A small fixed core the guest can change without a `WRMSR`, so Hyperlight |
| 18 | + always saves it: `KERNEL_GS_BASE` (via `SWAPGS`), `TSC`, and active SSP on |
| 19 | + Hyper-V (via CET instructions). |
| 20 | + |
| 21 | +`restore()` writes those saved values back and resets every other MSR to a |
| 22 | +clean default. Nothing the guest did to an MSR after the snapshot carries |
| 23 | +across restore. |
| 24 | + |
| 25 | +On KVM, listing an MSR also lets the guest use it. The guest faults if it reads |
| 26 | +or writes an MSR that is not listed. MSHV and WHP cannot enforce this, so there |
| 27 | +the list only controls what is saved and restored. |
| 28 | + |
| 29 | +## The reset set |
| 30 | + |
| 31 | +Each backend provides the MSR indices it resets. VM creation reads them before |
| 32 | +guest execution and stores the values as the destination baseline. Restore |
| 33 | +writes the snapshot's value for each index it captured and the baseline for the |
| 34 | +rest. |
| 35 | + |
| 36 | +The backend owns discovery because register mappings and capabilities differ. |
| 37 | +Sorting, deduplication, baseline capture, snapshot validation, and the fallback |
| 38 | +to the destination baseline are shared in `MsrResetState`. |
| 39 | + |
| 40 | +Every reset entry must represent guest-writable retained state the host can read |
| 41 | +and write. On the Hyper-V backends the candidate table is derived from the |
| 42 | +Hyper-V source and must be audited when that source, register mappings, or |
| 43 | +feature exposure changes. VM creation probes each entry with a host read and a |
| 44 | +host write. A candidate whose read fails is a feature absent on this host and is |
| 45 | +dropped. A candidate that reads but cannot be written fails VM creation. |
| 46 | + |
| 47 | +## Snapshot validation |
| 48 | + |
| 49 | +Snapshot state is untrusted. Every `MsrEntry` in a snapshot must name a |
| 50 | +declared guest MSR or a core MSR the destination resets. That set is |
| 51 | +backend-independent, so a snapshot a KVM destination rejects is rejected the |
| 52 | +same way on MSHV and WHP. A snapshot cannot carry an MSR the destination does |
| 53 | +not restore. A read, validation, or write failure aborts restore and poisons |
| 54 | +the sandbox. |
| 55 | + |
| 56 | +The snapshot stores captured values only. It does not store the declared MSR |
| 57 | +set. Restore applies each captured value and scrubs the rest of the reset set to |
| 58 | +the destination baseline, so the destination configuration alone governs guest |
| 59 | +MSR access. |
| 60 | + |
| 61 | +`SandboxConfiguration::guest_msrs` accepts at most 16 distinct indices. KVM |
| 62 | +also supports at most 16 contiguous filter ranges. Each declared index must be |
| 63 | +resettable, host-readable, and host-writable. Write-only command MSRs such as |
| 64 | +`PRED_CMD` and `FLUSH_CMD` hold no resettable state and cannot be declared. |
| 65 | + |
| 66 | +Some reset MSRs cannot be declared. Active SSP (`0x7A0`) survives restore |
| 67 | +on the Hyper-V backends but cannot be declared. It has no architectural |
| 68 | +`RDMSR`/`WRMSR` and is reachable only through the VP register API, so no guest |
| 69 | +`WRMSR` sets it and no filter range names it. |
| 70 | + |
| 71 | +MSHV and WHP cannot enforce declared MSRs during guest execution. There the |
| 72 | +declared set governs only which MSRs survive restore. KVM additionally enforces |
| 73 | +it as the guest's access filter. |
| 74 | + |
| 75 | +## State restored elsewhere |
| 76 | + |
| 77 | +Not all architecturally MSR-backed state uses the MSR reset path. |
| 78 | + |
| 79 | +| State | Restore owner | |
| 80 | +| --- | --- | |
| 81 | +| `EFER`, `APIC_BASE`, `FS_BASE`, `GS_BASE` | Special-register snapshot state. | |
| 82 | +| PASID (`0xD93`) on MSHV | XSAVE state. | |
| 83 | + |
| 84 | +Keeping one owner avoids restoring the same state through two backend APIs. |
| 85 | + |
| 86 | +## KVM |
| 87 | + |
| 88 | +KVM installs a default-deny MSR filter. The declared guest MSRs supply the |
| 89 | +only permitted filter ranges. VM creation rejects a declared index unless KVM |
| 90 | +lists it and the host can read and write it. |
| 91 | + |
| 92 | +The KVM reset set contains: |
| 93 | + |
| 94 | +* Every declared guest MSR. |
| 95 | +* `KERNEL_GS_BASE`, because `WRGSBASE` followed by `SWAPGS` can change it |
| 96 | + without `WRMSR`. |
| 97 | +* `TSC`, so restore rewinds guest time on every backend. |
| 98 | + |
| 99 | +The default-deny filter also covers KVM's custom MSR namespace |
| 100 | +`0x4B56_4D00..=0x4B56_4DFF`. |
| 101 | + |
| 102 | +Some CPU state does not pass through the filter. Hyperlight addresses those |
| 103 | +paths separately: |
| 104 | + |
| 105 | +* VMX and SVM are absent from guest CPUID, so the guest cannot enter VMX or |
| 106 | + SVM operation. Nested VMCS and VMCB state, including MSRs the CPU loads or |
| 107 | + stores through dedicated VMCS fields that the filter does not cover, is |
| 108 | + unreachable. Their setup MSRs remain denied. |
| 109 | +* CET is absent from guest CPUID, so the guest cannot enable shadow stacks and |
| 110 | + cannot move active SSP. Active SSP has no architectural MSR, so it is absent |
| 111 | + from the KVM reset set and the backend never restores it. The CET MSRs stay |
| 112 | + denied and cannot be declared. |
| 113 | +* x2APIC is absent from guest CPUID, so a guest never uses it. `IA32_APIC_BASE` |
| 114 | + (`0x1B`) is not declared, so the default-deny filter denies a `WRMSR` that |
| 115 | + enables x2APIC. Snapshot restore rejects the same value. The x2APIC range |
| 116 | + `0x800..=0x8FF` is exempt from the filter, so KVM permits it. Every access |
| 117 | + raises `#GP` because the guest is never in x2APIC mode. By default there is no |
| 118 | + in-kernel LAPIC to serve those MSRs. With the `hw-interrupts` feature the LAPIC |
| 119 | + stays in xAPIC mode. Either case leaves x2APIC unreachable. |
| 120 | +* `FS_BASE` and `GS_BASE` are restored through special-register state. |
| 121 | + |
| 122 | +A denied guest access raises `#GP`. Hyperlight reports `GuestAborted` and |
| 123 | +poisons the sandbox. |
| 124 | + |
| 125 | +## Hyper-V backends |
| 126 | + |
| 127 | +MSHV and WHP cannot filter guest MSR access, so restore scrubs a broad set of |
| 128 | +retained MSRs to the destination baseline before writing the saved core and |
| 129 | +declared-MSR values. The scrub set is built from: |
| 130 | + |
| 131 | +* Retained-state candidates the backend maps and can read, including state |
| 132 | + reachable only through the VP register API such as active SSP. |
| 133 | +* MTRRs required by the virtual CPU's `MTRRCAP`. |
| 134 | + |
| 135 | +The candidate table is not an intercept list. Hyper-V also intercepts read-only, |
| 136 | +command, and host-derived MSRs that retain no guest-controlled value. An entry |
| 137 | +belongs in the table only when guest execution can leave state that affects |
| 138 | +later execution. |
| 139 | + |
| 140 | +A future partition-scrub hypercall will reset all MSRs directly and replace the |
| 141 | +candidate table. WHP has kernel support. MSHV support is in progress. The saved |
| 142 | +set stays the core plus the declared MSRs across that migration, so restore |
| 143 | +behavior does not change. |
| 144 | + |
| 145 | +### MSHV |
| 146 | + |
| 147 | +MSHV enables the processor features supported by the host unless a partition |
| 148 | +feature mask disables them. Its guest-visible MSR surface therefore varies by |
| 149 | +host CPU. |
| 150 | + |
| 151 | +MSHV maps `IA32_XSS` through `MSR_IA32_REGISTER_U_XSS`. It maps `IA32_MPERF` |
| 152 | +and `IA32_APERF` through the per-VP `MCount` and `ACount` registers. TSX, |
| 153 | +WAITPKG, CET, XFD, MPX, and deadline-timer state enter the reset set when the |
| 154 | +host exposes and maps them. |
| 155 | + |
| 156 | +The host's enumerated MSR index list does not identify retained state, so it |
| 157 | +does not define the reset set. |
| 158 | + |
| 159 | +On capable Intel hosts MSHV can expose ENQCMD and PASID. PASID is a supervisor |
| 160 | +XSAVE component, so XSAVE restore owns it. |
| 161 | + |
| 162 | +### WHP |
| 163 | + |
| 164 | +WHP's default feature banks disable speculation control, experimental |
| 165 | +`DEBUGCTL` bits, and performance monitoring. Its supported feature mask omits |
| 166 | +ENQCMD and defines no FRED feature, so WHP does not expose PASID or FRED. |
| 167 | + |
| 168 | +WHP maps supported MSRs to `WHV_REGISTER_NAME` values. The same mapping is used |
| 169 | +for snapshot reads and restore writes. |
| 170 | + |
| 171 | +### MTRRs |
| 172 | + |
| 173 | +MSHV and WHP read `IA32_MTRRCAP` during VM creation. The reset set includes |
| 174 | +`MTRR_DEF_TYPE`, every variable base and mask pair reported by `VCNT`, and all |
| 175 | +fixed MTRRs. Hyper-V accepts fixed-MTRR writes even when `MTRRCAP.FIX` is |
| 176 | +clear. |
| 177 | + |
| 178 | +VM creation fails if `VCNT` exceeds the supported maximum of 16 or required |
| 179 | +MTRRs cannot be read. |
| 180 | + |
| 181 | +### TSC |
| 182 | + |
| 183 | +Hyper-V stores `TSC` and `TSC_ADJUST` independently. While time runs, it |
| 184 | +preserves `TSC - TSC_ADJUST`: writing `TSC` also changes `TSC_ADJUST`, while |
| 185 | +writing `TSC_ADJUST` changes the internal TSC offset. |
| 186 | + |
| 187 | +Restore writes `TSC` before `TSC_ADJUST`. The two writes remove the guest's |
| 188 | +delta without freezing partition time. KVM also restores `TSC` so all backends |
| 189 | +use the same guest-time semantics. |
| 190 | + |
| 191 | +## Retained-state inventory |
| 192 | + |
| 193 | +These MSRs are reset when supported by the selected backend and host. |
| 194 | + |
| 195 | +| MSR (index) | Retained state | |
| 196 | +| --- | --- | |
| 197 | +| SYSENTER CS, ESP, EIP (`0x174`-`0x176`) | System-call entry state. | |
| 198 | +| STAR, LSTAR, CSTAR, SFMASK (`0xC000_0081`-`0xC000_0084`) | System-call target state. | |
| 199 | +| KERNEL_GS_BASE (`0xC000_0102`) | Kernel GS base, including `SWAPGS` changes. | |
| 200 | +| PAT (`0x277`) | Page attribute state. | |
| 201 | +| DEBUGCTL (`0x1D9`) | Debug control state. | |
| 202 | +| SPEC_CTRL (`0x48`), VIRT_SPEC_CTRL (`0xC001_011F`) | Speculation control state. | |
| 203 | +| CET (`0x6A0`, `0x6A2`, `0x6A4`-`0x6A8`) | CET control and shadow-stack state. | |
| 204 | +| Active SSP (`0x7A0`) | Shadow-stack pointer. Reset on Hyper-V backends through the VP register API. Cannot be declared: no architectural `RDMSR`/`WRMSR`. | |
| 205 | +| XSS (`0xDA0`) | Extended supervisor state mask. | |
| 206 | +| TSC, TSC_ADJUST, TSC_AUX (`0x10`, `0x3B`, `0xC000_0103`) | Guest clock state. | |
| 207 | +| MTRRs (`0x2FF`, `0x200`-`0x21F`, `0x250`, `0x258`-`0x259`, `0x268`-`0x26F`) | Memory-type state. | |
| 208 | +| TSX_CTRL (`0x122`) | TSX control state. | |
| 209 | +| XFD, XFD_ERR (`0x1C4`, `0x1C5`) | Extended-feature disable state. | |
| 210 | +| UMWAIT_CONTROL (`0xE1`) | WAITPKG control state. | |
| 211 | +| TSC_DEADLINE (`0x6E0`) | Deadline-timer state. | |
| 212 | +| BNDCFGS (`0xD90`) | MPX bounds configuration. | |
| 213 | +| MPERF, APERF (`0xE7`, `0xE8`) | Per-VP performance counters. | |
| 214 | + |
| 215 | +These classes do not need MSR reset entries. |
| 216 | + |
| 217 | +| MSR (index) | Reason | |
| 218 | +| --- | --- | |
| 219 | +| PRED_CMD (`0x49`) | Write-only command. Issues a prediction barrier. | |
| 220 | +| FLUSH_CMD (`0x10B`) | Write-only command. Flushes caches. | |
| 221 | +| MISC_ENABLE (`0x1A0`) | Hyper-V discards writes. AMD faults the access. | |
| 222 | +| FRED (`0x1CC`-`0x1D4`) | Hyperlight exposes no FRED feature. | |
| 223 | +| PMU (`0xC1`, `0x186`, `0x38D`, `0x38F`) | Hyperlight leaves perfmon disabled. | |
| 224 | +| LBR (`0x1C8`, `0x1C9`, `0x14CE`, `0x14CF`) | Hyperlight leaves perfmon disabled. | |
| 225 | + |
| 226 | +## Testing |
| 227 | + |
| 228 | +Focused tests cover: |
| 229 | + |
| 230 | +* Guest-written MSR values across snapshot, restore, and clone lifecycles. |
| 231 | +* Saving and restoring the core and declared MSRs, and scrubbing of undeclared |
| 232 | + MSRs on the Hyper-V backends. |
| 233 | +* Backend reset-set discovery and snapshot index validation. |
| 234 | +* `SWAPGS`, TSC, MTRR, and feature-gated Hyper-V state. |
| 235 | +* KVM nested-virtualization, x2APIC, and custom-MSR denial. |
| 236 | + |
| 237 | +The ignored full-window audit probes additional Hyper-V MSR ranges on the CI |
| 238 | +CPU. It is a regression tool, not a complete inventory of vendor MSRs. |
| 239 | + |
| 240 | +## Future work |
| 241 | + |
| 242 | +* Replace the Hyper-V candidate table with the partition-scrub hypercall as |
| 243 | + MSHV gains kernel support. WHP already has it. |
| 244 | +* Exercise MSHV and WHP on more CPU models. Their reachable MSR surfaces depend |
| 245 | + on host features. |
| 246 | +* Disable optional stateful features the guest does not need, so the guest |
| 247 | + cannot retain that state. This shrinks the reset set toward the always-present |
| 248 | + core (SYSENTER, syscall targets, KERNEL_GS_BASE, PAT, DEBUGCTL, TSC, MTRRs), |
| 249 | + which every host can read. The reset set then becomes fixed, so the discovery |
| 250 | + probe that drops host-unsupported entries becomes unnecessary, and the audit |
| 251 | + surface shrinks. Do not add a minimum feature set unless the goal is to bar |
| 252 | + some host CPUs from running Hyperlight. |
| 253 | +* Extend the inventory when Hyperlight enables new CPU features such as |
| 254 | + perfmon, FRED, or nested virtualization. |
0 commit comments