Recovering an ESXi Host After Boot Media Corruption

A routine reboot of one of my ESXi hosts recently turned into a useful reminder that backing up virtual machines is only part of a reliable recovery strategy.

The virtual machine data was never lost. The storage remained healthy and the server hardware itself was fine. What failed was much smaller: the USB flash drive containing the ESXi installation had developed corruption in one of the files required during boot.

Fortunately, I had taken an ESXi configuration backup. That changed the recovery from manually reconstructing a hypervisor into reinstalling ESXi, restoring its configuration and reconnecting the existing storage.

The Failure

The host had been placed into maintenance mode and rebooted as part of routine work.

During boot, ESXi started loading its normal set of compressed modules but stopped when attempting to decompress vim.v00:

error 33 (inconsistent data)
gzip_extract failed for vim.v00
fatal error: 33

vim.v00 is part of the ESXi boot image. The error therefore occurred before the hypervisor itself had started and before any of the virtual machine datastores became relevant.

At first, storage was an obvious thing to consider because the host uses iSCSI for part of its environment. However, the point at which ESXi failed made it clear that this was not an iSCSI or VMFS problem. The hypervisor simply could not read its own boot image correctly.

Separating the Hypervisor from the Data

The ESXi installation was located on a USB flash drive, while the virtual machines and their data lived elsewhere. That separation turned out to be important.

Conceptually, the system looked something like this:

server
├── usb flash
│   └── esxi installation
│
├── local storage
│
└── iscsi
    └── vmfs datastores
        └── virtual machines

The failure of the USB drive prevented ESXi from booting, but it did not imply corruption of the VMFS datastores.

This distinction changes the recovery problem considerably. Rather than attempting to recover virtual machine data, the objective becomes restoring a functioning hypervisor with the configuration required to find that data again.

Recovery Options

The first recovery attempt was to boot from ESXi’s alternate bootbank.

ESXi maintains two bootbanks so that an updated image can be written to the inactive bank while the previous image remains available as a rollback path. That makes the alternate bank the least destructive recovery option when the active image becomes unusable. If the corruption is isolated to the active bootbank, the previous image can potentially boot without reinstalling the host, changing its configuration or touching any of the datastores.

In this case, the alternate bank did not provide a usable recovery path. That also increased my suspicion of the USB device itself: two bootbanks provide useful protection against a bad update or corruption isolated to one image, but both still depend on the same physical medium.

Another possibility was to clone the existing USB to a new device and repair the cloned installation.

The failing module was known, and a clean copy of vim.v00 could be obtained from matching ESXi installation media. In principle, replacing that file on the clone could have produced a bootable copy while preserving the existing installation, including any installed drivers and host-specific changes.

However, repairing individual files on failing flash media has an obvious limitation: finding one corrupted file does not establish that it is the only corrupted file. A successful repair could simply move the failure to the next damaged module.

Since I already had a recent configuration backup, rebuilding the hypervisor became the more predictable option. The original USB was preserved rather than modified further.

The Configuration Backup

ESXi can export a host configuration bundle containing the host-level configuration required to rebuild an installation.

The backup is tiny compared with the data stored by the virtual machines, but in this incident it was arguably the most useful recovery artefact. It contains configuration required to restore areas such as:

• management networking;
• standard virtual switches and port groups;
• vmkernel interfaces;
• network addressing;
• storage configuration;
• software iscsi configuration;
• dns and ntp settings;
• host services and other esxi settings.

Recent ESXi versions store the main configuration payload in encrypted form, so the bundle should still be treated as sensitive infrastructure data even though its contents are not simply exposed as plaintext configuration files.

The important distinction is that this is a host configuration backup, not an image of the ESXi installation.

It does not contain the ESXi bootbanks, the hypervisor installation image, the contents of the VMFS datastores or the virtual disks themselves. A compatible ESXi installation still has to exist first. The configuration bundle then restores the host-level state that tells that installation how to interact with the surrounding network and storage environment.

In this incident, the recovery depended on three separate things being healthy:

esxi installation
        +
esxi configuration
        +
existing vmfs datastores

The installation provided a functioning hypervisor, the configuration bundle restored how that hypervisor was connected and configured, and the datastores still contained the workloads.

That separation is probably the most technically interesting part of the incident. Without the configuration backup, the virtual machine data would still have been safe, but the surrounding networking, storage discovery and host configuration would have needed to be reconstructed manually.

VM registration is another layer again. The VM files remain on the datastore, but if inventory entries are missing they can be registered again from their existing .vmx files once the datastore is available.

Moving ESXi to NVMe

Rather than putting the new installation back onto another USB flash drive, I installed ESXi onto an NVMe SSD.

The original arrangement was not particularly unusual. Where virtual machine storage is provided separately, booting ESXi from USB avoids dedicating a local disk or valuable drive slot to the hypervisor itself.

The trade-off has become less attractive with modern ESXi releases. System storage now includes ESX-OSData, scratch space, logs and other runtime state that benefits from persistent storage designed for regular I/O. In this host, moving ESXi to NVMe therefore trades a dedicated NVMe slot for a more robust system-storage device.

The new layout is closer to:

server
├── nvme
│   └── esxi
│
└── iscsi
    └── vmfs datastores
        └── virtual machines

The move matters for more than simply replacing the device that failed. The NVMe provides persistent local storage for the hypervisor, while the VM datastores remain independently located on iSCSI. It is not being introduced as another location for workload data; its role is specifically to provide reliable local system storage for ESXi.

The architecture is cleaner even if the original USB corruption had never happened.

Restoring the Host

A fresh ESXi installation was performed on the NVMe device.

Only enough configuration was initially added to establish management access to the host. The existing configuration bundle could then be copied to ESXi and restored using the built-in configuration restore mechanism.

The recovery process was effectively:

install esxi on nvme
        │
        ▼
configure temporary management access
        │
        ▼
restore esxi configuration bundle
        │
        ▼
reboot
        │
        ▼
networking and storage configuration returns
        │
        ▼
rescan storage
        │
        ▼
existing datastores become available

Once the restored networking and storage configuration was active, the existing datastores could be discovered again and the workloads returned.

No VM datastore data needed to be restored. This was a recovery of the hypervisor and its configuration, not a recovery of the virtual machines themselves.

What Actually Failed?

The immediate failure was straightforward: ESXi could not successfully decompress and validate vim.v00 from its boot device. What caused the underlying corruption is less certain.

Possible causes include:

• ageing or failing nand flash;
• filesystem corruption;
• an interrupted write;
• controller failure within the usb device.

I did not perform sufficient forensic analysis of the original flash device to distinguish between those possibilities, so attributing the incident specifically to worn NAND would be speculation.

The practical conclusion is simpler: the boot medium produced inconsistent data and can no longer be considered reliable. Replacing it rather than attempting to return it to production was therefore the appropriate fix.

The failed USB has been retained for the moment, but it is no longer part of the production boot path.

What Went Well

The most important part of the recovery had happened before the incident: I had a recent ESXi configuration backup.

That meant the recovery did not depend on remembering exactly how every virtual switch, VMkernel interface or iSCSI target had been configured. Several other design choices helped as well.

The VM datastores were separate from the hypervisor boot device, so failure of the ESXi installation did not also mean failure of the workloads’ storage. The server also had functioning out-of-band management through IPMI, which allowed the physical machine to be inspected and controlled while the hypervisor itself was unavailable.

Finally, the original USB could be preserved rather than modified during recovery. If additional information had been required later, the failed installation was still available for inspection.

What Could Have Gone Better

The weakest part of the system was the boot medium.

USB flash had been a practical boot option because it avoided consuming a local drive bay or storage device solely for the hypervisor. On this host, however, the failure demonstrated the other side of that trade-off. Given the system-storage requirements of modern ESXi and the availability of a suitable NVMe device, retaining USB as the boot medium no longer provided enough benefit to justify it.

The other issue exposed by the incident was the remote recovery path.

The server uses an older ATEN-based IPMI implementation whose Java KVM client predates modern Apple Silicon systems. Getting the viewer running required a combination of OpenWebStart, Java 8, an x86_64 runtime and Rosetta. Even then, the remote console infrastructure itself proved less dependable than expected.

That is a useful distinction: having IPMI configured is not the same as having a tested remote recovery procedure. Out-of-band access should occasionally be tested from the machines that are actually expected to use it during an incident.

Changes Made

The main corrective action is complete: ESXi now runs from NVMe rather than USB flash.

I have also kept installation media available for recovery while avoiding making removable flash storage part of the normal boot path. The ESXi configuration backup will remain part of the normal infrastructure backup process.

In particular, I want to keep:

• a recent esxi configuration bundle;
• matching installation media;
• multiple copies of the configuration backup;
• enough documentation to rebuild management connectivity if required.

I also want the recovery procedure itself to remain testable. A backup that has never been inspected and remote management that has never been exercised are both assumptions rather than recovery mechanisms.

Lessons

The obvious lesson is to back up the hypervisor configuration, but the more interesting lesson is that there are several different kinds of recovery involved in a virtualised environment.

Backing up a VM protects its data and application state, but it does not necessarily preserve the network configuration, storage discovery, host services and other infrastructure required to run that VM. Those are separate layers:

application data
        │
virtual machine
        │
datastore
        │
hypervisor configuration
        │
hypervisor installation
        │
physical hardware

A resilient recovery strategy should consider each of those layers independently. In this incident, the VM data itself was never in danger. The failure occurred several layers below it, and the important recovery task was reconnecting a functioning hypervisor to configuration and storage that were already intact.

There was also a less technical dependency that became unexpectedly relevant: physical access to the datacentre. The facility had changed its access-control procedures since my previous visit, so my existing access was no longer sufficient. Re-registration had to be completed in person during business hours before I could reach the server, and the new process now includes fingerprint and iris verification.

Yes, really.

None of that contributed to the technical failure itself, but it did increase the recovery time. It is a useful reminder that disaster recovery depends on more than backups, spare hardware and remote-management interfaces. Physical access procedures, credentials and site-specific security requirements are part of the recovery path as well.

For infrastructure that may require hands-on intervention, those dependencies should be treated in the same way as any other operational prerequisite: documented, periodically verified and assumed to change over time.

In this case, a configuration bundle only a few megabytes in size was what made reconnecting those layers straightforward. It took very little effort to create, but saved considerably more effort when it mattered.