System recovery

IncusOS is designed to be fairly resilient to failures, but there may be times when your system misbehaves. Here are some suggestions that might be useful if you ever need to recover an IncusOS system.

Fallback API endpoint

When IncusOS boots from the prior installed version, or if the primary application fails to start, the system will automatically attempt to activate a fallback API endpoint. This fallback endpoint allows trusted clients to maintain access to IncusOS via the network even if the primary application is unavailable for some reason.

By default, IncusOS will pick a random port and attempt to listen on all available network interfaces. The chosen port number will be displayed on the system’s terminal. A specific IP address and/or port can be set to limit where the fallback API endpoint is configured to listen, either as an IP:port such as 10.0.0.1:8445 or as a :port such as :8445 to listen on all interfaces.

The fallback API endpoint can also be configured to always run by setting the persistent option, providing a stable way to reach IncusOS regardless of the primary application’s state.

On first boot, IncusOS will attempt to extract any trusted client certificates present in seed data and set them as trusted client certificates for the fallback API endpoint. However, no further automatic synchronization of trusted client certificates is performed. The list of trusted client certificates for the fallback API endpoint can be updated at any time via

incus admin os system fallback-listener edit

Try booting into the previous image

IncusOS uses an A/B update mechanism to reboot onto the newer version while keeping the previous version available should a revert be needed. You can reboot your server and select the prior version at the boot menu. If that works, it means that something went wrong with the latest update – please report a bug!

Encryption recovery key(s)

IncusOS binds encryption of the install drive to the system’s TPM state and stores any additional pool encryption keys on that encrypted drive. You can retrieve an encryption recovery passphrase for the install drive as well as any pool encryption keys via the API. (You did do that and saved those somewhere safe before we ended up here, right?)

If something unexpectedly changes the TPM state of your system, you can still boot but will need to manually provide an encryption recovery passphrase. After IncusOS starts up, you can use the API to force-reset the TPM encryption bindings which should allow automatic decryption of the install drive at boot time.

Note

When manually entering the recovery, keep in mind that the keyboard layout is a traditional United States layout and that the recovery key must be typed as is, including the dashes.

Alternatively, with the recovery key(s), you can remove the affected drive(s) to a different machine and unlock them to access/migrate any data they contain.

Unlocking with a recovery key on RESCUE_DATA

If the TPM is unable to unlock the install drive and typing the recovery passphrase on the console isn’t practical (for example when no keyboard is available), the passphrase can be placed in a file named recovery.txt at the root of a FAT-formatted partition labeled RESCUE_DATA. During boot, IncusOS will display a warning, wait one minute so the process can be aborted by powering off the system, securely destroy the file and then use the passphrase to unlock the install drive.

The drive is only unlocked if the file was successfully destroyed, so a system won’t keep booting with the passphrase available on an attached device.

Tip

IncusOS has some basic recovery key complexity rules:

  • Must be at least 15 characters long

  • Must contain at least one special character

  • Must consist of at least five unique characters

Drive failure

If your install drive fails, sorry but there’s not much that can be done other than a new install. :(

If a drive in a storage pool fails, and the pool has sufficient redundancy, you can remove the failed drive and replace it with a new one via the API. The underlying pool driver will begin data recovery process(es), which you can monitor via querying the status of the storage endpoint.

Recovery mode

A special “recovery mode” can be triggered early in the IncusOS boot sequence if a data partition labeled RESCUE_DATA and formatted as FAT or ISO is present. IncusOS will automatically attempt to find and run a hot-fix script named hotfix.sh.sig at the root of that partition, followed by any OS or application updates contained in an update/ directory also at the root of the recovery partition.

Both the hot-fix script and update metadata JSON file must be properly signed by the same certificate used to distribute normal IncusOS updates. This prevents an attacker from simply being able to connect a random USB stick and then running arbitrary commands with full system access.

The recovery mode is intended as an option of last resort.