(disaster-recovery)= # How to recover instances in case of disaster Incus provides a tool for disaster recovery in case the {ref}`Incus database ` is corrupted or otherwise lost. The tool scans the storage pools for instances and imports the instances that it finds back into the database. You need to re-create the required entities that are missing (usually profiles, projects, and networks). ```{important} This tool should be used for disaster recovery only. Do not rely on this tool as an alternative to proper backups; you will lose data like profiles, network definitions, or server configuration. The tool must be run interactively and cannot be used in automated scripts. ``` ## Recovery process When you run the tool, it scans all storage pools that still exist in the database, looking for missing volumes that can be recovered. You can also specify the details of any unknown storage pools (those that exist on disk but do not exist in the database), and the tool attempts to scan those too. After mounting the specified storage pools (if not already mounted), the tool scans them for unknown volumes that look like they are associated with Incus. Incus maintains a `backup.yaml` file in each instance's storage volume, which contains all necessary information to recover a given instance (including instance configuration, attached devices, storage volume, and pool configuration). This data can be used to rebuild the instance, storage volume, and storage pool database records. Before recovering an instance, the tool performs some consistency checks to compare what is in the `backup.yaml` file with what is actually on disk (such as matching snapshots). If all checks out, the database records are re-created. If the storage pool database record also needs to be created, the tool uses the information from an instance's `backup.yaml` file as the basis of its configuration, rather than what the user provided during the discovery phase. However, if this information is not available, the tool falls back to restoring the pool's database record with what was provided by the user. The tool asks you to re-create missing entities like networks. However, the tool does not know how the instance was configured. That means that if some configuration was specified through the `default` profile, you must also re-add the required configuration to the profile. For example, if the `incusbr0` bridge is used in an instance and you are prompted to re-create it, you must add it back to the `default` profile so that the recovered instance uses it. ## Example This is how a recovery process could look: ```{terminal} :input: incus admin recover This Incus server currently has the following storage pools: Would you like to recover another storage pool? (yes/no) [default=no]: yes Name of the storage pool: default Name of the storage backend (btrfs, ceph, cephfs, cephobject, dir, lvm, lvmcluster, zfs): zfs Source of the storage pool (block device, volume group, dataset, path, ... as applicable): /var/lib/incus/storage-pools/default/containers Additional storage pool configuration property (KEY=VALUE, empty when done): zfs.pool_name=default Additional storage pool configuration property (KEY=VALUE, empty when done): Would you like to recover another storage pool? (yes/no) [default=no]: The recovery process will be scanning the following storage pools: - NEW: "default" (backend="zfs", source="/var/lib/incus/storage-pools/default/containers") Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: yes Scanning for unknown volumes... The following unknown volumes have been found: - Container "u1" on pool "default" in project "default" (includes 0 snapshots) - Container "u2" on pool "default" in project "default" (includes 0 snapshots) You are currently missing the following: - Network "incusbr0" in project "default" Please create those missing entries and then hit ENTER: ^Z [1]+ Stopped incus admin recover :input: incus network create incusbr0 Network incusbr0 created :input: fg incus admin recover The following unknown volumes have been found: - Container "u1" on pool "default" in project "default" (includes 0 snapshots) - Container "u2" on pool "default" in project "default" (includes 0 snapshots) Would you like those to be recovered? (yes/no) [default=no]: yes Starting recovery... :input: incus list +------+---------+------+------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+---------+------+------+-----------+-----------+ | u1 | STOPPED | | | CONTAINER | 0 | +------+---------+------+------+-----------+-----------+ | u2 | STOPPED | | | CONTAINER | 0 | +------+---------+------+------+-----------+-----------+ :input: incus profile device add default eth0 nic network=incusbr0 name=eth0 Device eth0 added to default :input: incus start u1 :input: incus list +------+---------+-------------------+---------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+---------+-------------------+---------------------------------------------+-----------+-----------+ | u1 | RUNNING | 192.0.2.49 (eth0) | 2001:db8:8b6:abfe:216:3eff:fe82:918e (eth0) | CONTAINER | 0 | +------+---------+-------------------+---------------------------------------------+-----------+-----------+ | u2 | STOPPED | | | CONTAINER | 0 | +------+---------+-------------------+---------------------------------------------+-----------+-----------+ ```