TrueNAS - truenas

The truenas storage driver in Incus

The truenas storage driver enables an Incus node to use a remote TrueNAS storage server to host one or more Incus storage pools. When the node is part of a cluster, all cluster members can access the storage pool simultaneously, making it ideal for use cases such as live migrating virtual machines (VMs) between nodes.

The driver operates in a block-based manner, meaning that all Incus volumes are created as ZFS Volume block devices on the remote TrueNAS server. These ZFS Volume block devices are accessed on the local Incus node via iSCSI.

Modeled after the existing ZFS driver, the truenas driver supports most standard ZFS functionality, but operates on remote TrueNAS servers. For instance, a local VM can be snapshotted and cloned, with the snapshot and clone operations performed on the remote server after synchronizing the local file system. The clone is then activated through iSCSI as necessary.

Each storage pool corresponds to a ZFS dataset on a remote TrueNAS host. The dataset is created automatically if it does not exist. The driver uses ZFS features available on the remote host to support efficient image handling, copy operations, and snapshot management without requiring nested ZFS (ZFS-on-ZFS).

To reference a remote dataset, the source property can be specified in the form: [<remote host>:]<remote pool>[[/<remote dataset>]...][/]

If the path ends with a trailing /, the dataset name will be derived from the Incus storage pool name (e.g., tank/pool1).

Requirements

The driver relies on the truenas_incus_ctl tool to interact with the TrueNAS API and perform actions on the remote server. This tool also manages the activation and deactivation of remote ZFS Volumes via open-iscsi. If truenas_incus_ctl is not installed or available in the system’s PATH, the driver will be disabled.

To install the required tool, download the latest version (v0.7.2+ is required) from the truenas\_incus\_ctl GitHub page. Additionally, ensure that open-iscsi is installed on the system, which can be done using:

sudo apt install open-iscsi

Logging in to the TrueNAS host

As an alternative to manually creating an API Key and supplying using the truenas.api_key property, you can instead login to the remote server using the truenas_incus_ctl tool.

sudo truenas_incus_ctl config login

This will prompt you to provide connection details for the TrueNAS server, including authentication details, and will save the configuration to a local file. After logging in, you can verify the iSCSI setup with:

sudo truenas_incus_ctl share iscsi setup --test

Once the tool is configured, you can use it to interact with remote datasets and create storage pools:

incus storage create <poolname> truenas source=[host:]<pool>[/<dataset>]/[remote-poolname]

In this command:

  • source refers to the location on the remote TrueNAS host where the storage pool will be created.

  • host is optional, and can be specified using the truenas.host property, or by specifying a configuration with truenas.config

  • If remote-poolname is not supplied, it will default to the name of the local pool.

Configuration options

The following configuration options are available for storage pools that use the truenas driver and for storage volumes in these pools.

Storage pool configuration

source

ZFS dataset to use on the remote TrueNAS host. Format: [<host>:]<pool>[/<dataset>][/]. If host is omitted here, it must be set via truenas.host.

Key: source
Type:

string
Default:
Scope:

local
truenas.allow_insecure

If set to true, allows insecure (non-TLS) connections to the TrueNAS API.

Key: truenas.allow_insecure
Type:

bool
Default:

false
Scope:

global
truenas.api_key

API key used to authenticate with the TrueNAS host.

Key: truenas.api_key
Type:

string
Default:
Scope:

global
truenas.clone_copy

Whether to use lightweight clones rather than full dataset copies.

Key: truenas.clone_copy
Type:

bool
Default:

true
Scope:

global
truenas.config

Path to a configuration file for the TrueNAS client tool.

Key: truenas.config
Type:

string
Default:
Scope:

global
truenas.dataset

Remote dataset name. Typically inferred from source, but can be overridden.

Key: truenas.dataset
Type:

string
Default:
Scope:

global
truenas.force_reuse

Allow to use an existing non-empty pool.

Key: truenas.force_reuse
Type:

bool
Default:

false
Scope:

global
truenas.host

Hostname or IP address of the remote TrueNAS system. Optional if included in the source, or a configuration is used.

Key: truenas.host
Type:

string
Default:
Scope:

global
truenas.initiator

iSCSI initiator name used during block volume attachment.

Key: truenas.initiator
Type:

string
Default:
Scope:

global
truenas.portal

iSCSI portal address to use for block volume connections.

Key: truenas.portal
Type:

string
Default:
Scope:

global

Tip

In addition to these configurations, you can also set default values for the storage volume configurations. See Configure default values for storage volumes.

Storage volume configuration

block.filesystem

File system of the storage volume: btrfs, ext4 or xfs (ext4 if not set)

Key: block.filesystem
Type:

string
Default:

same as volume.block.filesystem
Condition:
block.mount_options

Mount options for block-backed file system volumes

Key: block.mount_options
Type:

string
Default:

same as volume.block.mount_options
Condition:
initial.gid

GID of the volume owner in the instance

Key: initial.gid
Type:

int
Default:

same as volume.initial.gid or 0
Condition:

custom volume with content type filesystem
initial.mode

Mode of the volume in the instance

Key: initial.mode
Type:

int
Default:

same as volume.initial.mode or 711
Condition:

custom volume with content type filesystem
initial.uid

UID of the volume owner in the instance

Key: initial.uid
Type:

int
Default:

same as volume.initial.uid or 0
Condition:

custom volume with content type filesystem
security.shared

Enable sharing the volume across multiple instances

Key: security.shared
Type:

bool
Default:

same as volume.security.shared or false
Condition:

custom block volume
security.shifted

Enable ID shifting overlay (allows attach by multiple isolated instances)

Key: security.shifted
Type:

bool
Default:

same as volume.security.shifted or false
Condition:

custom volume
security.unmapped

Disable ID mapping for the volume

Key: security.unmapped
Type:

bool
Default:

same as volume.security.unmapped or false
Condition:

custom volume
size

Size/quota of the storage volume

Key: size
Type:

string
Default:

same as volume.size
Condition:

appropriate driver
snapshots.expiry

Controls when snapshots are to be deleted (expects an expression like 1M 2H 3d 4w 5m 6y)

Key: snapshots.expiry
Type:

string
Default:

same as volume.snapshot.expiry
Condition:

custom volume
snapshots.expiry.manual

Controls when snapshots are to be deleted (expects an expression like 1M 2H 3d 4w 5m 6y)

Key: snapshots.expiry.manual
Type:

string
Default:

same as volume.snapshot.expiry.manual
Condition:

custom volume
snapshots.pattern

Pongo2 template string that represents the snapshot name (used for scheduled snapshots and unnamed snapshots)

Key: snapshots.pattern
Type:

string
Default:

same as volume.snapshot.pattern or snap%d
Condition:

custom volume
snapshots.schedule

Cron expression (<minute> <hour> <dom> <month> <dow>), a comma-separated list of schedule aliases (@hourly, @daily, @midnight, @weekly, @monthly, @annually, @yearly), or empty to disable automatic snapshots (the default)

Key: snapshots.schedule
Type:

string
Default:

same as volume.snapshot.schedule
Condition:

custom volume
truenas.blocksize

Size of the ZFS block in range from 512 bytes to 16 MiB (must be power of 2) - for block volume, a maximum value of 128 KiB will be used even if a higher value is set

Key: truenas.blocksize
Type:

string
Default:

same as volume.truenas.blocksize
Condition:
truenas.remove_snapshots

Remove snapshots as needed

Key: truenas.remove_snapshots
Type:

bool
Default:

same as volume.truenas.remove_snapshots or false
Condition:
truenas.use_refquota

Use refquota instead of quota for space

Key: truenas.use_refquota
Type:

bool
Default:

same as volume.truenas.use_refquota or false
Condition: