Ceph - ceph#

  • Uses RBD images for images, then snapshots and clones to create instances and snapshots.

  • Due to the way copy-on-write works in RBD, parent filesystems can’t be removed until all children are gone. As a result, LXD will automatically prefix any removed but still referenced object with “zombie_” and keep it until such time the references are gone and it can safely be removed.

  • Note that LXD will assume it has full control over the osd storage pool. It is recommended to not maintain any non-LXD owned filesystem entities in a LXD OSD storage pool since LXD might delete them.

  • Note that sharing the same osd storage pool between multiple LXD instances is not supported. LXD only allows sharing of an OSD storage pool between multiple LXD instances only for backup purposes of existing instances via lxd import. In line with this, LXD requires the “ceph.osd.force_reuse” property to be set to true. If not set, LXD will refuse to reuse an osd storage pool it detected as being in use by another LXD instance.

  • When setting up a Ceph cluster that LXD is going to use we recommend using xfs as the underlying filesystem for the storage entities that are used to hold OSD storage pools. Using ext4 as the underlying filesystem for the storage entities is not recommended by Ceph upstream. You may see unexpected and erratic failures which are unrelated to LXD itself.

  • To use Ceph osd pool of type “erasure” you must have the osd pool created beforehand, as well as a separate osd pool of type “replicated” that will be used for storing metadata. This is required as RBD & CephFS do not support omap. To specify which pool is “earasure coded” you need to use the ceph.osd.data_pool_name=<erasure-coded-pool-name> and source=<replicated-pool-name> for the replicated pool.

Storage pool configuration#

Key

Type

Default

Description

ceph.cluster_name

string

ceph

Name of the Ceph cluster in which to create new storage pools

ceph.osd.data_pool_name

string

-

Name of the osd data pool

ceph.osd.force_reuse

bool

false

Force using an osd storage pool that is already in use by another LXD instance

ceph.osd.pg_num

string

32

Number of placement groups for the osd storage pool

ceph.osd.pool_name

string

name of the pool

Name of the osd storage pool

ceph.rbd.clone_copy

bool

true

Whether to use RBD lightweight clones rather than full dataset copies

ceph.rbd.du

bool

true

Whether to use rbd du to obtain disk usage data for stopped instances.

ceph.rbd.features

string

layering

Comma separate list of RBD features to enable on the volumes

ceph.user.name

string

admin

The Ceph user to use when creating storage pools and volumes

source

string

-

Existing OSD storage pool to use

volatile.pool.pristine

string

true

Whether the pool has been empty on creation time

Storage volume configuration#

Key

Type

Condition

Default

Description

block.filesystem

string

block based driver

same as volume.block.filesystem

Filesystem of the storage volume

block.mount_options

string

block based driver

same as volume.block.mount_options

Mount options for block devices

security.shifted

bool

custom volume

false

Enable id shifting overlay (allows attach by multiple isolated instances)

security.unmapped

bool

custom volume

false

Disable id mapping for the volume

size

string

appropriate driver

same as volume.size

Size of the storage volume

snapshots.expiry

string

custom volume

-

Controls when snapshots are to be deleted (expects expression like 1M 2H 3d 4w 5m 6y)

snapshots.pattern

string

custom volume

snap%d

Pongo2 template string which represents the snapshot name (used for scheduled snapshots and unnamed snapshots)

snapshots.schedule

string

custom volume

-

Cron expression (<minute> <hour> <dom> <month> <dow>), or a comma separated list of schedule aliases (@hourly, @daily, @midnight, @weekly, @monthly, @annually, @yearly)