Back to the news overview

LXD 4.14 has been released

7th of May 2021


The LXD team is very excited to announce the release of LXD 4.14!

This was a slightly shorted development cycle for us but a pretty busy one with nice improvements to both standalone and cluster users.


New features and highlights

ACL support on managed bridges

Network ACLs can now be used with traditional managed bridges now.

The feature works a bit differently than when used with OVN networks though as there was no practical way to do the port-based ACLs with traffic labeling with normal bridges.

Instead what's available is tying the managed bridge with one or more ACLs with those ACLs applying to egress/ingress on the bridge. Internal bridge traffic will not be affected by those ACLs.

Cluster member certificates

LXD now makes sure that each cluster member has its own distinct server certificate and that this server certificate is used for member to member communication.

This doesn't change anything to the client facing API as that one will always be served the cluster-wide certificate. Instead this change allows for easy validation of what cluster member issued a paritcular request and also allows preventing any further communication from a previous cluster when removed.

NOTE: If you were directly consuming server.crt and expecting that to match the public TLS API, this won't be the case anymore on clusters, instead use cluster.crt.

Cluster member description

Cluster members can now have a description which is editable with lxc cluster edit and displayed in lxc cluster list.

stgraber@castiana:~$ lxc cluster list
|  NAME   |                 URL                 | DATABASE | ARCHITECTURE | FAILURE DOMAIN |     DESCRIPTION      | STATE  |      MESSAGE      |
| abydos  | https://[2602:fd23:8:200::100]:8443 | YES      | x86_64       | default        | HIVE - top server    | ONLINE | Fully operational |
| langara | https://[2602:fd23:8:200::101]:8443 | YES      | x86_64       | default        | HIVE - middle server | ONLINE | Fully operational |
| orilla  | https://[2602:fd23:8:200::102]:8443 | YES      | x86_64       | default        | HIVE - bottom server | ONLINE | Fully operational |

Cluster token based join

Instead of having to pre-add the joining server's certificate to the cluster ahead of time or have to rely on a shared trust password, it's now possible to generate a single-use token for a new cluster member.

This is done using lxc cluster add:

root@vm01:~# lxc cluster add vm02

Then at joining time, this can be provided like so:

root@vm02:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What name should be used to identify this node in the cluster? [default=vm02]: 
What IP address or DNS name should be used to reach this node? [default=]: 
Are you joining an existing cluster? (yes/no) [default=no]: yes
Do you have a join token? (yes/no) [default=no]: yes
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes

Server warnings

Instead of just burying warnings in our log file, we're also now starting to expose them through our API. Currently only cgroup warnings are exposed but we expect this to slowly grow in use over the next few releases.

Warnings are tracked cluster-wide, can self-resolve when corrected (if LXD can detect it to be solved) and they can also be silenced by acknowledging them.

stgraber@castiana:~$ lxc warning list
|                 UUID                 |                 TYPE                  | STATUS | SEVERITY | COUNT | PROJECT | LOCATION |          LAST SEEN          |
| 51526df0-b71b-41ca-8936-5c8c9298111c | Couldn't find the CGroup blkio.weight | NEW    | LOW      | 1     |         |          | May 7, 2021 at 5:40pm (UTC) |

stgraber@castiana:~$ lxc warning ack 51526df0-b71b-41ca-8936-5c8c9298111c
stgraber@castiana:~$ lxc warning list

stgraber@castiana:~$ lxc warning list -a
|                 UUID                 |                 TYPE                  |    STATUS    | SEVERITY | COUNT | PROJECT | LOCATION |          LAST SEEN          |
| 51526df0-b71b-41ca-8936-5c8c9298111c | Couldn't find the CGroup blkio.weight | ACKNOWLEDGED | LOW      | 1     |         |          | May 7, 2021 at 5:40pm (UTC) |

Backup and snapshot project restrictions

A new set of project restrictions was added to control backups and snapshots.

Backups and snapshots can be resource expensive, both in CPU time and disk usage, so you may not want every project user to be allowed to use them.

The restricted.snapshots and restricted.backups config keys can be used to control this. Those apply to both instance and custom volume snapshots/backups and also affect scheduled snapshots.

User keys in device configuration

It's now possible to user user.XYZ type keys directly in device configs.

stgraber@castiana:~$ lxc config device set cgroup2 root bar
stgraber@castiana:~$ lxc config device get cgroup2 root

We believe this was the last place in LXD's API where the user namespace of config keys wasn't allowed yet. This being added now should make it easy for 3rd party tools to store their metadata directly on LXD objects.

More auto-generated REST-API documentation

We made some more progress on adding swagger metadata to our API.
A bit over half of the instance endpoints are now covered with the rest expected to land in the coming days. This will leave us with just the storage API to document.

We're now expecting this to be fully complete in LXD 4.15.

A temporary rendering of this can be found here:

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • lxd/firewwall/drivers/drivers/nftables: Changes nftables to use a single inet table rather than separate ip and ip6 tables
  • test: Updates proxy tests to check nftables inet table
  • tests: Fix failure on 5.11 kernel
  • lxd/network/acl/driver/common: Improve error messages in d.validateRuleSubjects
  • lxd/network/acl/driver/common: Allow single IP addresses in source/destination subjects
  • lxd/network/driver/ovn: Renames logName var to logPrefix
  • lxd/network/acl/acl/ovn: Handle single IP in source/destination subjects
  • lxd/network/acl/acl/ovn: Renames logName arg to logPrefix in OVNApplyInstanceNICDefaultRules
  • lxd/firewall/drivers/drivers/xtables: Don't check for existing rule in iptablesAdd
  • lxd/firewall/drivers/drivers/xtables: Updates d.iptablesChainExists to return if chain has any rules
  • shared/validate/validate: Check IsNetworkPortRange range starts lower than end
  • forkexec: log more failures
  • Revert "lxd/firewall/drivers/drivers/xtables: Don't check for existing rule in iptablesAdd"
  • lxd/daemon/images: Adds imageDownloadLock function
  • lxd/daemon/images: Use d.imageDownloadLock in ImageDownload
  • lxd/daemon/images: Improve error messages in ImageDownload
  • lxd/instance: Improve error messages in instanceCreateFromImage
  • lxd/instance: Use d.imageDownloadLock in instanceCreateFromImage
  • lxd/firewall/drivers/drivers/xtables: Only add multiple instance <> instance NAT rules for proxy if connect port range used in InstanceSetupProxyNAT
  • lxd/firewall/drivers/drivers/xtables: Don't look for existing rules in iptablesAdd
  • lxd/firewall/drivers/drivers/nftables: Only add multiple instance <> instance NAT rules for proxy if connect port range used in InstanceSetupProxyNAT
  • lxd/network/acl/driver/common: Add check for incorrect use of mixed IP family subjects in validateRule
  • lxd/network/acl/driver/common: Improve IP family validation for ICMP
  • lxd/networks: Updates doNetworksCreate to skip network validation during pre-join phase
  • lxd/network/acl/driver/common: Fix typo
  • lxd/project: Only consider syscall interception as low-level
  • lxd/network/driver/bridge: Validate ACL options
  • lxd/firewall/firewall/interface: Adds NetworkApplyACLRules function
  • lxd/firewall/drivers/drivers/consts: Adds ACL field to Opts to indicate we should ready firewall setup for ACL rules
  • lxd/firewall/drivers/drivers/consts: Define ACLRule type
  • lxd/network/acl/acl/firewall: Adds ACL firewall helper functions
  • lxd/network/acl/acl/interface: Adds clientType to Update
  • lxd/network/acl/acl/load: Add Config field to NetworkACLUsage
  • lxd/network/acl/acl/load: Update NetworkUsage to return bridge and ovn networks
  • lxd/network/acl/driver/common: Modifies Update to support clientType arg
  • lxd/network/acls: Pass clientType to netACL.Update
  • lxd/network/driver/ovn: Populate acl.NetworkACLUsage Config field
  • lxd/network/driver/bridge: Set ACL field in fwOpts if ACLs specified
  • lxd/network/driver/bridge: Apply ACLs on network start
  • lxd/firewall/drivers/drivers/xtables: Add ACL support
  • lxd/firewall/drivers/drivers/nftables: NetworkApplyACLRules WIP
  • test: Adds container_devices_nic_bridged_acl test for bridge ACL tests with iptables and nftables
  • api: Adds network_bridge_acl extension
  • doc/networks: Adds security.acls* config keys for bridge networks
  • doc: Adds network ACLs limitations section and links to it from bridge security.acls config key
  • shared/simplestreams: Improve error messages
  • lxd/daemon/images: Improve error messages
  • client/simplestreams/images: Improve error messages
  • test/godeps.list: Adds
  • Renames container_devices_nic_bridged_acl test file to align with other tests
  • test: Fix Udhcpc6 detection for nic bridged acl tests
  • lxd: don't set device cgroup values for unpriv containers
  • lxd/storage: Re-introduce cluster distributino of volume snapshots
  • lxc/remote: Only update URL in set-url
  • lxd/instance/drivers: Don't overwrite template triggers
  • lxd/daemon/images: Removes unnecessary imagesDownloadingLock mutex
  • lxc: Fix help for string arguments
  • tests: Fix apply_template check
  • lxd/cluster/membership: Updates Bootstrap to generate new cluster certificate
  • lxd/util/encryption: Comment on LoadCert
  • lxd/util/encryption: Adds LoadClusterCert function
  • test: load cluster.crt not server.crt when bootstrapping a cluster
  • lxc-to-lxd: Fix TestConvertNetworkConfig loopback only test
  • lxd/db: Add cluster ToAPI
  • lxd: Switch to using GetNodes
  • lxd/cluster: Drop List function
  • tests: Update for current cluster messages
  • lxd/lxd: Prevent multiple routed NIC devices from using "auto" gateway mode
  • lxd/db: Add node ID to StorageVolumeArgs
  • lxd/storage/volumes/snapshots: Updates autoCreateCustomVolumeSnapshotsTask to always snapshot local custom volumes
  • lxd/storage/volumes/snapshots: Update autoCreateCustomVolumeSnapshots return value
  • lxd/storage/volumes/snapshots: Adds comments to autoCreateCustomVolumeSnapshots
  • lxd: Move image-refresh to /internal/testing/
  • test/suites: Use /internal/testing/image-refresh
  • lxc/storage_volume: Properly use cluster target
  • lxc/storage_volume: Add missing target
  • vm/qemu: configure spice using -spice parameter
  • lxd/storage_volume_snapshots: Fix cluster redirection
  • lxd/db/storage: Properly increment snapshots
  • lxd/storage_pools: Fix ordering of pool delete
  • lxd/endpoint: Retry binding on startup
  • lxd/instance/qemu: Move to query-cpus-fast
  • shared/api/certificate: Adds certificate type constants
  • lxd/db/certificates: Adds CertificateAPITypeToDBType and ToAPIType functions
  • lxd/db/certificates: Comment on Certificate
  • lxc: Switch to CertificateTypeClient constant
  • lxd: Switch to CertificateTypeClient constant
  • lxd-p2c/utils: Switch to CertificateTypeClient constant
  • lxd/util/encryption: Adds LoadServerCert function
  • lxd/certificates: updateCertificateCache error quoting consistency
  • lxd/certificates: Store certificateCache by certificate type
  • lxd/certificates: db.CertificateAPITypeToDBType usage and set certificate type to server
  • lxd/certificates: Comment ending consistency
  • lxd/daemon: Adds getTrustedCertificates function
  • lxd/daemon: Update Authenticate to use d.getTrusterCertificates
  • lxd/daemon: Comment ending consistency in Authenticate
  • lxd/daemon: Ensure d.clientCerts.Projects isn't accessed without a lock
  • tests: Removes use of -v flag for nc inside busybox
  • lxd/network/driver/bridge: Don't attempt to setup ipv6 firewall when no ipv6.address
  • lxd/swagger: Add NotFound response
  • lxd/snapshots: Fix multiple schedules
  • lxd/images: Ignore intervals on manual refreshes
  • api: Add warnings
  • shared/api: Add warning structs
  • lxd/db/cluster: Create warnings table
  • lxd/db: Add operation type OperationWarningsPruneResolved
  • lxd/db: Allow object retrieval by ID
  • lxd/db: Add getURIFromEntity
  • lxd/db: Add warning severity
  • lxd/db: Add warning types
  • lxd/db: Add warning status
  • lxd/db: Add warning functions
  • lxd: Add warnings endpoints
  • lxd: Prune resolved warnings
  • lxd: Add internal /testing/warnings endpoint
  • client: Add warning functions
  • lxc: Add warning sub-command
  • test: Add warnings test
  • po: Update translations
  • lxd/db: Fix typo in error
  • doc/rest-api: Refresh swagger YAML
  • doc/rest-api: Add warnings
  • lxd: Add function to resolve warnings by their type code
  • lxd/db: Add GetWarningsByType
  • lxd/db: Add CGroup warning types
  • lxd/cgroup: Replace Log() with Warnings()
  • lxd: Handle CGroup logging and warnings in daemon
  • lxd/util/http: Fix CheckTrustState to block access for revoked certificates that were formerly trusted
  • lxd/certificates: Adds comment about the importance of a check related to CA mode in certificatesPost
  • test: Remove trusted remote before checking adding untrusted remote
  • test: Don't use CA PKI generated server certs for LXD
  • test/deps: Removes alternative pre-generated server cert and key
  • test: Remove LXD_ALT_CERT
  • test: Update PKI tests to comply with expectations of revocation behaviour
  • lxd/ip: Add ip package
  • lxd/device: Replace ip command with ip package
  • lxd: Replace ip command with ip package
  • lxd/network: Replace ip command with ip package
  • lxd/openvswitch: Replace ip command with ip package
  • tests: Add ip package to static_analysis test
  • lxd/networks/utils: Log forkdns refresh task starting in networkUpdateForkdnsServersTask
  • lxd/db/node: Adds certificates table to local database
  • lxd/db/certificates: Adds GetCertificates function
  • lxd/db/certificates: Adds ReplaceCertificates function
  • lxd/db/certificates: Adds DeleteCertificateByNameAndType function
  • lxd/certificates: Fix import ordering
  • lxd/certificates: Updates updateCertificateCache to handle per-certificate upgrade
  • Revert "lxd/instance/qemu: Move to query-cpus-fast"
  • lxd/certificates: Adds updateCertificateCacheFromLocal function
  • lxd/certificates: Notify other cluster members of certificate update in doCertificateUpdate
  • lxd/certificates: Notify other cluster members of certificate deletion in certificateDelete
  • lxd/certificates: Allow certificate type change in doCertificateUpdate
  • lxd/certificates: cluster.ErrCertificateExists and serverCert usage in certificatesPost
  • lxd/daemon: Adds serverCert and serverCertInt vars
  • lxd/daemon: Updates State to populate serverCert
  • lxd/daemon: Load trusted server certs from local DB on startup using updateCertificateCacheFromLocal
  • lxd/daemon: Refresh cached trusted certificates when heartbeat node count changes in NodeRefreshTask
  • lxd/daemon: Pass d.serverCert and networkCert to startClusterTasks Add
  • lxd/daemon: Updates Authenticate to check trusted server certs
  • lxd/state: Updates NewState to have a serverCert and updateCertificateCache arg
  • lxd/state: Update tests with NewState usage
  • lxd/util/http: Updates CheckTrustState to use networkCert argument
  • lxd/cluster/notify: Update NewNotifier to accept networkCert and serverCert args
  • lxd/cluster/tls: Update tlsClientConfig to accept networkCert and serverCert
  • lxd/cluster/tls: Updates tlsCheckCert to accept networkCert and serverCert
  • lxd/cluster/connect: Adds ErrCertificateExists var
  • lxd/cluster/connect: Updates Connect to accept networkCert and serverCert args
  • lxd/cluster/connect: Updates SetupTrust to accept serverName arg
  • lxd/cluster/connect: Adds UpdateTrust function
  • lxd/cluster/connect: Updates HasConnectivity to accept networkCert and serverCert
  • lxd/cluster/events: Updates events functions to accept networkCert and serverCert
  • lxd/cluster/gateway: Store networkCert and serverCert in Gateway and update NewGateway
  • lxd/cluster/gateway: Updates HandlerFuncs to accept trustedCerts function
  • lxd/cluster/gateway: HasConnectivity usage
  • lxd/cluster/gateway: Update Reset to handle networkCert
  • lxd/cluster/gateway: tlsClientConfig usage
  • lxd/cluster/gateway: loadInfo usage
  • lxd/cluster/heartbeat: tlsClientConfig in Send and Heartbeat
  • lxd/cluster/upgrade: Updates NotifyUpgradeCompleted with networkCert and serverCert args
  • lxd/cluster/membership: Adds EnsureServerCertificateTrusted function
  • lxd/cluster/membership: Updates Bootstrap to store serverCert in trusted certificates table
  • lxd/cluster/membership: Update Join to handle per-server certificates
  • lxd/cluster/membership: Updates notifyNodesUpdate to handle serverCert
  • lxd/cluster/membership: HasConnectivity usage
  • lxd/cluster/membership: Update Purge to remove trusted server certificate
  • lxd/cluster: Update tests to work with changes
  • lxd/api: d.gateway.HandlerFuncs usage
  • lxd/api/cluster: Updates clusterPutJoin to handle per server certificates
  • lxd/api/cluster: d.gateway.Reset usage
  • lxd/api/cluster: Call updateCertificateCache in clusterNodeDelete after certificate removed
  • lxd/api/cluster/test: server name as cert name
  • lxd/main/init: state.NewState usage
  • lxd/main/init/interactive: cluster.SetupTrust usage and serverCert naming for consistency
  • lxd: cluster.NewNotifier usage
  • lxd: cluster.Connect and related function usage
  • lxd/patches: Adds patchClusteringServerCertTrust
  • doc/clustering: Update guide to show that cluster.crt on bootstrap member should be used
  • test: Add check for trusted server certificate removal on cluster member removal
  • test: Update table count check to account for local certificates table
  • lxd/firewall/drivers/drivers/nftables: Require kernel version >= 5.2 to allow support for inet table NAT rules
  • test/suites: Update warning tests
  • lxd/images: Specify image type during distribution
  • client/connection: Correct HTTPs to HTTPS in ConnectPublicLXD
  • lxd/operations: Clarify return values in comment on Render
  • lxd/db/operations: Adds GetOnlineNodesWithRunningOperationsOfType function
  • lxd/operations: Adds operationCancel function
  • lxd/operations: Adds operationsGetByType function
  • lxd/images: Updates imageValidSecret to accept projectName and opType arguments
  • lxd/images: projectName argument in createTokenResponse
  • lxd/images: imageValidSecret usage
  • lxd/operations: Updates operationsGet to use projectName when retrieving remote operations
  • lxd/operations: Updates operationsGetByType to use projectName when retrieving remote operations
  • lxd/instances: Swagger for logs
  • lxd/instances: Update error message
  • lxd/instances: Swagger for files
  • doc/rest-api: Refresh swagger YAML
  • lxc/warning: Fix argument parsing
  • lxc/warning: Fix usage and comments
  • i18n: Update translation templates
  • lxd/project: Add AllowBackupCreation and AllowSnapshotCreation
  • lxd/projects: Add restricted.backups and restricted.snapshots
  • lxd: Support for restricted.backups and restricted.snapshots
  • api: projects_restricted_backups_and_snapshots
  • doc/projects: Add restricted.backups and restricted.snapshots
  • shared/api: Add swagger metadata for instance exec
  • lxd/instances: Swagger for exec
  • lxd/swagger: Fix json name of metadata
  • shared/api: Add swagger metadata for instance state
  • lxd/instances: Swagger for state
  • shared/api: Add swagger metadata for instance console
  • lxd/instances: Swagger for console
  • shared/api: Add swagger metadata for instances
  • lxd/instances: Swagger for instance
  • doc/rest-api: Refresh swagger YAML
  • lxd/instance/qmp: Switch to query-cpus-fast
  • lxd/apparmor: Respect LXD_OVMF_PATH
  • lxd/daemon: Improved logging in NodeRefreshTask
  • lxd/db/operations: Import ordering
  • lxd/db/operations/types: Adds OperationClusterJoinToken type
  • lxd/db/operations: Replace GetOnlineNodesWithRunningOperationsOfType with GetOperationsOfType
  • lxd/operations: Updates operationCancel with correct remote address
  • lxd/operations: Fixes operationsGetByType to filter operations by type correctly
  • lxd/node/raft/test: Corrects typo
  • api: Adds clustering_join_token extension
  • shared/api/cluster: Adds ClusterMembersPost type
  • shared/api/cluster: Adds ClusterMemberJoinToken type
  • lxd/api/cluster: Adds clusterNodesPost handler
  • client/interfaces: Adds CreateClusterMember function to interface
  • client/lxd/cluster: Adds CreateClusterMember function
  • lxc/cluster: Add lxc cluster add command
  • lxd/certificates: Adds clusterMemberJoinTokenValid and clusterMemberJoinTokenDecode functions
  • lxd/certificates: Updates certificatesPost to check supplied password against active cluster join token operations
  • lxd/main/init/interactive: Adds join token support to askClustering
  • lxc/cluster: Adds cluster list-tokens command
  • lxc/cluster: Adds clusterJoinTokenOperationToAPI function
  • lxd/operations: Updates OperationClass.String() to use constants from shared/api
  • shared/api/operations: Adds operation class name constants
  • doc/clustering: Adds details on using the join token during adding cluster members
  • i18n: Update translation templates
  • test: Adds overridable join secret to spawn_lxd_and_join_cluster
  • test: Adds join token tests to clustering_membership
  • test: Increase the offline thresholds to above 12 as heartbeat interval is hardcoded to 10
  • doc/rest-api: Refresh swagger YAML
  • Makefile: Set GO111MODULE=on for update-api swagger build
  • shared/api: Fix snapshot structs
  • lxc/config: Update following InstanceSnapshotPut fix
  • shared/api: Add swagger metadata for instance snapshots
  • lxd/instances: Swagger for snapshots
  • doc/rest-api: Refresh swagger YAML
  • i18n: Update translations from weblate
  • shared/units: Add GetByteSizeStringIEC
  • lxc/project: Use IEC units in info
  • api: clustering_description
  • shared/api: Add cluster member description
  • lxd: Expose cluster member description
  • lxc/cluster: Tweak cluster list, add description
  • Revert "test: Increase the offline thresholds to above 12 as heartbeat interval is hardcoded to 10"
  • api: Adds description back for clustering_join_token extension
  • lxd/images: Dont log error in autoSyncImagesTask when not clustered
  • lxd/images: Make logging consistent in autoSyncImagesTask
  • lxd/db/node: Display last heartbeat time in ToAPI
  • tests: Update for project info change
  • lxc: Add -f as shorthand for --format
  • lxd/devices: Allow user.XYZ
  • lxd/db/node: Updates SetNodeHeartbeat to return ErrNoSuchObject if row doesn't exist to be updated
  • lxd/db/query/retry: Use errors.Cause in Retry
  • lxd/cluster/heartbeat: Single call to time.Now() in heartbeat
  • lxd/cluster/heartbeat: Fixes bug in heartbeat that causes heartbeat round to be discarded if member removed during round
  • lxd/cluster/heartbeat: Keep error handling from g.currentRaftNodes together
  • lxd/cluster/heartbeat: Error logging consistency
  • lxd/cluster/heartbeat: Use contextual logging
  • lxd/cluster/events: Improve logging consistency in eventsUpdateListeners
  • lxd/task/group: Adds context arg to Start
  • lxd/task/start: Add context arg to Start
  • lxd/task: Start context usage
  • lxd/daemon: Updates Start functions usage by passing daemon context
  • lxd/images: Improve logging in imageSyncBetweenNodes
  • test: Add lxc cluster list before comparison in test_clustering_handover for visability into cluster state
  • test: Separate stop and publish commands in test_clustering_image_replication

Try it for yourself

This new LXD release is already available for you to try on our demo service.


The release tarballs can be found on our download page.

Binary builds are also available for:

  • Linux: snap install lxd
  • MacOS: brew install lxc
  • Windows: choco install lxc