News

Incus 6.2 has been released

31st of May 2024

Introduction

The Incus team is pleased to announce the release of Incus 6.2!

This release contains the second wave of changes contributed by students of the University of Texas at Austin and a few other features and improvements.

image|690x454

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features

New incus top command

A new incus top command was added. This builds on top of Incus' built-in OpenMetrics endpoint and allows for a refreshing view of the instance list, including CPU, memory and disk usage.

+---------------+-------------+-----------+-----------+
| INSTANCE NAME | CPU TIME(S) |  MEMORY   |   DISK    |
+---------------+-------------+-----------+-----------+
| foo           | 6.73        | 12.44MiB  | 341.88MiB |
+---------------+-------------+-----------+-----------+
| speedtest     | 32.79       | 23.84MiB  | 373.50MiB |
+---------------+-------------+-----------+-----------+
| v1            | 67130.91    | 254.54MiB | 1.25GiB   |
+---------------+-------------+-----------+-----------+
Press 'd' + ENTER to change delay
Press 's' + ENTER to change sorting method
Press CTRL-C to exit

Delay: 5s
Sorting Method: Alphabetical

This work was contributed by University of Texas at Austin students.

System load information in resources API

A new section was added to the resources API to expose server load information (1min, 5min, 10min) as well as total process count.

This is particularly useful for placement and auto-balancing logic as it allows for getting a good glimpse at how busy the various servers are solely from the Incus API.

stgraber@castiana:~$ incus info --resources
System:
  UUID: 05006c9c-7863-ee11-9e1b-224425600022
  Vendor: Framework
  Product: Laptop 13 (AMD Ryzen 7040Series)
  Family: Laptop
  Version: A5
  SKU: FRANDGCP05
  Serial: FRANDGCPA5340500AZ
  Type: physical
  Chassis:
      Vendor: Framework
      Type: Notebook
      Version: A5
      Serial: FRANDGCPA5340500AZ
  Motherboard:
      Vendor: Framework
      Product: FRANMDCP05
      Serial: FRANMDCPA534040120
      Version: A5
  Firmware:
      Vendor: INSYDE Corp.
      Version: 03.05
      Date: 03/29/2024

Load:
  Processes: 519
  Average: 0.80 0.77 0.71

[snip...]

This work was contributed by University of Texas at Austin students.

Ability to query access information for instances and projects

Two new APIs were added to allow querying the access list of a project or even a specific instance.

This integrates with our OpenFGA support and provided a sufficiently recent version of OpenFGA, will show you exactly who can access an instance and what role they have.

stgraber@castiana:~$ incus info --show-access foo
- identifier: stgraber@stgraber.org
  role: admin
  provider: openfga

stgraber@castiana:~$ incus project info --show-access default
- identifier: stgraber@stgraber.org
  role: admin
  provider: openfga

This work was contributed by University of Texas at Austin students.

Forceful deletion of projects

When dealing with a lot of busy projects, deleting them can become rather frustrating due to having to track down and delete everything they contain in the right order.

To address that, we now have incus project delete --force which will instruct Incus itself to delete everything in the correct order before deleting the project itself.

This is obviously an extremely dangerous thing to do. The command line tool will always ask for confirmation that you indeed want this project fully gone.

stgraber@castiana:~$ incus project delete demo
Error: Only empty projects can be removed.

stgraber@castiana:~$ incus project delete demo --force
Remove demo and everything it contains (instances, images, volumes, networks, ...) (yes/no): yes
Project demo deleted

New get_project scriptlet function

For those using our scriplet instance placement feature (instances.placement.scriptlet), a new function has now been added, get_project.

This allows retrieving all the details (api.Project) for a specific project and is particularly useful if you want project restrictions or limits to impact the placement decision.

Documentation: https://linuxcontainers.org/incus/docs/main/explanation/clustering/#instance-placement-scriptlet
This work was contributed by University of Texas at Austin students.

Querying objects across projects

Incus has long supported listing all instances regardless of projects.
Then recently this was extended to also cover storage volumes, images, profiles, network zones and operations.

With Incus 6.2, all remaining object collections now support this, adding:
- Storage buckets
- Networks
- Network ACLs

The CLI was updated to match, so all list commands interacting with objects that can be project-specific now also support --all-projects.

This work was contributed by University of Texas at Austin students.

PCI devices in incus info --resources

All PCI devices are now included in the incus info --resources output.
In the past, only those devices that were included in the GPU or disk sections were readily available.

This work was contributed by University of Texas at Austin students.

Improved alias handling in incus-simplestreams

The initial incus-simplestreams implementation would automatically generate our standard looking alias, basically DISTRIBUTION/RELEASE/VARIANT but that's not suitable for all environments and so you now have two new arguments to incus-simplestreams add:

  • --no-default-alias to disable the above alias
  • --alias to define a custom alias (can be passed multiple times)

Feeding YAML to create commands in the incus CLI

This work was started with Incus 6.1 and is now complete.

All create commands as well as incus init and incus launch now support reading an initial configuration as YAML from stdin.

This enables much easier scripting of complex deployments.

Customizable column lists in the CLI

Another piece of work which started with Incus 6.1 and is now complete.

All CLI commands that have a list function now support the --column/-c flag.

This work was contributed by University of Texas at Austin students.

More automatically generated documentation

Not something that should be generally noticeable to most users, but we've been slowly moving our documentation to be generated directly from comments in our code, limiting the risk of it getting outdated or out of sync.

With Incus 6.2, the following are now generated in that way:

  • Network zones
  • Image restrictions
  • Kernel limits
  • Devices
    • disk
    • unix-block
    • unix-char
    • unix-hotplug
    • usb

This work was contributed by University of Texas at Austin students.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • incusd/instance/lxc: Add gendoc comments for image restrictions
  • incusd/instance/qemu: Add gendoc comments for image restrictions
  • doc: Update configs
  • doc/image-handling: Use gendoc data
  • incus/storage_bucket: Add yaml support for key create
  • i18n: Update translation templates
  • incusd/instance/qemu: Fix StartedAt handling
  • incus/storage: Customizable columns in list
  • i18n: Update translation templates
  • incusd/network/ovn: Port LogicalRouterSNATAdd and LogicalRouterDNATSNATAdd to libovsdb
  • incusd/network: Update for OVN function changes
  • api: instances_scriptlet_get_project
  • doc/instances/scriptlet: Add get_project
  • incusd/scriptlet: Add get_project
  • api: networks_all_projects
  • shared/api: Add Project field to Network
  • client: Add GetNetworksAllProjects
  • incus/network: Add all-projects
  • incusd/db: Add GetNetworksAllProjects
  • incusd/networks: Add all-projects
  • internal/instance: Add gendoc for limits.kernel
  • doc: Update configs
  • doc: Use gendoc for limits.kernel
  • api: network_acls_all_projects
  • shared/api: Add Project field to NetworkACL
  • client: Add GetNetworkACLsAllProjects
  • incus/network/acl: Add all-projects
  • incusd/db: Add GetNetworkACLsAllProjects
  • incusd/network/acl: Set Project field
  • incusd/network_acls: Add all-projects
  • api: storage_buckets_all_projects
  • shared/api: Add Project field to StorageBucket
  • client: Add GetStoragePoolBucketsAllProjects
  • incus/storage/bucket: Add all-projects
  • incusd/db/storage_buckets: Fill Project field
  • incusd/storage_buckets: Add all-projects
  • i18n: Update translation templates
  • incusd/networks: Fix import shadowing
  • doc/rest-api: Refresh swagger YAML
  • client: Align GetProfilesAllProjects with GetProfiles
  • client: Align GetNetworkZonesAllProjects with GetNetworkZones
  • client: Standardize the GetNetworkAllocation functions
  • incus/network_allocations: Update for client changes
  • incusd/device/usb: Add gendoc for the USB device
  • doc: Update configs
  • doc: Use gendoc for USB devices
  • api: resources_load
  • shared/api: Add Load to resources API
  • doc/rest-api: Refresh swagger YAML
  • incusd/resources: Add load information
  • incus/info: Add load information
  • i18n: Update translation templates
  • incusd/device/unix: Add gendoc comments
  • doc: Update configs
  • doc/devices_unix_block.md: Use gendoc data
  • doc/devices_unix_char.md: Use gendoc data
  • doc/devices_unix_hotplug.md: Use gendoc data
  • incus/top: Add new command
  • i18n: Update translation templates
  • incusd/network/zone: add gendoc for network zone
  • doc: Update configs
  • doc: Use gen doc for network zones
  • incusd/device/unix: Run gofmt
  • incus/info: Add PCI devices to --resources
  • i18n: Update translation templates
  • incusd/device/disk: Add gendoc comments
  • doc: Update configs
  • doc/devices/disk: Use gendoc
  • incus/network: Customizable columns in list
  • i18n: Update translation templates
  • incusd/network_zones: Fix listing of zones within a project
  • incusd/instance/lxc: Fix handling of migration.stateful
  • gomod: Update dependencies
  • incusd/network/ovs: Fix bad VLANMode value
  • fix: close resources
  • incusd/instance: Fix building on 32bit architectures
  • incus/network_zone: Add example for network zone record create
  • i18n: Update translation template
  • incus/storage_volume: Add yaml support for create
  • i18n: Update translation templates
  • cmd/incus/info: Fix runtime error when chassis, motherboard and firwmare information is not available
  • Translated using Weblate (German)
  • incusd/instance/qemu: Allow setting protection.delete when running
  • doc/api-extension: Fix typo
  • shared/api: Introduce Access structs
  • api: instance_access
  • incusd/auth: Introduce GetInstanceAccess
  • incusd/instance: Add access endpoint
  • api: project_access
  • incusd/auth: Introduce GetProjectAccess
  • incusd/project: Add access endpoint
  • doc/rest-api: Refresh swagger YAML
  • client: Add GetInstanceAccess
  • client: Add GetProjectAccess
  • incus/info: Fix description of --show-log
  • incus/info: Add --show-access
  • incus/project: Add --show-access to info
  • i18n: Update translation templates
  • incusd/auth/fga: Rename manager by admin in model
  • incusd/auth/fga: Rework permission model
  • incusd/auth/fga: Rebuild model
  • tests: Fix for permission changes
  • incusd/instance/agent-loader: Support installing to /etc
  • incusd/apparmor/lxc: Fix rule syntax
  • incus-simplestreams add: added flags: --no-default-alias, --alias. #875
  • incus/storage_volume/snapshot: Support YAML for creation
  • i18n: Update translation templates
  • shared/idmap: Make get_userns_fd configure the userns
  • incus-migrate: Handle valid CA certificates
  • incusd/instances_post: Fix migrating into remote cluster
  • incusd/apparmor: Detect nosymfollow support
  • incusd: Set SELinux label on socket
  • incus/network: Align attach-profile with attach
  • create_detached_idmapped_mount: avoid double close
  • incusd/instance/qemu: Extend missing QEMU error
  • doc/installing: Mention extra packages for VMs
  • incusd/storage/btrfs: Fix btrfs argument order
  • incusd/seccomp/sysinfo: Handle 32bit on 64bit
  • api: projects_force_delete
  • incusd/api_project: Add force delete endpoint
  • doc/rest-api: Refresh swagger YAML
  • client: Introduce DeleteProjectForce
  • cmd/incus/project: Add --force to delete
  • i18n: Update translation templates
  • incusd/project: Simplify projectIsEmpty
  • incusd/db: Introduce GetNetworkZoneURIs
  • incusd/db: Introduce GetStorageBucketURIs
  • incusd/api_project: Fix UsedBy
  • incusd/api_project: Add force deletion logic
  • incus/completion: Reduce API calls
  • incus/publish: Complete snapshot names
  • incus/completion: Fix import shadowing
  • Translated using Weblate (French)
  • Makefile: Pin go-acme/lego for Go 1.21
  • Update dependencies
  • cmd/incus/console: Cleanup --show-log
  • incusd/instance_console: Remove old check
  • incusd/instance_console: Handle missing log file
  • incusd/instance_console: Don't fail on empty logs
  • incusd/instance_console: Cleanup error message
  • i18n: Update translation templates
  • incusd/device/sriov: Line up code with comment

Documentation

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.2.0

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support

Monthly feature releases are only supported up until the next release comes out. Users needing a longer support length and less frequent changes should consider using Incus 6.0 LTS instead.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 6.1 has been released

7th of May 2024

Introduction

The Incus team is pleased to announce the release of Incus 6.1!

This is our first feature release following Incus 6.0 LTS.
As a reminder, feature releases are only supported until the next one comes out, usually on a monthly cadence. Critical production environments should stay on the LTS release instead.

In this release, we have a lot of small quality of life improvements throughout. A lot of those being first contributions from students of the University of Texas at Austin. Expect a lot more of those in Incus 6.2!

image|690x459

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features

Creation of complex ZFS pools

The source key when creating storage pools using our zfs driver has now been extended to allow the creation of more complex vdevs including striping, mirroring, raidz1 and raidz2.

Example syntax:

  • /dev/sda,/dev/sdb (striping, RAID0)
  • mirror=/dev/sda,/dev/sdb (mirroring, RAID1)
  • raidz1=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde (raidz1, RAID5)
  • raidz2=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde (raidz2, RAID6)

This combined with the data from incus info --resources will now make it possible to deploy complex storage pools all through the API.

Listing of profiles across projects

As part of an effort to add cross-project querying of all API objects, it is now possible to list profiles across all projects.

At the API level, this is support for all-projects=true on the /1.0/profiles API endpoint, at the CLI level this looks like:

stgraber@dakara:~$ incus profile list --all-projects
+-----------------+---------+---------------------------------------+---------+
|     PROJECT     |  NAME   |              DESCRIPTION              | USED BY |
+-----------------+---------+---------------------------------------+---------+
| default         | default | Default Incus profile                 | 10      |
+-----------------+---------+---------------------------------------+---------+
| demo            | default | Default Incus profile                 | 12      |
+-----------------+---------+---------------------------------------+---------+
| lab-cgroup      | default | Default Incus profile                 | 2       |
+-----------------+---------+---------------------------------------+---------+
| lab-lvm-cluster | default | Default Incus profile                 | 3       |
+-----------------+---------+---------------------------------------+---------+
| lab-ovn-ic      | default | Default Incus profile                 | 10      |
+-----------------+---------+---------------------------------------+---------+
| vpn             | default | Default Incus profile for project vpn | 2       |
+-----------------+---------+---------------------------------------+---------+

This feature was contributed by University of Texas at Austin students.

Listing of network zones across projects

As part of an effort to add cross-project querying of all API objects, it is now possible to list network zones across all projects.

At the API level, this is support for all-projects=true on the /1.0/network-zones API endpoint, at the CLI level this looks like:

stgraber@dakara:~$ incus network zone list --all-projects
+---------+--------------------------+-------------+---------+
| PROJECT |           NAME           | DESCRIPTION | USED BY |
+---------+--------------------------+-------------+---------+
| default | default.demo.example.net |             | 0       |
+---------+--------------------------+-------------+---------+
| foo     | foo.demo.example.net     |             | 0       |
+---------+--------------------------+-------------+---------+

This feature was contributed by University of Texas at Austin students.

Additional functions made available to the instance placement scriptlet

Incus supports customizing instance placement through the use of a python-like script called a scriptlet. When used, the scriptlet is exposed some information about the instance, potential targets and the reason for the request.

On top of those arguments, a number of functions are also exposed to those scriptlets.
That includes the ability to log information, the function to actually make the final placement decision and the ability to fetch some basic load information about the candidate servers.

Now this is being extended through two additional functions:

  • get_instances(location, project) => []api.Instance
  • get_cluster_members(group) => []api.ClusterMember

Those are all optional arguments, so they can be used to list all instances or all cluster members as well, allowing a lot of flexibility in placement scripts.

Documentation: https://linuxcontainers.org/incus/docs/main/explanation/clustering/#instance-placement-scriptlet
This feature was contributed by University of Texas at Austin students.

Feeding YAML to create commands in the incus CLI

A number of Incus commands already support reading a YAML file through their standard input as part of a create command, but this isn't very consistent nor well documented, we're now in the process of making things consistent and this release now has support for reading a YAML object definition in the following commands:

  • incus create & incus launch
  • incus cluster group create
  • incus network acl create
  • incus network forward create
  • incus network integration create
  • incus network load-balance create
  • incus network peer create
  • incus network zone create
  • incus profile create
  • incus project create
  • incus snapshot create
  • incus storage create
  • incus storage bucket create

For all of those, YAML data similar to what's showed in the matching show command can be fed through stdin at creation time to configure the object as part of its creation.

Customizable columns in the incus CLI

Something else we're slowly making consistent in the CLI is the ability to choose what columns to display in our list commands.

This has also been expanded with the following commands now supporting it:

  • incus cluster list
  • incus config trust list
  • incus image list
  • incus list
  • incus profile list
  • incus project list
  • incus storage volume list
  • incus storage volume snapshot list
  • incus warning list

This feature was contributed by University of Texas at Austin students.

migration.stateful configuration key for containers

The migration.stateful configuration key has been expanded to also apply to containers now.

It is now required to have it set to true to access any feature requiring the recording and restoration of process state in containers (CRIU), which basically means stateful stop, stateful snapshots and live migration.

This change is unlikely to affect many users as CRIU's ability to live-migrate or perform stateful dumps of Incus containers is extremely limited and so generally seen as not functional.
The change does have the benefit of providing clearer errors to users who accidentally request an action which would make use of CRIU.

This feature was contributed by University of Texas at Austin students.

Stateless ACLs on OVN

A new allow-stateless action has now been added to Incus' network ACL rules.

As the name implies, this leads to the creation of a stateless rule inside of OVN.
This is great for situations where stateful rules may come with a heavy cost and where a matching stateless rule is possible (e.g. DNS interactions).

This feature was contributed by University of Texas at Austin students.

Instance uptime (startup time) tracking

A new StartedAt field has been added to the instance state data.
This exposes the timestamp at which the instance was started and is also available in incus info and incus list.

stgraber@dakara:~$ incus info speedtest | grep Started
Started: 2024/04/29 11:03 EDT

stgraber@dakara:~$ incus list -cnstU
+-------------+---------+-----------------+----------------------+
|    NAME     |  STATE  |      TYPE       |      STARTED AT      |
+-------------+---------+-----------------+----------------------+
| centos3     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| centos4     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| fga         | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+
| incus-ui    | RUNNING | CONTAINER       | 2024/05/07 16:54 EDT |
+-------------+---------+-----------------+----------------------+
| kernel-test | RUNNING | VIRTUAL-MACHINE | 2024/05/07 15:43 EDT |
+-------------+---------+-----------------+----------------------+
| keybase     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| ovn-test    | RUNNING | VIRTUAL-MACHINE | 2024/05/07 15:43 EDT |
+-------------+---------+-----------------+----------------------+
| speedtest   | RUNNING | CONTAINER       | 2024/04/29 11:03 EDT |
+-------------+---------+-----------------+----------------------+
| void        | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+
| win11       | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+

This feature was contributed by University of Texas at Austin students.

Improvement to network handling during evacuation

When performing a cluster evacuation, all the networks will now be shut down at the end of the evacuation and only started back as part of the restoration action.

This is particularly useful in OVN environments as it ensures that an evacuated Incus server doesn't act as a virtual router for any of the defined networks, making system shutdown/reboot less likely to cause network glitches.

This feature was contributed by University of Texas at Austin students.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • doc/support: Update for LTS
  • incusd/network: Remove bridge.driver=native requirement for extended external_interfaces syntax
  • doc/network/bridge: Update extended external_interfaces documentation
  • incusd/storage/drivers/zfs: Simplify dataset receive and fix progress handling
  • gomod: Update dependencies
  • mini-oidc: Merge into main gomod
  • gomod: Update dependencies
  • go.mod: Bump package major version
  • global: Update Go package to v6
  • test: Update godeps.list
  • README: Update for godoc URLs
  • doc/rest-api: Refresh swagger YAML
  • cmd/incus: Fix import ordering
  • incusd: Remove unneeded import renames
  • incusd/instance: Fix duplicate import
  • doc/projects: Tweak examples
  • shared/api: Remove container resources (deprecated)
  • doc/rest-api: Refresh swagger YAML
  • incus/create: Remove dead code
  • i18n: Update translation templates
  • incusd/daemon: Remove old migration logic
  • incusd: Stop mentioning containers in resources
  • doc/rest-api: Use instances API in example
  • incusd/db/cluster: containers URLs aren't valid in Incus
  • incusd/instances: Don't start instances when evacuated
  • doc/installing: Sort source-build distro instructions
  • doc/installing: Add OpenSUSE source instructions
  • doc/installing: Add Alpine instructions
  • incus/aliases: fix completion regression
  • incus/snapshot: Fix deletion of snapshots containing colons
  • incusd/instance/drivers: Have SR-IOV get stable MACs
  • incusd/device/nic_sriov: Use stable MAC
  • incus/profile: Add support for creating from YAML
  • i18n: Update translation templates
  • incusd/instance/lxc: Only apply soft cgroup limits on cgroup1
  • incus/admin: Don't hide the sql command
  • grafana: Refresh dashboard
  • doc/metrics: Mention Loki in Grafana setup
  • grafana: Better filter Loki events by project
  • incusd/loki: Use hostname as default instance property on standalone systems
  • incusd/loki: Re-order config fields
  • incusd/loki: Allow overriding the location field
  • incusd/loki: Set location field to local hostname on standalone systems
  • incus/projct: Add support for creating project from yaml
  • i18n: Update translation templates
  • incusd/network/ovs: Port GetOVNEncapIP to libovsdb
  • incusd/network/ovs: Add some comments to GetOVNBridgeMappings
  • incusd/network/ovs: Port AddOVNBridgeMapping to libovsdb
  • incusd/network/ovs: Port RemoveOVNBridgeMapping to libovsdb
  • incusd/network/ovs: Port GetHardwareOffload to libovsdb
  • incusd/network/ovs: Port GetBridgePorts to libovsdb
  • incusd/network/ovs: Port UpdateBridgePortVLANs to libovsdb
  • incusd/network/ovs: Port AssociateInterfaceOVNSwitchPort to libovsdb
  • incusd/network/ovs: Switch Installed to checking for unix socket
  • incusd/network: Update for OVS function changes
  • Add missing SecureBoot firmware names
  • incus/snapshot: Add support for creating project from yaml
  • i18n: Update translation templates
  • api: network_zones_all_projects
  • shared/api: Add Project field to NetworkZone
  • incusd/network_zones: Add support for all-projects
  • doc/rest-api: Refresh swagger YAML
  • client: Add GetNetworkZonesAllProjects
  • incus/network_zone: Add --all-projects flag to list
  • i18n: Update translation templates
  • tests: Add all-projects network zone test
  • incusd/network/ovn: Port LogicalRouterAdd to libovsdb
  • incusd/network/ovn: Remove LogicalRouterAdd
  • incusd/network: Replace LogicalRouterAdd usage with CreateLogicalRouter
  • incusd/network/ovn: gofmt
  • incusd/cluster: Disable networks during evacuation
  • incusd/cgroup: Set hugepages reserved limits
  • incusd/storage/drivers: Introduce SparseFileWrapper
  • incusd/storage/drivers/vfs: Use SparseFileWrapper on backup import
  • incusd/storage/drivers/vfs: Use SparseFileWrapper on volume migration
  • incus/storage: Support creating storage pool from yaml
  • i18n: Update translation templates
  • incus/info: Show CPU architecture as separate line
  • i18n: Update translation templates
  • incus/cluster_group: Add yaml support for cluster group create
  • i18n: Update translation templates
  • gitignore: Add JetBrains
  • api: storage_zfs_vdev
  • doc: Update ZFS support for multiple block devices and vdev types in doc/reference/storage_zfs.md
  • incusd/storage/zfs: Add support for vdev type and multiple block devices
  • incus/info: Sorting network interfaces
  • incus/network_acl: Add cmd.Example for network acl create
  • i18n: Update translation templates
  • incus/network_forward: Add yaml example for create
  • i18n: Update translation templates
  • incus/config_template: Add file support for create
  • i18n: Update translation templates
  • incus/network_integrations: Add yaml support for create
  • i18n: Update translation templates
  • incusd/storage/s3: Use 'mc' client
  • incusd/storage: Switch to use minio's 'mc' client
  • gomod: Update dependencies
  • github: Download MinIO client
  • doc: Avoid MyST 3.0.0
  • incus/doc/installing.md: Add Docker information
  • doc: Add Podman to wordlist
  • incus/cluster: Add columns to list
  • i18n: Update translation templates
  • incus/project: Add customazible columns to list
  • i18n: Update translation templates
  • api: container_migration_stateful
  • internal/instance: Add migration.stateful to containers
  • incusd/instance/lxc: Add checks for migration.stateful
  • doc: Update configs
  • incus/network_load_balancer: add Example to create
  • i18n: Update translation templates
  • incus/network_zone: Add example for create command
  • i18n: Update translation templates
  • doc: Fix bad sphinx requirements
  • incusd/instances/qemu: Tweak secureboot firmware list
  • cmd/incus/admin_cluster: Add libexec path for incusd
  • incus/storage: Show usage when no driver passed
  • incusd/storage/drivers/dir: Tweak path validation
  • incusd/backup: Show profile list on lookup error
  • incusd/apparmor/lxc: Allow access to binfmt_misc
  • incusd/apparmor/lxc: Refresh generated rules
  • incusd/storage: Handle instance volume size on import
  • incus/profile: Add customizable columns to list
  • i18n: Update translation templates
  • incus/project: Fix help message for list
  • i18n: Update translation templates
  • api: profiles_all_projects
  • shared/api: Add Project field to Profile
  • client: Add GetProfilesAllProjects
  • incus/profile: Add all-projects to list
  • incusd/db/cluster: Add Project field to Profile
  • incusd/profile: Add all-projects support
  • doc/rest-api: Refresh swagger YAML
  • i18n: Update translation templates
  • incus/storage_volume: Clarify volume errors
  • incusd/apparmor/lxc: Fix access to kernel/security/apparmor
  • api: instances_scriptlet_get_instances
  • doc/instances/scriptlet: Add get_instances
  • incusd/scriptlet: Add get_instances
  • api: instances_scriptlet_get_cluster_members
  • doc/instances/scriptlet: Add get_cluster_members
  • incusd/scriptlet: Add get_cluster_members
  • api: Add network_acl_stateless
  • doc/network_acl: Add allow-stateless action
  • incusd/network/acl: Add allow-stateless action
  • incusd/network/ovn/nb: Port DeleteLogicalRouter to libovsdb
  • incusd/network/ovn/nb: Port CreateLogicalRouterSNAT to libovsdb
  • incusd/network: Update for OVS function changes
  • incusd/network/acl: Properly run instance counting outside of ACL loop
  • incusd/network/ovn: Wait up to 1s for dynamic IPs
  • incusd/network/ovn/nb: Port DeleteLogicalRouterNAT to libovsdb
  • incusd/network: Update for OVS function changes
  • shared/archive: Fix typo
  • incusd/cluster: Re-organize joining logic
  • incusd/cluster: Ignore OVN networks during joining
  • shared/archive: Properly anchor exclude rules
  • incusd/project: Rework low-level permission checks
  • incus/storage_bucket: Add example for storage bucket create
  • i18n: Update translation templates
  • incus/network_peer: Add example for create command
  • i18n: Update trasnlation templates
  • api: instance_state_started_at
  • shared/api: Add StartedAt to InstanceState
  • doc/rest-api: Refresh swagger YAML
  • incusd/instance: Add StartedAt to InstanceState
  • incus/info: Add Started field
  • incus/list: Add started at column
  • i18n: Update translation templates
  • Makefile: Cleanup gomod update
  • gomod: Update dependencies
  • tests/mini-oidc: Bump to go-jose/v4
  • client/connection: Add support for the socket existing in /run/incus

Documentation

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.1.0

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 6.0 LTS has been released

4th of April 2024

Introduction

It's with great pride and pleasure that the Incus team is announcing the release of Incus 6.0 LTS!

Incus is a modern system container and virtual machine manager developed and maintained by the same team that first created LXD. It's released under the Apache 2.0 license and is run as a community led Open Source project as part of the Linux Containers organization.

Incus provides a cloud-like environment, creating instances from premade images and offers a wide variety of features, including the ability to seamlessly cluster up to 50 servers together.

It supports multiple different local or remote storage options, traditional or fully distributed networking and offers most common cloud features, including a full REST API and integrations with common tooling like Ansible, Terraform/OpenTofu and more!

image|690x489

This is a major milestone for Incus as it marks our first release with extended support, suitable for use in production environments where monthly feature releases aren't suitable.

It joins LXC 6.0 LTS and LXCFS 6.0 LTS in wrapping up this round of LTS releases.

Just like its sister projects, Incus 6.0 LTS will be supported until June 2029.
The first 2 years will feature bug and security fixes as well as minor usability improvements, delivered through occasional point releases (6.0.x). After that initial two years, Incus 6.0 LTS will move to security only maintenance for the remaining of its 5 years of support.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

PS: Incus was made possible thanks to the work of over 70 individual contributors!

Changes since Incus 0.7

Swap limits for containers

The existing limits.memory.swap configuration key for containers has been extended to also allow for byte amounts.

This now makes its behavior be as follows:

  • limits.memory.swap=true => Container memory may be swapped (default)
  • limits.memory.swap=false => Container shouldn't get swapped (minimal swappiness)
  • limits.memory.swap=256MiB => Container can use up to 256MiB of swap space (in addition to its memory limit set through limits.memory)

Example (cgroup2 system):

stgraber@dakara:~$ incus launch images:debian/12 d12 -c limits.memory=1GiB
Launching d12
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m
               total        used        free      shared  buff/cache   available           
Mem:            1024          21         983           0          19        1002
Swap:              0           0           0
root@d12:~#
exit
stgraber@dakara:~$ incus config set d12 limits.memory.swap=128MiB
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m      
               total        used        free      shared  buff/cache   available
Mem:            1024          21         983           0          19        1002
Swap:            128           0         128
root@d12:~#
exit

New shell completion mechanism

With this release, we complete the migration away from a hand-maintained bash completion script and over to generate completion scripts directly in our command line tool.

Completion profiles are now available for:

  • bash
  • fish
  • powershell
  • zsh

The profile can be retrieved by calling incus completion <shell> (e.g. incus completion bash) though this will generally be done by packagers as part of the Incus package build process.

Creation of external bridge interfaces

The managed network bridge configuration syntax for external interfaces, bridge.external_interfaces has now been extended to allow for the creation and attachment of VLAN interfaces.

stgraber@dakara:~$ incus network set incusbr0 bridge.external_interfaces=vlan60/enp35s0/60
stgraber@dakara:~$ ip link show dev vlan60
269: vlan60@enp35s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master incusbr0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 00:23:a4:01:01:6f brd ff:ff:ff:ff:ff:ff
stgraber@dakara:~$ incus network unset incusbr0 bridge.external_interfaces
stgraber@dakara:~$ ip link show dev vlan60
Device "vlan60" does not exist.
stgraber@dakara:~$

Live-migration of VMs with attached disks (from remote storage)

As an extension to our ever improving VM live-migration support, virtual-machines with additional disks attached to them which come from a "remote" storage pool (ceph or lvmcluster) will now be live-migratable alongside the virtual machine.

No user action is required for this to happen, you'll simply now notice that virtual machines that previously would have refused to live-migrate through either a manual incus move --target or a cluster evacuation will now happily live-migrate to another server.

System information in incus info --resources

A new System section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
System:
  UUID: 88eecd60-34fc-9f97-48f5-fc34979f48f6
  Vendor: ASUS
  Product: System Product Name
  Family: To be filled by O.E.M.
  Version: System Version
  SKU: SKU
  Serial: System Serial Number
  Type: physical
  Chassis:
      Vendor: Default string
      Type: Desktop
      Version: Default string
      Serial: Default string
  Motherboard:
      Vendor: ASUSTeK COMPUTER INC.
      Product: ProArt B550-CREATOR
      Serial: 210382121300122
      Version: Rev X.0x
  Firmware:
      Vendor: American Megatrends Inc.
      Version: 2803
      Date: 04/28/2022

 [snip...]

Having access to this information is particularly useful in clustered environments where incus info --resources can be used with the --target argument to query specific servers, check that all firmwares are up to date and check what machines one is dealing with.

This feature was contributed by University of Texas at Austin students.

USB devices in incus info --resources

A new USB devices section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
[snip...]

USB devices:
  Device 0:
    Vendor: Intel Corp.
    Vendor ID: 8087
    Product: AX200 Bluetooth
    Product ID: 0029
    Bus Address: 1
    Device Address: 6
  Device 1:
    Vendor: Corsair
    Vendor ID: 1b1c
    Product: H150iRGBPROXT
    Product ID: 0c22
    Bus Address: 1
    Device Address: 5
  Device 2:
    Vendor: ASUSTek Computer, Inc.
    Vendor ID: 0b05
    Product: AURA LED Controller
    Product ID: 19af
    Bus Address: 1
    Device Address: 2
  Device 3:
    Vendor: Realtek Semiconductor Corp.
    Vendor ID: 0bda
    Product: TX42C500
    Product ID: 4933
    Bus Address: 5
    Device Address: 2
  Device 4:
    Vendor: Blue Microphones
    Vendor ID: b58e
    Product: Yeti Stereo Microphone
    Product ID: 9e84
    Bus Address: 5
    Device Address: 15
  Device 5:
    Vendor: Yubico.com
    Vendor ID: 1050
    Product: YubiKey FIDO+CCID
    Product ID: 0406
    Bus Address: 5
    Device Address: 29
  Device 6:
    Vendor: Logitech, Inc.
    Vendor ID: 046d
    Product: HD Pro Webcam C920
    Product ID: 082d
    Bus Address: 5
    Device Address: 17
  Device 7:
    Vendor: Powerware Corp.
    Vendor ID: 0592
    Product: Powerware UPS
    Product ID: 0002
    Bus Address: 7
    Device Address: 2

That information comes in very handy when adding a USB device to a container or virtual machine.

This feature was contributed by University of Texas at Austin students.

Changes since LXD 5.0 LTS

For those coming from the LXD 5.0 LTS release, here is a concise list of what to expect as far as features having been removed and what has been added both in subsequent LXD feature releases and then through Incus.

Feature removal

A number of features that were Ubuntu or Canonical specific were removed as part of the creation of the Incus project. A number of legacy APIs have also been removed at the same time.
You'll find the full list in the Incus 0.1 announcement.

Highlights:

  • shiftfs has been removed in favor of VFS idmap shifting
  • Canonical Candid authentication has been removed in favor of OpenID Connect
  • Canonical RBAC authorization has been removed in favor of OpenFGA
  • Canonical MAAS network integration has been removed (under/unused feature)
  • Ubuntu Fan networking has been removed in favor of OVN
  • core.trust_password has been removed in favor of trust tokens for security reasons

Feature additions

Here are a few highlights from the many new features introduced within the 2 years since the release of LXD 5.0 LTS.

Complete changelog

Here is a complete list of all changes since Incus 0.7:

Full commit list
  • Translated using Weblate (Japanese)
  • Translated using Weblate (Japanese)
  • incus/image: Fix column handling with --all-projects
  • Replace util.ValueInSlice with slices.Contains
  • shared/util: Delete ValueInSlice function
  • incus/image: Fix column handling with --all-projects
  • incusd/instance/qemu: Relocate image requirement checks
  • doc/images: Add requirements.cdrom_agent
  • incusd/instance/qemu: Add support for requirements.cdrom_agent
  • incusd/device/disk: Fix incorrect block volume usage
  • Translated using Weblate (Japanese)
  • incusd/network/ovn: Use ParseIPToNet instead of manual IPToNet and net.ParseIP
  • incusd/network/ovn: Use listenAddressNet in family check
  • incusd/instance/drivers: Disable architecture check on incus cp with snapshots
  • Translated using Weblate (French)
  • incusd/network/bridge: Set local address on all VXLAN tunnels
  • incus/instance/qemu: Fix RecordOutput
  • incus: add completions for instance actions and snapshots
  • incus: add completions for profiles
  • incusd/network/ovn: Introduce get helper
  • incusd/network/ovn: Add some missing indices
  • incusd/network/ovn: Use get helper
  • incusd/network/ovn: Fix LogicalSwitchPortIPs logic
  • incusd/network/bridge: Fix gofmt
  • incusd/network/ovn: Fix gofmt
  • cmd/incus: Use proper timestamp check
  • cmd/incus: Use consistent date format and timezone
  • client: Rename network_peer for consistency
  • cmd/incusd: Rename network_peer to network_peers
  • shared/api: Rename network_allocation for consistency
  • incusd/db: Fix comment typoes
  • incusd/db/generate: Fix bad camel case handling
  • incusd/db/network_peers: Fix duplicate type definitions
  • incusd/auth: Drop Permission type
  • incusd/auth: Add boilerplate doc strings
  • incusd/images: Properly handle null creation and expiry dates
  • incus: add completions for remotes
  • incus: add completions for projects
  • incusd/images: Fix reporting of images in multiple projects
  • github: Add static build of lxd-to-incus
  • lxd-to-incus: Add support for Alpine service name
  • lxd-to-incus: Re-organize target list
  • lxd-to-incus: Add support for APK
  • Makefile: Add OVN IC to update-ovsdb
  • incusd/network: Update OVS/OVN schemas
  • incusd/network/ovn: Add IC clients
  • incusd/network/ovn: Add GetName to NB client
  • incusd/network/ovn: Add GetGateways to ICSB
  • incusd/network/ovn: Introduce new errors
  • incusd/network/ovn: Add CreateTransitSwitch and DeleteTransitSwitch to ICNB
  • incusd/device/gpu_sriov: Add locking
  • incusd/device/gpu_sriov: Re-locate vfio-pci loading
  • incusd/device/gpu_sriov: Rework VF allocation logic
  • incus/remote: Add a generate-certificate sub-command
  • i18n: Update translation templates
  • incusd/drivers/qmp: Add SetBlockThrottle
  • incusd/device/disk/config: Add DiskLimits
  • incusd/device/disk: Re-shuffle limit parsing
  • incusd/device/disk: Add disk limits on VMs
  • incusd/device/disk: Support live limits update for VMs
  • incusd/instance/qemu: Support disk I/O limits
  • incus/remote: Add missing docstrings
  • incusd/certificates: Improve token handling when clustered
  • cmd/incusd/api_1.0: Update context
  • cmd/incusd/api_cluster: Update context
  • cmd/incusd/api_internal: Update context
  • cmd/incusd/daemon: Update context
  • cmd/incusd/api_project: Update context
  • cmd/incusd/certificates: Update context
  • cmd/incusd/images: Update context
  • cmd/incusd/instance: Update context
  • cmd/incusd/network: Update context
  • cmd/incusd/operations: Update context
  • cmd/incusd/profiles: Update context
  • cmd/incusd/storage: Update context
  • cmd/incusd/warnings: Update context
  • incusd/devices: Skip isolated threads from NUMA CPUs
  • incusd/devices: Restrict CPU threads by NUMA node
  • incusd/instance/qemu: Add support for limits.cpu.nodes
  • incusd/device/gpu: Add support for limits.cpu.nodes for VF selection
  • incusd: Fix import shadowing
  • incusd/images: Fix potential race condition
  • incusd/instance/qemu: Add support for NUMA node restrictions for memory
  • incusd/apparmor/qemu: Silence apparmor failures
  • incusd/network/ovs: Introduce new errors
  • incusd/network/ovn/nb: Move SetChassisGroupPriority to new function signature
  • incusd/network/ovn/sb: Move GetLogicalRouterPortActiveChassisHostname to new function signature
  • incusd/network/ovs: Move GetBridge to new function signature
  • incusd/network/ovs: Move CreateBridge to new function signature
  • incusd/network/ovs: Move DeleteBridge to new function signature
  • incusd/network/ovs: Move CreateBridgePort to new function signature
  • incusd/network/ovs: Move GetChassisID to new function signature
  • incusd/network/ovs: Move GetOVNBridgeMappings to new function signature
  • incusd/network: Update for function changes
  • incusd/device/nic: Update for function changes
  • incusd: Update for function changes
  • doc: Fix bad snapshot syntax
  • Translated using Weblate (French)
  • doc: Fix token creation procedure
  • incusd/network/ovn/nb: Add GetLogicalSwitch
  • incusd/network/ovn/nb: Replace ChassisGroupChassisDelete with SetChassisGroupPriority
  • incusd/network/ovn/nb: Port CreateLogicalRouterPort to OVSDB
  • incusd/network/ovn/nb: Replace LogicalRouterPortLinkChassisGroup with CreateLogicalRouterPort
  • incusd/network/ovn/nb: Port CreateChassisGroup to OVSDB
  • incusd/network/ovn/nb: Port CreateLogicalSwitch to OVSDB
  • incusd/network/ovn: Update for function changes
  • incusd/network/ovn: Remove state references
  • incusd/state: Add OVNNB and OVNSB handles
  • incusd: Update to use state for OVN
  • incusd/device: Make init function return error
  • incusd/device: Add OVN check on nicOVN
  • client: Still return response on RawQuery error
  • incus/query: Respect --raw for errors
  • incusd/network/acl: Add OVN check
  • incusd/network: Make init function return error
  • incusd/network: Add OVN check on ovn driver
  • incusd/api: Re-order config checks
  • incusd: Add OVN loader
  • Translated using Weblate (French)
  • incusd/network/ovn/nb: Port CreateLogicalSwitchPort to OVSDB
  • incusd/network/ovn/nb: Port DeleteLogicalSwitchPort to OVSDB
  • incusd/network/ovn/nb: Port DeleteLogicalRouterPort to OVSDB
  • incusd/network/ovn: Update for function changes
  • incusd/network/ovs: Port GetOVNSouthboundDBRemoteAddress to OVSDB
  • incusd/network/ovs: Port DeleteBridgePort to OVSDB
  • incusd/network/ovs: Port GetInterfaceAssociatedOVNSwitchPort to OVSDB
  • incusd/network/ovs: Align GetChassisID with other functions
  • incusd: Update for OVS function changes
  • incusd/network/ovn/icsb: Fix bad DB schema
  • incusd/network/ovn/nb: Introduce GetLogicalRouterPort
  • incusd/network/ovn/nb: Extend OVNSwitchPortOpts to handle router ports
  • incusd/network/ovn/nb: Change type of RouterPort field to OVNRouterPort
  • incusd/network/ovn/nb: Port DeleteChassisGroup to OVSDB
  • incusd/network/ovn/icnb: Update DeleteTransitSwitch to handle missing switches
  • incusd/network/ovn: Update for function changes
  • Translated using Weblate (French)
  • incus/completion: do not add a space after remote names completion
  • incusd/device/disk: Disable virtiofsd caching
  • incus-agent: Cleanup mount logic
  • Translated using Weblate (French)
  • incus: expose parseVolume to entire package
  • incus: add completions for storage pools and volumes
  • incusd/device/gpu_sriov: Fix default handling
  • doc/packaging: Add mention of documentation
  • incusd/auth: Fix --all-projects for restricted users
  • doc: Add third party tools page
  • gomod: Update dependencies
  • incusd/auth/tls: Prevent project modifications
  • doc: Update wordlist
  • internal/usbid: allow path override of usb.ids path
  • incus/completion: fix image names completion
  • doc/environment: document INCUS_USBIDS_PATH
  • incusd/instance/qemu/agent: Check for semanage
  • incusd/project: Fix config name in ImageProjectFromRecord
  • incus/restart: Fix long description
  • i18n: Update translations
  • lxd-to-incus: Handle common existing bridges
  • shared/simplestreams: Remove defaultOS
  • shared/simplestreams: Add NewLocalClient
  • incus-simplestreams: Introduce new command
  • incus-simplestreams: Simplify delete logic
  • doc: Re-organize image server doc
  • doc: Add section for incus-simplestreams
  • incusd/seccomp: Add support for pidfd threads
  • incus: add completions for clusters
  • incus: add completions for cluster groups
  • incus: add completions for cluster roles
  • incus: add completions for config devices
  • incus: add completions for config templates
  • update translations
  • doc: Update references to mage docs
  • doc/backup: Remove bad reference
  • incus: add completions for network acls
  • shared/api: Add new structs to support configuration metadata
  • client: Add GetMetadataConfiguration
  • incusd: Rename documentation.go -> metadata.go
  • doc/rest-api: Refresh swagger YAML
  • shared/api/metadata: Add GetKeys to simplify usage
  • incusd: Add support for JWT authentication
  • gomod: Update dependencies
  • tests: Add tls2jwt tool
  • tests: Add JWT authentication test
  • api: auth_tls_jwt
  • doc/authentication: Add section on JWT
  • doc/instances: Remove size.state requirement for live migration
  • incusd/instance/qemu: Allow live migration without size.state
  • shared/idmap: Support uid/gid in subuid/subgid
  • shared/cliconfig: Copy clientcerts on remote copy
  • shared/cliconfig: Add HasRemoteClientCertificate
  • shared/cliconfig: Support per-remote client certificates
  • doc: Add clientcerts
  • incusd/cluster/config: Add oidc.claim
  • incusd/auth/oidc: Add support for using a specific claim as username
  • incusd: Pass OIDC claim to verifier
  • api: oidc_claim
  • doc: Update configs
  • doc/howto/instances: Mention extra resources in ISO guidea
  • doc/installing: Add Debian backport
  • doc: Add backported to dictionary
  • lxd-to-incus: Add support for LXD 5.21
  • shared/cliconfig: Ensure client certificate key is 0600
  • api: device_usb_serial
  • doc: Add busnum, devnum and serial to USB devices
  • shared/api: Add Serial to ResourcesUSBDevice
  • incusd/resources: Add USB Serial
  • incusd/devices/usb: Add serial, busnum and devnum options
  • doc/rest-api: Refresh swagger YAML
  • incusd/instance/qemu: Fix handling of > 64 limits.cpu
  • incusd/device/gpu_sriov: Implement NUMA fallback
  • incus: add completions for network forwards
  • incus: add completions for network load balancers
  • shared/validate: Remove stringInSlice
  • shared/validate: Add And and Or functions
  • shared/util: Move ParseUint32Range
  • incusd/project: Update for ParseUint32Range
  • doc/instance_options: Remove mention of limits.cpu.nodes from container-only section
  • incusd/devices: Better handle bad config
  • api: numa_cpu_balanced
  • internal/instance: Add support for balanced NUMA nodes
  • doc: Update configs
  • incusd/instance/common: Add NUMA balancing
  • incusd/instance/lxc: Add support for balanced NUMA allocation
  • incusd/instance/qemu: Add support for balanced NUMA allocation
  • incusd/devices: Add support for balanced NUMA allocation
  • incusd/device/gpu_sriov: Simplify NUMA logic
  • doc/cloud-init: Don't mention non-existing remotes
  • doc/howto/images_remote: Fix wording around image servers
  • doc/benchmark: Fix install command
  • incusd/instance/common: Fix CanMigrate mutating devices
  • incusd/instance/qemu: Reduce agent queries
  • incusd/metrics: Don't filter out all server metrics
  • incusd/auth/tls: Include project restrictions for metrics certificates
  • incusd/auth/tls: Return project-aware checker for metrics
  • incusd/metrics: Use project-specific checker if no global access
  • internal/server/instance/lxd: add support for image.requirments.nesting
  • api: add image_restriction_nesting
  • doc/images: introduce requirements.nesting
  • Show the count values in snapshot count mismatch error
  • incus/admin/init: Use btrfs subvol in --auto
  • incus-migrate: Clarify that disk image files must be raw
  • incusd/network/ovn/icnb: Fix comment
  • incusd/project: Re-format the comments
  • incusd/project: Fix bad default value
  • doc: Update configs
  • incus/migrate: Add CSM support
  • incusd/storage/backend: Better handle name conflicts
  • incus-migrate: Support using the local server
  • api: network_integrations
  • shared/api: Add type and target_integration fields to NetworkPeersPost
  • incusd/db/cluster: Add networks_integrations
  • incusd/db/cluster: Re-generate schema
  • incusd/db/cluster: Add generated DB code for network integrations
  • incusd/db: Update network peer DB query functions
  • client: Add check for network_integrations in CreateNetworkPeer
  • incus/network/peer: Add support for network peer types
  • shared/api: Add network integrations
  • client: Add network integration functions
  • incus/network: Introduce support for integrations
  • incusd/auth: Add network integration functions
  • shared/api: Add lifecycle events for network integrations
  • incusd/lifecycle: Add network integration events
  • incusd: Add network integration API
  • incusd/db: Add GetNetworkPeersURLByIntegration
  • incusd/network_integration: Add UsedBy field
  • incusd/network_integrations: Add validator
  • incusd/network/ovn: Add support for peering with OVN IC
  • incusd/project: Add restricted.networks.integrations
  • incusd/project: Add NetworkIntegrationAllowed
  • incusd/network/integrations: Respect project restrictions
  • incusd/network/ovn: Add support for integration restrictions
  • incusd/auth/openfga: Update the model
  • incusd/auth/openfga: Update the generated model
  • incusd/auth/openfga: Handle model updates
  • incusd: Remove openfga.store.model_id
  • incusd/db/cluster: Remove openfga.store.model_id
  • doc/ovn_peers: Add remote peering
  • doc: Add documentation for network integrations
  • doc/rest-api: Refresh swagger YAML
  • i18n: Update translation templates
  • doc: Update configs
  • gomod: Update dependencies

Documentation

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Installation

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Linux packages

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.0.0

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Migrating from LXD

A lxd-to-incus migration tool allows for in-place migration from LXD to Incus.
It's been tested with LXD versions as low as 4.0 LTS and as high as the latest LXD 5.21 bugfix release.

It allows for a very quick migration from LXD over to Incus, automatically checking for potential conflicts ahead of time.

asciicast

More details can be found here: https://linuxcontainers.org/incus/docs/main/howto/server_migrate_lxd/

Support

Incus 6.0 LTS will be supported for a total of 5 years (until June 2029).

During the first 2 years, new point releases will be issued including a mix of bug and security fixes as well as some minor usabiltiy improvements. After that initial 2 years (after Incus 7.0 LTS is released), Incus 6.0 LTS will transition to security fixes only for the remaining 3 years.

This matches what we've been doing for our other projects (LXC and LXCFS) over the past 10 years.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.7 has been released

26th of March 2024

Introduction

The Incus team is pleased to announce the release of Incus 0.7!

This is going to be our last release before Incus 6.0 LTS which is now scheduled to be released next week. As releases go, this is quite a busy one, which is how we like it before releasing an LTS, trying to keep the amount of new features in the LTS release itself to a minimum.


image|690x404

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features

Network integrations

A new top-level concept, network integrations are a way to connect an Incus deployment to networks outside of its own control.

Currently the only implementation of the concept is OVN interconnection which makes it possible for an Incus cluster to directly peer its OVN networks with equivalent networks running on other Incus clusters or even other OVN users like OpenStack or Kubernetes.

Here is an example of creating a new network integration using an OVN interconnection gateway, then peering an existing network through it:

root@az01-server01:~# incus network integration create ovn-region ovn
Network integration ovn-region created
root@az01-server01:~# incus network integration set ovn-region ovn.northbound_connection tcp:[10.50.1.12]:6645,tcp:[10.50.2.13]:6645,tcp:[10.50.3.19]:6645
root@az01-server01:~# incus network integration set ovn-region ovn.southbound_connection tcp:[10.50.1.12]:6646,tcp:[10.50.2.13]:6646,tcp:[10.50.3.19]:6646
root@az01-server01:~# incus network peer create default region ovn-region --type=remote
Network peer region created

Documentation: https://linuxcontainers.org/incus/docs/main/howto/network_integrations/

Image server management tool

A common way to run an Incus image server, be it for some internal servers or as a publicly available image server is through a static web server providing Incus images using simplestreams.

To make this easier to set up, we're now introducing a new tool, incus-simplestreams which can easily manage a simple image server, listing the images available, adding and removing images as well as generating the needed metadata files.

stgraber@dakara:~$ mkdir image-server
stgraber@dakara:~$ cd image-server/
stgraber@dakara:~/image-server$ incus-simplestreams generate-metadata ~/Downloads/incus.tar.xz
Operating system name: Red Hat Enterprise Linux
Release name: 9
Variant name [default="default"]:
Architecture name: x86_64
Description [default="Red Hat Enterprise Linux 9 (default) (x86_64) (202403260239)"]:·
stgraber@dakara:~/image-server$ incus-simplestreams add ~/Downloads/incus.tar.xz ~/Downloads/rhel9.qcow2·
stgraber@dakara:~/image-server$ incus-simplestreams list
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
|                           FINGERPRINT                            |                   DESCRIPTION                    |            OS            | RELEASE | VARIANT | ARCHITECTURE |      TYPE       |       CREATED        |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
| 7d256e4fac6fc63fb47bc1e07e1c6ee234281cdf1ed21788c920d763b7bd93ba | Red Hat Enterprise Linux 9 x86_64 (202403252239) | Red Hat Enterprise Linux | 9       | default | x86_64       | virtual-machine | 2024/03/25 00:00 UTC |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
stgraber@dakara:~/image-server$ find . | sort
.
./images
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.incus.tar.xz
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.qcow2
./streams
./streams/v1
./streams/v1/images.json
./streams/v1/index.json

Put that on an HTTPS capable web server and then add it with:

incus remote add my-server https://xyz.example.net --protocol=simplestreams

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_servers/#tooling-to-manage-a-simplestreams-server

JSON Web Token authentication

Incus basically supports two mechanisms for remote authentication:

  • TLS client certificates (added to the local trust store with or without restrictions)
  • OpenID Connect external authentication (with or without OpenFGA for authorization)

The former is the most common for simple interactions with a remote Incus server.
Our own CLI tool and most 3rd party tools don't have any problems using a TLS keypair to establish the HTTPS connection and get authenticated that way.

But there are some situations, like running Incus behind a reverse HTTP(S) proxy where TLS client certificates can become a bit problematic.

To address that, we now support using a JSON Web Token (JWT) bearer token through the HTTP Authorization field. That token can be generated by any user with a valid TLS client certificate by setting the Subject field to the certificate fingerprint, setting applicable NotBefore/NotAfter values and signing the JWT with their private key.

Incus will treat any such connections as equivalent to using the TLS client certificate.

stgraber@dakara:~$ openssl req -x509 -newkey rsa:4096 -sha384 -keyout client.key -nodes -out client.crt -days 1 -subj "/CN=test.local"
.+.........+...+...+..+....+......+........+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+......+...+......+......+..+...+...+...............+.......+...+......+...+.....+......+....+......+..+...+......+....+............+.....+....+.....+.+............+..+.........+......+....+......+...........+....+........+...+...+.+...+..+..........+.....+...+......+............+...+.......+........+....+.....+.+..+.......+......+..+....+........+..........+...+..+.+.....+.+......+..+.......+.....+.+..+..........+..+....+..............+.+..+...........................+...+....+......+...+..............+.+..+....+.....+.+.........+...+..+....+..+.............+.........+.....+...+..........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...+..+...+.........+.+...+............+..+............+.+.......................................+.....+...+......+.........+......+.+.....+...+.+...........+......+.......+.....+.......+......+.....+..........+...+..+.........+....+.........+...........+......+.+..................+..+....+...........+.............+.....+....+..+......+............+..........+......+......+......+..+.............+.....+...+.+........+............+....+.................+.........+......+.......+...+.........+.....+....+......+........+.+..+....+......+........+....+...+.................+.+..+.........+....+............+.....+.........+....+.....+.......+.....+...............+..............................+....+........+.......+...........+.......+.....+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+...............+.+...+...+...+.........+..+.+...+..+.......+.....+.+..+...+...................+...........+......+....+..+............+...+.......+..+.........+....+...+.........+.....+...+...+....+...........+...+.+.........+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+......+..+...+......+.+..+.............+..+.......+........+...+...+....+...............+..+....+..+....+...+.....+.+.....+..........+.....+.+.....+....+............+.........+...+.....+......+......+.............+..+.+..+.......+...+........+...+.......+.........+......+..+.+......+...........+...+.........+...+...+....+..+.........+....+.....+.+......+.........+..+..........+..+...+...+....+..+...+.+.....+......+.+...+......+.....+.+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
stgraber@dakara:~$ incus config trust add-certificate client.crt --restricted --projects demo
stgraber@dakara:~$ tls2jwt client.key client.crt now 120
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0stgraber@dakara:~$·
stgraber@dakara:~$ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0' https://localhost:8443/1.0/projects | jq
{
  "type": "sync",
  "status": "Success",
  "status_code": 200,
  "operation": "",
  "error_code": 0,
  "error": "",
  "metadata": [
    "/1.0/projects/demo"
  ]
}

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#using-json-web-token-jwt-to-perform-tls-authentication

Configurable OIDC username field

For those using OpenID Connect, you may have noticed that Incus will use the e-mail claim when available as the user's identifier. Then if missing, it will rely on the Subject.

As different deployments may make different information available through OIDC claims, it's now possible to set oidc.claim to the claim to use as the user identifier.

stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org
stgraber@dakara:~$ incus config set oidc.claim=name
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
Stéphane Graber
stgraber@dakara:~$ incus config set oidc.claim=sub
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
99cb8caa-3640-45b9-b87a-55266366aaf3
stgraber@dakara:~$ incus config set oidc.claim=email
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org

Improved NUMA handling

With this release, we spent a fair amount of time trying to improve both the container and virtual-machine performance on large systems. This obviously includes multi-socket systems but also AMD systems running in NPS4 or similar mode where each CPU is exposed as multiple NUMA nodes.

In general, our goal has been to make it easy to distribute workloads across NUMA nodes while keeping their CPU and memory properly pinned and also selecting PCIe resources that are closest to their NUMA node(s).

As part of that, a few things were done:
- limits.cpu.nodes is now supported for virtual-machines too
- A new balanced value has been added to limits.cpu.nodes which will have Incus pick the NUMA node with the least instances configured to use it
- SR-IOV GPU selection now also considers NUMA nodes as part of the selection logic and when no match is found, will prefer PCIe devices that are attached to the same CPU socket

For example:

stgraber@gputest:~$ incus list stgraber-gpu -cns4,limits.cpu.nodes,volatile.cpu.nodes,volatile.gpu.last_state.pci.parent,volatile.gpu.last_state.vf.id
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
|      NAME      |  STATE  |         IPV4          | LIMITS CPU NODES | VOLATILE CPU NODES | VOLATILE GPU LAST STATE PCI PARENT | VOLATILE GPU LAST STATE VF ID |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu01 | RUNNING | 10.232.44.8 (enp5s0)  | balanced         | 0                  | 0000:63:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu02 | RUNNING | 10.232.44.9 (enp5s0)  | balanced         | 2                  | 0000:03:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu03 | RUNNING | 10.232.44.10 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu04 | RUNNING | 10.232.44.11 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu05 | RUNNING | 10.232.44.12 (enp5s0) | balanced         | 6                  | 0000:c3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu06 | RUNNING | 10.232.44.13 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu07 | RUNNING | 10.232.44.15 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu08 | RUNNING | 10.232.44.16 (enp5s0) | balanced         | 2                  | 0000:03:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu09 | RUNNING | 10.232.44.17 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu10 | RUNNING | 10.232.44.18 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu11 | RUNNING | 10.232.44.19 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu12 | RUNNING | 10.232.44.20 (enp5s0) | balanced         | 6                  | 0000:83:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu13 | RUNNING | 10.232.44.21 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu14 | RUNNING | 10.232.44.22 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu15 | RUNNING | 10.232.44.23 (enp5s0) | balanced         | 2                  | 0000:43:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu16 | RUNNING | 10.232.44.24 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+

In this case we can see 16 VMs each using the new balanced option for NUMA nodes and getting scheduled across 8 NUMA nodes (2 sockets AMD NPS4) with GPUs being selected to match.

More options to select USB devices

USB device passhtrough for both containers and virtual-machines has so far been using the vendorid and productid fields. This works fine as long as there is only one USB device of any one type connected to the system.

When multiple identical devices are present, the inability to distinguish them has been a problem.

To address this, three new fields have now been added to usb devices:
- busnum referring to the USB bus number
- devnum referring to the USB device number (on its bus)
- serial referring to the USB device serial number (not present on all devices)

The same fields can be found in the full Incus resources list through:

incus query /1.0/resources

Disk I/O throttling for VMs

One more feature gap between containers and virtual-machines is now gone.

The limits.write and limits.read properties on disk devices are now properly enforced on virtual-machines by having Incus setup an I/O throttle in QEMU.

Both bytes per second and I/O per second type limits are supported.

Per-remote client certificates

It's now possible to put a <remote>.crt and <remote>.key file in a new clientcerts folder within the Incus command line client config directory (typically ~/.config/incus/) and have those certificates be used when interacting with that particular remote.

While this may be useful on its own, it becomes a lot more useful when combined with global remotes, which can be added in /etc/incus/config.yml. Now with this feature, those global remotes can also have a client certificate made available in /etc/incus/clientcerts/ which will then be used by all users on the system.

Manual generation of the client certificate keypair

A new command to manually trigger the generation of the main client.crt and client.key keypair is now available.

This is done by running incus remote generate-certificate

Improvements to lxd-to-incus

The lxd-to-incus tool keeps evolving with every release.

In this one, it gains support for migrating users from the newly released LXD 5.21 LTS as well as handling Alpine installations.

Additionally, a static binary version of the tool is now available on Github, making it easier for users to fetch the latest version of the tool, useful as bugs get fixed in between Incus releases.

Improvements to incus-migrate

The workload migration tool incus-migrate has also seen a couple of small improvements.

It can now use the local Incus system as the target of the migration, useful when importing virtual-machine images from another virtualization tool.

And it's also now prompting for whether the imported virtual machine should be using a UEFI firmware or instead use a legacy BIOS.

Additional image restrictions

A bit of an internal detail, or at least only relevant to public image server operators, but two new image restrictions have been added:

  • requirements.nesting which will require the container have security.nesting=true
  • requirements.cdrom_agent which will require that a source=agent:config disk device be added to the virtual-machine

Those two can be used to flag specific images that need extra user interaction to work properly, resulting in a clear client-side error rather than starting a potentially broken instance.

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • Translated using Weblate (Japanese)
  • Translated using Weblate (Japanese)
  • incus/image: Fix column handling with --all-projects
  • Replace util.ValueInSlice with slices.Contains
  • shared/util: Delete ValueInSlice function
  • incus/image: Fix column handling with --all-projects
  • incusd/instance/qemu: Relocate image requirement checks
  • doc/images: Add requirements.cdrom_agent
  • incusd/instance/qemu: Add support for requirements.cdrom_agent
  • incusd/device/disk: Fix incorrect block volume usage
  • Translated using Weblate (Japanese)
  • incusd/network/ovn: Use ParseIPToNet instead of manual IPToNet and net.ParseIP
  • incusd/network/ovn: Use listenAddressNet in family check
  • incusd/instance/drivers: Disable architecture check on incus cp with snapshots
  • Translated using Weblate (French)
  • incusd/network/bridge: Set local address on all VXLAN tunnels
  • incus/instance/qemu: Fix RecordOutput
  • incus: add completions for instance actions and snapshots
  • incus: add completions for profiles
  • incusd/network/ovn: Introduce get helper
  • incusd/network/ovn: Add some missing indices
  • incusd/network/ovn: Use get helper
  • incusd/network/ovn: Fix LogicalSwitchPortIPs logic
  • incusd/network/bridge: Fix gofmt
  • incusd/network/ovn: Fix gofmt
  • cmd/incus: Use proper timestamp check
  • cmd/incus: Use consistent date format and timezone
  • client: Rename network_peer for consistency
  • cmd/incusd: Rename network_peer to network_peers
  • shared/api: Rename network_allocation for consistency
  • incusd/db: Fix comment typoes
  • incusd/db/generate: Fix bad camel case handling
  • incusd/db/network_peers: Fix duplicate type definitions
  • incusd/auth: Drop Permission type
  • incusd/auth: Add boilerplate doc strings
  • incusd/images: Properly handle null creation and expiry dates
  • incus: add completions for remotes
  • incus: add completions for projects
  • incusd/images: Fix reporting of images in multiple projects
  • github: Add static build of lxd-to-incus
  • lxd-to-incus: Add support for Alpine service name
  • lxd-to-incus: Re-organize target list
  • lxd-to-incus: Add support for APK
  • Makefile: Add OVN IC to update-ovsdb
  • incusd/network: Update OVS/OVN schemas
  • incusd/network/ovn: Add IC clients
  • incusd/network/ovn: Add GetName to NB client
  • incusd/network/ovn: Add GetGateways to ICSB
  • incusd/network/ovn: Introduce new errors
  • incusd/network/ovn: Add CreateTransitSwitch and DeleteTransitSwitch to ICNB
  • incusd/device/gpu_sriov: Add locking
  • incusd/device/gpu_sriov: Re-locate vfio-pci loading
  • incusd/device/gpu_sriov: Rework VF allocation logic
  • incus/remote: Add a generate-certificate sub-command
  • i18n: Update translation templates
  • incusd/drivers/qmp: Add SetBlockThrottle
  • incusd/device/disk/config: Add DiskLimits
  • incusd/device/disk: Re-shuffle limit parsing
  • incusd/device/disk: Add disk limits on VMs
  • incusd/device/disk: Support live limits update for VMs
  • incusd/instance/qemu: Support disk I/O limits
  • incus/remote: Add missing docstrings
  • incusd/certificates: Improve token handling when clustered
  • cmd/incusd/api_1.0: Update context
  • cmd/incusd/api_cluster: Update context
  • cmd/incusd/api_internal: Update context
  • cmd/incusd/daemon: Update context
  • cmd/incusd/api_project: Update context
  • cmd/incusd/certificates: Update context
  • cmd/incusd/images: Update context
  • cmd/incusd/instance: Update context
  • cmd/incusd/network: Update context
  • cmd/incusd/operations: Update context
  • cmd/incusd/profiles: Update context
  • cmd/incusd/storage: Update context
  • cmd/incusd/warnings: Update context
  • incusd/devices: Skip isolated threads from NUMA CPUs
  • incusd/devices: Restrict CPU threads by NUMA node
  • incusd/instance/qemu: Add support for limits.cpu.nodes
  • incusd/device/gpu: Add support for limits.cpu.nodes for VF selection
  • incusd: Fix import shadowing
  • incusd/images: Fix potential race condition
  • incusd/instance/qemu: Add support for NUMA node restrictions for memory
  • incusd/apparmor/qemu: Silence apparmor failures
  • incusd/network/ovs: Introduce new errors
  • incusd/network/ovn/nb: Move SetChassisGroupPriority to new function signature
  • incusd/network/ovn/sb: Move GetLogicalRouterPortActiveChassisHostname to new function signature
  • incusd/network/ovs: Move GetBridge to new function signature
  • incusd/network/ovs: Move CreateBridge to new function signature
  • incusd/network/ovs: Move DeleteBridge to new function signature
  • incusd/network/ovs: Move CreateBridgePort to new function signature
  • incusd/network/ovs: Move GetChassisID to new function signature
  • incusd/network/ovs: Move GetOVNBridgeMappings to new function signature
  • incusd/network: Update for function changes
  • incusd/device/nic: Update for function changes
  • incusd: Update for function changes
  • doc: Fix bad snapshot syntax
  • Translated using Weblate (French)
  • doc: Fix token creation procedure
  • incusd/network/ovn/nb: Add GetLogicalSwitch
  • incusd/network/ovn/nb: Replace ChassisGroupChassisDelete with SetChassisGroupPriority
  • incusd/network/ovn/nb: Port CreateLogicalRouterPort to OVSDB
  • incusd/network/ovn/nb: Replace LogicalRouterPortLinkChassisGroup with CreateLogicalRouterPort
  • incusd/network/ovn/nb: Port CreateChassisGroup to OVSDB
  • incusd/network/ovn/nb: Port CreateLogicalSwitch to OVSDB
  • incusd/network/ovn: Update for function changes
  • incusd/network/ovn: Remove state references
  • incusd/state: Add OVNNB and OVNSB handles
  • incusd: Update to use state for OVN
  • incusd/device: Make init function return error
  • incusd/device: Add OVN check on nicOVN
  • client: Still return response on RawQuery error
  • incus/query: Respect --raw for errors
  • incusd/network/acl: Add OVN check
  • incusd/network: Make init function return error
  • incusd/network: Add OVN check on ovn driver
  • incusd/api: Re-order config checks
  • incusd: Add OVN loader
  • Translated using Weblate (French)
  • incusd/network/ovn/nb: Port CreateLogicalSwitchPort to OVSDB
  • incusd/network/ovn/nb: Port DeleteLogicalSwitchPort to OVSDB
  • incusd/network/ovn/nb: Port DeleteLogicalRouterPort to OVSDB
  • incusd/network/ovn: Update for function changes
  • incusd/network/ovs: Port GetOVNSouthboundDBRemoteAddress to OVSDB
  • incusd/network/ovs: Port DeleteBridgePort to OVSDB
  • incusd/network/ovs: Port GetInterfaceAssociatedOVNSwitchPort to OVSDB
  • incusd/network/ovs: Align GetChassisID with other functions
  • incusd: Update for OVS function changes
  • incusd/network/ovn/icsb: Fix bad DB schema
  • incusd/network/ovn/nb: Introduce GetLogicalRouterPort
  • incusd/network/ovn/nb: Extend OVNSwitchPortOpts to handle router ports
  • incusd/network/ovn/nb: Change type of RouterPort field to OVNRouterPort
  • incusd/network/ovn/nb: Port DeleteChassisGroup to OVSDB
  • incusd/network/ovn/icnb: Update DeleteTransitSwitch to handle missing switches
  • incusd/network/ovn: Update for function changes
  • Translated using Weblate (French)
  • incus/completion: do not add a space after remote names completion
  • incusd/device/disk: Disable virtiofsd caching
  • incus-agent: Cleanup mount logic
  • Translated using Weblate (French)
  • incus: expose parseVolume to entire package
  • incus: add completions for storage pools and volumes
  • incusd/device/gpu_sriov: Fix default handling
  • doc/packaging: Add mention of documentation
  • incusd/auth: Fix --all-projects for restricted users
  • doc: Add third party tools page
  • gomod: Update dependencies
  • incusd/auth/tls: Prevent project modifications
  • doc: Update wordlist
  • internal/usbid: allow path override of usb.ids path
  • incus/completion: fix image names completion
  • doc/environment: document INCUS_USBIDS_PATH
  • incusd/instance/qemu/agent: Check for semanage
  • incusd/project: Fix config name in ImageProjectFromRecord
  • incus/restart: Fix long description
  • i18n: Update translations
  • lxd-to-incus: Handle common existing bridges
  • shared/simplestreams: Remove defaultOS
  • shared/simplestreams: Add NewLocalClient
  • incus-simplestreams: Introduce new command
  • incus-simplestreams: Simplify delete logic
  • doc: Re-organize image server doc
  • doc: Add section for incus-simplestreams
  • incusd/seccomp: Add support for pidfd threads
  • incus: add completions for clusters
  • incus: add completions for cluster groups
  • incus: add completions for cluster roles
  • incus: add completions for config devices
  • incus: add completions for config templates
  • update translations
  • doc: Update references to mage docs
  • doc/backup: Remove bad reference
  • incus: add completions for network acls
  • shared/api: Add new structs to support configuration metadata
  • client: Add GetMetadataConfiguration
  • incusd: Rename documentation.go -> metadata.go
  • doc/rest-api: Refresh swagger YAML
  • shared/api/metadata: Add GetKeys to simplify usage
  • incusd: Add support for JWT authentication
  • gomod: Update dependencies
  • tests: Add tls2jwt tool
  • tests: Add JWT authentication test
  • api: auth_tls_jwt
  • doc/authentication: Add section on JWT
  • doc/instances: Remove size.state requirement for live migration
  • incusd/instance/qemu: Allow live migration without size.state
  • shared/idmap: Support uid/gid in subuid/subgid
  • shared/cliconfig: Copy clientcerts on remote copy
  • shared/cliconfig: Add HasRemoteClientCertificate
  • shared/cliconfig: Support per-remote client certificates
  • doc: Add clientcerts
  • incusd/cluster/config: Add oidc.claim
  • incusd/auth/oidc: Add support for using a specific claim as username
  • incusd: Pass OIDC claim to verifier
  • api: oidc_claim
  • doc: Update configs
  • doc/howto/instances: Mention extra resources in ISO guidea
  • doc/installing: Add Debian backport
  • doc: Add backported to dictionary
  • lxd-to-incus: Add support for LXD 5.21
  • shared/cliconfig: Ensure client certificate key is 0600
  • api: device_usb_serial
  • doc: Add busnum, devnum and serial to USB devices
  • shared/api: Add Serial to ResourcesUSBDevice
  • incusd/resources: Add USB Serial
  • incusd/devices/usb: Add serial, busnum and devnum options
  • doc/rest-api: Refresh swagger YAML
  • incusd/instance/qemu: Fix handling of > 64 limits.cpu
  • incusd/device/gpu_sriov: Implement NUMA fallback
  • incus: add completions for network forwards
  • incus: add completions for network load balancers
  • shared/validate: Remove stringInSlice
  • shared/validate: Add And and Or functions
  • shared/util: Move ParseUint32Range
  • incusd/project: Update for ParseUint32Range
  • doc/instance_options: Remove mention of limits.cpu.nodes from container-only section
  • incusd/devices: Better handle bad config
  • api: numa_cpu_balanced
  • internal/instance: Add support for balanced NUMA nodes
  • doc: Update configs
  • incusd/instance/common: Add NUMA balancing
  • incusd/instance/lxc: Add support for balanced NUMA allocation
  • incusd/instance/qemu: Add support for balanced NUMA allocation
  • incusd/devices: Add support for balanced NUMA allocation
  • incusd/device/gpu_sriov: Simplify NUMA logic
  • doc/cloud-init: Don't mention non-existing remotes
  • doc/howto/images_remote: Fix wording around image servers
  • doc/benchmark: Fix install command
  • incusd/instance/common: Fix CanMigrate mutating devices
  • incusd/instance/qemu: Reduce agent queries
  • incusd/metrics: Don't filter out all server metrics
  • incusd/auth/tls: Include project restrictions for metrics certificates
  • incusd/auth/tls: Return project-aware checker for metrics
  • incusd/metrics: Use project-specific checker if no global access
  • internal/server/instance/lxd: add support for image.requirments.nesting
  • api: add image_restriction_nesting
  • doc/images: introduce requirements.nesting
  • Show the count values in snapshot count mismatch error
  • incus/admin/init: Use btrfs subvol in --auto
  • incus-migrate: Clarify that disk image files must be raw
  • incusd/network/ovn/icnb: Fix comment
  • incusd/project: Re-format the comments
  • incusd/project: Fix bad default value
  • doc: Update configs
  • incus/migrate: Add CSM support
  • incusd/storage/backend: Better handle name conflicts
  • incus-migrate: Support using the local server
  • api: network_integrations
  • shared/api: Add type and target_integration fields to NetworkPeersPost
  • incusd/db/cluster: Add networks_integrations
  • incusd/db/cluster: Re-generate schema
  • incusd/db/cluster: Add generated DB code for network integrations
  • incusd/db: Update network peer DB query functions
  • client: Add check for network_integrations in CreateNetworkPeer
  • incus/network/peer: Add support for network peer types
  • shared/api: Add network integrations
  • client: Add network integration functions
  • incus/network: Introduce support for integrations
  • incusd/auth: Add network integration functions
  • shared/api: Add lifecycle events for network integrations
  • incusd/lifecycle: Add network integration events
  • incusd: Add network integration API
  • incusd/db: Add GetNetworkPeersURLByIntegration
  • incusd/network_integration: Add UsedBy field
  • incusd/network_integrations: Add validator
  • incusd/network/ovn: Add support for peering with OVN IC
  • incusd/project: Add restricted.networks.integrations
  • incusd/project: Add NetworkIntegrationAllowed
  • incusd/network/integrations: Respect project restrictions
  • incusd/network/ovn: Add support for integration restrictions
  • incusd/auth/openfga: Update the model
  • incusd/auth/openfga: Update the generated model
  • incusd/auth/openfga: Handle model updates
  • incusd: Remove openfga.store.model_id
  • incusd/db/cluster: Remove openfga.store.model_id
  • doc/ovn_peers: Add remote peering
  • doc: Add documentation for network integrations
  • doc/rest-api: Refresh swagger YAML
  • i18n: Update translation templates
  • doc: Update configs
  • gomod: Update dependencies

Documentation

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.7

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.6 has been released

23rd of February 2024

Introduction

The Incus team is pleased to announce the release of Incus 0.6!

This second release of 2024 features a number of improvements, both large and small.

It comes with a completely new storage driver for cluster users, import/export support for storage buckets, a number of improvements for OVN users, support for new container kernel features, quite a few improvements to the migration tooling and more!

image|690x459

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features

Clustered LVM storage driver

A new storage driver named lvmcluster has been added.

As the name implies, this is for clustered use of the LVM storage driver.
This relies on llvmlockd and a lock manager like sanlock to provide concurrent access to a shared storage device across a cluster.

This is aimed at anyone who wants to run an Incus cluster and use a shared storage device like a FiberChannel SAN, iSCSI export, NVMEoF/NVMEoTCP disk, ...

With the ability of having the same storage available on all servers comes support for much faster live migrations, server evacuations and the ability to automatically recover should a server suddenly die.

image|622x500

Documentation: https://linuxcontainers.org/incus/docs/main/reference/storage_lvm/#lvmcluster-driver-in-incus

Storage bucket backup and import

It is now possible to backup a full storage bucket and later re-import back into Incus.

stgraber@dakara:~$ incus storage bucket create default foo
Storage bucket foo created
Admin access key: HD2GWC1KX693MFVT3P30
Admin secret key: tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv

stgraber@dakara:~$ dd if=/dev/random of=out.img bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0975758 s, 430 MB/s
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv put out.img s3://foo
upload: 'out.img' -> 's3://foo/out.img'  [part 1 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   262.42 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 2 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   241.54 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 3 of 3, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s   241.99 MB/s  done
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:26     41943040  s3://foo/out.img

stgraber@dakara:~$ incus storage bucket export default foo
Backup exported successfully!
stgraber@dakara:~$ incus storage bucket delete default foo
Storage bucket foo deleted

stgraber@dakara:~$ incus storage bucket import default backup.tar.gz
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:27     41943040  s3://foo/out.img

API: https://linuxcontainers.org/incus/docs/main/rest-api-spec/#/storage/storage_pool_buckets_backups_post

Listing image across all projects

Just as it's possible to list instances across all projects using the --all-projects, it's now possible to do the same but for images.

stgraber@dakara:~$ incus image list --all-projects
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| PROJECT | ALIAS | FINGERPRINT  | PUBLIC |               DESCRIPTION                | ARCHITECTURE |      TYPE       |   SIZE    |          UPLOAD DATE          |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 256f59a72af5 | no     | Ubuntu jammy amd64 (20240222_07:42)      | x86_64       | VIRTUAL-MACHINE | 267.19MiB | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 0941e441dbb9 | no     | Alpine edge amd64 (20240222_13:00)       | x86_64       | CONTAINER       | 2.93MiB   | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | d5fc6024f0fa | no     | Openwrt snapshot amd64 (20240222_11:57)  | x86_64       | CONTAINER       | 3.50MiB   | Feb 23, 2024 at 2:55am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| demo    |       | f44a6b4e56f4 | no     | Archlinux current amd64 (20240222_04:18) | x86_64       | CONTAINER       | 188.86MiB | Feb 23, 2024 at 3:32am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+

Over the API, this is done by passing ?all-projects=true.

binfmt_misc in unprivileged containers

Linux 6.7 added support for mounting of binfmt_misc inside of unprivileged containers.

Incus 0.6 will detect kernels that support this feature and when they do, it will no longer bind-mount binfmt_misc from the host system but instead allow it to be mounted from within the container.

stgraber@castiana:~$ incus launch images:ubuntu/22.04 foo
Launching foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# uname -a
Linux foo 6.7.4-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Mon Feb  5 23:37:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@foo:~# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/
root@foo:~# ls -lh /proc/sys/fs/binfmt_misc/
total 0
--w------- 1 root root 0 Feb 23 03:46 register
-rw-r--r-- 1 root root 0 Feb 23 03:46 status

The binfmt_misc filesystem is used to define custom binary formats for emulation.
It's most notably used by qemu-user-static as a way to run binaries of other architectures.

Control over shared block storage volumes

A new security.shared volume storage option has been introduced to control whether a custom block volume should be allowed to be used by multiple instances.

It is now required to set that configuration key prior to adding a custom block volume to a profile or add it directly to multiple virtual machines.

stgraber@castiana:~$ incus config device add lvm01 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm01

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Error: Failed add validation for device "shared": Cannot add un-shared custom storage block volume to more than one instance

stgraber@castiana:~$ incus storage volume set default shared-lvm security.shared=true

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm02
stgraber@castiana:~$ incus config device add lvm03 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm03
stgraber@castiana:~$ incus start lvm01 lvm02 lvm03

OVN logical router name in network info

To make it easier to see what's going on within OVN, incus network info now gives you the name of the logical router for a particular network.

root@abydos:~# incus network info default
Name: default
MAC address: 00:16:3e:38:dd:28
MTU: 1500
State: up
Type: broadcast

IP addresses:
  inet  10.180.103.1/24 (link)
  inet6 2602:fc62:a:1004::1/64 (link)

Network usage:
  Bytes received: 0B
  Bytes sent: 0B
  Packets received: 0
  Packets sent: 0

OVN:
  Chassis: abydos
  Logical router: incus-net6-lr

File ownership and permissions in image templates

Template files can now have a uid, gid and mode set on them.
This can be particularly useful if a template is meant to be an executable shell script.

stgraber@castiana:~$ incus config metadata show foo
architecture: amd64
creation_date: 1708588077
expiry_date: 1711180077
properties:
  architecture: amd64
  description: Ubuntu jammy amd64 (20240222_07:42)
  name: ubuntu-jammy-amd64-default-20240222_07:42
  os: ubuntu
  release: jammy
  serial: "20240222_07:42"
  variant: default
templates:
  /etc/hostname:
    when:
    - create
    - copy
    create_only: false
    template: hostname.tpl
    properties: {}
  /etc/hosts:
    when:
    - create
    - copy
    create_only: false
    template: hosts.tpl
    properties: {}
  /root/hello.sh:
    when:
    - start
    create_only: false
    template: hello.tpl
    properties: {}
    uid: "1000"
    gid: "2000"
    mode: "0755"

stgraber@castiana:~$ incus config template show foo hello.tpl
#!/bin/sh
echo "Hello world!"

stgraber@castiana:~$ incus start foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# ls -lh /root/hello.sh
-rwxr-xr-x 1 ubuntu 2000 30 Feb 23 04:07 /root/hello.sh
root@foo:~# /root/hello.sh
Hello world!

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_format/#template-rules

Encrypted EC client certificate keys

For those interacting with remote Incus servers, you may not know that it's possible to protect your Incus private key with a password.

Up until now, this was only possible for RSA keys, but with Incus 0.6, we're now adding support for EC keys too. That's particularly relevant as EC keys have been the default for a while now.

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#encrypting-local-keys

Worth noting that if using this feature, you'll likely also want to make use of the recently introduced "keepalive mode" as this will then significantly reduce the number of password prompts you'll get while using Incus.

lxd-to-incus improvements

Lastly, lxd-to-incus saw a number of improvements:

  • Support for Void Linux
  • Detection of the boot.debug_edk2 configuration key
  • Handling of OVN SSL database connections
  • Automatic clearing of the simplestreams cache during migration

Complete changelog

Here is a complete list of all changes in this release:

Full commit list
  • incusd/instance/qemu/qmp: Don't risk blocking QMP on eject
  • github: Switch to minio deb and pin working version
  • github: Use stable MicroCeph
  • incusd/db: Fix error handling in CreateNetworkPeer
  • incusd/apparmor: Allow more mounts in unprivileged containers
  • Translated using Weblate (Portuguese (Brazil))
  • incusd/instance/qemu/agent-loader: Handle legacy lxd-agent-loader
  • incusd/device/disk: Use genisoimage when mkisofs can't be found
  • shared/api: Add example of Chassis in NetworkStateOVN
  • doc/rest-api: Refresh swagger YAML
  • api: network_state_ovn_lr
  • shared/api: Add LogicalRouter field to NetworkStateOVN
  • incusd/network/ovn: Expose LogicalRouter name
  • incus: Add OVN logical router name
  • i18n: Update translation templates
  • doc/rest-api: Refresh swagger YAML
  • scripts/bash: Don't follow aliases
  • Translated using Weblate (Japanese)
  • test: Adapt test_database_no_disk_space() to newer libraft versions
  • incus/launch: Fix --console
  • api: image_template_permissions
  • doc: Add uid/gid/mode to image template metadata
  • shared/api: Add owner/permission fields to ImageMetadataTemplate
  • doc/rest-api: Refresh swagger YAML
  • incusd/instance/lxc: Add support for template ownership/mode
  • incus-agent: Add support for template ownership/mode
  • lxd-to-incus: Add comment
  • lxd-to-incus: Add cache cleanup logic
  • .github: Bump minimum Go version to 1.21
  • Makefile: Bump minimum Go version to 1.21
  • gomod: Update dependencies
  • doc: Bump minimum Go version to 1.21
  • Translated using Weblate (French)
  • incusd/daemon: Fix log format
  • incusd/sys: Sort kernel features
  • incusd/sys: Add doc strings
  • incusd/checkfeature: Add unprivileged binfmt detection
  • incusd/sys: Add UnprivBinfmt
  • incusd/daemon: Detect unprivileged binfmt_misc
  • incusd/api_1.0: Add unpriv_binfmt kernel feature
  • incusd/instance/lxc: Support unpriv binfmt_misc
  • incusd/apparmor: Support unpriv binfmt_misc
  • Translated using Weblate (Italian)
  • completion: support returning non-incus remotes
  • incus/image: add dynamic command line completions
  • shared/cliconfig: Support SSH encrypted keys
  • tests: Add crypto/ssh
  • doc/remotes: Add mention of keepalive
  • doc: Add Ansible to wordlist
  • doc/authentication: Hints on encrypting client key
  • api: images_all_projects
  • lxd-to-incus: Indicate what existing configuration was found
  • shared/api: Add project property to Image
  • client: Add GetImagesAllProjects
  • incusd/images: Add support for all_projects
  • doc/rest-api: Refresh swagger YAML
  • incus/image: Add --all-projects flag to list
  • i18n: Update translation templates
  • shared/cliconfig: Fix static analysis
  • gomod: Update dependencies
  • Makefile: Pin rpc2 version due to OVN issue
  • incusd/instance/qemu: Re-shuffle agent NIC handling
  • incusd/instance/qemu/agent-loader: Handle SELinux
  • incus/config: add completions
  • incus-agent: Load virtio_net before configuring NICs
  • incus: Run gofmt
  • doc/storage/lvm: Cleanup tables
  • incusd/patches: Move lvm.vg.force_reuse to be server-specific
  • incusd/db: Mark lvm.vg.force_reuse as server-specific
  • doc/faq: Add mention of the -mtu network interfaces
  • incusd/device: Bump base VM filesystem volume to 500MiB
  • incusd/storage: Create rootfs dir on empty instances
  • incusd/storage/quota: Don't fail on missing project
  • incusd/instances: Use correct project on cross-project copy
  • incusd/patches: Fix bad SQL query
  • incusd/instances: Don't bypass instance limit check
  • Added translation using Weblate (Portuguese)
  • incus-agent: Re-order imports
  • api: Add storage_bucket_backup extension
  • shared/api: Add storage bucket backup
  • incusd/db: Add storage bucket backup functions
  • incusd/db/operation: Add storage volume backup types
  • incusd/lifecycle: Add storage bucket backup events
  • incusd/project: Add StorageBucket function
  • incusd/storage/s3: Add transfer manager
  • incusd: Add storage bucket backup
  • client: Add storage bucket backup
  • incus: Add storage bucket import/export
  • doc/rest-api: Refresh swagger YAML
  • i18n: Update translation templates
  • test: Add storage bucket backup
  • client: Remove ceph-specific logic
  • incusd/storage/s3: Fix typo
  • incus: Fix import shadowing
  • incus: Fix comments on exported functions
  • client: Fix comments on exported functions
  • incusd: Fix import shadowing
  • incusd: Fix comments on exported functions
  • incusd/cluster: Make remote storage volume logic generic
  • incusd/db: Don't hardcode remote storage drivers
  • incusd/storage_volumes: Don't hardcode ceph
  • incusd/instances: Update ceph-specific comment
  • incusd/cluster: Update ceph-specific comment
  • incusd/instances: Generalize Ceph logic
  • lxd-to-incus: Support SSL authentication in OVN
  • lxd-to-incus: Don't fail on missing OVN bridge mapping
  • lxd-to-incus: Ignore OVN chassis external_ids
  • lxd-to-incus: Add boot.debug_edk2 to deprecated keys
  • lxd-to-incus: Support Void Linux
  • cmd/incusd: Disable the architecture check on incus cp/mv
  • incusd/instance: Disable the architecture check on incus cp/mv
  • api: storage_lvm_cluster
  • incus/admin/init: Add LVM cluster logic
  • i18n: Update translation templates
  • lxd-to-incus: Add LVM cluster
  • doc: Add LVM cluster
  • shared/api: Add cephobject
  • shared/api: Add lvmcluster
  • doc/rest-api: Refresh swagger YAML
  • incusd/storage/lvm: Add clustered LVM
  • incusd/storage/lvm: Fix import shadowing
  • incusd/storage/lvm: Tweak locking in cluster
  • internal/server/device/config: Don't include empty values
  • internal/server/device/config: Fix return values of Update function
  • api: Add shared_custom_block_volumes API extension
  • doc/reference: Add security.shared config key
  • instance/server/device: Validate shared block devices
  • internal/server/storage: Handle security.shared update
  • internal/server/storage: Allow security.shared key for custom block volumes
  • internal/server/storage/drivers: Handle security.shared in fillVolumeConfig
  • incusd/instance/qemu: Cap hotplug CPU slots to 64
  • incusd/storage/lvm: Make gofmt happy
  • shared/subprocess: Improve error handling
  • incusd/instance/qemu: Improve error handling
  • incusd/network/ovs: Wait for bridge interface to appear
  • incusd/storage/zfs: Fix refresh of VM volumes
  • internal/instance: Fix volatile key definitions
  • doc: Update configs
  • incusd/network/ovn: Extend validateExternalSubnet to allow uplink subnets
  • incusd/network/ovn: Add static routes to load-balancer and forwards
  • gomod: Update dependencies

Documentation

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.6

Winget package for the Incus client

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Older news