News¶

Incus 6.0 LTS has been released¶

4th of April 2024

Introduction¶

It's with great pride and pleasure that the Incus team is announcing the release of Incus 6.0 LTS!

Incus is a modern system container and virtual machine manager developed and maintained by the same team that first created LXD. It's released under the Apache 2.0 license and is run as a community led Open Source project as part of the Linux Containers organization.

Incus provides a cloud-like environment, creating instances from premade images and offers a wide variety of features, including the ability to seamlessly cluster up to 50 servers together.

It supports multiple different local or remote storage options, traditional or fully distributed networking and offers most common cloud features, including a full REST API and integrations with common tooling like Ansible, Terraform/OpenTofu and more!

This is a major milestone for Incus as it marks our first release with extended support, suitable for use in production environments where monthly feature releases aren't suitable.

It joins LXC 6.0 LTS and LXCFS 6.0 LTS in wrapping up this round of LTS releases.

Just like its sister projects, Incus 6.0 LTS will be supported until June 2029.
The first 2 years will feature bug and security fixes as well as minor usability improvements, delivered through occasional point releases (6.0.x). After that initial two years, Incus 6.0 LTS will move to security only maintenance for the remaining of its 5 years of support.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

PS: Incus was made possible thanks to the work of over 70 individual contributors!

Changes since Incus 0.7¶

Swap limits for containers¶

The existing limits.memory.swap configuration key for containers has been extended to also allow for byte amounts.

This now makes its behavior be as follows:

• limits.memory.swap=true => Container memory may be swapped (default)
• limits.memory.swap=false => Container shouldn't get swapped (minimal swappiness)
• limits.memory.swap=256MiB => Container can use up to 256MiB of swap space (in addition to its memory limit set through limits.memory)

Example (cgroup2 system):

stgraber@dakara:~$ incus launch images:debian/12 d12 -c limits.memory=1GiB
Launching d12
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m
               total        used        free      shared  buff/cache   available           
Mem:            1024          21         983           0          19        1002
Swap:              0           0           0
root@d12:~#
exit
stgraber@dakara:~$ incus config set d12 limits.memory.swap=128MiB
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m      
               total        used        free      shared  buff/cache   available
Mem:            1024          21         983           0          19        1002
Swap:            128           0         128
root@d12:~#
exit

New shell completion mechanism¶

With this release, we complete the migration away from a hand-maintained bash completion script and over to generate completion scripts directly in our command line tool.

Completion profiles are now available for:

• bash
• fish
• powershell
• zsh

The profile can be retrieved by calling incus completion <shell> (e.g. incus completion bash) though this will generally be done by packagers as part of the Incus package build process.

Creation of external bridge interfaces¶

The managed network bridge configuration syntax for external interfaces, bridge.external_interfaces has now been extended to allow for the creation and attachment of VLAN interfaces.

stgraber@dakara:~$ incus network set incusbr0 bridge.external_interfaces=vlan60/enp35s0/60
stgraber@dakara:~$ ip link show dev vlan60
269: vlan60@enp35s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master incusbr0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 00:23:a4:01:01:6f brd ff:ff:ff:ff:ff:ff
stgraber@dakara:~$ incus network unset incusbr0 bridge.external_interfaces
stgraber@dakara:~$ ip link show dev vlan60
Device "vlan60" does not exist.
stgraber@dakara:~$

Live-migration of VMs with attached disks (from remote storage)¶

As an extension to our ever improving VM live-migration support, virtual-machines with additional disks attached to them which come from a "remote" storage pool (ceph or lvmcluster) will now be live-migratable alongside the virtual machine.

No user action is required for this to happen, you'll simply now notice that virtual machines that previously would have refused to live-migrate through either a manual incus move --target or a cluster evacuation will now happily live-migrate to another server.

System information in incus info --resources¶

A new System section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
System:
  UUID: 88eecd60-34fc-9f97-48f5-fc34979f48f6
  Vendor: ASUS
  Product: System Product Name
  Family: To be filled by O.E.M.
  Version: System Version
  SKU: SKU
  Serial: System Serial Number
  Type: physical
  Chassis:
      Vendor: Default string
      Type: Desktop
      Version: Default string
      Serial: Default string
  Motherboard:
      Vendor: ASUSTeK COMPUTER INC.
      Product: ProArt B550-CREATOR
      Serial: 210382121300122
      Version: Rev X.0x
  Firmware:
      Vendor: American Megatrends Inc.
      Version: 2803
      Date: 04/28/2022

 [snip...]

Having access to this information is particularly useful in clustered environments where incus info --resources can be used with the --target argument to query specific servers, check that all firmwares are up to date and check what machines one is dealing with.

This feature was contributed by University of Texas at Austin students.

USB devices in incus info --resources¶

A new USB devices section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
[snip...]

USB devices:
  Device 0:
    Vendor: Intel Corp.
    Vendor ID: 8087
    Product: AX200 Bluetooth
    Product ID: 0029
    Bus Address: 1
    Device Address: 6
  Device 1:
    Vendor: Corsair
    Vendor ID: 1b1c
    Product: H150iRGBPROXT
    Product ID: 0c22
    Bus Address: 1
    Device Address: 5
  Device 2:
    Vendor: ASUSTek Computer, Inc.
    Vendor ID: 0b05
    Product: AURA LED Controller
    Product ID: 19af
    Bus Address: 1
    Device Address: 2
  Device 3:
    Vendor: Realtek Semiconductor Corp.
    Vendor ID: 0bda
    Product: TX42C500
    Product ID: 4933
    Bus Address: 5
    Device Address: 2
  Device 4:
    Vendor: Blue Microphones
    Vendor ID: b58e
    Product: Yeti Stereo Microphone
    Product ID: 9e84
    Bus Address: 5
    Device Address: 15
  Device 5:
    Vendor: Yubico.com
    Vendor ID: 1050
    Product: YubiKey FIDO+CCID
    Product ID: 0406
    Bus Address: 5
    Device Address: 29
  Device 6:
    Vendor: Logitech, Inc.
    Vendor ID: 046d
    Product: HD Pro Webcam C920
    Product ID: 082d
    Bus Address: 5
    Device Address: 17
  Device 7:
    Vendor: Powerware Corp.
    Vendor ID: 0592
    Product: Powerware UPS
    Product ID: 0002
    Bus Address: 7
    Device Address: 2

That information comes in very handy when adding a USB device to a container or virtual machine.

This feature was contributed by University of Texas at Austin students.

Changes since LXD 5.0 LTS¶

For those coming from the LXD 5.0 LTS release, here is a concise list of what to expect as far as features having been removed and what has been added both in subsequent LXD feature releases and then through Incus.

Feature removal¶

A number of features that were Ubuntu or Canonical specific were removed as part of the creation of the Incus project. A number of legacy APIs have also been removed at the same time.
You'll find the full list in the Incus 0.1 announcement.

Highlights:

• shiftfs has been removed in favor of VFS idmap shifting
• Canonical Candid authentication has been removed in favor of OpenID Connect
• Canonical RBAC authorization has been removed in favor of OpenFGA
• Canonical MAAS network integration has been removed (under/unused feature)
• Ubuntu Fan networking has been removed in favor of OVN
• core.trust_password has been removed in favor of trust tokens for security reasons

Feature additions¶

Here are a few highlights from the many new features introduced within the 2 years since the release of LXD 5.0 LTS.

• API
• Abiltiy to list objects across projects (?all-projects=true or --all-projects in CLI)
• JWT authentication (derived from TLS certificate)
• Instances
• Placement scriptlet
• Instance rebuilding
• READY instance state
• NUMA aware instance placement (limits.cpu.nodes)
• (CONTAINER) sysinfo system call interception (security.syscalls.intercept.sysinfo)
• (VM) CPU hotplug support (limits.cpu)
• (VM) "Online" live-migration support
• (VM) AMD SEV support (security.sev)
• (VM) Legacy (BIOS) support (security.csm)
• (VM) Ability to hot-plug directories backed disks
• (VM) NVME and VirtIO block I/O bus options
• Integrations
• Grafana Loki log and event streaming
• ACME / Let's Encrypt certificate generation/signing
• OpenID Connect authentication support
• OpenFGA authorization support
• Image server management tool
• Networking
• Network integrations (OVN interconnect support)
• Load-balancers (OVN)
• IPAM data export API
• VDPA for offloaded OVN networks
• Storage
• Clustered LVM storage driver
• Storage buckets (S3 API)
• ISO image custom volumes
• ZFS delegation
• ZFS block mode

Complete changelog¶

Here is a complete list of all changes since Incus 0.7:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Installation¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Linux packages¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.0.0

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Migrating from LXD¶

A lxd-to-incus migration tool allows for in-place migration from LXD to Incus.
It's been tested with LXD versions as low as 4.0 LTS and as high as the latest LXD 5.21 bugfix release.

It allows for a very quick migration from LXD over to Incus, automatically checking for potential conflicts ahead of time.

More details can be found here: https://linuxcontainers.org/incus/docs/main/howto/server_migrate_lxd/

Support¶

Incus 6.0 LTS will be supported for a total of 5 years (until June 2029).

During the first 2 years, new point releases will be issued including a mix of bug and security fixes as well as some minor usabiltiy improvements. After that initial 2 years (after Incus 7.0 LTS is released), Incus 6.0 LTS will transition to security fixes only for the remaining 3 years.

This matches what we've been doing for our other projects (LXC and LXCFS) over the past 10 years.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.7 has been released¶

26th of March 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.7!

This is going to be our last release before Incus 6.0 LTS which is now scheduled to be released next week. As releases go, this is quite a busy one, which is how we like it before releasing an LTS, trying to keep the amount of new features in the LTS release itself to a minimum.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features¶

Network integrations¶

A new top-level concept, network integrations are a way to connect an Incus deployment to networks outside of its own control.

Currently the only implementation of the concept is OVN interconnection which makes it possible for an Incus cluster to directly peer its OVN networks with equivalent networks running on other Incus clusters or even other OVN users like OpenStack or Kubernetes.

Here is an example of creating a new network integration using an OVN interconnection gateway, then peering an existing network through it:

root@az01-server01:~# incus network integration create ovn-region ovn
Network integration ovn-region created
root@az01-server01:~# incus network integration set ovn-region ovn.northbound_connection tcp:[10.50.1.12]:6645,tcp:[10.50.2.13]:6645,tcp:[10.50.3.19]:6645
root@az01-server01:~# incus network integration set ovn-region ovn.southbound_connection tcp:[10.50.1.12]:6646,tcp:[10.50.2.13]:6646,tcp:[10.50.3.19]:6646
root@az01-server01:~# incus network peer create default region ovn-region --type=remote
Network peer region created

Documentation: https://linuxcontainers.org/incus/docs/main/howto/network_integrations/

Image server management tool¶

A common way to run an Incus image server, be it for some internal servers or as a publicly available image server is through a static web server providing Incus images using simplestreams.

To make this easier to set up, we're now introducing a new tool, incus-simplestreams which can easily manage a simple image server, listing the images available, adding and removing images as well as generating the needed metadata files.

stgraber@dakara:~$ mkdir image-server
stgraber@dakara:~$ cd image-server/
stgraber@dakara:~/image-server$ incus-simplestreams generate-metadata ~/Downloads/incus.tar.xz
Operating system name: Red Hat Enterprise Linux
Release name: 9
Variant name [default="default"]:
Architecture name: x86_64
Description [default="Red Hat Enterprise Linux 9 (default) (x86_64) (202403260239)"]:·
stgraber@dakara:~/image-server$ incus-simplestreams add ~/Downloads/incus.tar.xz ~/Downloads/rhel9.qcow2·
stgraber@dakara:~/image-server$ incus-simplestreams list
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
|                           FINGERPRINT                            |                   DESCRIPTION                    |            OS            | RELEASE | VARIANT | ARCHITECTURE |      TYPE       |       CREATED        |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
| 7d256e4fac6fc63fb47bc1e07e1c6ee234281cdf1ed21788c920d763b7bd93ba | Red Hat Enterprise Linux 9 x86_64 (202403252239) | Red Hat Enterprise Linux | 9       | default | x86_64       | virtual-machine | 2024/03/25 00:00 UTC |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
stgraber@dakara:~/image-server$ find . | sort
.
./images
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.incus.tar.xz
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.qcow2
./streams
./streams/v1
./streams/v1/images.json
./streams/v1/index.json

Put that on an HTTPS capable web server and then add it with:

incus remote add my-server https://xyz.example.net --protocol=simplestreams

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_servers/#tooling-to-manage-a-simplestreams-server

JSON Web Token authentication¶

Incus basically supports two mechanisms for remote authentication:

• TLS client certificates (added to the local trust store with or without restrictions)
• OpenID Connect external authentication (with or without OpenFGA for authorization)

The former is the most common for simple interactions with a remote Incus server.
Our own CLI tool and most 3rd party tools don't have any problems using a TLS keypair to establish the HTTPS connection and get authenticated that way.

But there are some situations, like running Incus behind a reverse HTTP(S) proxy where TLS client certificates can become a bit problematic.

To address that, we now support using a JSON Web Token (JWT) bearer token through the HTTP Authorization field. That token can be generated by any user with a valid TLS client certificate by setting the Subject field to the certificate fingerprint, setting applicable NotBefore/NotAfter values and signing the JWT with their private key.

Incus will treat any such connections as equivalent to using the TLS client certificate.

stgraber@dakara:~$ openssl req -x509 -newkey rsa:4096 -sha384 -keyout client.key -nodes -out client.crt -days 1 -subj "/CN=test.local"
.+.........+...+...+..+....+......+........+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+......+...+......+......+..+...+...+...............+.......+...+......+...+.....+......+....+......+..+...+......+....+............+.....+....+.....+.+............+..+.........+......+....+......+...........+....+........+...+...+.+...+..+..........+.....+...+......+............+...+.......+........+....+.....+.+..+.......+......+..+....+........+..........+...+..+.+.....+.+......+..+.......+.....+.+..+..........+..+....+..............+.+..+...........................+...+....+......+...+..............+.+..+....+.....+.+.........+...+..+....+..+.............+.........+.....+...+..........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...+..+...+.........+.+...+............+..+............+.+.......................................+.....+...+......+.........+......+.+.....+...+.+...........+......+.......+.....+.......+......+.....+..........+...+..+.........+....+.........+...........+......+.+..................+..+....+...........+.............+.....+....+..+......+............+..........+......+......+......+..+.............+.....+...+.+........+............+....+.................+.........+......+.......+...+.........+.....+....+......+........+.+..+....+......+........+....+...+.................+.+..+.........+....+............+.....+.........+....+.....+.......+.....+...............+..............................+....+........+.......+...........+.......+.....+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+...............+.+...+...+...+.........+..+.+...+..+.......+.....+.+..+...+...................+...........+......+....+..+............+...+.......+..+.........+....+...+.........+.....+...+...+....+...........+...+.+.........+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+......+..+...+......+.+..+.............+..+.......+........+...+...+....+...............+..+....+..+....+...+.....+.+.....+..........+.....+.+.....+....+............+.........+...+.....+......+......+.............+..+.+..+.......+...+........+...+.......+.........+......+..+.+......+...........+...+.........+...+...+....+..+.........+....+.....+.+......+.........+..+..........+..+...+...+....+..+...+.+.....+......+.+...+......+.....+.+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
stgraber@dakara:~$ incus config trust add-certificate client.crt --restricted --projects demo
stgraber@dakara:~$ tls2jwt client.key client.crt now 120
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0stgraber@dakara:~$·
stgraber@dakara:~$ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0' https://localhost:8443/1.0/projects | jq
{
  "type": "sync",
  "status": "Success",
  "status_code": 200,
  "operation": "",
  "error_code": 0,
  "error": "",
  "metadata": [
    "/1.0/projects/demo"
  ]
}

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#using-json-web-token-jwt-to-perform-tls-authentication

Configurable OIDC username field¶

For those using OpenID Connect, you may have noticed that Incus will use the e-mail claim when available as the user's identifier. Then if missing, it will rely on the Subject.

As different deployments may make different information available through OIDC claims, it's now possible to set oidc.claim to the claim to use as the user identifier.

stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org
stgraber@dakara:~$ incus config set oidc.claim=name
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
Stéphane Graber
stgraber@dakara:~$ incus config set oidc.claim=sub
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
99cb8caa-3640-45b9-b87a-55266366aaf3
stgraber@dakara:~$ incus config set oidc.claim=email
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org

Improved NUMA handling¶

With this release, we spent a fair amount of time trying to improve both the container and virtual-machine performance on large systems. This obviously includes multi-socket systems but also AMD systems running in NPS4 or similar mode where each CPU is exposed as multiple NUMA nodes.

In general, our goal has been to make it easy to distribute workloads across NUMA nodes while keeping their CPU and memory properly pinned and also selecting PCIe resources that are closest to their NUMA node(s).

As part of that, a few things were done:
- limits.cpu.nodes is now supported for virtual-machines too
- A new balanced value has been added to limits.cpu.nodes which will have Incus pick the NUMA node with the least instances configured to use it
- SR-IOV GPU selection now also considers NUMA nodes as part of the selection logic and when no match is found, will prefer PCIe devices that are attached to the same CPU socket

For example:

stgraber@gputest:~$ incus list stgraber-gpu -cns4,limits.cpu.nodes,volatile.cpu.nodes,volatile.gpu.last_state.pci.parent,volatile.gpu.last_state.vf.id
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
|      NAME      |  STATE  |         IPV4          | LIMITS CPU NODES | VOLATILE CPU NODES | VOLATILE GPU LAST STATE PCI PARENT | VOLATILE GPU LAST STATE VF ID |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu01 | RUNNING | 10.232.44.8 (enp5s0)  | balanced         | 0                  | 0000:63:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu02 | RUNNING | 10.232.44.9 (enp5s0)  | balanced         | 2                  | 0000:03:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu03 | RUNNING | 10.232.44.10 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu04 | RUNNING | 10.232.44.11 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu05 | RUNNING | 10.232.44.12 (enp5s0) | balanced         | 6                  | 0000:c3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu06 | RUNNING | 10.232.44.13 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu07 | RUNNING | 10.232.44.15 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu08 | RUNNING | 10.232.44.16 (enp5s0) | balanced         | 2                  | 0000:03:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu09 | RUNNING | 10.232.44.17 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu10 | RUNNING | 10.232.44.18 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu11 | RUNNING | 10.232.44.19 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu12 | RUNNING | 10.232.44.20 (enp5s0) | balanced         | 6                  | 0000:83:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu13 | RUNNING | 10.232.44.21 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu14 | RUNNING | 10.232.44.22 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu15 | RUNNING | 10.232.44.23 (enp5s0) | balanced         | 2                  | 0000:43:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu16 | RUNNING | 10.232.44.24 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+

In this case we can see 16 VMs each using the new balanced option for NUMA nodes and getting scheduled across 8 NUMA nodes (2 sockets AMD NPS4) with GPUs being selected to match.

More options to select USB devices¶

USB device passhtrough for both containers and virtual-machines has so far been using the vendorid and productid fields. This works fine as long as there is only one USB device of any one type connected to the system.

When multiple identical devices are present, the inability to distinguish them has been a problem.

To address this, three new fields have now been added to usb devices:
- busnum referring to the USB bus number
- devnum referring to the USB device number (on its bus)
- serial referring to the USB device serial number (not present on all devices)

The same fields can be found in the full Incus resources list through:

incus query /1.0/resources

Disk I/O throttling for VMs¶

One more feature gap between containers and virtual-machines is now gone.

The limits.write and limits.read properties on disk devices are now properly enforced on virtual-machines by having Incus setup an I/O throttle in QEMU.

Both bytes per second and I/O per second type limits are supported.

Per-remote client certificates¶

It's now possible to put a <remote>.crt and <remote>.key file in a new clientcerts folder within the Incus command line client config directory (typically ~/.config/incus/) and have those certificates be used when interacting with that particular remote.

While this may be useful on its own, it becomes a lot more useful when combined with global remotes, which can be added in /etc/incus/config.yml. Now with this feature, those global remotes can also have a client certificate made available in /etc/incus/clientcerts/ which will then be used by all users on the system.

Manual generation of the client certificate keypair¶

A new command to manually trigger the generation of the main client.crt and client.key keypair is now available.

This is done by running incus remote generate-certificate

Improvements to lxd-to-incus¶

The lxd-to-incus tool keeps evolving with every release.

In this one, it gains support for migrating users from the newly released LXD 5.21 LTS as well as handling Alpine installations.

Additionally, a static binary version of the tool is now available on Github, making it easier for users to fetch the latest version of the tool, useful as bugs get fixed in between Incus releases.

Improvements to incus-migrate¶

The workload migration tool incus-migrate has also seen a couple of small improvements.

It can now use the local Incus system as the target of the migration, useful when importing virtual-machine images from another virtualization tool.

And it's also now prompting for whether the imported virtual machine should be using a UEFI firmware or instead use a legacy BIOS.

Additional image restrictions¶

A bit of an internal detail, or at least only relevant to public image server operators, but two new image restrictions have been added:

• requirements.nesting which will require the container have security.nesting=true
• requirements.cdrom_agent which will require that a source=agent:config disk device be added to the virtual-machine

Those two can be used to flag specific images that need extra user interaction to work properly, resulting in a clear client-side error rather than starting a potentially broken instance.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.7

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.6 has been released¶

23rd of February 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.6!

This second release of 2024 features a number of improvements, both large and small.

It comes with a completely new storage driver for cluster users, import/export support for storage buckets, a number of improvements for OVN users, support for new container kernel features, quite a few improvements to the migration tooling and more!

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features¶

Clustered LVM storage driver¶

A new storage driver named lvmcluster has been added.

As the name implies, this is for clustered use of the LVM storage driver.
This relies on llvmlockd and a lock manager like sanlock to provide concurrent access to a shared storage device across a cluster.

This is aimed at anyone who wants to run an Incus cluster and use a shared storage device like a FiberChannel SAN, iSCSI export, NVMEoF/NVMEoTCP disk, ...

With the ability of having the same storage available on all servers comes support for much faster live migrations, server evacuations and the ability to automatically recover should a server suddenly die.

Documentation: https://linuxcontainers.org/incus/docs/main/reference/storage_lvm/#lvmcluster-driver-in-incus

Storage bucket backup and import¶

It is now possible to backup a full storage bucket and later re-import back into Incus.

stgraber@dakara:~$ incus storage bucket create default foo
Storage bucket foo created
Admin access key: HD2GWC1KX693MFVT3P30
Admin secret key: tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv

stgraber@dakara:~$ dd if=/dev/random of=out.img bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0975758 s, 430 MB/s
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv put out.img s3://foo
upload: 'out.img' -> 's3://foo/out.img'  [part 1 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   262.42 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 2 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   241.54 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 3 of 3, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s   241.99 MB/s  done
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:26     41943040  s3://foo/out.img

stgraber@dakara:~$ incus storage bucket export default foo
Backup exported successfully!
stgraber@dakara:~$ incus storage bucket delete default foo
Storage bucket foo deleted

stgraber@dakara:~$ incus storage bucket import default backup.tar.gz
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:27     41943040  s3://foo/out.img

API: https://linuxcontainers.org/incus/docs/main/rest-api-spec/#/storage/storage_pool_buckets_backups_post

Listing image across all projects¶

Just as it's possible to list instances across all projects using the --all-projects, it's now possible to do the same but for images.

stgraber@dakara:~$ incus image list --all-projects
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| PROJECT | ALIAS | FINGERPRINT  | PUBLIC |               DESCRIPTION                | ARCHITECTURE |      TYPE       |   SIZE    |          UPLOAD DATE          |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 256f59a72af5 | no     | Ubuntu jammy amd64 (20240222_07:42)      | x86_64       | VIRTUAL-MACHINE | 267.19MiB | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 0941e441dbb9 | no     | Alpine edge amd64 (20240222_13:00)       | x86_64       | CONTAINER       | 2.93MiB   | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | d5fc6024f0fa | no     | Openwrt snapshot amd64 (20240222_11:57)  | x86_64       | CONTAINER       | 3.50MiB   | Feb 23, 2024 at 2:55am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| demo    |       | f44a6b4e56f4 | no     | Archlinux current amd64 (20240222_04:18) | x86_64       | CONTAINER       | 188.86MiB | Feb 23, 2024 at 3:32am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+

Over the API, this is done by passing ?all-projects=true.

binfmt_misc in unprivileged containers¶

Linux 6.7 added support for mounting of binfmt_misc inside of unprivileged containers.

Incus 0.6 will detect kernels that support this feature and when they do, it will no longer bind-mount binfmt_misc from the host system but instead allow it to be mounted from within the container.

stgraber@castiana:~$ incus launch images:ubuntu/22.04 foo
Launching foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# uname -a
Linux foo 6.7.4-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Mon Feb  5 23:37:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@foo:~# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/
root@foo:~# ls -lh /proc/sys/fs/binfmt_misc/
total 0
--w------- 1 root root 0 Feb 23 03:46 register
-rw-r--r-- 1 root root 0 Feb 23 03:46 status

The binfmt_misc filesystem is used to define custom binary formats for emulation.
It's most notably used by qemu-user-static as a way to run binaries of other architectures.

Control over shared block storage volumes¶

A new security.shared volume storage option has been introduced to control whether a custom block volume should be allowed to be used by multiple instances.

It is now required to set that configuration key prior to adding a custom block volume to a profile or add it directly to multiple virtual machines.

stgraber@castiana:~$ incus config device add lvm01 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm01

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Error: Failed add validation for device "shared": Cannot add un-shared custom storage block volume to more than one instance

stgraber@castiana:~$ incus storage volume set default shared-lvm security.shared=true

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm02
stgraber@castiana:~$ incus config device add lvm03 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm03
stgraber@castiana:~$ incus start lvm01 lvm02 lvm03

OVN logical router name in network info¶

To make it easier to see what's going on within OVN, incus network info now gives you the name of the logical router for a particular network.

root@abydos:~# incus network info default
Name: default
MAC address: 00:16:3e:38:dd:28
MTU: 1500
State: up
Type: broadcast

IP addresses:
  inet  10.180.103.1/24 (link)
  inet6 2602:fc62:a:1004::1/64 (link)

Network usage:
  Bytes received: 0B
  Bytes sent: 0B
  Packets received: 0
  Packets sent: 0

OVN:
  Chassis: abydos
  Logical router: incus-net6-lr

File ownership and permissions in image templates¶

Template files can now have a uid, gid and mode set on them.
This can be particularly useful if a template is meant to be an executable shell script.

stgraber@castiana:~$ incus config metadata show foo
architecture: amd64
creation_date: 1708588077
expiry_date: 1711180077
properties:
  architecture: amd64
  description: Ubuntu jammy amd64 (20240222_07:42)
  name: ubuntu-jammy-amd64-default-20240222_07:42
  os: ubuntu
  release: jammy
  serial: "20240222_07:42"
  variant: default
templates:
  /etc/hostname:
    when:
    - create
    - copy
    create_only: false
    template: hostname.tpl
    properties: {}
  /etc/hosts:
    when:
    - create
    - copy
    create_only: false
    template: hosts.tpl
    properties: {}
  /root/hello.sh:
    when:
    - start
    create_only: false
    template: hello.tpl
    properties: {}
    uid: "1000"
    gid: "2000"
    mode: "0755"

stgraber@castiana:~$ incus config template show foo hello.tpl
#!/bin/sh
echo "Hello world!"

stgraber@castiana:~$ incus start foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# ls -lh /root/hello.sh
-rwxr-xr-x 1 ubuntu 2000 30 Feb 23 04:07 /root/hello.sh
root@foo:~# /root/hello.sh
Hello world!

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_format/#template-rules

Encrypted EC client certificate keys¶

For those interacting with remote Incus servers, you may not know that it's possible to protect your Incus private key with a password.

Up until now, this was only possible for RSA keys, but with Incus 0.6, we're now adding support for EC keys too. That's particularly relevant as EC keys have been the default for a while now.

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#encrypting-local-keys

Worth noting that if using this feature, you'll likely also want to make use of the recently introduced "keepalive mode" as this will then significantly reduce the number of password prompts you'll get while using Incus.

lxd-to-incus improvements¶

Lastly, lxd-to-incus saw a number of improvements:

• Support for Void Linux
• Detection of the boot.debug_edk2 configuration key
• Handling of OVN SSL database connections
• Automatic clearing of the simplestreams cache during migration

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.6

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.5.1 has been released¶

29th of January 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.5.1!

This is an unusual release as we normally do not issue point releases on top of the monthly feature releases. But we felt this was needed this time due to some pretty important bugfixes and a minor feature addition needed to accommodate those running CentOS/Alma/Rocky virtual machines.

Most changes are on the server side, so if you're only using the command line client, there is no strong reason to upgrade from 0.5 to 0.5.1.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

Highlights¶

Alternative way to get the VM agent¶

With Incus 0.5, the distribution mechanism for the Incus VM agent changed a bit.
In the past, we had a single share named config which would include both the instance-specific agent configuration and the incus-agent binary.

This was a bit wasteful, requiring a copy of the 15-20MB large incus-agent for every VM but was still somewhat manageable. This share was also exposed as both 9p and virtiofs. Leading to two processes running on the host system for every Incus VM.

With support for multiple agent binaries, copying them for every VM really wasn't an option anymore, so a separate share was introduced just for the binaries. As we really didn't want to end up with another two processes running on the host per VM, we made the decision to only make those internal shares be available over 9p.

Testing on a variety of images, including CentOS 7 showed that this would be fine.
9p is lower performance than virtiofs but as those shares are only use for a couple of seconds on every VM boot, that really wasn't a concern. User defined shares would still be exposed over virtiofs so those would still get the high performance option.

What we failed to notice is that for some reason, CentOS 8-Stream, CentOS 9-Stream and other distributions that are derivatives of RHEL 8/9, do not ship the 9p kernel driver at all...

This means that those instances no longer had a way to fetch an agent, leading to broken incus exec and incus file.

We still don't feel like running 4 host processes for every single Incus VM just to make things work on those few images. Instead, what we're introducing with Incus 0.5.1 is a new agent drive, effectively an extra disk which can be attached to those specific VMs, providing those files through what looks like a CD-ROM drive rather than being retrieved over a networked filesystem.

So to run CentOS 9-Stream, one now needs to do:

incus create images:centos/9-Stream centos --vm
incus config device add centos agent disk source=agent:config
incus start centos

If you run many such VMs, a better option is likely by creating a profile for it:

incus profile create vm-agent
incus profile device add vm-agent agent disk source=agent:config

At which point you can do:

incus launch images:centos/9-Stream centos --vm -p default -p vm-agent

This is obviously not ideal and adds a few more steps when creating VMs for those distributions but this new mechanism now offers a way to get the agent up and running in just about any environment.

NOTE: We're not considering always providing that extra device as it takes some resources to generate the cdrom device and uses some extra disk on the host. So it's best added only when needed.

Fixed handling of stopped instances during evacuation¶

A bug introduced with Incus 0.5 was causing stopped instances to get relocated to other systems during evacuation, even if the instance was configured to remain where it was.

This has now been corrected and instances using stopped, force-stop or stateful-stop are now guaranteed to remain on their current server.

Database performance fixes¶

Database improvements in Incus 0.5 accidentally caused some nested database transactions to occur when fetching network information details for a large number of instances.

This would only really become visible when using an Incus cluster that also serves DNS zones and has its metrics scraped by Prometheus. This combination would cause large spikes in API requests every 15s or so, which would then start triggering timeouts and retries, eventually leading to other API requests piling up and timing out.

The logic has now been changed to remove such nested transactions and further optimizations were also made to save some database interactions during very command API interactions like executing commands instance of instances.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.5.1

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.5 has been released¶

26th of January 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.5!

This is our first release of 2024 and it's quite a busy one! It's also the first release to feature no change coming from LXD following their decision to re-license to AGPLv3.

This release comes with a number of welcome improvements to the Incus CLI, a number of new virtual machine features, more options to handle cluster evacuations and host shutdown and some other smaller features and improvements!

On top of that, we've got quite a lot of bugfixes as well as a number of database improvements which should yield noticeable performance improvements especially in clusters.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

Highlights¶

Ansible, Terraform/OpenTofu and Packer¶

Over the past few months, Incus support has grown quite a bit in common tools!

• A connection plugin for Incus can now be found in Ansible.
• We have an official provider for Terraform and OpenTofu
• There is an available Packer plugin for Incus

Linux distribution packages¶

Since the last release of Incus, additional packages are now available for:

• Arch Linux
• Debian (testing/unstable)
• Ubuntu (noble)
• Void Linux

You'll find all instructions in our installation guide.

Translations¶

We've spent a bit of time cleaning up translations and setting up Weblate for Incus.
It's now easier than ever to log into Weblate and translate the Incus CLI into your language.
All changes are automatically submitted for inclusion through Github.

Upgrade notes¶

subuid/subgid entries¶

A longstanding bug in the idmap parser was causing everything but the first large entry for the root user to be discarded when parsing /etc/subuid and /etc/subgid.

This was then causing issues for the few users that have a legitimate reason to split their uid/gid allocation in half, mostly those using remote authentication on the host system.

This bug has been resolved, but this has a side effect of making Incus containers to fail to start on systems with an invalid subuid/subgid configuration.

If you notice that your containers won't start anymore, go look at /etc/subuid and /etc/subgid and make sure that there is one large entry for the root user, it must be at least 65536 uid/gid large. More importantly, make sure that there is no conflict/overlap in allocations given to the root user.

In most cases, the easiest is to remove all the root entries from those two files and replace them with a single very large entry:

root:1000000:1000000000

New features¶

New incus file create command¶

A new incus file create command was added which provides an easy way to create empty files, symlinks and directories without having to transfer an existing local directory tree.

stgraber@dakara:~$ incus file create demo/root/file
stgraber@dakara:~$ incus file create --type=symlink demo/root/symlink /etc/hosts
stgraber@dakara:~$ incus file create --type=directory demo/root/dir
stgraber@dakara:~$ incus exec demo -- ls -lh /root
total 2.5K
drwxr-xr-x 2 root root  2 Jan 26 03:38 dir
-rw-r--r-- 1 root root  0 Jan 26 03:37 file
lrwxrwxrwx 1 root root 10 Jan 26 03:38 symlink -> /etc/hosts

New incus snapshot show command¶

A new incus snapshot show command makes it easy to look at the configuration data that's included as part of an Incus instance snapshot.

As a reminder, Incus snapshots don't only contain the filesystem state, but also include all the instance configuration (config keys, devices, ...) at the time of the snapshot.

stgraber@dakara:~$ incus snapshot create demo s1
stgraber@dakara:~$ incus snapshot list demo
+------+----------------------+----------------------+----------+
| NAME |       TAKEN AT       |      EXPIRES AT      | STATEFUL |
+------+----------------------+----------------------+----------+
| s1   | 2024/01/25 22:39 EST | 0000/12/31 19:03 LMT | NO       |
+------+----------------------+----------------------+----------+
stgraber@dakara:~$ incus snapshot show demo s1
expires_at: 0001-01-01T00:00:00Z
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20240125_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20240125_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: f9e9abeb4fc8691edf48078616a1aae628c6d5938b715e361c6b47cda0474679
  volatile.cloud-init.instance-id: f724feba-245a-424b-bc51-43167258dc2a
  volatile.eth0.host_name: vethecbb346e
  volatile.eth0.hwaddr: 00:16:3e:06:67:f0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8b8a1c75-977b-4197-9ad7-507b899432e9
  volatile.uuid.generation: 8b8a1c75-977b-4197-9ad7-507b899432e9
created_at: 2024-01-26T03:39:09.583020489Z
devices: {}
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20240125_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20240125_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: f9e9abeb4fc8691edf48078616a1aae628c6d5938b715e361c6b47cda0474679
  volatile.cloud-init.instance-id: f724feba-245a-424b-bc51-43167258dc2a
  volatile.eth0.host_name: vethecbb346e
  volatile.eth0.hwaddr: 00:16:3e:06:67:f0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8b8a1c75-977b-4197-9ad7-507b899432e9
  volatile.uuid.generation: 8b8a1c75-977b-4197-9ad7-507b899432e9
expanded_devices:
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: s1
profiles:
- default
stateful: false
size: 53760

More shell completion options¶

We're slowly transitioning from a single hand-written bash completion script for the incus command line tool, to instead using a much more dynamic way of handling shell completion.

Initial shell completion profiles can be retrieved with:

• incus completion bash
• incus completion fish
• incus completion powershell
• incus completion zsh

Bash users are probably still better off using the hand-written completion script at this point, but we're hopeful that the new dynamically generated completion profiles will take over in the next release or two.

Support for multiple VM agent binaries¶

It's now possible for Incus to provide multiple agent binaries to its virtual machines.

This is useful in two scenarios:

• Handling multiple operating systems
• Handling multiple architectures

At this stage, the focus is on multiple architectures. With this new ability, you can now have 32bit virtual machines running on your system and have them fetch a 32bit of the agent binary.

stgraber@castiana:~$ incus exec debian32 bash
root@debian32:~# uname -a
Linux debian32 6.1.0-17-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) i686 GNU/Linux
root@debian32:~# 
exit
stgraber@castiana:~$ incus exec debian32 bash
root@debian32:~# uname -m
i686
root@debian32:~# mount -t 9p agent /mnt
root@debian32:~# ls -lh /mnt
total 34M
-rwxr-xr-x 1 root root 17M Jan 24 10:10 incus-agent.linux.i686
-rwxr-xr-x 1 root root 18M Jan 24 10:10 incus-agent.linux.x86_64

Support for virtio-blk as a disk io.bus¶

After adding NVME support in Incus 0.2, we're now expanding that mechanism to also offering virtio-blk as a disk I/O bus in our virtual machines.

To use it, set the io.bus property on the disk device to be virtio-blk.

stgraber@dakara:~$ incus launch images:debian/12 demo --vm
Launching demo
stgraber@dakara:~$ incus storage volume create default demo size=5GiB --type=block
Storage volume demo created
stgraber@dakara:~$ incus config device add demo extra disk pool=default source=demo io.bus=virtio-blk
Device extra added to demo
stgraber@dakara:~$ incus exec demo bash
root@demo:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk 
├─sda1   8:1    0  100M  0 part /boot/efi
└─sda2   8:2    0  3.9G  0 part /
vda    253:0    0    5G  0 disk

Support for USB network device pass-through in VMs¶

When using nictype=physical for a virtual machine with the parent network device being connected over the USB bus, Incus will now detect the situation and internally convert this into a USB device pass-through to the virtual machine.

stgraber@castiana:~$ incus launch images:debian/12 demo --vm
Launching demo
stgraber@castiana:~$ incus config device add demo eth1 nic nictype=physical parent=enx207bd2a0f9eb
Device eth1 added to demo
stgraber@castiana:~$ incus exec demo bash
root@demo:~# apt install usbutils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libusb-1.0-0
The following NEW packages will be installed:
  libusb-1.0-0 usbutils
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 142 kB of archives.
After this operation, 492 kB of additional disk space will be used.
Do you want to continue? [Y/n]·
Get:1 http://deb.debian.org/debian bookworm/main amd64 libusb-1.0-0 amd64 2:1.0.26-1 [62.6 kB]
Get:2 http://deb.debian.org/debian bookworm/main amd64 usbutils amd64 1:014-1 [79.7 kB]
Fetched 142 kB in 1s (124 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libusb-1.0-0:amd64.
(Reading database ... 20425 files and directories currently installed.)
Preparing to unpack .../libusb-1.0-0_2%3a1.0.26-1_amd64.deb ...
Unpacking libusb-1.0-0:amd64 (2:1.0.26-1) ...
Selecting previously unselected package usbutils.
Preparing to unpack .../usbutils_1%3a014-1_amd64.deb ...
Unpacking usbutils (1:014-1) ...
Setting up libusb-1.0-0:amd64 (2:1.0.26-1) ...
Setting up usbutils (1:014-1) ...
Processing triggers for libc-bin (2.36-9+deb12u3) ...
root@demo:~# lsusb -tv
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 4: Dev 2, If 0, Class=Communications, Driver=cdc_ncm, 5000M
        ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
    |__ Port 4: Dev 2, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
        ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
root@demo:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:e7:f7:2d brd ff:ff:ff:ff:ff:ff
3: enx207bd2a0f9eb: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 20:7b:d2:a0:f9:eb brd ff:ff:ff:ff:ff:ff

New cluster evacuation options (force-stop and stateful-stop)¶

A couple of new cluster evacuation options have been added.

Those can be selected on a per-instance basis through the cluster.evacuate instance configuration key.

force-stop causes the instance to be immediately stopped without giving it a chance at a clean shutdown. This only really makes sense in cases where the instance is effectively stateless as it won't have any chance to flush any ongoing state to disk.

stateful-stop causes the instance's state to be written to disk before stopping the instance. On restore, the instance state is restored too, causing the instance to just continue where it left off.
This option is currently primarily targeted at virtual machines as stateful stop for containers is quite difficult to achieve.

Ability to configure the host instance shutdown action¶

A new instance configuration key, boot.host_shutdown_action, has been introduced which supports:

• stop (normal shutdown behavior)
• force-stop (see above)
• stateful-stop (see above)

This makes it particularly easy to have a number of virtual machines going through stateful stop on host shutdown and then being restored on boot.

Ability to start instances as part of creation¶

A small API optimization was made which now allows for instances to be started as part of the creation request, saving an API call and making it easier for those scripting the Incus API.

incus launch now makes use of this too.

Configurable Loki instance name¶

When sending events to Loki, Incus provides a set of default labels.

Those include both an instance and a location label .Worth noting that here instance refers to the Loki event source instance, not an Incus instance.

So far, those would only differ in the somewhat unlikely event that a server would be forwarding an event originating from another server in a cluster.

Instead, in clustered environments, it makes a lot more sense to have a way to provide a cluster name of some kind, so that if multiple clusters use the same Loki instance, they can easily be filtered.

To that effect, we've introduced a new loki.instance server configuration key which, when set, will override the instance label.

The default Grafana dashboard has also been updated to filter the Loki events under the assumption that the Loki instance label will match the Prometheus job name.

Extended HEAD support on files¶

The HEAD method on the Incus instance file API now returns the file size through the Content-Length header.

The primary use for this is for those building some kind of file manager on top of the Incus instance file API as it now allows for not just showing the name and file type but also the size of any regular files.

Use of /run/incus for runtime data¶

Up until now, Incus has stored some amount of runtime data in the instance log directories under /var/log/incus/. Other than it obviously not being the correct location for it, it was also causing some issues with systems that aggressively rotate and expire log files.

To solve this, Incus 0.5 will now place runtime data in /run/incus, keeping /var/log/incus only for actual log files.

In the future, more data will likely be relocated from /var/lib/incus to /run/incus as well.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.5

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Older news¶

• 21st of December 2023
• 27th of November 2023
• 28th of October 2023
• 7th of October 2023