News¶

Incus 0.3 has been released¶

27th of November 2023

Introduction¶

The Incus team is pleased to announce the release of Incus 0.3!

This isn't a very busy release as a good chunk of the Incus team has been traveling to the Linux Plumbers Conference in Richmond, VA.

The most exciting new feature is likely the addition of OpenFGA support as when combined with an OpenID connect provider, this now allows for a fully open source identity and authorization stack. This also removes the last blocker for some waiting to migrate to Incus from LXD (with Canonical RBAC).

On top of that, a lot of improvements have gone into the lxd-to-incus migration tool and we've also added support for hot-plug/hot-remove of shared paths in virtual machines!

You can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features and highlights¶

OpenFGA support for authorization control¶

OpenFGA is an open source authorization solution which is designed to be very easy to integrate with while still offering extremely good performance.

It's basically an external daemon that you run on your network and which will get asked by Incus whether to allow a specific user to perform a specific action.

You can learn more about OpenFGA here: https://openfga.dev/

On the Incus side of things, OpenFGA is enabled through a new set of server configuration keys:

• openfga.api.token
• openfga.api.url
• openfga.store.id
• openfga.store.model_id

You'll want to set those to point to your OpenFGA instance and then configure an OIDC provider for authentication. Once done, OpenFGA will be queried whenever a user request is received.

Documentation: https://linuxcontainers.org/incus/docs/main/authorization/#open-fine-grained-authorization-openfga

This feature was first introduced in LXD.

lxd-to-incus improvements¶

The lxd-to-incus migration tool has seen a lot of improvements:

• Support for OpenRC target systems
• Detection and handling of mounts on the daemon path
• Support for migrating Ceph storage pools
• Support for migrating OVN networks
• Generation of a log file
• Generation of backups (database, OVN data)

Note that as LXD is late in releasing LXD 5.20, the migration tool only supports up to LXD 5.19 as a source release. Packagers should update that to 5.20 once LXD releases and assuming no last minute changes that would break the migration.

Hot-plug/hot-remove of paths in virtual machines¶

Incus has had support for hot-plug and hot-remove of disks for a little while.
With those, you see a virtual disk appearing or disappearing inside the VM.

But Incus also supports sharing just a path from the host system or passing in a shared custom volume (filesystem) to the instance.

This is handled through virtio-fs or 9p and up until now, required the VM to be stopped, the device added and then the VM started.

But that's now a thing of the past and Incus now supports hot-plug and hot-remove of those paths through a combination of PCI hotplug in QEMU and communication with the incus-agent in the guest to perform the actual mount as part of the hot-plug process.

Worth noting that the agent will not automatically unmount the filesystem prior to hot-remove. If the filesystem is mounted in the guest, you'll get an error during hot-remove.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Zabbly packages for Debian and Ubuntu¶

Zabbly provides both daily and stable builds of Incus to Debian and Ubuntu users:
https://github.com/zabbly/incus

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool will soon be available through Chocolatey for Windows users.

Until then, binaries can be found here: https://github.com/lxc/incus/releases/tag/v0.3.0

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.2 has been released¶

28th of October 2023

Introduction¶

The Incus team is pleased to announce the release of Incus 0.2!

This version incorporates most changes that went into LXD 5.19 as well as introduce a few additional features and improvements.

You can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features and highlights¶

NVME storage support in virtual machines¶

A new io.bus configuration key was added to disk type devices of virtual-machines.

This defaults to virtio-scsi but can also now be set to nvme in order to have the disk appear as an NVME SSD inside the virtual machine.

Cluster support for migration from LXD¶

The lxd-to-incus migration tool now supports clustered environments.
Additionally, it's also been updated to support LXD 5.19 as a source release.

This means that anyone on LXD version 4.0 and higher (up until 5.19) can now easily move over to Incus by installing Incus and running lxd-to-incus!

https://media.hachyderm.io/media_attachments/files/111/299/784/894/279/245/original/980eafd9c3450216.mp4

New image requirement for unprivileged containers¶

When adding support for NixOS as a container image, it came out that this particular image cannot currently work inside of a privileged container.

Rather than just let it silently fail for those users, a new image requirement was added.
requirements.privileged can be set to false in order to prevent the image from being used with a privileged container.

stgraber@dakara:~$ incus launch images:nixos nixos-priv -c security.privileged=true
Creating nixos-priv
Starting nixos-priv
Error: The image used by this instance is incompatible with privileged containers. Please unset security.privileged on the instance
Try `incus info --show-log local:nixos-priv` for more info
stgraber@dakara:~$

Server-side custom volume copy¶

Incus now supports server-side copies of custom volumes. This significantly speeds up copies of custom volumes by eliminating the need for the client to act as a relay.

The command line tool automatically detects support for this and uses it when available.

This feature was first introduced in LXD.

Static binaries now available for 64-bit Arm¶

All static binaries provided as part of our releases and tests are now provided for both Intel 64-bit as well as Arm 64-bit.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Zabbly packages for Debian and Ubuntu¶

Zabbly provides both daily and stable builds of Incus to Debian and Ubuntu users:
https://github.com/zabbly/incus

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool will soon be available through Chocolatey for Windows users.

Until then, binaries can be found here: https://github.com/lxc/incus/releases/tag/v0.2.0

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.1 has been released¶

7th of October 2023

Introduction¶

The Linux Containers team is very excited to announce the initial release of Incus!

Incus is a community fork of Canonical LXD, created by @cyphar and now part of the Linux Containers project.

This initial release is roughly equivalent to LXD 5.18 but with a number of breaking changes on top of the obvious rename.

You can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Changes¶

With this initial release of Incus, we took the opportunity to remove a lot of unused or problematic features from LXD. Most of those changes are things we would have liked to do in LXD but couldn't due to having strong guarantees around backward compatibility.

Incus will be similarly strict with backward compatibility in the future, but as this is the first release of the fork, it was our one big opportunity to change things.

That said, the API and CLI are still extremely close to what LXD has, making it trivial if not completely seamless to port from LXD to Incus.

API¶

Removal of /1.0/containers and /1.0/virtual-machines¶

LXD started as a container-only project and therefore its REST API used /1.0/containers.

When virtual-machines were then introduced, a new /1.0/instances endpoint took over for all operations across both containers and virtual-machines, but /1.0/containers was kept around for legacy clients. On top of that, a convenience /1.0/virtual-machines endpoint was also added, though never used.

With Incus, those two legacy endpoints are now removed and the only supported way to interact with instances is through /1.0/instances.

Replacement of /dev/lxd with /dev/incus¶

As part of the renaming of everything from LXD to Incus, the guest API was also renamed from /dev/lxd to /dev/incus.

Convenience symlinks are currently provided so existing workloads will keep working after a migration from LXD.

Type change for the server configuration¶

The LXD server configuration has always been a bit oddly typed as map[string]any rather than our usual map[string]string. The reason for that one inconsistency was because of core.trust_password, the legacy authentication method used by LXD in the early days.

That's because LXD never stored the password in clear text but instead would hash it and store the hash instead. This made it impossible to return the full configuration to the user. Instead when a password would be set, a true boolean would be returned rather than a string.

Now as mentioned, core.trust_password is a legacy way to handle authentication and should be replaced by either token based authentication, TLS certificate trust or external authentication.

Incus has removed support for core.trust_password completely (see below) and so all server config keys can now fit in a standard map[string]string.

Removal of legacy Container functions from the Go API¶

Following the above change of removing /1.0/containers and /1.0/virtual-machines, the matching functions in the Go client package are also all gone now.

Instead the Instance equivalents of those Container functions should be used.
(e.g. CreateInstance instead of CreateContainer)

CLI¶

Introduction of incus snapshot sub-command¶

A long term annoyance and inconsistency in the LXD CLI has been around snapshot management. While most other operations get their own sub-command (file, config, ...), snapshots were kept at the top-level with snapshot and restore.

This was then forcing us to handle things like renaming or deleting snapshots through the lxc rename and lxc delete functions. Instead having an lxc snapshot sub-command with lxc snapshot create would have made a lot more sense but couldn't be done without breaking all scripts using lxc snapshot.

Now is time to fix this, so incus snapshot is now a sub-command, featuring:

• incus snapshot create
• incus snapshot delete
• incus snapshot list
• incus snapshot rename
• incus snapshot restore

Handling of incus config trust add and incus cluster add¶

With Incus doing away with password based authentication and focusing a lot more on tokens, it was important to make the experience of issuing tokens be easy and consistent.

As a result, both incus config trust add and incus cluster add now simply take a name as argument and return a valid token.

The certificate handling aspect of incus config trust add has now been moved to incus config trust add-certificate instead.

Introduction of incus admin sub-command¶

Another long time annoyance in the LXD world was the fact that both lxc and lxd had to be used to set up a server. With incus, the incusd binary can be safely hidden away as nobody should ever have to directly interact with it.

Instead, we now have the incus admin sub-command:

• incus admin cluster
• incus admin init
• incus admin recover
• incus admin shutdown
• incus admin waitready

Dependencies¶

Transition to Cowsql¶

https://github.com/cowsql/cowsql

Shortly after @cyphar forked LXD as Incus, @freeekanayaka, the original author of Dqlite, also decided to fork Dqlite as Cowsql. Given that @freeekanayaka is also a maintainer on Incus, it made sense to port incus over to Cowsql.

Similarly to Incus itself, Cowsql is a community fork of Canonical Dqlite, largely compatible with Dqlite, allowing Incus to easily import existing data during the migration stage.

Focus on Cowsql so far has been on dramatically improving performance testing and making a number of changes across the Raft and Cowsql layers to make it as performant as possible for Incus' usage pattern.

Minimum Go version¶

For those building Incus from source, the minimum Go version required is now Go 1.20.
In general, we'll attempt to keep supporting building on the current and previous Go version.

Feature removal¶

A number of LXD features have been removed from Incus.
Those primarily focus around features that were added because of ecosystem reasons and/or depend on now deprecated or poorly maintained software.

Removal of Ubuntu Fan bridges¶

The Ubuntu Fan bridges depend on custom Ubuntu-only kernel patches and user-space changes to iproute2. While this feature is still maintained in the Ubuntu kernel, it can't be found in the mainline kernel nor in any other distro's kernel.

While convenient in some environments, Ubuntu Fan bridges are dwarfed by what's possible through Incus' integration with OVN.

The following network config keys have therefore been removed:

• bridge.mode
• fan.overlay_subnet
• fan.underlay_subnet
• fan.type

Removal of Ubuntu shiftfs¶

Another feature that's unique to the Ubuntu kernel, shiftfs was an initial attempt at flexible uid/gid shifting at the kernel level.

While LXD supported shiftfs for years, it was never enabled by default due to a variety of kernel issues. Instead a cleaner in-kernel solution to this problem was developed, VFS idmap shifting. That's now available in recent Linux kernels and automatically used by Incus when present.

Removal of Canonical Candid authentication¶

Back in the days, Canonical developed its own authentication system based on the use of Macaroons. The central authentication server for this was Candid.

LXD got Candid support, allowing it to authenticate users through a variety of providers through it.

These days, Candid is mostly unmaintained and the focus has been on moving towards industry standards, namely Open ID Connect.

Therefore, the following server configuration keys have been removed:

• candid.api.key
• candid.api.url
• candid.domains
• candid.expiry

Incus already has OpenID Connect support for external authentication and there are widely available OIDC servers that match and often exceed what Candid supported.

Removal of Canonical RBAC authorization¶

Canonical RBAC was a proprietary Role Based Access Control solution built on top of Macaroons and Candid.

It was only ever supported by MAAS and LXD and has seen extremely little adoption, mostly due to it being proprietary and generally difficult to get access to.

It's mostly unmaintained and the focus these days is to transition to something more industry standard, namely OpenFGA.

Therefore the following server configuration keys have been removed:

• rbac.agent.url
• rbac.agent.username
• rbac.agent.private_key
• rbac.agent.public_key
• rbac.api.expiry
• rbac.api.key
• rbac.api.url
• rbac.expiry

Incus currently doesn't have OpenFGA support, so any existing user of Canonical RBAC should stick with LXD until such time as an OIDC + OpenFGA alternative is available.

Removal of Canonical MAAS integration¶

LXD supports integrating with the MAAS API to use MAAS as some kind of IPAM (IP Address Management) solution.

This allows mapping specific LXD networks to MAAS subnet and then having MAAS create records for each instance on that network, assigning it a static address and DNS record.

Unfortunately, this integration has seen very little adoption and MAAS itself has a number of constraints that makes this integration problematic:

• MAC addresses in MAAS are globally unique whereas in LXD they have to be unique among running instances on the same network. This different makes some instance copy operations impossible.
• Instance names are expected to be globally unique in MAAS, even if attached to completely different DNS zones. This effectively makes it impossible to use LXD projects as instances can now easily conflict.

Between those issues and the lack of adoption of this feature, we've made the decision to remove MAAS support from Incus, the following configuration keys have therefore been removed:

• maas.api.key
• maas.api.url
• maas.subnet.ipv4
• maas.subnet.ipv6

Removal of the concept of trust password¶

Back in the early LXD days, the only way to connect to a remote server was to use a trust password, basically a fixed secret that would then allow a client to add its TLS certificate to the server's trust store.

This effectively allowed anyone who knew or could find the trust password to add their client into the trust store after which the trust store was no longer required for further operations.

The trust password therefore had to be kept very safely, or unset immediately after adding a new client. This was unfortunately rarely the case, opening up the door for a brute force attack on the trust password and ultimately an attacker gaining full control over LXD and the server it runs on.

To address this, support for using one-time trust tokens has been added a little while back and all documentation updated to discourage users from using trust passwords.

In Incus, we went one step further and completely removed support for trust passwords. Adding new clients or new servers into a cluster should now be done through the use of one-time tokens or by using external authentication through OIDC.

Therefore, the following server configuration key has been removed:

• core.trust_password

Other¶

Change in DMI information¶

Inside of Incus virtual machines, the vendor is now set to Linux Containers and the product is set to Incus.

Change in virtio-serial marker¶

The virtio-serial device used for limited communication with Incus prior to establishing full agent access through vsock is now org.linuxcontainers.incus.

Migration from LXD¶

Incus comes with a convenient lxd-to-incus tool which can be used to transition a system from LXD to Incus.

The main restrictions at this stage are:

• Minimum LXD version of 4.0
• Maximum LXD version of 5.18
• No support for migrating clusters at this stage (this is being worked on)

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Zabbly packages for Debian and Ubuntu¶

Zabbly provides both daily and stable builds of Incus to Debian and Ubuntu users:
https://github.com/zabbly/incus

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool will soon be available through Chocolatey for Windows users.
Until then, binaries can be found here: https://github.com/lxc/incus/releases/tag/v0.1.0

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Bugs can be reported at: https://github.com/lxc/incus/issues