News¶

Incus 6.1 has been released¶

7th of May 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 6.1!

This is our first feature release following Incus 6.0 LTS.
As a reminder, feature releases are only supported until the next one comes out, usually on a monthly cadence. Critical production environments should stay on the LTS release instead.

In this release, we have a lot of small quality of life improvements throughout. A lot of those being first contributions from students of the University of Texas at Austin. Expect a lot more of those in Incus 6.2!

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features¶

Creation of complex ZFS pools¶

The source key when creating storage pools using our zfs driver has now been extended to allow the creation of more complex vdevs including striping, mirroring, raidz1 and raidz2.

Example syntax:

• /dev/sda,/dev/sdb (striping, RAID0)
• mirror=/dev/sda,/dev/sdb (mirroring, RAID1)
• raidz1=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde (raidz1, RAID5)
• raidz2=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde (raidz2, RAID6)

This combined with the data from incus info --resources will now make it possible to deploy complex storage pools all through the API.

Listing of profiles across projects¶

As part of an effort to add cross-project querying of all API objects, it is now possible to list profiles across all projects.

At the API level, this is support for all-projects=true on the /1.0/profiles API endpoint, at the CLI level this looks like:

stgraber@dakara:~$ incus profile list --all-projects
+-----------------+---------+---------------------------------------+---------+
|     PROJECT     |  NAME   |              DESCRIPTION              | USED BY |
+-----------------+---------+---------------------------------------+---------+
| default         | default | Default Incus profile                 | 10      |
+-----------------+---------+---------------------------------------+---------+
| demo            | default | Default Incus profile                 | 12      |
+-----------------+---------+---------------------------------------+---------+
| lab-cgroup      | default | Default Incus profile                 | 2       |
+-----------------+---------+---------------------------------------+---------+
| lab-lvm-cluster | default | Default Incus profile                 | 3       |
+-----------------+---------+---------------------------------------+---------+
| lab-ovn-ic      | default | Default Incus profile                 | 10      |
+-----------------+---------+---------------------------------------+---------+
| vpn             | default | Default Incus profile for project vpn | 2       |
+-----------------+---------+---------------------------------------+---------+

This feature was contributed by University of Texas at Austin students.

Listing of network zones across projects¶

As part of an effort to add cross-project querying of all API objects, it is now possible to list network zones across all projects.

At the API level, this is support for all-projects=true on the /1.0/network-zones API endpoint, at the CLI level this looks like:

stgraber@dakara:~$ incus network zone list --all-projects
+---------+--------------------------+-------------+---------+
| PROJECT |           NAME           | DESCRIPTION | USED BY |
+---------+--------------------------+-------------+---------+
| default | default.demo.example.net |             | 0       |
+---------+--------------------------+-------------+---------+
| foo     | foo.demo.example.net     |             | 0       |
+---------+--------------------------+-------------+---------+

This feature was contributed by University of Texas at Austin students.

Additional functions made available to the instance placement scriptlet¶

Incus supports customizing instance placement through the use of a python-like script called a scriptlet. When used, the scriptlet is exposed some information about the instance, potential targets and the reason for the request.

On top of those arguments, a number of functions are also exposed to those scriptlets.
That includes the ability to log information, the function to actually make the final placement decision and the ability to fetch some basic load information about the candidate servers.

Now this is being extended through two additional functions:

• get_instances(location, project) => []api.Instance
• get_cluster_members(group) => []api.ClusterMember

Those are all optional arguments, so they can be used to list all instances or all cluster members as well, allowing a lot of flexibility in placement scripts.

Documentation: https://linuxcontainers.org/incus/docs/main/explanation/clustering/#instance-placement-scriptlet
This feature was contributed by University of Texas at Austin students.

Feeding YAML to create commands in the incus CLI¶

A number of Incus commands already support reading a YAML file through their standard input as part of a create command, but this isn't very consistent nor well documented, we're now in the process of making things consistent and this release now has support for reading a YAML object definition in the following commands:

• incus create & incus launch
• incus cluster group create
• incus network acl create
• incus network forward create
• incus network integration create
• incus network load-balance create
• incus network peer create
• incus network zone create
• incus profile create
• incus project create
• incus snapshot create
• incus storage create
• incus storage bucket create

For all of those, YAML data similar to what's showed in the matching show command can be fed through stdin at creation time to configure the object as part of its creation.

Customizable columns in the incus CLI¶

Something else we're slowly making consistent in the CLI is the ability to choose what columns to display in our list commands.

This has also been expanded with the following commands now supporting it:

• incus cluster list
• incus config trust list
• incus image list
• incus list
• incus profile list
• incus project list
• incus storage volume list
• incus storage volume snapshot list
• incus warning list

This feature was contributed by University of Texas at Austin students.

migration.stateful configuration key for containers¶

The migration.stateful configuration key has been expanded to also apply to containers now.

It is now required to have it set to true to access any feature requiring the recording and restoration of process state in containers (CRIU), which basically means stateful stop, stateful snapshots and live migration.

This change is unlikely to affect many users as CRIU's ability to live-migrate or perform stateful dumps of Incus containers is extremely limited and so generally seen as not functional.
The change does have the benefit of providing clearer errors to users who accidentally request an action which would make use of CRIU.

This feature was contributed by University of Texas at Austin students.

Stateless ACLs on OVN¶

A new allow-stateless action has now been added to Incus' network ACL rules.

As the name implies, this leads to the creation of a stateless rule inside of OVN.
This is great for situations where stateful rules may come with a heavy cost and where a matching stateless rule is possible (e.g. DNS interactions).

This feature was contributed by University of Texas at Austin students.

Instance uptime (startup time) tracking¶

A new StartedAt field has been added to the instance state data.
This exposes the timestamp at which the instance was started and is also available in incus info and incus list.

stgraber@dakara:~$ incus info speedtest | grep Started
Started: 2024/04/29 11:03 EDT

stgraber@dakara:~$ incus list -cnstU
+-------------+---------+-----------------+----------------------+
|    NAME     |  STATE  |      TYPE       |      STARTED AT      |
+-------------+---------+-----------------+----------------------+
| centos3     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| centos4     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| fga         | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+
| incus-ui    | RUNNING | CONTAINER       | 2024/05/07 16:54 EDT |
+-------------+---------+-----------------+----------------------+
| kernel-test | RUNNING | VIRTUAL-MACHINE | 2024/05/07 15:43 EDT |
+-------------+---------+-----------------+----------------------+
| keybase     | STOPPED | CONTAINER       |                      |
+-------------+---------+-----------------+----------------------+
| ovn-test    | RUNNING | VIRTUAL-MACHINE | 2024/05/07 15:43 EDT |
+-------------+---------+-----------------+----------------------+
| speedtest   | RUNNING | CONTAINER       | 2024/04/29 11:03 EDT |
+-------------+---------+-----------------+----------------------+
| void        | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+
| win11       | STOPPED | VIRTUAL-MACHINE |                      |
+-------------+---------+-----------------+----------------------+

This feature was contributed by University of Texas at Austin students.

Improvement to network handling during evacuation¶

When performing a cluster evacuation, all the networks will now be shut down at the end of the evacuation and only started back as part of the restoration action.

This is particularly useful in OVN environments as it ensures that an evacuated Incus server doesn't act as a virtual router for any of the defined networks, making system shutdown/reboot less likely to cause network glitches.

This feature was contributed by University of Texas at Austin students.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.1.0

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 6.0 LTS has been released¶

4th of April 2024

Introduction¶

It's with great pride and pleasure that the Incus team is announcing the release of Incus 6.0 LTS!

Incus is a modern system container and virtual machine manager developed and maintained by the same team that first created LXD. It's released under the Apache 2.0 license and is run as a community led Open Source project as part of the Linux Containers organization.

Incus provides a cloud-like environment, creating instances from premade images and offers a wide variety of features, including the ability to seamlessly cluster up to 50 servers together.

It supports multiple different local or remote storage options, traditional or fully distributed networking and offers most common cloud features, including a full REST API and integrations with common tooling like Ansible, Terraform/OpenTofu and more!

This is a major milestone for Incus as it marks our first release with extended support, suitable for use in production environments where monthly feature releases aren't suitable.

It joins LXC 6.0 LTS and LXCFS 6.0 LTS in wrapping up this round of LTS releases.

Just like its sister projects, Incus 6.0 LTS will be supported until June 2029.
The first 2 years will feature bug and security fixes as well as minor usability improvements, delivered through occasional point releases (6.0.x). After that initial two years, Incus 6.0 LTS will move to security only maintenance for the remaining of its 5 years of support.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

PS: Incus was made possible thanks to the work of over 70 individual contributors!

Changes since Incus 0.7¶

Swap limits for containers¶

The existing limits.memory.swap configuration key for containers has been extended to also allow for byte amounts.

This now makes its behavior be as follows:

• limits.memory.swap=true => Container memory may be swapped (default)
• limits.memory.swap=false => Container shouldn't get swapped (minimal swappiness)
• limits.memory.swap=256MiB => Container can use up to 256MiB of swap space (in addition to its memory limit set through limits.memory)

Example (cgroup2 system):

stgraber@dakara:~$ incus launch images:debian/12 d12 -c limits.memory=1GiB
Launching d12
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m
               total        used        free      shared  buff/cache   available           
Mem:            1024          21         983           0          19        1002
Swap:              0           0           0
root@d12:~#
exit
stgraber@dakara:~$ incus config set d12 limits.memory.swap=128MiB
stgraber@dakara:~$ incus exec d12 bash
root@d12:~# free -m      
               total        used        free      shared  buff/cache   available
Mem:            1024          21         983           0          19        1002
Swap:            128           0         128
root@d12:~#
exit

New shell completion mechanism¶

With this release, we complete the migration away from a hand-maintained bash completion script and over to generate completion scripts directly in our command line tool.

Completion profiles are now available for:

• bash
• fish
• powershell
• zsh

The profile can be retrieved by calling incus completion <shell> (e.g. incus completion bash) though this will generally be done by packagers as part of the Incus package build process.

Creation of external bridge interfaces¶

The managed network bridge configuration syntax for external interfaces, bridge.external_interfaces has now been extended to allow for the creation and attachment of VLAN interfaces.

stgraber@dakara:~$ incus network set incusbr0 bridge.external_interfaces=vlan60/enp35s0/60
stgraber@dakara:~$ ip link show dev vlan60
269: vlan60@enp35s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master incusbr0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 00:23:a4:01:01:6f brd ff:ff:ff:ff:ff:ff
stgraber@dakara:~$ incus network unset incusbr0 bridge.external_interfaces
stgraber@dakara:~$ ip link show dev vlan60
Device "vlan60" does not exist.
stgraber@dakara:~$

Live-migration of VMs with attached disks (from remote storage)¶

As an extension to our ever improving VM live-migration support, virtual-machines with additional disks attached to them which come from a "remote" storage pool (ceph or lvmcluster) will now be live-migratable alongside the virtual machine.

No user action is required for this to happen, you'll simply now notice that virtual machines that previously would have refused to live-migrate through either a manual incus move --target or a cluster evacuation will now happily live-migrate to another server.

System information in incus info --resources¶

A new System section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
System:
  UUID: 88eecd60-34fc-9f97-48f5-fc34979f48f6
  Vendor: ASUS
  Product: System Product Name
  Family: To be filled by O.E.M.
  Version: System Version
  SKU: SKU
  Serial: System Serial Number
  Type: physical
  Chassis:
      Vendor: Default string
      Type: Desktop
      Version: Default string
      Serial: Default string
  Motherboard:
      Vendor: ASUSTeK COMPUTER INC.
      Product: ProArt B550-CREATOR
      Serial: 210382121300122
      Version: Rev X.0x
  Firmware:
      Vendor: American Megatrends Inc.
      Version: 2803
      Date: 04/28/2022

 [snip...]

Having access to this information is particularly useful in clustered environments where incus info --resources can be used with the --target argument to query specific servers, check that all firmwares are up to date and check what machines one is dealing with.

This feature was contributed by University of Texas at Austin students.

USB devices in incus info --resources¶

A new USB devices section is now visible in incus info --resources

stgraber@dakara:~$ incus info --resources
[snip...]

USB devices:
  Device 0:
    Vendor: Intel Corp.
    Vendor ID: 8087
    Product: AX200 Bluetooth
    Product ID: 0029
    Bus Address: 1
    Device Address: 6
  Device 1:
    Vendor: Corsair
    Vendor ID: 1b1c
    Product: H150iRGBPROXT
    Product ID: 0c22
    Bus Address: 1
    Device Address: 5
  Device 2:
    Vendor: ASUSTek Computer, Inc.
    Vendor ID: 0b05
    Product: AURA LED Controller
    Product ID: 19af
    Bus Address: 1
    Device Address: 2
  Device 3:
    Vendor: Realtek Semiconductor Corp.
    Vendor ID: 0bda
    Product: TX42C500
    Product ID: 4933
    Bus Address: 5
    Device Address: 2
  Device 4:
    Vendor: Blue Microphones
    Vendor ID: b58e
    Product: Yeti Stereo Microphone
    Product ID: 9e84
    Bus Address: 5
    Device Address: 15
  Device 5:
    Vendor: Yubico.com
    Vendor ID: 1050
    Product: YubiKey FIDO+CCID
    Product ID: 0406
    Bus Address: 5
    Device Address: 29
  Device 6:
    Vendor: Logitech, Inc.
    Vendor ID: 046d
    Product: HD Pro Webcam C920
    Product ID: 082d
    Bus Address: 5
    Device Address: 17
  Device 7:
    Vendor: Powerware Corp.
    Vendor ID: 0592
    Product: Powerware UPS
    Product ID: 0002
    Bus Address: 7
    Device Address: 2

That information comes in very handy when adding a USB device to a container or virtual machine.

This feature was contributed by University of Texas at Austin students.

Changes since LXD 5.0 LTS¶

For those coming from the LXD 5.0 LTS release, here is a concise list of what to expect as far as features having been removed and what has been added both in subsequent LXD feature releases and then through Incus.

Feature removal¶

A number of features that were Ubuntu or Canonical specific were removed as part of the creation of the Incus project. A number of legacy APIs have also been removed at the same time.
You'll find the full list in the Incus 0.1 announcement.

Highlights:

• shiftfs has been removed in favor of VFS idmap shifting
• Canonical Candid authentication has been removed in favor of OpenID Connect
• Canonical RBAC authorization has been removed in favor of OpenFGA
• Canonical MAAS network integration has been removed (under/unused feature)
• Ubuntu Fan networking has been removed in favor of OVN
• core.trust_password has been removed in favor of trust tokens for security reasons

Feature additions¶

Here are a few highlights from the many new features introduced within the 2 years since the release of LXD 5.0 LTS.

• API
• Abiltiy to list objects across projects (?all-projects=true or --all-projects in CLI)
• JWT authentication (derived from TLS certificate)
• Instances
• Placement scriptlet
• Instance rebuilding
• READY instance state
• NUMA aware instance placement (limits.cpu.nodes)
• (CONTAINER) sysinfo system call interception (security.syscalls.intercept.sysinfo)
• (VM) CPU hotplug support (limits.cpu)
• (VM) "Online" live-migration support
• (VM) AMD SEV support (security.sev)
• (VM) Legacy (BIOS) support (security.csm)
• (VM) Ability to hot-plug directories backed disks
• (VM) NVME and VirtIO block I/O bus options
• Integrations
• Grafana Loki log and event streaming
• ACME / Let's Encrypt certificate generation/signing
• OpenID Connect authentication support
• OpenFGA authorization support
• Image server management tool
• Networking
• Network integrations (OVN interconnect support)
• Load-balancers (OVN)
• IPAM data export API
• VDPA for offloaded OVN networks
• Storage
• Clustered LVM storage driver
• Storage buckets (S3 API)
• ISO image custom volumes
• ZFS delegation
• ZFS block mode

Complete changelog¶

Here is a complete list of all changes since Incus 0.7:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Installation¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Linux packages¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/6.0.0

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Migrating from LXD¶

A lxd-to-incus migration tool allows for in-place migration from LXD to Incus.
It's been tested with LXD versions as low as 4.0 LTS and as high as the latest LXD 5.21 bugfix release.

It allows for a very quick migration from LXD over to Incus, automatically checking for potential conflicts ahead of time.

More details can be found here: https://linuxcontainers.org/incus/docs/main/howto/server_migrate_lxd/

Support¶

Incus 6.0 LTS will be supported for a total of 5 years (until June 2029).

During the first 2 years, new point releases will be issued including a mix of bug and security fixes as well as some minor usabiltiy improvements. After that initial 2 years (after Incus 7.0 LTS is released), Incus 6.0 LTS will transition to security fixes only for the remaining 3 years.

This matches what we've been doing for our other projects (LXC and LXCFS) over the past 10 years.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.7 has been released¶

26th of March 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.7!

This is going to be our last release before Incus 6.0 LTS which is now scheduled to be released next week. As releases go, this is quite a busy one, which is how we like it before releasing an LTS, trying to keep the amount of new features in the LTS release itself to a minimum.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features¶

Network integrations¶

A new top-level concept, network integrations are a way to connect an Incus deployment to networks outside of its own control.

Currently the only implementation of the concept is OVN interconnection which makes it possible for an Incus cluster to directly peer its OVN networks with equivalent networks running on other Incus clusters or even other OVN users like OpenStack or Kubernetes.

Here is an example of creating a new network integration using an OVN interconnection gateway, then peering an existing network through it:

root@az01-server01:~# incus network integration create ovn-region ovn
Network integration ovn-region created
root@az01-server01:~# incus network integration set ovn-region ovn.northbound_connection tcp:[10.50.1.12]:6645,tcp:[10.50.2.13]:6645,tcp:[10.50.3.19]:6645
root@az01-server01:~# incus network integration set ovn-region ovn.southbound_connection tcp:[10.50.1.12]:6646,tcp:[10.50.2.13]:6646,tcp:[10.50.3.19]:6646
root@az01-server01:~# incus network peer create default region ovn-region --type=remote
Network peer region created

Documentation: https://linuxcontainers.org/incus/docs/main/howto/network_integrations/

Image server management tool¶

A common way to run an Incus image server, be it for some internal servers or as a publicly available image server is through a static web server providing Incus images using simplestreams.

To make this easier to set up, we're now introducing a new tool, incus-simplestreams which can easily manage a simple image server, listing the images available, adding and removing images as well as generating the needed metadata files.

stgraber@dakara:~$ mkdir image-server
stgraber@dakara:~$ cd image-server/
stgraber@dakara:~/image-server$ incus-simplestreams generate-metadata ~/Downloads/incus.tar.xz
Operating system name: Red Hat Enterprise Linux
Release name: 9
Variant name [default="default"]:
Architecture name: x86_64
Description [default="Red Hat Enterprise Linux 9 (default) (x86_64) (202403260239)"]:·
stgraber@dakara:~/image-server$ incus-simplestreams add ~/Downloads/incus.tar.xz ~/Downloads/rhel9.qcow2·
stgraber@dakara:~/image-server$ incus-simplestreams list
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
|                           FINGERPRINT                            |                   DESCRIPTION                    |            OS            | RELEASE | VARIANT | ARCHITECTURE |      TYPE       |       CREATED        |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
| 7d256e4fac6fc63fb47bc1e07e1c6ee234281cdf1ed21788c920d763b7bd93ba | Red Hat Enterprise Linux 9 x86_64 (202403252239) | Red Hat Enterprise Linux | 9       | default | x86_64       | virtual-machine | 2024/03/25 00:00 UTC |
+------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+
stgraber@dakara:~/image-server$ find . | sort
.
./images
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.incus.tar.xz
./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.qcow2
./streams
./streams/v1
./streams/v1/images.json
./streams/v1/index.json

Put that on an HTTPS capable web server and then add it with:

incus remote add my-server https://xyz.example.net --protocol=simplestreams

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_servers/#tooling-to-manage-a-simplestreams-server

JSON Web Token authentication¶

Incus basically supports two mechanisms for remote authentication:

• TLS client certificates (added to the local trust store with or without restrictions)
• OpenID Connect external authentication (with or without OpenFGA for authorization)

The former is the most common for simple interactions with a remote Incus server.
Our own CLI tool and most 3rd party tools don't have any problems using a TLS keypair to establish the HTTPS connection and get authenticated that way.

But there are some situations, like running Incus behind a reverse HTTP(S) proxy where TLS client certificates can become a bit problematic.

To address that, we now support using a JSON Web Token (JWT) bearer token through the HTTP Authorization field. That token can be generated by any user with a valid TLS client certificate by setting the Subject field to the certificate fingerprint, setting applicable NotBefore/NotAfter values and signing the JWT with their private key.

Incus will treat any such connections as equivalent to using the TLS client certificate.

stgraber@dakara:~$ openssl req -x509 -newkey rsa:4096 -sha384 -keyout client.key -nodes -out client.crt -days 1 -subj "/CN=test.local"
.+.........+...+...+..+....+......+........+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+......+...+......+......+..+...+...+...............+.......+...+......+...+.....+......+....+......+..+...+......+....+............+.....+....+.....+.+............+..+.........+......+....+......+...........+....+........+...+...+.+...+..+..........+.....+...+......+............+...+.......+........+....+.....+.+..+.......+......+..+....+........+..........+...+..+.+.....+.+......+..+.......+.....+.+..+..........+..+....+..............+.+..+...........................+...+....+......+...+..............+.+..+....+.....+.+.........+...+..+....+..+.............+.........+.....+...+..........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...+..+...+.........+.+...+............+..+............+.+.......................................+.....+...+......+.........+......+.+.....+...+.+...........+......+.......+.....+.......+......+.....+..........+...+..+.........+....+.........+...........+......+.+..................+..+....+...........+.............+.....+....+..+......+............+..........+......+......+......+..+.............+.....+...+.+........+............+....+.................+.........+......+.......+...+.........+.....+....+......+........+.+..+....+......+........+....+...+.................+.+..+.........+....+............+.....+.........+....+.....+.......+.....+...............+..............................+....+........+.......+...........+.......+.....+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...+...............+.+...+...+...+.........+..+.+...+..+.......+.....+.+..+...+...................+...........+......+....+..+............+...+.......+..+.........+....+...+.........+.....+...+...+....+...........+...+.+.........+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+......+..+...+......+.+..+.............+..+.......+........+...+...+....+...............+..+....+..+....+...+.....+.+.....+..........+.....+.+.....+....+............+.........+...+.....+......+......+.............+..+.+..+.......+...+........+...+.......+.........+......+..+.+......+...........+...+.........+...+...+....+..+.........+....+.....+.+......+.........+..+..........+..+...+...+....+..+...+.+.....+......+.+...+......+.....+.+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
stgraber@dakara:~$ incus config trust add-certificate client.crt --restricted --projects demo
stgraber@dakara:~$ tls2jwt client.key client.crt now 120
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0stgraber@dakara:~$·
stgraber@dakara:~$ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0' https://localhost:8443/1.0/projects | jq
{
  "type": "sync",
  "status": "Success",
  "status_code": 200,
  "operation": "",
  "error_code": 0,
  "error": "",
  "metadata": [
    "/1.0/projects/demo"
  ]
}

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#using-json-web-token-jwt-to-perform-tls-authentication

Configurable OIDC username field¶

For those using OpenID Connect, you may have noticed that Incus will use the e-mail claim when available as the user's identifier. Then if missing, it will rely on the Subject.

As different deployments may make different information available through OIDC claims, it's now possible to set oidc.claim to the claim to use as the user identifier.

stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org
stgraber@dakara:~$ incus config set oidc.claim=name
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
Stéphane Graber
stgraber@dakara:~$ incus config set oidc.claim=sub
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
99cb8caa-3640-45b9-b87a-55266366aaf3
stgraber@dakara:~$ incus config set oidc.claim=email
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name
stgraber@stgraber.org

Improved NUMA handling¶

With this release, we spent a fair amount of time trying to improve both the container and virtual-machine performance on large systems. This obviously includes multi-socket systems but also AMD systems running in NPS4 or similar mode where each CPU is exposed as multiple NUMA nodes.

In general, our goal has been to make it easy to distribute workloads across NUMA nodes while keeping their CPU and memory properly pinned and also selecting PCIe resources that are closest to their NUMA node(s).

As part of that, a few things were done:
- limits.cpu.nodes is now supported for virtual-machines too
- A new balanced value has been added to limits.cpu.nodes which will have Incus pick the NUMA node with the least instances configured to use it
- SR-IOV GPU selection now also considers NUMA nodes as part of the selection logic and when no match is found, will prefer PCIe devices that are attached to the same CPU socket

For example:

stgraber@gputest:~$ incus list stgraber-gpu -cns4,limits.cpu.nodes,volatile.cpu.nodes,volatile.gpu.last_state.pci.parent,volatile.gpu.last_state.vf.id
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
|      NAME      |  STATE  |         IPV4          | LIMITS CPU NODES | VOLATILE CPU NODES | VOLATILE GPU LAST STATE PCI PARENT | VOLATILE GPU LAST STATE VF ID |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu01 | RUNNING | 10.232.44.8 (enp5s0)  | balanced         | 0                  | 0000:63:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu02 | RUNNING | 10.232.44.9 (enp5s0)  | balanced         | 2                  | 0000:03:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu03 | RUNNING | 10.232.44.10 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu04 | RUNNING | 10.232.44.11 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu05 | RUNNING | 10.232.44.12 (enp5s0) | balanced         | 6                  | 0000:c3:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu06 | RUNNING | 10.232.44.13 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu07 | RUNNING | 10.232.44.15 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu08 | RUNNING | 10.232.44.16 (enp5s0) | balanced         | 2                  | 0000:03:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu09 | RUNNING | 10.232.44.17 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu10 | RUNNING | 10.232.44.18 (enp5s0) | balanced         | 4                  | 0000:e3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu11 | RUNNING | 10.232.44.19 (enp5s0) | balanced         | 5                  | 0000:c3:00.0                       | 0                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu12 | RUNNING | 10.232.44.20 (enp5s0) | balanced         | 6                  | 0000:83:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu13 | RUNNING | 10.232.44.21 (enp5s0) | balanced         | 7                  | 0000:83:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu14 | RUNNING | 10.232.44.22 (enp5s0) | balanced         | 1                  | 0000:43:00.0                       | 1                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu15 | RUNNING | 10.232.44.23 (enp5s0) | balanced         | 2                  | 0000:43:00.0                       | 2                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
| stgraber-gpu16 | RUNNING | 10.232.44.24 (enp5s0) | balanced         | 3                  | 0000:03:00.0                       | 3                             |
+----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+

In this case we can see 16 VMs each using the new balanced option for NUMA nodes and getting scheduled across 8 NUMA nodes (2 sockets AMD NPS4) with GPUs being selected to match.

More options to select USB devices¶

USB device passhtrough for both containers and virtual-machines has so far been using the vendorid and productid fields. This works fine as long as there is only one USB device of any one type connected to the system.

When multiple identical devices are present, the inability to distinguish them has been a problem.

To address this, three new fields have now been added to usb devices:
- busnum referring to the USB bus number
- devnum referring to the USB device number (on its bus)
- serial referring to the USB device serial number (not present on all devices)

The same fields can be found in the full Incus resources list through:

incus query /1.0/resources

Disk I/O throttling for VMs¶

One more feature gap between containers and virtual-machines is now gone.

The limits.write and limits.read properties on disk devices are now properly enforced on virtual-machines by having Incus setup an I/O throttle in QEMU.

Both bytes per second and I/O per second type limits are supported.

Per-remote client certificates¶

It's now possible to put a <remote>.crt and <remote>.key file in a new clientcerts folder within the Incus command line client config directory (typically ~/.config/incus/) and have those certificates be used when interacting with that particular remote.

While this may be useful on its own, it becomes a lot more useful when combined with global remotes, which can be added in /etc/incus/config.yml. Now with this feature, those global remotes can also have a client certificate made available in /etc/incus/clientcerts/ which will then be used by all users on the system.

Manual generation of the client certificate keypair¶

A new command to manually trigger the generation of the main client.crt and client.key keypair is now available.

This is done by running incus remote generate-certificate

Improvements to lxd-to-incus¶

The lxd-to-incus tool keeps evolving with every release.

In this one, it gains support for migrating users from the newly released LXD 5.21 LTS as well as handling Alpine installations.

Additionally, a static binary version of the tool is now available on Github, making it easier for users to fetch the latest version of the tool, useful as bugs get fixed in between Incus releases.

Improvements to incus-migrate¶

The workload migration tool incus-migrate has also seen a couple of small improvements.

It can now use the local Incus system as the target of the migration, useful when importing virtual-machine images from another virtualization tool.

And it's also now prompting for whether the imported virtual machine should be using a UEFI firmware or instead use a legacy BIOS.

Additional image restrictions¶

A bit of an internal detail, or at least only relevant to public image server operators, but two new image restrictions have been added:

• requirements.nesting which will require the container have security.nesting=true
• requirements.cdrom_agent which will require that a source=agent:config disk device be added to the virtual-machine

Those two can be used to flag specific images that need extra user interaction to work properly, resulting in a clear client-side error rather than starting a potentially broken instance.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.7

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.6 has been released¶

23rd of February 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.6!

This second release of 2024 features a number of improvements, both large and small.

It comes with a completely new storage driver for cluster users, import/export support for storage buckets, a number of improvements for OVN users, support for new container kernel features, quite a few improvements to the migration tooling and more!

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

New features¶

Clustered LVM storage driver¶

A new storage driver named lvmcluster has been added.

As the name implies, this is for clustered use of the LVM storage driver.
This relies on llvmlockd and a lock manager like sanlock to provide concurrent access to a shared storage device across a cluster.

This is aimed at anyone who wants to run an Incus cluster and use a shared storage device like a FiberChannel SAN, iSCSI export, NVMEoF/NVMEoTCP disk, ...

With the ability of having the same storage available on all servers comes support for much faster live migrations, server evacuations and the ability to automatically recover should a server suddenly die.

Documentation: https://linuxcontainers.org/incus/docs/main/reference/storage_lvm/#lvmcluster-driver-in-incus

Storage bucket backup and import¶

It is now possible to backup a full storage bucket and later re-import back into Incus.

stgraber@dakara:~$ incus storage bucket create default foo
Storage bucket foo created
Admin access key: HD2GWC1KX693MFVT3P30
Admin secret key: tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv

stgraber@dakara:~$ dd if=/dev/random of=out.img bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.0975758 s, 430 MB/s
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv put out.img s3://foo
upload: 'out.img' -> 's3://foo/out.img'  [part 1 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   262.42 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 2 of 3, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s   241.54 MB/s  done
upload: 'out.img' -> 's3://foo/out.img'  [part 3 of 3, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s   241.99 MB/s  done
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:26     41943040  s3://foo/out.img

stgraber@dakara:~$ incus storage bucket export default foo
Backup exported successfully!
stgraber@dakara:~$ incus storage bucket delete default foo
Storage bucket foo deleted

stgraber@dakara:~$ incus storage bucket import default backup.tar.gz
stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo
2024-02-23 03:27     41943040  s3://foo/out.img

API: https://linuxcontainers.org/incus/docs/main/rest-api-spec/#/storage/storage_pool_buckets_backups_post

Listing image across all projects¶

Just as it's possible to list instances across all projects using the --all-projects, it's now possible to do the same but for images.

stgraber@dakara:~$ incus image list --all-projects
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| PROJECT | ALIAS | FINGERPRINT  | PUBLIC |               DESCRIPTION                | ARCHITECTURE |      TYPE       |   SIZE    |          UPLOAD DATE          |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 256f59a72af5 | no     | Ubuntu jammy amd64 (20240222_07:42)      | x86_64       | VIRTUAL-MACHINE | 267.19MiB | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | 0941e441dbb9 | no     | Alpine edge amd64 (20240222_13:00)       | x86_64       | CONTAINER       | 2.93MiB   | Feb 23, 2024 at 12:27am (UTC) |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| default |       | d5fc6024f0fa | no     | Openwrt snapshot amd64 (20240222_11:57)  | x86_64       | CONTAINER       | 3.50MiB   | Feb 23, 2024 at 2:55am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| demo    |       | f44a6b4e56f4 | no     | Archlinux current amd64 (20240222_04:18) | x86_64       | CONTAINER       | 188.86MiB | Feb 23, 2024 at 3:32am (UTC)  |
+---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+

Over the API, this is done by passing ?all-projects=true.

binfmt_misc in unprivileged containers¶

Linux 6.7 added support for mounting of binfmt_misc inside of unprivileged containers.

Incus 0.6 will detect kernels that support this feature and when they do, it will no longer bind-mount binfmt_misc from the host system but instead allow it to be mounted from within the container.

stgraber@castiana:~$ incus launch images:ubuntu/22.04 foo
Launching foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# uname -a
Linux foo 6.7.4-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Mon Feb  5 23:37:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@foo:~# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/
root@foo:~# ls -lh /proc/sys/fs/binfmt_misc/
total 0
--w------- 1 root root 0 Feb 23 03:46 register
-rw-r--r-- 1 root root 0 Feb 23 03:46 status

The binfmt_misc filesystem is used to define custom binary formats for emulation.
It's most notably used by qemu-user-static as a way to run binaries of other architectures.

Control over shared block storage volumes¶

A new security.shared volume storage option has been introduced to control whether a custom block volume should be allowed to be used by multiple instances.

It is now required to set that configuration key prior to adding a custom block volume to a profile or add it directly to multiple virtual machines.

stgraber@castiana:~$ incus config device add lvm01 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm01

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Error: Failed add validation for device "shared": Cannot add un-shared custom storage block volume to more than one instance

stgraber@castiana:~$ incus storage volume set default shared-lvm security.shared=true

stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm02
stgraber@castiana:~$ incus config device add lvm03 shared disk pool=default source=shared-lvm io.bus=nvme
Device shared added to lvm03
stgraber@castiana:~$ incus start lvm01 lvm02 lvm03

OVN logical router name in network info¶

To make it easier to see what's going on within OVN, incus network info now gives you the name of the logical router for a particular network.

root@abydos:~# incus network info default
Name: default
MAC address: 00:16:3e:38:dd:28
MTU: 1500
State: up
Type: broadcast

IP addresses:
  inet  10.180.103.1/24 (link)
  inet6 2602:fc62:a:1004::1/64 (link)

Network usage:
  Bytes received: 0B
  Bytes sent: 0B
  Packets received: 0
  Packets sent: 0

OVN:
  Chassis: abydos
  Logical router: incus-net6-lr

File ownership and permissions in image templates¶

Template files can now have a uid, gid and mode set on them.
This can be particularly useful if a template is meant to be an executable shell script.

stgraber@castiana:~$ incus config metadata show foo
architecture: amd64
creation_date: 1708588077
expiry_date: 1711180077
properties:
  architecture: amd64
  description: Ubuntu jammy amd64 (20240222_07:42)
  name: ubuntu-jammy-amd64-default-20240222_07:42
  os: ubuntu
  release: jammy
  serial: "20240222_07:42"
  variant: default
templates:
  /etc/hostname:
    when:
    - create
    - copy
    create_only: false
    template: hostname.tpl
    properties: {}
  /etc/hosts:
    when:
    - create
    - copy
    create_only: false
    template: hosts.tpl
    properties: {}
  /root/hello.sh:
    when:
    - start
    create_only: false
    template: hello.tpl
    properties: {}
    uid: "1000"
    gid: "2000"
    mode: "0755"

stgraber@castiana:~$ incus config template show foo hello.tpl
#!/bin/sh
echo "Hello world!"

stgraber@castiana:~$ incus start foo
stgraber@castiana:~$ incus exec foo bash
root@foo:~# ls -lh /root/hello.sh
-rwxr-xr-x 1 ubuntu 2000 30 Feb 23 04:07 /root/hello.sh
root@foo:~# /root/hello.sh
Hello world!

Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_format/#template-rules

Encrypted EC client certificate keys¶

For those interacting with remote Incus servers, you may not know that it's possible to protect your Incus private key with a password.

Up until now, this was only possible for RSA keys, but with Incus 0.6, we're now adding support for EC keys too. That's particularly relevant as EC keys have been the default for a while now.

Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#encrypting-local-keys

Worth noting that if using this feature, you'll likely also want to make use of the recently introduced "keepalive mode" as this will then significantly reduce the number of password prompts you'll get while using Incus.

lxd-to-incus improvements¶

Lastly, lxd-to-incus saw a number of improvements:

• Support for Void Linux
• Detection of the boot.debug_edk2 configuration key
• Handling of OVN SSL database connections
• Automatic clearing of the simplestreams cache during migration

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.6

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

Incus 0.5.1 has been released¶

29th of January 2024

Introduction¶

The Incus team is pleased to announce the release of Incus 0.5.1!

This is an unusual release as we normally do not issue point releases on top of the monthly feature releases. But we felt this was needed this time due to some pretty important bugfixes and a minor feature addition needed to accommodate those running CentOS/Alma/Rocky virtual machines.

Most changes are on the server side, so if you're only using the command line client, there is no strong reason to upgrade from 0.5 to 0.5.1.

As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/

Enjoy!

Highlights¶

Alternative way to get the VM agent¶

With Incus 0.5, the distribution mechanism for the Incus VM agent changed a bit.
In the past, we had a single share named config which would include both the instance-specific agent configuration and the incus-agent binary.

This was a bit wasteful, requiring a copy of the 15-20MB large incus-agent for every VM but was still somewhat manageable. This share was also exposed as both 9p and virtiofs. Leading to two processes running on the host system for every Incus VM.

With support for multiple agent binaries, copying them for every VM really wasn't an option anymore, so a separate share was introduced just for the binaries. As we really didn't want to end up with another two processes running on the host per VM, we made the decision to only make those internal shares be available over 9p.

Testing on a variety of images, including CentOS 7 showed that this would be fine.
9p is lower performance than virtiofs but as those shares are only use for a couple of seconds on every VM boot, that really wasn't a concern. User defined shares would still be exposed over virtiofs so those would still get the high performance option.

What we failed to notice is that for some reason, CentOS 8-Stream, CentOS 9-Stream and other distributions that are derivatives of RHEL 8/9, do not ship the 9p kernel driver at all...

This means that those instances no longer had a way to fetch an agent, leading to broken incus exec and incus file.

We still don't feel like running 4 host processes for every single Incus VM just to make things work on those few images. Instead, what we're introducing with Incus 0.5.1 is a new agent drive, effectively an extra disk which can be attached to those specific VMs, providing those files through what looks like a CD-ROM drive rather than being retrieved over a networked filesystem.

So to run CentOS 9-Stream, one now needs to do:

incus create images:centos/9-Stream centos --vm
incus config device add centos agent disk source=agent:config
incus start centos

If you run many such VMs, a better option is likely by creating a profile for it:

incus profile create vm-agent
incus profile device add vm-agent agent disk source=agent:config

At which point you can do:

incus launch images:centos/9-Stream centos --vm -p default -p vm-agent

This is obviously not ideal and adds a few more steps when creating VMs for those distributions but this new mechanism now offers a way to get the agent up and running in just about any environment.

NOTE: We're not considering always providing that extra device as it takes some resources to generate the cdrom device and uses some extra disk on the host. So it's best added only when needed.

Fixed handling of stopped instances during evacuation¶

A bug introduced with Incus 0.5 was causing stopped instances to get relocated to other systems during evacuation, even if the instance was configured to remain where it was.

This has now been corrected and instances using stopped, force-stop or stateful-stop are now guaranteed to remain on their current server.

Database performance fixes¶

Database improvements in Incus 0.5 accidentally caused some nested database transactions to occur when fetching network information details for a large number of instances.

This would only really become visible when using an Incus cluster that also serves DNS zones and has its metrics scraped by Prometheus. This combination would cause large spikes in API requests every 15s or so, which would then start triggering timeouts and retries, eventually leading to other API requests piling up and timing out.

The logic has now been changed to remove such nested transactions and further optimizations were also made to save some database interactions during very command API interactions like executing commands instance of instances.

Complete changelog¶

Here is a complete list of all changes in this release:

Documentation¶

The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/

Packages¶

There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.

Installing the Incus server on Linux¶

Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.

https://linuxcontainers.org/incus/docs/main/installing/

Homebrew package for the Incus client¶

The client tool is available through HomeBrew for both Linux and MacOS.

https://formulae.brew.sh/formula/incus

Chocolatey package for the Incus client¶

The client tool is available through Chocolatey for Windows users.

https://community.chocolatey.org/packages/incus/0.5.1

Winget package for the Incus client¶

The client tool is also available through Winget for Windows users.

https://winstall.app/apps/LinuxContainers.Incus

Support¶

At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.

Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues

旧闻¶

• 26th of January 2024
• 21st of December 2023
• 27th of November 2023
• 28th of October 2023
• 7th of October 2023