News¶
Incus 6.0 LTS has been released¶
4th of April 2024
Introduction¶
It's with great pride and pleasure that the Incus team is announcing the release of Incus 6.0 LTS!
Incus is a modern system container and virtual machine manager developed and maintained by the same team that first created LXD. It's released under the Apache 2.0 license and is run as a community led Open Source project as part of the Linux Containers organization.
Incus provides a cloud-like environment, creating instances from premade images and offers a wide variety of features, including the ability to seamlessly cluster up to 50 servers together.
It supports multiple different local or remote storage options, traditional or fully distributed networking and offers most common cloud features, including a full REST API and integrations with common tooling like Ansible, Terraform/OpenTofu and more!
This is a major milestone for Incus as it marks our first release with extended support, suitable for use in production environments where monthly feature releases aren't suitable.
It joins LXC 6.0 LTS and LXCFS 6.0 LTS in wrapping up this round of LTS releases.
Just like its sister projects, Incus 6.0 LTS will be supported until June 2029.
The first 2 years will feature bug and security fixes as well as minor usability improvements, delivered through occasional point releases (6.0.x). After that initial two years, Incus 6.0 LTS will move to security only maintenance for the remaining of its 5 years of support.
As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/
Enjoy!
PS: Incus was made possible thanks to the work of over 70 individual contributors!
Changes since Incus 0.7¶
Swap limits for containers¶
The existing limits.memory.swap
configuration key for containers has been extended to also allow for byte amounts.
This now makes its behavior be as follows:
limits.memory.swap=true
=> Container memory may be swapped (default)limits.memory.swap=false
=> Container shouldn't get swapped (minimal swappiness)limits.memory.swap=256MiB
=> Container can use up to 256MiB of swap space (in addition to its memory limit set throughlimits.memory
)
Example (cgroup2 system):
stgraber@dakara:~$ incus launch images:debian/12 d12 -c limits.memory=1GiB Launching d12 stgraber@dakara:~$ incus exec d12 bash root@d12:~# free -m total used free shared buff/cache available Mem: 1024 21 983 0 19 1002 Swap: 0 0 0 root@d12:~# exit stgraber@dakara:~$ incus config set d12 limits.memory.swap=128MiB stgraber@dakara:~$ incus exec d12 bash root@d12:~# free -m total used free shared buff/cache available Mem: 1024 21 983 0 19 1002 Swap: 128 0 128 root@d12:~# exit
New shell completion mechanism¶
With this release, we complete the migration away from a hand-maintained bash completion script and over to generate completion scripts directly in our command line tool.
Completion profiles are now available for:
- bash
- fish
- powershell
- zsh
The profile can be retrieved by calling incus completion <shell>
(e.g. incus completion bash
) though this will generally be done by packagers as part of the Incus package build process.
Creation of external bridge interfaces¶
The managed network bridge configuration syntax for external interfaces, bridge.external_interfaces
has now been extended to allow for the creation and attachment of VLAN interfaces.
stgraber@dakara:~$ incus network set incusbr0 bridge.external_interfaces=vlan60/enp35s0/60 stgraber@dakara:~$ ip link show dev vlan60 269: vlan60@enp35s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master incusbr0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000 link/ether 00:23:a4:01:01:6f brd ff:ff:ff:ff:ff:ff stgraber@dakara:~$ incus network unset incusbr0 bridge.external_interfaces stgraber@dakara:~$ ip link show dev vlan60 Device "vlan60" does not exist. stgraber@dakara:~$
Live-migration of VMs with attached disks (from remote storage)¶
As an extension to our ever improving VM live-migration support, virtual-machines with additional disks attached to them which come from a "remote" storage pool (ceph
or lvmcluster
) will now be live-migratable alongside the virtual machine.
No user action is required for this to happen, you'll simply now notice that virtual machines that previously would have refused to live-migrate through either a manual incus move --target
or a cluster evacuation will now happily live-migrate to another server.
System information in incus info --resources
¶
A new System
section is now visible in incus info --resources
stgraber@dakara:~$ incus info --resources System: UUID: 88eecd60-34fc-9f97-48f5-fc34979f48f6 Vendor: ASUS Product: System Product Name Family: To be filled by O.E.M. Version: System Version SKU: SKU Serial: System Serial Number Type: physical Chassis: Vendor: Default string Type: Desktop Version: Default string Serial: Default string Motherboard: Vendor: ASUSTeK COMPUTER INC. Product: ProArt B550-CREATOR Serial: 210382121300122 Version: Rev X.0x Firmware: Vendor: American Megatrends Inc. Version: 2803 Date: 04/28/2022 [snip...]
Having access to this information is particularly useful in clustered environments where incus info --resources
can be used with the --target
argument to query specific servers, check that all firmwares are up to date and check what machines one is dealing with.
This feature was contributed by University of Texas at Austin students.
USB devices in incus info --resources
¶
A new USB devices
section is now visible in incus info --resources
stgraber@dakara:~$ incus info --resources [snip...] USB devices: Device 0: Vendor: Intel Corp. Vendor ID: 8087 Product: AX200 Bluetooth Product ID: 0029 Bus Address: 1 Device Address: 6 Device 1: Vendor: Corsair Vendor ID: 1b1c Product: H150iRGBPROXT Product ID: 0c22 Bus Address: 1 Device Address: 5 Device 2: Vendor: ASUSTek Computer, Inc. Vendor ID: 0b05 Product: AURA LED Controller Product ID: 19af Bus Address: 1 Device Address: 2 Device 3: Vendor: Realtek Semiconductor Corp. Vendor ID: 0bda Product: TX42C500 Product ID: 4933 Bus Address: 5 Device Address: 2 Device 4: Vendor: Blue Microphones Vendor ID: b58e Product: Yeti Stereo Microphone Product ID: 9e84 Bus Address: 5 Device Address: 15 Device 5: Vendor: Yubico.com Vendor ID: 1050 Product: YubiKey FIDO+CCID Product ID: 0406 Bus Address: 5 Device Address: 29 Device 6: Vendor: Logitech, Inc. Vendor ID: 046d Product: HD Pro Webcam C920 Product ID: 082d Bus Address: 5 Device Address: 17 Device 7: Vendor: Powerware Corp. Vendor ID: 0592 Product: Powerware UPS Product ID: 0002 Bus Address: 7 Device Address: 2
That information comes in very handy when adding a USB device to a container or virtual machine.
This feature was contributed by University of Texas at Austin students.
Changes since LXD 5.0 LTS¶
For those coming from the LXD 5.0 LTS release, here is a concise list of what to expect as far as features having been removed and what has been added both in subsequent LXD feature releases and then through Incus.
Feature removal¶
A number of features that were Ubuntu or Canonical specific were removed as part of the creation of the Incus project. A number of legacy APIs have also been removed at the same time.
You'll find the full list in the Incus 0.1 announcement.
Highlights:
shiftfs
has been removed in favor of VFS idmap shiftingCanonical Candid
authentication has been removed in favor of OpenID ConnectCanonical RBAC
authorization has been removed in favor of OpenFGACanonical MAAS
network integration has been removed (under/unused feature)Ubuntu Fan
networking has been removed in favor of OVNcore.trust_password
has been removed in favor of trust tokens for security reasons
Feature additions¶
Here are a few highlights from the many new features introduced within the 2 years since the release of LXD 5.0 LTS.
- API
- Abiltiy to list objects across projects (
?all-projects=true
or--all-projects
in CLI) - JWT authentication (derived from TLS certificate)
- Instances
- Placement scriptlet
- Instance rebuilding
READY
instance state- NUMA aware instance placement (
limits.cpu.nodes
) - (CONTAINER)
sysinfo
system call interception (security.syscalls.intercept.sysinfo
) - (VM) CPU hotplug support (
limits.cpu
) - (VM) "Online" live-migration support
- (VM) AMD SEV support (
security.sev
) - (VM) Legacy (BIOS) support (
security.csm
) - (VM) Ability to hot-plug directories backed disks
- (VM) NVME and VirtIO block I/O bus options
- Integrations
- Grafana Loki log and event streaming
- ACME / Let's Encrypt certificate generation/signing
- OpenID Connect authentication support
- OpenFGA authorization support
- Image server management tool
- Networking
- Network integrations (OVN interconnect support)
- Load-balancers (OVN)
- IPAM data export API
- VDPA for offloaded OVN networks
- Storage
- Clustered LVM storage driver
- Storage buckets (S3 API)
- ISO image custom volumes
- ZFS delegation
- ZFS block mode
Complete changelog¶
Here is a complete list of all changes since Incus 0.7:
Full commit list
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- incus/image: Fix column handling with --all-projects
- Replace util.ValueInSlice with slices.Contains
- shared/util: Delete ValueInSlice function
- incus/image: Fix column handling with --all-projects
- incusd/instance/qemu: Relocate image requirement checks
- doc/images: Add requirements.cdrom_agent
- incusd/instance/qemu: Add support for requirements.cdrom_agent
- incusd/device/disk: Fix incorrect block volume usage
- Translated using Weblate (Japanese)
- incusd/network/ovn: Use ParseIPToNet instead of manual IPToNet and net.ParseIP
- incusd/network/ovn: Use listenAddressNet in family check
- incusd/instance/drivers: Disable architecture check on incus cp with snapshots
- Translated using Weblate (French)
- incusd/network/bridge: Set local address on all VXLAN tunnels
- incus/instance/qemu: Fix RecordOutput
- incus: add completions for instance actions and snapshots
- incus: add completions for profiles
- incusd/network/ovn: Introduce get helper
- incusd/network/ovn: Add some missing indices
- incusd/network/ovn: Use get helper
- incusd/network/ovn: Fix LogicalSwitchPortIPs logic
- incusd/network/bridge: Fix gofmt
- incusd/network/ovn: Fix gofmt
- cmd/incus: Use proper timestamp check
- cmd/incus: Use consistent date format and timezone
- client: Rename network_peer for consistency
- cmd/incusd: Rename network_peer to network_peers
- shared/api: Rename network_allocation for consistency
- incusd/db: Fix comment typoes
- incusd/db/generate: Fix bad camel case handling
- incusd/db/network_peers: Fix duplicate type definitions
- incusd/auth: Drop Permission type
- incusd/auth: Add boilerplate doc strings
- incusd/images: Properly handle null creation and expiry dates
- incus: add completions for remotes
- incus: add completions for projects
- incusd/images: Fix reporting of images in multiple projects
- github: Add static build of lxd-to-incus
- lxd-to-incus: Add support for Alpine service name
- lxd-to-incus: Re-organize target list
- lxd-to-incus: Add support for APK
- Makefile: Add OVN IC to update-ovsdb
- incusd/network: Update OVS/OVN schemas
- incusd/network/ovn: Add IC clients
- incusd/network/ovn: Add GetName to NB client
- incusd/network/ovn: Add GetGateways to ICSB
- incusd/network/ovn: Introduce new errors
- incusd/network/ovn: Add CreateTransitSwitch and DeleteTransitSwitch to ICNB
- incusd/device/gpu_sriov: Add locking
- incusd/device/gpu_sriov: Re-locate vfio-pci loading
- incusd/device/gpu_sriov: Rework VF allocation logic
- incus/remote: Add a generate-certificate sub-command
- i18n: Update translation templates
- incusd/drivers/qmp: Add SetBlockThrottle
- incusd/device/disk/config: Add DiskLimits
- incusd/device/disk: Re-shuffle limit parsing
- incusd/device/disk: Add disk limits on VMs
- incusd/device/disk: Support live limits update for VMs
- incusd/instance/qemu: Support disk I/O limits
- incus/remote: Add missing docstrings
- incusd/certificates: Improve token handling when clustered
- cmd/incusd/api_1.0: Update context
- cmd/incusd/api_cluster: Update context
- cmd/incusd/api_internal: Update context
- cmd/incusd/daemon: Update context
- cmd/incusd/api_project: Update context
- cmd/incusd/certificates: Update context
- cmd/incusd/images: Update context
- cmd/incusd/instance: Update context
- cmd/incusd/network: Update context
- cmd/incusd/operations: Update context
- cmd/incusd/profiles: Update context
- cmd/incusd/storage: Update context
- cmd/incusd/warnings: Update context
- incusd/devices: Skip isolated threads from NUMA CPUs
- incusd/devices: Restrict CPU threads by NUMA node
- incusd/instance/qemu: Add support for limits.cpu.nodes
- incusd/device/gpu: Add support for limits.cpu.nodes for VF selection
- incusd: Fix import shadowing
- incusd/images: Fix potential race condition
- incusd/instance/qemu: Add support for NUMA node restrictions for memory
- incusd/apparmor/qemu: Silence apparmor failures
- incusd/network/ovs: Introduce new errors
- incusd/network/ovn/nb: Move SetChassisGroupPriority to new function signature
- incusd/network/ovn/sb: Move GetLogicalRouterPortActiveChassisHostname to new function signature
- incusd/network/ovs: Move GetBridge to new function signature
- incusd/network/ovs: Move CreateBridge to new function signature
- incusd/network/ovs: Move DeleteBridge to new function signature
- incusd/network/ovs: Move CreateBridgePort to new function signature
- incusd/network/ovs: Move GetChassisID to new function signature
- incusd/network/ovs: Move GetOVNBridgeMappings to new function signature
- incusd/network: Update for function changes
- incusd/device/nic: Update for function changes
- incusd: Update for function changes
- doc: Fix bad snapshot syntax
- Translated using Weblate (French)
- doc: Fix token creation procedure
- incusd/network/ovn/nb: Add GetLogicalSwitch
- incusd/network/ovn/nb: Replace ChassisGroupChassisDelete with SetChassisGroupPriority
- incusd/network/ovn/nb: Port CreateLogicalRouterPort to OVSDB
- incusd/network/ovn/nb: Replace LogicalRouterPortLinkChassisGroup with CreateLogicalRouterPort
- incusd/network/ovn/nb: Port CreateChassisGroup to OVSDB
- incusd/network/ovn/nb: Port CreateLogicalSwitch to OVSDB
- incusd/network/ovn: Update for function changes
- incusd/network/ovn: Remove state references
- incusd/state: Add OVNNB and OVNSB handles
- incusd: Update to use state for OVN
- incusd/device: Make init function return error
- incusd/device: Add OVN check on nicOVN
- client: Still return response on RawQuery error
- incus/query: Respect --raw for errors
- incusd/network/acl: Add OVN check
- incusd/network: Make init function return error
- incusd/network: Add OVN check on ovn driver
- incusd/api: Re-order config checks
- incusd: Add OVN loader
- Translated using Weblate (French)
- incusd/network/ovn/nb: Port CreateLogicalSwitchPort to OVSDB
- incusd/network/ovn/nb: Port DeleteLogicalSwitchPort to OVSDB
- incusd/network/ovn/nb: Port DeleteLogicalRouterPort to OVSDB
- incusd/network/ovn: Update for function changes
- incusd/network/ovs: Port GetOVNSouthboundDBRemoteAddress to OVSDB
- incusd/network/ovs: Port DeleteBridgePort to OVSDB
- incusd/network/ovs: Port GetInterfaceAssociatedOVNSwitchPort to OVSDB
- incusd/network/ovs: Align GetChassisID with other functions
- incusd: Update for OVS function changes
- incusd/network/ovn/icsb: Fix bad DB schema
- incusd/network/ovn/nb: Introduce GetLogicalRouterPort
- incusd/network/ovn/nb: Extend OVNSwitchPortOpts to handle router ports
- incusd/network/ovn/nb: Change type of RouterPort field to OVNRouterPort
- incusd/network/ovn/nb: Port DeleteChassisGroup to OVSDB
- incusd/network/ovn/icnb: Update DeleteTransitSwitch to handle missing switches
- incusd/network/ovn: Update for function changes
- Translated using Weblate (French)
- incus/completion: do not add a space after remote names completion
- incusd/device/disk: Disable virtiofsd caching
- incus-agent: Cleanup mount logic
- Translated using Weblate (French)
- incus: expose parseVolume to entire package
- incus: add completions for storage pools and volumes
- incusd/device/gpu_sriov: Fix default handling
- doc/packaging: Add mention of documentation
- incusd/auth: Fix --all-projects for restricted users
- doc: Add third party tools page
- gomod: Update dependencies
- incusd/auth/tls: Prevent project modifications
- doc: Update wordlist
- internal/usbid: allow path override of usb.ids path
- incus/completion: fix image names completion
- doc/environment: document INCUS_USBIDS_PATH
- incusd/instance/qemu/agent: Check for semanage
- incusd/project: Fix config name in ImageProjectFromRecord
- incus/restart: Fix long description
- i18n: Update translations
- lxd-to-incus: Handle common existing bridges
- shared/simplestreams: Remove defaultOS
- shared/simplestreams: Add NewLocalClient
- incus-simplestreams: Introduce new command
- incus-simplestreams: Simplify delete logic
- doc: Re-organize image server doc
- doc: Add section for incus-simplestreams
- incusd/seccomp: Add support for pidfd threads
- incus: add completions for clusters
- incus: add completions for cluster groups
- incus: add completions for cluster roles
- incus: add completions for config devices
- incus: add completions for config templates
- update translations
- doc: Update references to mage docs
- doc/backup: Remove bad reference
- incus: add completions for network acls
- shared/api: Add new structs to support configuration metadata
- client: Add GetMetadataConfiguration
- incusd: Rename documentation.go -> metadata.go
- doc/rest-api: Refresh swagger YAML
- shared/api/metadata: Add GetKeys to simplify usage
- incusd: Add support for JWT authentication
- gomod: Update dependencies
- tests: Add tls2jwt tool
- tests: Add JWT authentication test
- api: auth_tls_jwt
- doc/authentication: Add section on JWT
- doc/instances: Remove size.state requirement for live migration
- incusd/instance/qemu: Allow live migration without size.state
- shared/idmap: Support uid/gid in subuid/subgid
- shared/cliconfig: Copy clientcerts on remote copy
- shared/cliconfig: Add HasRemoteClientCertificate
- shared/cliconfig: Support per-remote client certificates
- doc: Add clientcerts
- incusd/cluster/config: Add oidc.claim
- incusd/auth/oidc: Add support for using a specific claim as username
- incusd: Pass OIDC claim to verifier
- api: oidc_claim
- doc: Update configs
- doc/howto/instances: Mention extra resources in ISO guidea
- doc/installing: Add Debian backport
- doc: Add backported to dictionary
- lxd-to-incus: Add support for LXD 5.21
- shared/cliconfig: Ensure client certificate key is 0600
- api: device_usb_serial
- doc: Add busnum, devnum and serial to USB devices
- shared/api: Add Serial to ResourcesUSBDevice
- incusd/resources: Add USB Serial
- incusd/devices/usb: Add serial, busnum and devnum options
- doc/rest-api: Refresh swagger YAML
- incusd/instance/qemu: Fix handling of > 64 limits.cpu
- incusd/device/gpu_sriov: Implement NUMA fallback
- incus: add completions for network forwards
- incus: add completions for network load balancers
- shared/validate: Remove stringInSlice
- shared/validate: Add And and Or functions
- shared/util: Move ParseUint32Range
- incusd/project: Update for ParseUint32Range
- doc/instance_options: Remove mention of limits.cpu.nodes from container-only section
- incusd/devices: Better handle bad config
- api: numa_cpu_balanced
- internal/instance: Add support for balanced NUMA nodes
- doc: Update configs
- incusd/instance/common: Add NUMA balancing
- incusd/instance/lxc: Add support for balanced NUMA allocation
- incusd/instance/qemu: Add support for balanced NUMA allocation
- incusd/devices: Add support for balanced NUMA allocation
- incusd/device/gpu_sriov: Simplify NUMA logic
- doc/cloud-init: Don't mention non-existing remotes
- doc/howto/images_remote: Fix wording around image servers
- doc/benchmark: Fix install command
- incusd/instance/common: Fix CanMigrate mutating devices
- incusd/instance/qemu: Reduce agent queries
- incusd/metrics: Don't filter out all server metrics
- incusd/auth/tls: Include project restrictions for metrics certificates
- incusd/auth/tls: Return project-aware checker for metrics
- incusd/metrics: Use project-specific checker if no global access
- internal/server/instance/lxd: add support for image.requirments.nesting
- api: add image_restriction_nesting
- doc/images: introduce requirements.nesting
- Show the count values in snapshot count mismatch error
- incus/admin/init: Use btrfs subvol in --auto
- incus-migrate: Clarify that disk image files must be raw
- incusd/network/ovn/icnb: Fix comment
- incusd/project: Re-format the comments
- incusd/project: Fix bad default value
- doc: Update configs
- incus/migrate: Add CSM support
- incusd/storage/backend: Better handle name conflicts
- incus-migrate: Support using the local server
- api: network_integrations
- shared/api: Add type and target_integration fields to NetworkPeersPost
- incusd/db/cluster: Add networks_integrations
- incusd/db/cluster: Re-generate schema
- incusd/db/cluster: Add generated DB code for network integrations
- incusd/db: Update network peer DB query functions
- client: Add check for network_integrations in CreateNetworkPeer
- incus/network/peer: Add support for network peer types
- shared/api: Add network integrations
- client: Add network integration functions
- incus/network: Introduce support for integrations
- incusd/auth: Add network integration functions
- shared/api: Add lifecycle events for network integrations
- incusd/lifecycle: Add network integration events
- incusd: Add network integration API
- incusd/db: Add GetNetworkPeersURLByIntegration
- incusd/network_integration: Add UsedBy field
- incusd/network_integrations: Add validator
- incusd/network/ovn: Add support for peering with OVN IC
- incusd/project: Add restricted.networks.integrations
- incusd/project: Add NetworkIntegrationAllowed
- incusd/network/integrations: Respect project restrictions
- incusd/network/ovn: Add support for integration restrictions
- incusd/auth/openfga: Update the model
- incusd/auth/openfga: Update the generated model
- incusd/auth/openfga: Handle model updates
- incusd: Remove openfga.store.model_id
- incusd/db/cluster: Remove openfga.store.model_id
- doc/ovn_peers: Add remote peering
- doc: Add documentation for network integrations
- doc/rest-api: Refresh swagger YAML
- i18n: Update translation templates
- doc: Update configs
- gomod: Update dependencies
Documentation¶
The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/
Installation¶
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Linux packages¶
Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.
https://linuxcontainers.org/incus/docs/main/installing/
Homebrew package for the Incus client¶
The client tool is available through HomeBrew for both Linux and MacOS.
https://formulae.brew.sh/formula/incus
Chocolatey package for the Incus client¶
The client tool is available through Chocolatey for Windows users.
https://community.chocolatey.org/packages/incus/6.0.0
Winget package for the Incus client¶
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Migrating from LXD¶
A lxd-to-incus
migration tool allows for in-place migration from LXD to Incus.
It's been tested with LXD versions as low as 4.0 LTS and as high as the latest LXD 5.21 bugfix release.
It allows for a very quick migration from LXD over to Incus, automatically checking for potential conflicts ahead of time.
More details can be found here: https://linuxcontainers.org/incus/docs/main/howto/server_migrate_lxd/
Support¶
Incus 6.0 LTS will be supported for a total of 5 years (until June 2029).
During the first 2 years, new point releases will be issued including a mix of bug and security fixes as well as some minor usabiltiy improvements. After that initial 2 years (after Incus 7.0 LTS is released), Incus 6.0 LTS will transition to security fixes only for the remaining 3 years.
This matches what we've been doing for our other projects (LXC and LXCFS) over the past 10 years.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues
Incus 0.7 has been released¶
26th of March 2024
Introduction¶
The Incus team is pleased to announce the release of Incus 0.7!
This is going to be our last release before Incus 6.0 LTS which is now scheduled to be released next week. As releases go, this is quite a busy one, which is how we like it before releasing an LTS, trying to keep the amount of new features in the LTS release itself to a minimum.
As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/
Enjoy!
New features¶
Network integrations¶
A new top-level concept, network integrations are a way to connect an Incus deployment to networks outside of its own control.
Currently the only implementation of the concept is OVN interconnection which makes it possible for an Incus cluster to directly peer its OVN networks with equivalent networks running on other Incus clusters or even other OVN users like OpenStack or Kubernetes.
Here is an example of creating a new network integration using an OVN interconnection gateway, then peering an existing network through it:
root@az01-server01:~# incus network integration create ovn-region ovn Network integration ovn-region created root@az01-server01:~# incus network integration set ovn-region ovn.northbound_connection tcp:[10.50.1.12]:6645,tcp:[10.50.2.13]:6645,tcp:[10.50.3.19]:6645 root@az01-server01:~# incus network integration set ovn-region ovn.southbound_connection tcp:[10.50.1.12]:6646,tcp:[10.50.2.13]:6646,tcp:[10.50.3.19]:6646 root@az01-server01:~# incus network peer create default region ovn-region --type=remote Network peer region created
Documentation: https://linuxcontainers.org/incus/docs/main/howto/network_integrations/
Image server management tool¶
A common way to run an Incus image server, be it for some internal servers or as a publicly available image server is through a static web server providing Incus images using simplestreams
.
To make this easier to set up, we're now introducing a new tool, incus-simplestreams
which can easily manage a simple image server, listing the images available, adding and removing images as well as generating the needed metadata files.
stgraber@dakara:~$ mkdir image-server stgraber@dakara:~$ cd image-server/ stgraber@dakara:~/image-server$ incus-simplestreams generate-metadata ~/Downloads/incus.tar.xz Operating system name: Red Hat Enterprise Linux Release name: 9 Variant name [default="default"]: Architecture name: x86_64 Description [default="Red Hat Enterprise Linux 9 (default) (x86_64) (202403260239)"]:· stgraber@dakara:~/image-server$ incus-simplestreams add ~/Downloads/incus.tar.xz ~/Downloads/rhel9.qcow2· stgraber@dakara:~/image-server$ incus-simplestreams list +------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+ | FINGERPRINT | DESCRIPTION | OS | RELEASE | VARIANT | ARCHITECTURE | TYPE | CREATED | +------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+ | 7d256e4fac6fc63fb47bc1e07e1c6ee234281cdf1ed21788c920d763b7bd93ba | Red Hat Enterprise Linux 9 x86_64 (202403252239) | Red Hat Enterprise Linux | 9 | default | x86_64 | virtual-machine | 2024/03/25 00:00 UTC | +------------------------------------------------------------------+--------------------------------------------------+--------------------------+---------+---------+--------------+-----------------+----------------------+ stgraber@dakara:~/image-server$ find . | sort . ./images ./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.incus.tar.xz ./images/ef6cf538776b05a64c789f16f235a757522724f2c490c7e118645be2eb920d30.qcow2 ./streams ./streams/v1 ./streams/v1/images.json ./streams/v1/index.json
Put that on an HTTPS capable web server and then add it with:
incus remote add my-server https://xyz.example.net --protocol=simplestreams
Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_servers/#tooling-to-manage-a-simplestreams-server
JSON Web Token authentication¶
Incus basically supports two mechanisms for remote authentication:
- TLS client certificates (added to the local trust store with or without restrictions)
- OpenID Connect external authentication (with or without OpenFGA for authorization)
The former is the most common for simple interactions with a remote Incus server.
Our own CLI tool and most 3rd party tools don't have any problems using a TLS keypair to establish the HTTPS connection and get authenticated that way.
But there are some situations, like running Incus behind a reverse HTTP(S) proxy where TLS client certificates can become a bit problematic.
To address that, we now support using a JSON Web Token (JWT) bearer token through the HTTP Authorization
field. That token can be generated by any user with a valid TLS client certificate by setting the Subject field to the certificate fingerprint, setting applicable NotBefore/NotAfter values and signing the JWT with their private key.
Incus will treat any such connections as equivalent to using the TLS client certificate.
stgraber@dakara:~$ openssl req -x509 -newkey rsa:4096 -sha384 -keyout client.key -nodes -out client.crt -days 1 -subj "/CN=test.local" .+.........+...+...+..+....+......+........+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+......+...+......+......+..+...+...+...............+.......+...+......+...+.....+......+....+......+..+...+......+....+............+.....+....+.....+.+............+..+.........+......+....+......+...........+....+........+...+...+.+...+..+..........+.....+...+......+............+...+.......+........+....+.....+.+..+.......+......+..+....+........+..........+...+..+.+.....+.+......+..+.......+.....+.+..+..........+..+....+..............+.+..+...........................+...+....+......+...+..............+.+..+....+.....+.+.........+...+..+....+..+.............+.........+.....+...+..........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...+..+...+.........+.+...+............+..+............+.+.......................................+.....+...+......+.........+......+.+.....+...+.+...........+......+.......+.....+.......+......+.....+..........+...+..+.........+....+.........+...........+......+.+..................+..+....+...........+.............+.....+....+..+......+............+..........+......+......+......+..+.............+.....+...+.+........+............+....+.................+.........+......+.......+...+.........+.....+....+......+........+.+..+....+......+........+....+...+.................+.+..+.........+....+............+.....+.........+....+.....+.......+.....+...............+..............................+....+........+.......+...........+.......+.....+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ...+...............+.+...+...+...+.........+..+.+...+..+.......+.....+.+..+...+...................+...........+......+....+..+............+...+.......+..+.........+....+...+.........+.....+...+...+....+...........+...+.+.........+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+......+..+...+......+.+..+.............+..+.......+........+...+...+....+...............+..+....+..+....+...+.....+.+.....+..........+.....+.+.....+....+............+.........+...+.....+......+......+.............+..+.+..+.......+...+........+...+.......+.........+......+..+.+......+...........+...+.........+...+...+....+..+.........+....+.....+.+......+.........+..+..........+..+...+...+....+..+...+.+.....+......+.+...+......+.....+.+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ----- stgraber@dakara:~$ incus config trust add-certificate client.crt --restricted --projects demo stgraber@dakara:~$ tls2jwt client.key client.crt now 120 eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0stgraber@dakara:~$· stgraber@dakara:~$ curl -s -k -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI2MzI3Y2Q5YmIxYTFmN2ExMWM3ODBkZjc4YjVkNjg5YzhkMGQ5YzcwZGQxOGQ1YTMyYzI1M2ZiODA0N2U2M2E0IiwiZXhwIjoxNzExNDcyMjE2LCJuYmYiOjE3MTE0NzIwOTYsImlhdCI6MTcxMTQ3MjA5Nn0.pNQ4AcgoymxWHROXVjcYX8QMKdf9QgRH3zex7qc16avX7_Ax1q_WFWzQWfP48Fh-ooeh9hBQKCQkZxjVxYx8Sy-cNqmkf1AI9KGh5uemHh3FYAbvebCTaIXan0B6glWHVnDSwLZKBWTDDai2VXOmUfntyV9yPJdTqxt1J0j8PNuIWzNVdFlcTxzpggcJMhbcqtf4GRwSMKx69HU5sP4AQ7GJ2cBvN7Im-nkRXTc7xiyYnIsFx0vIWJzojC4zwg0-C1LHKQD4DyEKhqOVISIKUSa3GhD6ajcDuGDS8af4Iz19sNPsSoSULBUG-a7E5lXx2vk802vOFFWV68ZHugsJHpdSpLFwTVixipQ1-QdKRozlMjNPguu-5CYxhZVR1p32lbN9D879xGbFXUgPJVwK25NILvbEMcrqnGPgKcRUjJlHtVljGOgXrjmG7dMiW5QOsyy1eIvJ1D1sNsG02fDTbchTzXHmIybxQTK0FXCyNDLOAl6xgW0Jundg7AN1uJU2cLEWy1x3TusqC7lyeTeF3WYT-G8xE2CU4GpLBeYWyLwuJgxRkaWcg9IXiivguPbWpcT0RMl1bmpn0TJ2VgEPCuSG0mJxMBp8HbAgxwgar8AHdpoZ43dCCwZnB0a0O_kmGkBE2xGKKvgTx_U6eSixZzyyNmHDC1KH1Vy1WW1ZcF0' https://localhost:8443/1.0/projects | jq { "type": "sync", "status": "Success", "status_code": 200, "operation": "", "error_code": 0, "error": "", "metadata": [ "/1.0/projects/demo" ] }
Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#using-json-web-token-jwt-to-perform-tls-authentication
Configurable OIDC username field¶
For those using OpenID Connect, you may have noticed that Incus will use the e-mail
claim when available as the user's identifier. Then if missing, it will rely on the Subject.
As different deployments may make different information available through OIDC claims, it's now possible to set oidc.claim
to the claim to use as the user identifier.
stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name stgraber@stgraber.org stgraber@dakara:~$ incus config set oidc.claim=name stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name Stéphane Graber stgraber@dakara:~$ incus config set oidc.claim=sub stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name 99cb8caa-3640-45b9-b87a-55266366aaf3 stgraber@dakara:~$ incus config set oidc.claim=email stgraber@dakara:~$ incus query s-dakara:/1.0 | jq -r .auth_user_name stgraber@stgraber.org
Improved NUMA handling¶
With this release, we spent a fair amount of time trying to improve both the container and virtual-machine performance on large systems. This obviously includes multi-socket systems but also AMD systems running in NPS4 or similar mode where each CPU is exposed as multiple NUMA nodes.
In general, our goal has been to make it easy to distribute workloads across NUMA nodes while keeping their CPU and memory properly pinned and also selecting PCIe resources that are closest to their NUMA node(s).
As part of that, a few things were done:
- limits.cpu.nodes
is now supported for virtual-machines too
- A new balanced
value has been added to limits.cpu.nodes
which will have Incus pick the NUMA node with the least instances configured to use it
- SR-IOV GPU selection now also considers NUMA nodes as part of the selection logic and when no match is found, will prefer PCIe devices that are attached to the same CPU socket
For example:
stgraber@gputest:~$ incus list stgraber-gpu -cns4,limits.cpu.nodes,volatile.cpu.nodes,volatile.gpu.last_state.pci.parent,volatile.gpu.last_state.vf.id +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | NAME | STATE | IPV4 | LIMITS CPU NODES | VOLATILE CPU NODES | VOLATILE GPU LAST STATE PCI PARENT | VOLATILE GPU LAST STATE VF ID | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu01 | RUNNING | 10.232.44.8 (enp5s0) | balanced | 0 | 0000:63:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu02 | RUNNING | 10.232.44.9 (enp5s0) | balanced | 2 | 0000:03:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu03 | RUNNING | 10.232.44.10 (enp5s0) | balanced | 4 | 0000:e3:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu04 | RUNNING | 10.232.44.11 (enp5s0) | balanced | 5 | 0000:c3:00.0 | 2 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu05 | RUNNING | 10.232.44.12 (enp5s0) | balanced | 6 | 0000:c3:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu06 | RUNNING | 10.232.44.13 (enp5s0) | balanced | 7 | 0000:83:00.0 | 0 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu07 | RUNNING | 10.232.44.15 (enp5s0) | balanced | 1 | 0000:43:00.0 | 3 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu08 | RUNNING | 10.232.44.16 (enp5s0) | balanced | 2 | 0000:03:00.0 | 0 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu09 | RUNNING | 10.232.44.17 (enp5s0) | balanced | 3 | 0000:03:00.0 | 2 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu10 | RUNNING | 10.232.44.18 (enp5s0) | balanced | 4 | 0000:e3:00.0 | 0 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu11 | RUNNING | 10.232.44.19 (enp5s0) | balanced | 5 | 0000:c3:00.0 | 0 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu12 | RUNNING | 10.232.44.20 (enp5s0) | balanced | 6 | 0000:83:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu13 | RUNNING | 10.232.44.21 (enp5s0) | balanced | 7 | 0000:83:00.0 | 2 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu14 | RUNNING | 10.232.44.22 (enp5s0) | balanced | 1 | 0000:43:00.0 | 1 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu15 | RUNNING | 10.232.44.23 (enp5s0) | balanced | 2 | 0000:43:00.0 | 2 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+ | stgraber-gpu16 | RUNNING | 10.232.44.24 (enp5s0) | balanced | 3 | 0000:03:00.0 | 3 | +----------------+---------+-----------------------+------------------+--------------------+------------------------------------+-------------------------------+
In this case we can see 16 VMs each using the new balanced
option for NUMA nodes and getting scheduled across 8 NUMA nodes (2 sockets AMD NPS4) with GPUs being selected to match.
More options to select USB devices¶
USB device passhtrough for both containers and virtual-machines has so far been using the vendorid
and productid
fields. This works fine as long as there is only one USB device of any one type connected to the system.
When multiple identical devices are present, the inability to distinguish them has been a problem.
To address this, three new fields have now been added to usb
devices:
- busnum
referring to the USB bus number
- devnum
referring to the USB device number (on its bus)
- serial
referring to the USB device serial number (not present on all devices)
The same fields can be found in the full Incus resources list through:
incus query /1.0/resources
Disk I/O throttling for VMs¶
One more feature gap between containers and virtual-machines is now gone.
The limits.write
and limits.read
properties on disk
devices are now properly enforced on virtual-machines by having Incus setup an I/O throttle in QEMU.
Both bytes per second and I/O per second type limits are supported.
Per-remote client certificates¶
It's now possible to put a <remote>.crt
and <remote>.key
file in a new clientcerts
folder within the Incus command line client config directory (typically ~/.config/incus/
) and have those certificates be used when interacting with that particular remote.
While this may be useful on its own, it becomes a lot more useful when combined with global remotes, which can be added in /etc/incus/config.yml
. Now with this feature, those global remotes can also have a client certificate made available in /etc/incus/clientcerts/
which will then be used by all users on the system.
Manual generation of the client certificate keypair¶
A new command to manually trigger the generation of the main client.crt
and client.key
keypair is now available.
This is done by running incus remote generate-certificate
Improvements to lxd-to-incus
¶
The lxd-to-incus
tool keeps evolving with every release.
In this one, it gains support for migrating users from the newly released LXD 5.21 LTS as well as handling Alpine installations.
Additionally, a static binary version of the tool is now available on Github, making it easier for users to fetch the latest version of the tool, useful as bugs get fixed in between Incus releases.
Improvements to incus-migrate
¶
The workload migration tool incus-migrate
has also seen a couple of small improvements.
It can now use the local Incus system as the target of the migration, useful when importing virtual-machine images from another virtualization tool.
And it's also now prompting for whether the imported virtual machine should be using a UEFI firmware or instead use a legacy BIOS.
Additional image restrictions¶
A bit of an internal detail, or at least only relevant to public image server operators, but two new image restrictions have been added:
requirements.nesting
which will require the container havesecurity.nesting=true
requirements.cdrom_agent
which will require that asource=agent:config
disk device be added to the virtual-machine
Those two can be used to flag specific images that need extra user interaction to work properly, resulting in a clear client-side error rather than starting a potentially broken instance.
Complete changelog¶
Here is a complete list of all changes in this release:
Full commit list
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- incus/image: Fix column handling with --all-projects
- Replace util.ValueInSlice with slices.Contains
- shared/util: Delete ValueInSlice function
- incus/image: Fix column handling with --all-projects
- incusd/instance/qemu: Relocate image requirement checks
- doc/images: Add requirements.cdrom_agent
- incusd/instance/qemu: Add support for requirements.cdrom_agent
- incusd/device/disk: Fix incorrect block volume usage
- Translated using Weblate (Japanese)
- incusd/network/ovn: Use ParseIPToNet instead of manual IPToNet and net.ParseIP
- incusd/network/ovn: Use listenAddressNet in family check
- incusd/instance/drivers: Disable architecture check on incus cp with snapshots
- Translated using Weblate (French)
- incusd/network/bridge: Set local address on all VXLAN tunnels
- incus/instance/qemu: Fix RecordOutput
- incus: add completions for instance actions and snapshots
- incus: add completions for profiles
- incusd/network/ovn: Introduce get helper
- incusd/network/ovn: Add some missing indices
- incusd/network/ovn: Use get helper
- incusd/network/ovn: Fix LogicalSwitchPortIPs logic
- incusd/network/bridge: Fix gofmt
- incusd/network/ovn: Fix gofmt
- cmd/incus: Use proper timestamp check
- cmd/incus: Use consistent date format and timezone
- client: Rename network_peer for consistency
- cmd/incusd: Rename network_peer to network_peers
- shared/api: Rename network_allocation for consistency
- incusd/db: Fix comment typoes
- incusd/db/generate: Fix bad camel case handling
- incusd/db/network_peers: Fix duplicate type definitions
- incusd/auth: Drop Permission type
- incusd/auth: Add boilerplate doc strings
- incusd/images: Properly handle null creation and expiry dates
- incus: add completions for remotes
- incus: add completions for projects
- incusd/images: Fix reporting of images in multiple projects
- github: Add static build of lxd-to-incus
- lxd-to-incus: Add support for Alpine service name
- lxd-to-incus: Re-organize target list
- lxd-to-incus: Add support for APK
- Makefile: Add OVN IC to update-ovsdb
- incusd/network: Update OVS/OVN schemas
- incusd/network/ovn: Add IC clients
- incusd/network/ovn: Add GetName to NB client
- incusd/network/ovn: Add GetGateways to ICSB
- incusd/network/ovn: Introduce new errors
- incusd/network/ovn: Add CreateTransitSwitch and DeleteTransitSwitch to ICNB
- incusd/device/gpu_sriov: Add locking
- incusd/device/gpu_sriov: Re-locate vfio-pci loading
- incusd/device/gpu_sriov: Rework VF allocation logic
- incus/remote: Add a generate-certificate sub-command
- i18n: Update translation templates
- incusd/drivers/qmp: Add SetBlockThrottle
- incusd/device/disk/config: Add DiskLimits
- incusd/device/disk: Re-shuffle limit parsing
- incusd/device/disk: Add disk limits on VMs
- incusd/device/disk: Support live limits update for VMs
- incusd/instance/qemu: Support disk I/O limits
- incus/remote: Add missing docstrings
- incusd/certificates: Improve token handling when clustered
- cmd/incusd/api_1.0: Update context
- cmd/incusd/api_cluster: Update context
- cmd/incusd/api_internal: Update context
- cmd/incusd/daemon: Update context
- cmd/incusd/api_project: Update context
- cmd/incusd/certificates: Update context
- cmd/incusd/images: Update context
- cmd/incusd/instance: Update context
- cmd/incusd/network: Update context
- cmd/incusd/operations: Update context
- cmd/incusd/profiles: Update context
- cmd/incusd/storage: Update context
- cmd/incusd/warnings: Update context
- incusd/devices: Skip isolated threads from NUMA CPUs
- incusd/devices: Restrict CPU threads by NUMA node
- incusd/instance/qemu: Add support for limits.cpu.nodes
- incusd/device/gpu: Add support for limits.cpu.nodes for VF selection
- incusd: Fix import shadowing
- incusd/images: Fix potential race condition
- incusd/instance/qemu: Add support for NUMA node restrictions for memory
- incusd/apparmor/qemu: Silence apparmor failures
- incusd/network/ovs: Introduce new errors
- incusd/network/ovn/nb: Move SetChassisGroupPriority to new function signature
- incusd/network/ovn/sb: Move GetLogicalRouterPortActiveChassisHostname to new function signature
- incusd/network/ovs: Move GetBridge to new function signature
- incusd/network/ovs: Move CreateBridge to new function signature
- incusd/network/ovs: Move DeleteBridge to new function signature
- incusd/network/ovs: Move CreateBridgePort to new function signature
- incusd/network/ovs: Move GetChassisID to new function signature
- incusd/network/ovs: Move GetOVNBridgeMappings to new function signature
- incusd/network: Update for function changes
- incusd/device/nic: Update for function changes
- incusd: Update for function changes
- doc: Fix bad snapshot syntax
- Translated using Weblate (French)
- doc: Fix token creation procedure
- incusd/network/ovn/nb: Add GetLogicalSwitch
- incusd/network/ovn/nb: Replace ChassisGroupChassisDelete with SetChassisGroupPriority
- incusd/network/ovn/nb: Port CreateLogicalRouterPort to OVSDB
- incusd/network/ovn/nb: Replace LogicalRouterPortLinkChassisGroup with CreateLogicalRouterPort
- incusd/network/ovn/nb: Port CreateChassisGroup to OVSDB
- incusd/network/ovn/nb: Port CreateLogicalSwitch to OVSDB
- incusd/network/ovn: Update for function changes
- incusd/network/ovn: Remove state references
- incusd/state: Add OVNNB and OVNSB handles
- incusd: Update to use state for OVN
- incusd/device: Make init function return error
- incusd/device: Add OVN check on nicOVN
- client: Still return response on RawQuery error
- incus/query: Respect --raw for errors
- incusd/network/acl: Add OVN check
- incusd/network: Make init function return error
- incusd/network: Add OVN check on ovn driver
- incusd/api: Re-order config checks
- incusd: Add OVN loader
- Translated using Weblate (French)
- incusd/network/ovn/nb: Port CreateLogicalSwitchPort to OVSDB
- incusd/network/ovn/nb: Port DeleteLogicalSwitchPort to OVSDB
- incusd/network/ovn/nb: Port DeleteLogicalRouterPort to OVSDB
- incusd/network/ovn: Update for function changes
- incusd/network/ovs: Port GetOVNSouthboundDBRemoteAddress to OVSDB
- incusd/network/ovs: Port DeleteBridgePort to OVSDB
- incusd/network/ovs: Port GetInterfaceAssociatedOVNSwitchPort to OVSDB
- incusd/network/ovs: Align GetChassisID with other functions
- incusd: Update for OVS function changes
- incusd/network/ovn/icsb: Fix bad DB schema
- incusd/network/ovn/nb: Introduce GetLogicalRouterPort
- incusd/network/ovn/nb: Extend OVNSwitchPortOpts to handle router ports
- incusd/network/ovn/nb: Change type of RouterPort field to OVNRouterPort
- incusd/network/ovn/nb: Port DeleteChassisGroup to OVSDB
- incusd/network/ovn/icnb: Update DeleteTransitSwitch to handle missing switches
- incusd/network/ovn: Update for function changes
- Translated using Weblate (French)
- incus/completion: do not add a space after remote names completion
- incusd/device/disk: Disable virtiofsd caching
- incus-agent: Cleanup mount logic
- Translated using Weblate (French)
- incus: expose parseVolume to entire package
- incus: add completions for storage pools and volumes
- incusd/device/gpu_sriov: Fix default handling
- doc/packaging: Add mention of documentation
- incusd/auth: Fix --all-projects for restricted users
- doc: Add third party tools page
- gomod: Update dependencies
- incusd/auth/tls: Prevent project modifications
- doc: Update wordlist
- internal/usbid: allow path override of usb.ids path
- incus/completion: fix image names completion
- doc/environment: document INCUS_USBIDS_PATH
- incusd/instance/qemu/agent: Check for semanage
- incusd/project: Fix config name in ImageProjectFromRecord
- incus/restart: Fix long description
- i18n: Update translations
- lxd-to-incus: Handle common existing bridges
- shared/simplestreams: Remove defaultOS
- shared/simplestreams: Add NewLocalClient
- incus-simplestreams: Introduce new command
- incus-simplestreams: Simplify delete logic
- doc: Re-organize image server doc
- doc: Add section for incus-simplestreams
- incusd/seccomp: Add support for pidfd threads
- incus: add completions for clusters
- incus: add completions for cluster groups
- incus: add completions for cluster roles
- incus: add completions for config devices
- incus: add completions for config templates
- update translations
- doc: Update references to mage docs
- doc/backup: Remove bad reference
- incus: add completions for network acls
- shared/api: Add new structs to support configuration metadata
- client: Add GetMetadataConfiguration
- incusd: Rename documentation.go -> metadata.go
- doc/rest-api: Refresh swagger YAML
- shared/api/metadata: Add GetKeys to simplify usage
- incusd: Add support for JWT authentication
- gomod: Update dependencies
- tests: Add tls2jwt tool
- tests: Add JWT authentication test
- api: auth_tls_jwt
- doc/authentication: Add section on JWT
- doc/instances: Remove size.state requirement for live migration
- incusd/instance/qemu: Allow live migration without size.state
- shared/idmap: Support uid/gid in subuid/subgid
- shared/cliconfig: Copy clientcerts on remote copy
- shared/cliconfig: Add HasRemoteClientCertificate
- shared/cliconfig: Support per-remote client certificates
- doc: Add clientcerts
- incusd/cluster/config: Add oidc.claim
- incusd/auth/oidc: Add support for using a specific claim as username
- incusd: Pass OIDC claim to verifier
- api: oidc_claim
- doc: Update configs
- doc/howto/instances: Mention extra resources in ISO guidea
- doc/installing: Add Debian backport
- doc: Add backported to dictionary
- lxd-to-incus: Add support for LXD 5.21
- shared/cliconfig: Ensure client certificate key is 0600
- api: device_usb_serial
- doc: Add busnum, devnum and serial to USB devices
- shared/api: Add Serial to ResourcesUSBDevice
- incusd/resources: Add USB Serial
- incusd/devices/usb: Add serial, busnum and devnum options
- doc/rest-api: Refresh swagger YAML
- incusd/instance/qemu: Fix handling of > 64 limits.cpu
- incusd/device/gpu_sriov: Implement NUMA fallback
- incus: add completions for network forwards
- incus: add completions for network load balancers
- shared/validate: Remove stringInSlice
- shared/validate: Add And and Or functions
- shared/util: Move ParseUint32Range
- incusd/project: Update for ParseUint32Range
- doc/instance_options: Remove mention of limits.cpu.nodes from container-only section
- incusd/devices: Better handle bad config
- api: numa_cpu_balanced
- internal/instance: Add support for balanced NUMA nodes
- doc: Update configs
- incusd/instance/common: Add NUMA balancing
- incusd/instance/lxc: Add support for balanced NUMA allocation
- incusd/instance/qemu: Add support for balanced NUMA allocation
- incusd/devices: Add support for balanced NUMA allocation
- incusd/device/gpu_sriov: Simplify NUMA logic
- doc/cloud-init: Don't mention non-existing remotes
- doc/howto/images_remote: Fix wording around image servers
- doc/benchmark: Fix install command
- incusd/instance/common: Fix CanMigrate mutating devices
- incusd/instance/qemu: Reduce agent queries
- incusd/metrics: Don't filter out all server metrics
- incusd/auth/tls: Include project restrictions for metrics certificates
- incusd/auth/tls: Return project-aware checker for metrics
- incusd/metrics: Use project-specific checker if no global access
- internal/server/instance/lxd: add support for image.requirments.nesting
- api: add image_restriction_nesting
- doc/images: introduce requirements.nesting
- Show the count values in snapshot count mismatch error
- incus/admin/init: Use btrfs subvol in --auto
- incus-migrate: Clarify that disk image files must be raw
- incusd/network/ovn/icnb: Fix comment
- incusd/project: Re-format the comments
- incusd/project: Fix bad default value
- doc: Update configs
- incus/migrate: Add CSM support
- incusd/storage/backend: Better handle name conflicts
- incus-migrate: Support using the local server
- api: network_integrations
- shared/api: Add type and target_integration fields to NetworkPeersPost
- incusd/db/cluster: Add networks_integrations
- incusd/db/cluster: Re-generate schema
- incusd/db/cluster: Add generated DB code for network integrations
- incusd/db: Update network peer DB query functions
- client: Add check for network_integrations in CreateNetworkPeer
- incus/network/peer: Add support for network peer types
- shared/api: Add network integrations
- client: Add network integration functions
- incus/network: Introduce support for integrations
- incusd/auth: Add network integration functions
- shared/api: Add lifecycle events for network integrations
- incusd/lifecycle: Add network integration events
- incusd: Add network integration API
- incusd/db: Add GetNetworkPeersURLByIntegration
- incusd/network_integration: Add UsedBy field
- incusd/network_integrations: Add validator
- incusd/network/ovn: Add support for peering with OVN IC
- incusd/project: Add restricted.networks.integrations
- incusd/project: Add NetworkIntegrationAllowed
- incusd/network/integrations: Respect project restrictions
- incusd/network/ovn: Add support for integration restrictions
- incusd/auth/openfga: Update the model
- incusd/auth/openfga: Update the generated model
- incusd/auth/openfga: Handle model updates
- incusd: Remove openfga.store.model_id
- incusd/db/cluster: Remove openfga.store.model_id
- doc/ovn_peers: Add remote peering
- doc: Add documentation for network integrations
- doc/rest-api: Refresh swagger YAML
- i18n: Update translation templates
- doc: Update configs
- gomod: Update dependencies
Documentation¶
The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/
Packages¶
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Installing the Incus server on Linux¶
Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.
https://linuxcontainers.org/incus/docs/main/installing/
Homebrew package for the Incus client¶
The client tool is available through HomeBrew for both Linux and MacOS.
https://formulae.brew.sh/formula/incus
Chocolatey package for the Incus client¶
The client tool is available through Chocolatey for Windows users.
https://community.chocolatey.org/packages/incus/0.7
Winget package for the Incus client¶
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Support¶
At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues
Incus 0.6 has been released¶
23rd of February 2024
Introduction¶
The Incus team is pleased to announce the release of Incus 0.6!
This second release of 2024 features a number of improvements, both large and small.
It comes with a completely new storage driver for cluster users, import/export support for storage buckets, a number of improvements for OVN users, support for new container kernel features, quite a few improvements to the migration tooling and more!
As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/
Enjoy!
New features¶
Clustered LVM storage driver¶
A new storage driver named lvmcluster
has been added.
As the name implies, this is for clustered use of the LVM storage driver.
This relies on llvmlockd
and a lock manager like sanlock
to provide concurrent access to a shared storage device across a cluster.
This is aimed at anyone who wants to run an Incus cluster and use a shared storage device like a FiberChannel SAN, iSCSI export, NVMEoF/NVMEoTCP disk, ...
With the ability of having the same storage available on all servers comes support for much faster live migrations, server evacuations and the ability to automatically recover should a server suddenly die.
Documentation: https://linuxcontainers.org/incus/docs/main/reference/storage_lvm/#lvmcluster-driver-in-incus
Storage bucket backup and import¶
It is now possible to backup a full storage bucket and later re-import back into Incus.
stgraber@dakara:~$ incus storage bucket create default foo Storage bucket foo created Admin access key: HD2GWC1KX693MFVT3P30 Admin secret key: tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv stgraber@dakara:~$ dd if=/dev/random of=out.img bs=4M count=10 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.0975758 s, 430 MB/s stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv put out.img s3://foo upload: 'out.img' -> 's3://foo/out.img' [part 1 of 3, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 262.42 MB/s done upload: 'out.img' -> 's3://foo/out.img' [part 2 of 3, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 241.54 MB/s done upload: 'out.img' -> 's3://foo/out.img' [part 3 of 3, 10MB] [1 of 1] 10485760 of 10485760 100% in 0s 241.99 MB/s done stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo 2024-02-23 03:26 41943040 s3://foo/out.img stgraber@dakara:~$ incus storage bucket export default foo Backup exported successfully! stgraber@dakara:~$ incus storage bucket delete default foo Storage bucket foo deleted stgraber@dakara:~$ incus storage bucket import default backup.tar.gz stgraber@dakara:~$ s3cmd --host=127.0.0.1:8445 --host-bucket=127.0.0.1:8445 --no-check-certificate --access_key=HD2GWC1KX693MFVT3P30 --secret_key=tX9+G9G5UlcuT21yatKIUImSGvcXzPyA7ONgkjUv ls s3://foo 2024-02-23 03:27 41943040 s3://foo/out.img
Listing image across all projects¶
Just as it's possible to list instances across all projects using the --all-projects
, it's now possible to do the same but for images.
stgraber@dakara:~$ incus image list --all-projects +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+ | PROJECT | ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE | +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+ | default | | 256f59a72af5 | no | Ubuntu jammy amd64 (20240222_07:42) | x86_64 | VIRTUAL-MACHINE | 267.19MiB | Feb 23, 2024 at 12:27am (UTC) | +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+ | default | | 0941e441dbb9 | no | Alpine edge amd64 (20240222_13:00) | x86_64 | CONTAINER | 2.93MiB | Feb 23, 2024 at 12:27am (UTC) | +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+ | default | | d5fc6024f0fa | no | Openwrt snapshot amd64 (20240222_11:57) | x86_64 | CONTAINER | 3.50MiB | Feb 23, 2024 at 2:55am (UTC) | +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+ | demo | | f44a6b4e56f4 | no | Archlinux current amd64 (20240222_04:18) | x86_64 | CONTAINER | 188.86MiB | Feb 23, 2024 at 3:32am (UTC) | +---------+-------+--------------+--------+------------------------------------------+--------------+-----------------+-----------+-------------------------------+
Over the API, this is done by passing ?all-projects=true
.
binfmt_misc in unprivileged containers¶
Linux 6.7 added support for mounting of binfmt_misc
inside of unprivileged containers.
Incus 0.6 will detect kernels that support this feature and when they do, it will no longer bind-mount binfmt_misc
from the host system but instead allow it to be mounted from within the container.
stgraber@castiana:~$ incus launch images:ubuntu/22.04 foo Launching foo stgraber@castiana:~$ incus exec foo bash root@foo:~# uname -a Linux foo 6.7.4-zabbly+ #debian12 SMP PREEMPT_DYNAMIC Mon Feb 5 23:37:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux root@foo:~# mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc/ root@foo:~# ls -lh /proc/sys/fs/binfmt_misc/ total 0 --w------- 1 root root 0 Feb 23 03:46 register -rw-r--r-- 1 root root 0 Feb 23 03:46 status
The binfmt_misc
filesystem is used to define custom binary formats for emulation.
It's most notably used by qemu-user-static as a way to run binaries of other architectures.
Control over shared block storage volumes¶
A new security.shared
volume storage option has been introduced to control whether a custom block volume should be allowed to be used by multiple instances.
It is now required to set that configuration key prior to adding a custom block volume to a profile or add it directly to multiple virtual machines.
stgraber@castiana:~$ incus config device add lvm01 shared disk pool=default source=shared-lvm io.bus=nvme Device shared added to lvm01 stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme Error: Failed add validation for device "shared": Cannot add un-shared custom storage block volume to more than one instance stgraber@castiana:~$ incus storage volume set default shared-lvm security.shared=true stgraber@castiana:~$ incus config device add lvm02 shared disk pool=default source=shared-lvm io.bus=nvme Device shared added to lvm02 stgraber@castiana:~$ incus config device add lvm03 shared disk pool=default source=shared-lvm io.bus=nvme Device shared added to lvm03 stgraber@castiana:~$ incus start lvm01 lvm02 lvm03
OVN logical router name in network info¶
To make it easier to see what's going on within OVN, incus network info
now gives you the name of the logical router for a particular network.
root@abydos:~# incus network info default Name: default MAC address: 00:16:3e:38:dd:28 MTU: 1500 State: up Type: broadcast IP addresses: inet 10.180.103.1/24 (link) inet6 2602:fc62:a:1004::1/64 (link) Network usage: Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0 OVN: Chassis: abydos Logical router: incus-net6-lr
File ownership and permissions in image templates¶
Template files can now have a uid
, gid
and mode
set on them.
This can be particularly useful if a template is meant to be an executable shell script.
stgraber@castiana:~$ incus config metadata show foo architecture: amd64 creation_date: 1708588077 expiry_date: 1711180077 properties: architecture: amd64 description: Ubuntu jammy amd64 (20240222_07:42) name: ubuntu-jammy-amd64-default-20240222_07:42 os: ubuntu release: jammy serial: "20240222_07:42" variant: default templates: /etc/hostname: when: - create - copy create_only: false template: hostname.tpl properties: {} /etc/hosts: when: - create - copy create_only: false template: hosts.tpl properties: {} /root/hello.sh: when: - start create_only: false template: hello.tpl properties: {} uid: "1000" gid: "2000" mode: "0755" stgraber@castiana:~$ incus config template show foo hello.tpl #!/bin/sh echo "Hello world!" stgraber@castiana:~$ incus start foo stgraber@castiana:~$ incus exec foo bash root@foo:~# ls -lh /root/hello.sh -rwxr-xr-x 1 ubuntu 2000 30 Feb 23 04:07 /root/hello.sh root@foo:~# /root/hello.sh Hello world!
Documentation: https://linuxcontainers.org/incus/docs/main/reference/image_format/#template-rules
Encrypted EC client certificate keys¶
For those interacting with remote Incus servers, you may not know that it's possible to protect your Incus private key with a password.
Up until now, this was only possible for RSA keys, but with Incus 0.6, we're now adding support for EC keys too. That's particularly relevant as EC keys have been the default for a while now.
Documentation: https://linuxcontainers.org/incus/docs/main/authentication/#encrypting-local-keys
Worth noting that if using this feature, you'll likely also want to make use of the recently introduced "keepalive mode" as this will then significantly reduce the number of password prompts you'll get while using Incus.
lxd-to-incus improvements¶
Lastly, lxd-to-incus
saw a number of improvements:
- Support for Void Linux
- Detection of the
boot.debug_edk2
configuration key - Handling of OVN SSL database connections
- Automatic clearing of the simplestreams cache during migration
Complete changelog¶
Here is a complete list of all changes in this release:
Full commit list
- incusd/instance/qemu/qmp: Don't risk blocking QMP on eject
- github: Switch to minio deb and pin working version
- github: Use stable MicroCeph
- incusd/db: Fix error handling in CreateNetworkPeer
- incusd/apparmor: Allow more mounts in unprivileged containers
- Translated using Weblate (Portuguese (Brazil))
- incusd/instance/qemu/agent-loader: Handle legacy lxd-agent-loader
- incusd/device/disk: Use genisoimage when mkisofs can't be found
- shared/api: Add example of Chassis in NetworkStateOVN
- doc/rest-api: Refresh swagger YAML
- api: network_state_ovn_lr
- shared/api: Add LogicalRouter field to NetworkStateOVN
- incusd/network/ovn: Expose LogicalRouter name
- incus: Add OVN logical router name
- i18n: Update translation templates
- doc/rest-api: Refresh swagger YAML
- scripts/bash: Don't follow aliases
- Translated using Weblate (Japanese)
- test: Adapt test_database_no_disk_space() to newer libraft versions
- incus/launch: Fix --console
- api: image_template_permissions
- doc: Add uid/gid/mode to image template metadata
- shared/api: Add owner/permission fields to ImageMetadataTemplate
- doc/rest-api: Refresh swagger YAML
- incusd/instance/lxc: Add support for template ownership/mode
- incus-agent: Add support for template ownership/mode
- lxd-to-incus: Add comment
- lxd-to-incus: Add cache cleanup logic
- .github: Bump minimum Go version to 1.21
- Makefile: Bump minimum Go version to 1.21
- gomod: Update dependencies
- doc: Bump minimum Go version to 1.21
- Translated using Weblate (French)
- incusd/daemon: Fix log format
- incusd/sys: Sort kernel features
- incusd/sys: Add doc strings
- incusd/checkfeature: Add unprivileged binfmt detection
- incusd/sys: Add UnprivBinfmt
- incusd/daemon: Detect unprivileged binfmt_misc
- incusd/api_1.0: Add unpriv_binfmt kernel feature
- incusd/instance/lxc: Support unpriv binfmt_misc
- incusd/apparmor: Support unpriv binfmt_misc
- Translated using Weblate (Italian)
- completion: support returning non-incus remotes
- incus/image: add dynamic command line completions
- shared/cliconfig: Support SSH encrypted keys
- tests: Add crypto/ssh
- doc/remotes: Add mention of keepalive
- doc: Add Ansible to wordlist
- doc/authentication: Hints on encrypting client key
- api: images_all_projects
- lxd-to-incus: Indicate what existing configuration was found
- shared/api: Add project property to Image
- client: Add GetImagesAllProjects
- incusd/images: Add support for all_projects
- doc/rest-api: Refresh swagger YAML
- incus/image: Add --all-projects flag to list
- i18n: Update translation templates
- shared/cliconfig: Fix static analysis
- gomod: Update dependencies
- Makefile: Pin rpc2 version due to OVN issue
- incusd/instance/qemu: Re-shuffle agent NIC handling
- incusd/instance/qemu/agent-loader: Handle SELinux
- incus/config: add completions
- incus-agent: Load virtio_net before configuring NICs
- incus: Run gofmt
- doc/storage/lvm: Cleanup tables
- incusd/patches: Move lvm.vg.force_reuse to be server-specific
- incusd/db: Mark lvm.vg.force_reuse as server-specific
- doc/faq: Add mention of the -mtu network interfaces
- incusd/device: Bump base VM filesystem volume to 500MiB
- incusd/storage: Create rootfs dir on empty instances
- incusd/storage/quota: Don't fail on missing project
- incusd/instances: Use correct project on cross-project copy
- incusd/patches: Fix bad SQL query
- incusd/instances: Don't bypass instance limit check
- Added translation using Weblate (Portuguese)
- incus-agent: Re-order imports
- api: Add storage_bucket_backup extension
- shared/api: Add storage bucket backup
- incusd/db: Add storage bucket backup functions
- incusd/db/operation: Add storage volume backup types
- incusd/lifecycle: Add storage bucket backup events
- incusd/project: Add StorageBucket function
- incusd/storage/s3: Add transfer manager
- incusd: Add storage bucket backup
- client: Add storage bucket backup
- incus: Add storage bucket import/export
- doc/rest-api: Refresh swagger YAML
- i18n: Update translation templates
- test: Add storage bucket backup
- client: Remove ceph-specific logic
- incusd/storage/s3: Fix typo
- incus: Fix import shadowing
- incus: Fix comments on exported functions
- client: Fix comments on exported functions
- incusd: Fix import shadowing
- incusd: Fix comments on exported functions
- incusd/cluster: Make remote storage volume logic generic
- incusd/db: Don't hardcode remote storage drivers
- incusd/storage_volumes: Don't hardcode ceph
- incusd/instances: Update ceph-specific comment
- incusd/cluster: Update ceph-specific comment
- incusd/instances: Generalize Ceph logic
- lxd-to-incus: Support SSL authentication in OVN
- lxd-to-incus: Don't fail on missing OVN bridge mapping
- lxd-to-incus: Ignore OVN chassis external_ids
- lxd-to-incus: Add boot.debug_edk2 to deprecated keys
- lxd-to-incus: Support Void Linux
- cmd/incusd: Disable the architecture check on incus cp/mv
- incusd/instance: Disable the architecture check on incus cp/mv
- api: storage_lvm_cluster
- incus/admin/init: Add LVM cluster logic
- i18n: Update translation templates
- lxd-to-incus: Add LVM cluster
- doc: Add LVM cluster
- shared/api: Add cephobject
- shared/api: Add lvmcluster
- doc/rest-api: Refresh swagger YAML
- incusd/storage/lvm: Add clustered LVM
- incusd/storage/lvm: Fix import shadowing
- incusd/storage/lvm: Tweak locking in cluster
- internal/server/device/config: Don't include empty values
- internal/server/device/config: Fix return values of Update function
- api: Add shared_custom_block_volumes API extension
- doc/reference: Add
security.shared
config key - instance/server/device: Validate shared block devices
- internal/server/storage: Handle
security.shared
update - internal/server/storage: Allow security.shared key for custom block volumes
- internal/server/storage/drivers: Handle security.shared in fillVolumeConfig
- incusd/instance/qemu: Cap hotplug CPU slots to 64
- incusd/storage/lvm: Make gofmt happy
- shared/subprocess: Improve error handling
- incusd/instance/qemu: Improve error handling
- incusd/network/ovs: Wait for bridge interface to appear
- incusd/storage/zfs: Fix refresh of VM volumes
- internal/instance: Fix volatile key definitions
- doc: Update configs
- incusd/network/ovn: Extend validateExternalSubnet to allow uplink subnets
- incusd/network/ovn: Add static routes to load-balancer and forwards
- gomod: Update dependencies
Documentation¶
The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/
Packages¶
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Installing the Incus server on Linux¶
Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.
https://linuxcontainers.org/incus/docs/main/installing/
Homebrew package for the Incus client¶
The client tool is available through HomeBrew for both Linux and MacOS.
https://formulae.brew.sh/formula/incus
Chocolatey package for the Incus client¶
The client tool is available through Chocolatey for Windows users.
https://community.chocolatey.org/packages/incus/0.6
Winget package for the Incus client¶
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Support¶
At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues
Incus 0.5.1 has been released¶
29th of January 2024
Introduction¶
The Incus team is pleased to announce the release of Incus 0.5.1!
This is an unusual release as we normally do not issue point releases on top of the monthly feature releases. But we felt this was needed this time due to some pretty important bugfixes and a minor feature addition needed to accommodate those running CentOS/Alma/Rocky virtual machines.
Most changes are on the server side, so if you're only using the command line client, there is no strong reason to upgrade from 0.5 to 0.5.1.
As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/
Enjoy!
Highlights¶
Alternative way to get the VM agent¶
With Incus 0.5, the distribution mechanism for the Incus VM agent changed a bit.
In the past, we had a single share named config
which would include both the instance-specific agent configuration and the incus-agent
binary.
This was a bit wasteful, requiring a copy of the 15-20MB large incus-agent
for every VM but was still somewhat manageable. This share was also exposed as both 9p
and virtiofs
. Leading to two processes running on the host system for every Incus VM.
With support for multiple agent binaries, copying them for every VM really wasn't an option anymore, so a separate share was introduced just for the binaries. As we really didn't want to end up with another two processes running on the host per VM, we made the decision to only make those internal shares be available over 9p
.
Testing on a variety of images, including CentOS 7 showed that this would be fine.
9p
is lower performance than virtiofs
but as those shares are only use for a couple of seconds on every VM boot, that really wasn't a concern. User defined shares would still be exposed over virtiofs
so those would still get the high performance option.
What we failed to notice is that for some reason, CentOS 8-Stream, CentOS 9-Stream and other distributions that are derivatives of RHEL 8/9, do not ship the 9p
kernel driver at all...
This means that those instances no longer had a way to fetch an agent, leading to broken incus exec
and incus file
.
We still don't feel like running 4 host processes for every single Incus VM just to make things work on those few images. Instead, what we're introducing with Incus 0.5.1 is a new agent drive, effectively an extra disk which can be attached to those specific VMs, providing those files through what looks like a CD-ROM drive rather than being retrieved over a networked filesystem.
So to run CentOS 9-Stream, one now needs to do:
incus create images:centos/9-Stream centos --vm incus config device add centos agent disk source=agent:config incus start centos
If you run many such VMs, a better option is likely by creating a profile for it:
incus profile create vm-agent incus profile device add vm-agent agent disk source=agent:config
At which point you can do:
incus launch images:centos/9-Stream centos --vm -p default -p vm-agent
This is obviously not ideal and adds a few more steps when creating VMs for those distributions but this new mechanism now offers a way to get the agent up and running in just about any environment.
NOTE: We're not considering always providing that extra device as it takes some resources to generate the cdrom device and uses some extra disk on the host. So it's best added only when needed.
Fixed handling of stopped instances during evacuation¶
A bug introduced with Incus 0.5 was causing stopped instances to get relocated to other systems during evacuation, even if the instance was configured to remain where it was.
This has now been corrected and instances using stopped
, force-stop
or stateful-stop
are now guaranteed to remain on their current server.
Database performance fixes¶
Database improvements in Incus 0.5 accidentally caused some nested database transactions to occur when fetching network information details for a large number of instances.
This would only really become visible when using an Incus cluster that also serves DNS zones and has its metrics scraped by Prometheus. This combination would cause large spikes in API requests every 15s or so, which would then start triggering timeouts and retries, eventually leading to other API requests piling up and timing out.
The logic has now been changed to remove such nested transactions and further optimizations were also made to save some database interactions during very command API interactions like executing commands instance of instances.
Complete changelog¶
Here is a complete list of all changes in this release:
Full commit list
- Translated using Weblate (German)
- Translated using Weblate (Dutch)
- incus/action: Fix resume
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- doc: Remove net_prio
- incusd/cgroup: Fully remove net_prio
- incusd/warningtype: Remove net_prio
- incusd/cgroup: Look for full cgroup controllers list at the root
- incusd/dns: Serialize DNS queries
- incusd/network: Optimize UsedByInstanceDevices
- incusd/backups: Simplify missing backup errors
- tests: Update for current backup errors
- incusd/cluster: Optimize ConnectIfInstanceIsRemote
- incusd/instance/qemu/agent-loader: Fix to work with busybox
- doc/installing.md: add a gentoo-wiki link under Gentoo section
- Translated using Weblate (French)
- Translated using Weblate (Dutch)
- incusd/device/disk: Better cleanup cloud-init ISO
- incusd/instance/qemu/qmp: Add Eject command
- incusd/instance/qemu/qmp: Handle eject requests
- api: agent_config_drive
- doc/devices/disk: Add agent:config drive
- incusd/device/disk: Add agent config drive
- incusd/project: Add support for agent config drive
- incusd/instance/qemu/agent-loader: Handle agent drive
- incusd/db/warningtype: gofmt
- incusd/loki: Sort lifecycle context keys
- incusd/instance/qemu/agent-loader: Don't hardcode paths
- incusd/cluster: Fix evacuation of stopped instances
Documentation¶
The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/
Packages¶
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Installing the Incus server on Linux¶
Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.
https://linuxcontainers.org/incus/docs/main/installing/
Homebrew package for the Incus client¶
The client tool is available through HomeBrew for both Linux and MacOS.
https://formulae.brew.sh/formula/incus
Chocolatey package for the Incus client¶
The client tool is available through Chocolatey for Windows users.
https://community.chocolatey.org/packages/incus/0.5.1
Winget package for the Incus client¶
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Support¶
At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues
Incus 0.5 has been released¶
26th of January 2024
Introduction¶
The Incus team is pleased to announce the release of Incus 0.5!
This is our first release of 2024 and it's quite a busy one! It's also the first release to feature no change coming from LXD following their decision to re-license to AGPLv3.
This release comes with a number of welcome improvements to the Incus CLI, a number of new virtual machine features, more options to handle cluster evacuations and host shutdown and some other smaller features and improvements!
On top of that, we've got quite a lot of bugfixes as well as a number of database improvements which should yield noticeable performance improvements especially in clusters.
As usual, you can try it for yourself online: https://linuxcontainers.org/incus/try-it/
Enjoy!
Highlights¶
Ansible, Terraform/OpenTofu and Packer¶
Over the past few months, Incus support has grown quite a bit in common tools!
- A connection plugin for Incus can now be found in Ansible.
- We have an official provider for Terraform and OpenTofu
- There is an available Packer plugin for Incus
Linux distribution packages¶
Since the last release of Incus, additional packages are now available for:
- Arch Linux
- Debian (testing/unstable)
- Ubuntu (noble)
- Void Linux
You'll find all instructions in our installation guide.
Translations¶
We've spent a bit of time cleaning up translations and setting up Weblate for Incus.
It's now easier than ever to log into Weblate and translate the Incus CLI into your language.
All changes are automatically submitted for inclusion through Github.
Upgrade notes¶
subuid/subgid entries¶
A longstanding bug in the idmap parser was causing everything but the first large entry for the root user to be discarded when parsing /etc/subuid
and /etc/subgid
.
This was then causing issues for the few users that have a legitimate reason to split their uid/gid allocation in half, mostly those using remote authentication on the host system.
This bug has been resolved, but this has a side effect of making Incus containers to fail to start on systems with an invalid subuid/subgid configuration.
If you notice that your containers won't start anymore, go look at /etc/subuid
and /etc/subgid
and make sure that there is one large entry for the root
user, it must be at least 65536
uid/gid large. More importantly, make sure that there is no conflict/overlap in allocations given to the root
user.
In most cases, the easiest is to remove all the root
entries from those two files and replace them with a single very large entry:
root:1000000:1000000000
New features¶
New incus file create
command¶
A new incus file create
command was added which provides an easy way to create empty files, symlinks and directories without having to transfer an existing local directory tree.
stgraber@dakara:~$ incus file create demo/root/file stgraber@dakara:~$ incus file create --type=symlink demo/root/symlink /etc/hosts stgraber@dakara:~$ incus file create --type=directory demo/root/dir stgraber@dakara:~$ incus exec demo -- ls -lh /root total 2.5K drwxr-xr-x 2 root root 2 Jan 26 03:38 dir -rw-r--r-- 1 root root 0 Jan 26 03:37 file lrwxrwxrwx 1 root root 10 Jan 26 03:38 symlink -> /etc/hosts
New incus snapshot show
command¶
A new incus snapshot show
command makes it easy to look at the configuration data that's included as part of an Incus instance snapshot.
As a reminder, Incus snapshots don't only contain the filesystem state, but also include all the instance configuration (config keys, devices, ...) at the time of the snapshot.
stgraber@dakara:~$ incus snapshot create demo s1 stgraber@dakara:~$ incus snapshot list demo +------+----------------------+----------------------+----------+ | NAME | TAKEN AT | EXPIRES AT | STATEFUL | +------+----------------------+----------------------+----------+ | s1 | 2024/01/25 22:39 EST | 0000/12/31 19:03 LMT | NO | +------+----------------------+----------------------+----------+ stgraber@dakara:~$ incus snapshot show demo s1 expires_at: 0001-01-01T00:00:00Z architecture: x86_64 config: image.architecture: amd64 image.description: Ubuntu jammy amd64 (20240125_07:42) image.os: Ubuntu image.release: jammy image.serial: "20240125_07:42" image.type: squashfs image.variant: default volatile.base_image: f9e9abeb4fc8691edf48078616a1aae628c6d5938b715e361c6b47cda0474679 volatile.cloud-init.instance-id: f724feba-245a-424b-bc51-43167258dc2a volatile.eth0.host_name: vethecbb346e volatile.eth0.hwaddr: 00:16:3e:06:67:f0 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING volatile.uuid: 8b8a1c75-977b-4197-9ad7-507b899432e9 volatile.uuid.generation: 8b8a1c75-977b-4197-9ad7-507b899432e9 created_at: 2024-01-26T03:39:09.583020489Z devices: {} ephemeral: false expanded_config: image.architecture: amd64 image.description: Ubuntu jammy amd64 (20240125_07:42) image.os: Ubuntu image.release: jammy image.serial: "20240125_07:42" image.type: squashfs image.variant: default volatile.base_image: f9e9abeb4fc8691edf48078616a1aae628c6d5938b715e361c6b47cda0474679 volatile.cloud-init.instance-id: f724feba-245a-424b-bc51-43167258dc2a volatile.eth0.host_name: vethecbb346e volatile.eth0.hwaddr: 00:16:3e:06:67:f0 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING volatile.uuid: 8b8a1c75-977b-4197-9ad7-507b899432e9 volatile.uuid.generation: 8b8a1c75-977b-4197-9ad7-507b899432e9 expanded_devices: eth0: name: eth0 network: incusbr0 type: nic root: path: / pool: default type: disk last_used_at: 0001-01-01T00:00:00Z name: s1 profiles: - default stateful: false size: 53760
More shell completion options¶
We're slowly transitioning from a single hand-written bash completion script for the incus
command line tool, to instead using a much more dynamic way of handling shell completion.
Initial shell completion profiles can be retrieved with:
incus completion bash
incus completion fish
incus completion powershell
incus completion zsh
Bash users are probably still better off using the hand-written completion script at this point, but we're hopeful that the new dynamically generated completion profiles will take over in the next release or two.
Support for multiple VM agent binaries¶
It's now possible for Incus to provide multiple agent binaries to its virtual machines.
This is useful in two scenarios:
- Handling multiple operating systems
- Handling multiple architectures
At this stage, the focus is on multiple architectures. With this new ability, you can now have 32bit virtual machines running on your system and have them fetch a 32bit of the agent binary.
stgraber@castiana:~$ incus exec debian32 bash root@debian32:~# uname -a Linux debian32 6.1.0-17-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) i686 GNU/Linux root@debian32:~# exit stgraber@castiana:~$ incus exec debian32 bash root@debian32:~# uname -m i686 root@debian32:~# mount -t 9p agent /mnt root@debian32:~# ls -lh /mnt total 34M -rwxr-xr-x 1 root root 17M Jan 24 10:10 incus-agent.linux.i686 -rwxr-xr-x 1 root root 18M Jan 24 10:10 incus-agent.linux.x86_64
Support for virtio-blk
as a disk io.bus
¶
After adding NVME support in Incus 0.2, we're now expanding that mechanism to also offering virtio-blk
as a disk I/O bus in our virtual machines.
To use it, set the io.bus
property on the disk
device to be virtio-blk
.
stgraber@dakara:~$ incus launch images:debian/12 demo --vm Launching demo stgraber@dakara:~$ incus storage volume create default demo size=5GiB --type=block Storage volume demo created stgraber@dakara:~$ incus config device add demo extra disk pool=default source=demo io.bus=virtio-blk Device extra added to demo stgraber@dakara:~$ incus exec demo bash root@demo:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 10G 0 disk ├─sda1 8:1 0 100M 0 part /boot/efi └─sda2 8:2 0 3.9G 0 part / vda 253:0 0 5G 0 disk
Support for USB network device pass-through in VMs¶
When using nictype=physical
for a virtual machine with the parent network device being connected over the USB bus, Incus will now detect the situation and internally convert this into a USB device pass-through to the virtual machine.
stgraber@castiana:~$ incus launch images:debian/12 demo --vm Launching demo stgraber@castiana:~$ incus config device add demo eth1 nic nictype=physical parent=enx207bd2a0f9eb Device eth1 added to demo stgraber@castiana:~$ incus exec demo bash root@demo:~# apt install usbutils Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: libusb-1.0-0 The following NEW packages will be installed: libusb-1.0-0 usbutils 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 142 kB of archives. After this operation, 492 kB of additional disk space will be used. Do you want to continue? [Y/n]· Get:1 http://deb.debian.org/debian bookworm/main amd64 libusb-1.0-0 amd64 2:1.0.26-1 [62.6 kB] Get:2 http://deb.debian.org/debian bookworm/main amd64 usbutils amd64 1:014-1 [79.7 kB] Fetched 142 kB in 1s (124 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package libusb-1.0-0:amd64. (Reading database ... 20425 files and directories currently installed.) Preparing to unpack .../libusb-1.0-0_2%3a1.0.26-1_amd64.deb ... Unpacking libusb-1.0-0:amd64 (2:1.0.26-1) ... Selecting previously unselected package usbutils. Preparing to unpack .../usbutils_1%3a014-1_amd64.deb ... Unpacking usbutils (1:014-1) ... Setting up libusb-1.0-0:amd64 (2:1.0.26-1) ... Setting up usbutils (1:014-1) ... Processing triggers for libc-bin (2.36-9+deb12u3) ... root@demo:~# lsusb -tv /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M ID 1d6b:0003 Linux Foundation 3.0 root hub |__ Port 4: Dev 2, If 0, Class=Communications, Driver=cdc_ncm, 5000M ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet |__ Port 4: Dev 2, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 480M ID 1d6b:0002 Linux Foundation 2.0 root hub root@demo:~# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:e7:f7:2d brd ff:ff:ff:ff:ff:ff 3: enx207bd2a0f9eb: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 20:7b:d2:a0:f9:eb brd ff:ff:ff:ff:ff:ff
New cluster evacuation options (force-stop
and stateful-stop
)¶
A couple of new cluster evacuation options have been added.
Those can be selected on a per-instance basis through the cluster.evacuate
instance configuration key.
force-stop
causes the instance to be immediately stopped without giving it a chance at a clean shutdown. This only really makes sense in cases where the instance is effectively stateless as it won't have any chance to flush any ongoing state to disk.
stateful-stop
causes the instance's state to be written to disk before stopping the instance. On restore, the instance state is restored too, causing the instance to just continue where it left off.
This option is currently primarily targeted at virtual machines as stateful stop for containers is quite difficult to achieve.
Ability to configure the host instance shutdown action¶
A new instance configuration key, boot.host_shutdown_action
, has been introduced which supports:
stop
(normal shutdown behavior)force-stop
(see above)stateful-stop
(see above)
This makes it particularly easy to have a number of virtual machines going through stateful stop on host shutdown and then being restored on boot.
Ability to start instances as part of creation¶
A small API optimization was made which now allows for instances to be started as part of the creation request, saving an API call and making it easier for those scripting the Incus API.
incus launch
now makes use of this too.
Configurable Loki instance name¶
When sending events to Loki, Incus provides a set of default labels.
Those include both an instance
and a location
label .Worth noting that here instance
refers to the Loki event source instance, not an Incus instance.
So far, those would only differ in the somewhat unlikely event that a server would be forwarding an event originating from another server in a cluster.
Instead, in clustered environments, it makes a lot more sense to have a way to provide a cluster name of some kind, so that if multiple clusters use the same Loki instance, they can easily be filtered.
To that effect, we've introduced a new loki.instance
server configuration key which, when set, will override the instance
label.
The default Grafana dashboard has also been updated to filter the Loki events under the assumption that the Loki instance
label will match the Prometheus job name.
Extended HEAD support on files¶
The HEAD
method on the Incus instance file API now returns the file size through the Content-Length
header.
The primary use for this is for those building some kind of file manager on top of the Incus instance file API as it now allows for not just showing the name and file type but also the size of any regular files.
Use of /run/incus
for runtime data¶
Up until now, Incus has stored some amount of runtime data in the instance log directories under /var/log/incus/. Other than it obviously not being the correct location for it, it was also causing some issues with systems that aggressively rotate and expire log files.
To solve this, Incus 0.5 will now place runtime data in /run/incus, keeping /var/log/incus only for actual log files.
In the future, more data will likely be relocated from /var/lib/incus to /run/incus as well.
Complete changelog¶
Here is a complete list of all changes in this release:
Full commit list
- incusd/instances/qemu: Don't fail event sending on missing agent
- incusd/network/ovsdb: Properly close the connections
- doc: Add Fedora installation instructions
- incusd/network/ovn: Use Mutate instead of Update
- incusd/network/ovs: Port BridgeDelete to OVSDB
- incusd/network/ovs: Port BridgeAdd to OVSDB
- incusd/network/ovs: Port BridgePortAdd to OVSDB
- incusd/network/ovn: Port LogicalSwitchPortIPs to OVSDB
- incus-doc: Don't try to guess types
- doc: Update configs
- doc/installing: Add Arch instructions
- lxd-to-incus: Clarify error messages
- doc/installing: Fix chocolatey link
- doc: Remove MicroCloud references
- incusd/network/ovn: Properly check transactions
- incusd/network/ovs: Properly check transactions
- incusd/network/ovn: Fix ChassisGroupChassisAdd
- incusd/network/ovn: Fix BridgeAdd
- incusd/network/ovn: Properly handle uplink with disabled DHCP/DHCPv6
- incus: Create config paths when no local daemons
- lxd-to-incus: Support alternative snap path
- incusd/device/pci: Detect USB bus
- incusd/device/nic: Support USB parents for physical NICs in VMs
- internal/instance: Add new volatile keys
- incusd/instance/qemu: Fix typo
- incusd/instance/qemu: Handle USB NIC hotremove
- incusd/networks: Don't Fill network configs when joining
- incus/create: Set Target after network/storage lookup
- doc/architectures: Add missing entries
- doc/architectures: Re-phrase headers
- doc/architectures: Fix typo
- doc/architectures: Add note about VM architectures
- doc/wordlist: Add Loongarch
- doc/migrate_lxd: Add mention of CLI configuration
- incus/snapshot: Fix format handling in list
- incus/snapshot: Add show sub-command
- i18n: Update translation templates
- incus: Enable cobra's completion support
- lxd-to-incus: Handle local OVN
- incusd/storage_pools: Don't crash on nil pool
- incusd/instance/lxc: Re-generate lxc.conf during Exec
- incusd/instance/qemu: Add ErrExecDisconnected
- incusd: Make VM shutdown/reboot exit cleanly
- lxd-to-incus: Fix incorrect directory name
- lxd-to-incus: Detect source daemon using symlink path
- lxd-to-incus: Remove trailing slashes from paths
- lxd-to-incus: touch completion file after migration
- incusd: Refresh the state on cluster put/join
- Makefile: Don't complain about shellcheck version
- golangci: Disable confusing-results
- shared/idmap: Remove Extend
- shared/idmap: Return all idmaps in DefaultIdmapSet
- shared/idmap: Split idrange
- shared/idmap: Split idmapset
- shared/idmap: Split idmap
- shared/idmap: Split ByHostid
- shared/idmap: Export non-Linux specific logic
- shared/idmap: Cleanup IdmapSet
- shared/idmap: Rename IdRange to IDRange
- shared/idmap: Cleanup IdmapEntry
- shared/idmap: Rename is_between to isBetween
- shared/idmap: Rename ByHostID and make it use IdmapSet
- shared/idmap: Update idmapset for modern standard
- shared/idmap: Rename the files
- shared/idmap: Move ByHostID back into set
- shared/idmap: Rename IdmapEntry to Entry
- shared/idmap: Rename IDRange to Range
- shared/idmap: Rename IdmapSet to Set
- shared/idmap: Fix import shadowing
- shared/idmap: Rename VFS3Fscaps to VFS3FSCaps
- shared/idmap: Add/tweak export function descriptions
- shared/idmap: Properly capitalize
- shared/idmap: Rename UIDShift to Shift
- shared/idmap: Update kernelDefaultMap to return multiple maps
- shared/idmap: Update comments
- shared/idmap: Add FilterPOSIX
- shared/idmap: Introduce NewSetFromIncusIDMap
- shared/idmap: Introduce NewSetFromJSON
- shared/idmap: Replace JSONMarshal with ToJSON
- shared/idmap: Introduce set_sort
- shared/idmap: Always use pointer receiver
- shared/idmap: Move remaining loaders to set_load
- shared/idmap: Run tests on all platforms
- shared/idmap: Introduce DefaultFullKernelSet
- shared/idmap: Introduce NewSetFromCurrentProcess
- shared/idmap: Remove GetSet
- shared/idmap: Introduce NewSetFromSystem
- shared/idmap: Remove kernelDefaultMap
- shared/idmap: Add Clone to Entry
- shared/idmap: Add Split to Set
- shared/idmap: Replace Shift functions with ShiftPath/UnshiftPath
- fuidshift: Update for idmap changes
- incusd: Update for idmap changes
- incus-user: Don't set raw.idmap when uid/gid aren't in system map
- shared/idmap: Add Includes to Set
- incusd: Simplify idmap serialization
- incusd/instance/lxc: Detect bad idmap and find new one
- shared/cliconfig: Improve error handling
- incusd/instance/qemu: Don't hardcode UEFI firmware in checkFeatures
- incusd/firewall/xtables: Fix iptablesClear on nft shim
- incus/network: add dynamic completions
- shared/idmap: Fix typo in comments
- incus/project: Get current project from connection info
- incusd/cluster: Ensure the cluster member config is always sorted
- Update madmin-go to support loong64
- server/seccomp: Add loongarch64
- shared/cgo: Add loongarch64
- shared/idmap: Don't change the json format
- shared/idmap: Document AddSafe and fix double records
- incusd: Update instance_test for shared/idmap fix
- incusd/instance/file: Add type and size to HEAD
- shared/idmap: Fix typo in comment
- api: disk_io_bus_virtio_blk
- doc: Add virtio-blk as option to io.bus
- incusd/device/disk: Add virtio-blk
- incusd/instance/qemu: Add virtio-blk support
- Move db backup functions to ClusterTx
- Move db image functions to ClusterTx
- Move db instance functions to ClusterTx
- Move db network ACL functions to ClusterTx
- Move db network forward functions to ClusterTx
- Move db network load balancer functions to ClusterTx
- Move db network peer functions to ClusterTx
- Move db profile functions to ClusterTx
- Move db network zone functions to ClusterTx
- Move db network functions to ClusterTx
- Move db snapshot functions to ClusterTx
- Move db storage bucket functions to ClusterTx
- Move db storage pool functions to ClusterTx
- Move db volume snapshot functions to ClusterTx
- Move db storage volume functions to ClusterTx
- Move db warning functions to ClusterTx
- cmd/incusd: Fix bulk unfreezing
- cmd/incus: Add
resume
command - i18n: Update translations
- incusd/loki: Replace complex backoff with simple loop
- gomod: Update dependencies
- incus-agent: Handle built-in vsock module
- gomod: Update dependencies
- README: Re-introduce weblate
- incusd/network/acl: Avoid nested DB transactions
- incusd/instance/qemu: Start using seabios as CSM firmware
- incusd/forknet: Handle wifi detach
- doc/CONTRIBUTING: Fix incorrect comamnd paths
- i18n: Manual update to french translation
- i18n: Update translation templates
- Translated using Weblate (French)
- tests: Add license check
- Revert "Update madmin-go to support loong64"
- gomod: Update dependencies
- incusd: Correctly update event location
- incusd/events: Upgrade to websocket as late as possible
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- api: loki_config_instance
- incusd/config: Add loki.instance
- incusd/loki: Add support for overriding instance name
- incusd: Add support for loki.instance
- doc: Update configs
- grafana: Add instance filters for Loki
- incusd/loki: Fix variable shadowing
- Translated using Weblate (Japanese)
- Translated using Weblate (Japanese)
- cmd/incusd/api_cluster: Join cluster transactions
- i18n: Remove empty translations
- api: instance_create_start
- shared/api: Add Start to InstnacesPost
- doc/rest-api: Refresh swagger YAML
- incusd/instance: Add support for Start property
- incus/launch: Use the Start property
- i18n: Update translation templates
- doc: Updates Windows install with Winget instructions
- doc: Add Winget to wordlist
- incusd/migration: Properly forward errors
- cmd/incus: Get owner mode only if
--gid
or--uid
is unset - cmd/incus: Add
incus file create
subcommand - test: Add tests for
incus file create
- i18n: Update translation templates
- Translated using Weblate (Japanese)
- Add note about scrape_interval and update examples
- cmd/incus: Remove unused
flagContent
variable inincus file create
- build(deps): bump actions/dependency-review-action from 3 to 4
- incusd/storage_volumes: Properly target refreshes
- incusd/storage_volumes: Use a single POST handler
- lxd-to-incus: Use Incus API client for LXD
- lxd-to-incus: Handle non-string LXD configs
- lxd-to-incus: Remove separate go package
- lxd-to-incus: Fix various issues
- Makefile: Update for lxd-to-incus
- gomod: Update dependencies
- incus/alias: Make default aliases visible
- incus: Mention aliases in help message
- i18n: Update translation templates
- incus: Handle non-existent home directory
- lxd-to-incus: Don't export internal functions
- lxd-to-incus: Fix error checking
- lxd-to-incus: Check that casting succeeded
- lxd-to-incus: Fix typo
- lxd-to-incus: Fix variable shadowing
- lxd-to-incus: Remove spurious printf
- lxd-to-incus: Add required comments
- lxd-to-incus: Simplify presence checks
- lxd-to-incus: Use field names in DottedVersion
- internal/util: Re-order path functions
- internal/util: Add RunPath
- incusd/sys: Add runtime directory
- incusd/seccomp: Move seccomp.socket to /run
- incusd/instance_logs: Drop conf files
- doc/rest-api: Refresh swagger YAML
- incusd/instance/common: Add RunPath
- incusd/instance/lxc: Move lxc.conf
- incusd/instance/qemu: Move qemu.conf
- doc: Update qemu.conf path
- incusd/apparmor: Add runtime directory
- incusd/instance/utils: Cleanup runtime path
- incusd/instance/lxc: Move files to runtime path
- incusd/instance/qemu: Move files to runtime path
- incusd/patches: Move files to runtime directory
- incusd/instance/qemu: Move agent loader to separate files
- incusd/apparmor/qemu: Remove mention of userns
- incusd/instance/qemu: Make config drive name configurable
- incusd/instance/qemu: Add new agent share
- incusd/apparmor/qemu: Allow access to agent path
- doc: Add INCUS_AGENT_PATH
- incusd/instance/qemu: Only expose config/agent drives over 9p
- incusd/instance/qemu/agent-loader: Remove virtiofs
- doc/getting_started: Point users to installing guide
- doc/installing: Cleanup distro instructions
- api_cluster: Optimize db transactions
- daemon_images: Optimize db transactions
- daemon_storage: Optimize db transactions
- images: Optimize db transactions
- storage_volumes_snapshot: Optimize db transactions
- instance/drivers: Optimize db transactions
- driver_ovn: Optimize db transactions
- network/acl: Optimize db transactions
- network/zone: Optimize db transactions
- storage_volumes: Optimize db transactions
- incusd/instance/qemu: Add some ArchLinux EDK2 filenames
- api_internal: Remove unreachable code
- doc/installing: Add Void Linux
- internal/instance: Don't use the node terminology
- doc: Update configs
- api: clustering_evacuation_stop_options
- internal/instance: Extend cluster.evacuate
- incusd/cluster: Add evacuation mode validation
- incusd/instance: Use a string for CanMigrate
- incusd/cluster: Update for CanMigrate
- incusd/cluster: Add stateful-stop and force-stop
- doc: Update configs
- api: boot_host_shutdown_action
- internal/instance: Add boot.host_shutdown_action
- doc: Update configs
- scripts/bash: Add boot.host_shutdown_action
- incusd/project: Add boot.host_shutdown_action
- incusd/instances: Add support for boot.host_shutdown_action
- incusd/instance: Fallback to stateless start when no state available
- internal/archive: Fix squashfs error handling
- gomod: Update dependencies
Documentation¶
The Incus documentation can be found at:
https://linuxcontainers.org/incus/docs/main/
Packages¶
There are no official Incus packages as Incus upstream only releases regular release tarballs. Below are some available options to get Incus up and running.
Installing the Incus server on Linux¶
Incus is available for most common Linux distributions. You'll find detailed installation instructions in our documentation.
https://linuxcontainers.org/incus/docs/main/installing/
Homebrew package for the Incus client¶
The client tool is available through HomeBrew for both Linux and MacOS.
https://formulae.brew.sh/formula/incus
Chocolatey package for the Incus client¶
The client tool is available through Chocolatey for Windows users.
https://community.chocolatey.org/packages/incus/0.5
Winget package for the Incus client¶
The client tool is also available through Winget for Windows users.
https://winstall.app/apps/LinuxContainers.Incus
Support¶
At this early stage, each Incus release will only be supported up until the next release comes out. This will change in a few months as we are planning an LTS release to coincide with the LTS releases of LXC and LXCFS.
Community support is provided at: https://discuss.linuxcontainers.org
Commercial support is available through: https://zabbly.com/incus
Bugs can be reported at: https://github.com/lxc/incus/issues
Older news¶
- Incus 6.0 LTS has been released
- Swap limits for containers
- New shell completion mechanism
- Creation of external bridge interfaces
- Live-migration of VMs with attached disks (from remote storage)
- System information in incus info --resources
- USB devices in incus info --resources
- Feature removal
- Feature additions
- Linux packages
- Homebrew package for the Incus client
- Chocolatey package for the Incus client
- Winget package for the Incus client
- Incus 0.7 has been released
- Network integrations
- Image server management tool
- JSON Web Token authentication
- Configurable OIDC username field
- Improved NUMA handling
- More options to select USB devices
- Disk I/O throttling for VMs
- Per-remote client certificates
- Manual generation of the client certificate keypair
- Improvements to lxd-to-incus
- Improvements to incus-migrate
- Additional image restrictions
- Installing the Incus server on Linux
- Homebrew package for the Incus client
- Chocolatey package for the Incus client
- Winget package for the Incus client
- Incus 0.6 has been released
- Clustered LVM storage driver
- Storage bucket backup and import
- Listing image across all projects
- binfmt_misc in unprivileged containers
- Control over shared block storage volumes
- OVN logical router name in network info
- File ownership and permissions in image templates
- Encrypted EC client certificate keys
- lxd-to-incus improvements
- Installing the Incus server on Linux
- Homebrew package for the Incus client
- Chocolatey package for the Incus client
- Winget package for the Incus client
- Incus 0.5.1 has been released
- Alternative way to get the VM agent
- Fixed handling of stopped instances during evacuation
- Database performance fixes
- Installing the Incus server on Linux
- Homebrew package for the Incus client
- Chocolatey package for the Incus client
- Winget package for the Incus client
- Incus 0.5 has been released
- Ansible, Terraform/OpenTofu and Packer
- Linux distribution packages
- Translations
- Upgrade notes
- New features
- Installing the Incus server on Linux
- Homebrew package for the Incus client
- Chocolatey package for the Incus client
- Winget package for the Incus client
- Older news