Server settings for an Incus production setup

To allow your Incus server to run a large number of instances, configure the following settings to avoid hitting server limits.

The Value column contains the suggested value for each parameter.

/etc/security/limits.conf

Domain

Type

Item

Value

Default

Description

*

soft

nofile

1048576

unset

Maximum number of open files

*

hard

nofile

1048576

unset

Maximum number of open files

root

soft

nofile

1048576

unset

Maximum number of open files

root

hard

nofile

1048576

unset

Maximum number of open files

*

soft

memlock

unlimited

unset

Maximum locked-in-memory address space (KB)

*

hard

memlock

unlimited

unset

Maximum locked-in-memory address space (KB)

root

soft

memlock

unlimited

unset

Maximum locked-in-memory address space (KB), only need with bpf syscall supervision

root

hard

memlock

unlimited

unset

Maximum locked-in-memory address space (KB), only need with bpf syscall supervision

/etc/sysctl.conf

Note

Reboot the server after changing any of these parameters.

Parameter

Value

Default

Description

fs.aio-max-nr

524288

65536

Maximum number of concurrent asynchronous I/O operations (you might need to increase this limit further if you have a lot of workloads that use the AIO subsystem, for example, MySQL)

fs.inotify.max_queued_events

1048576

16384

Upper limit on the number of events that can be queued to the corresponding inotify instance (see inotify)

fs.inotify.max_user_instances

1048576

128

Upper limit on the number of inotify instances that can be created per real user ID (see inotify)

fs.inotify.max_user_watches

1048576

8192

Upper limit on the number of watches that can be created per real user ID (see inotify)

kernel.dmesg_restrict

1

0

Whether to deny container access to the messages in the kernel ring buffer (note that this will also deny access to non-root users on the host system)

kernel.keys.maxbytes

2000000

20000

Maximum size of the key ring that non-root users can use

kernel.keys.maxkeys

2000

200

Maximum number of keys that a non-root user can use (the value should be higher than the number of instances)

net.core.bpf_jit_limit

1000000000

varies

Limit on the size of eBPF JIT allocations (on kernels < 5.15 that are compiled with CONFIG_BPF_JIT_ALWAYS_ON=y, this value might limit the amount of instances that can be created)

net.ipv4.neigh.default.gc_thresh3

8192

1024

Maximum number of entries in the IPv4 ARP table (increase this value if you plan to create over 1024 instances - otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration; see ip-sysctl)

net.ipv6.neigh.default.gc_thresh3

8192

1024

Maximum number of entries in IPv6 ARP table (increase this value if you plan to create over 1024 instances - otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration; see ip-sysctl)

vm.max_map_count

262144

65530

Maximum number of memory map areas a process may have (memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries)