Frequently Asked Questions

General issues

How to enable LXD server for remote access?

By default LXD server is not accessible from the networks as it only listens on a local unix socket. You can make LXD available from the network by specifying additional addresses to listen to. This is done with the core.https_address config variable.

To see the current server configuration, run:

lxc config show

To set the address to listen to, find out what addresses are available and use the config set command on the server:

ip addr
lxc config set core.https_address 192.168.1.15

When I do a lxc remote add over https, it asks for a password?

By default, LXD has no password for security reasons, so you can’t do a remote add this way. In order to set a password, do:

lxc config set core.trust_password SECRET

on the host LXD is running on. This will set the remote password that you can then use to do lxc remote add.

You can also access the server without setting a password by copying the client certificate from .config/lxc/client.crt to the server and adding it with:

lxc config trust add client.crt

How do I configure LXD storage?

LXD supports btrfs, ceph, directory, lvm and zfs based storage.

First make sure you have the relevant tools for your filesystem of choice installed on the machine (btrfs-progs, lvm2 or zfsutils-linux).

By default, LXD comes with no configured network or storage. You can get a basic configuration done with:

    lxd init

lxd init supports both directory based storage and ZFS. If you want something else, you’ll need to use the lxc storage command:

lxc storage create default BACKEND [OPTIONS...]
lxc profile device add default root disk path=/ pool=default

BACKEND is one of btrfs, ceph, dir, lvm or zfs.

Unless specified otherwise, LXD will setup loop based storage with a sane default size.

For production environments, you should be using block backed storage instead both for performance and reliability reasons.

How can I live migrate a container using LXD?

Live migration requires a tool installed on both hosts called CRIU, which is available in Ubuntu via:

sudo apt install criu

Then, launch your container with the following,

lxc launch ubuntu SOME-NAME
sleep 5s # let the container get to an interesting state
lxc move host1:SOME-NAME host2:SOME-NAME

And with luck you’ll have migrated the container :). Migration is still in experimental stages and may not work for all workloads. Please report bugs on lxc-devel, and we can escalate to CRIU lists as necessary.

Can I bind mount my home directory in a container?

Yes. This can be done using a disk device:

lxc config device add container-name home disk source=/home/${USER} path=/home/ubuntu

For unprivileged containers, you will also need one of:

  • Pass shift=true to the lxc config device add call. This depends on shiftfs being supported (see lxc info)

  • raw.idmap entry (see Idmaps for user namespace)

  • Recursive POSIX ACLs placed on your home directory

Either of those can be used to allow the user in the container to have working read/write permissions. When not setting one of those, everything will show up as the overflow uid/gid (65536:65536) and access to anything that’s not world readable will fail.

Privileged containers do not have this issue as all uid/gid inthe container are the same outside. But that’s also the cause of most of the security issues with such privileged containers.

How can I run docker inside a LXD container?

In order to run Docker inside a LXD container the security.nesting property of the container should be set to true.

lxc config set <container> security.nesting true

Note that LXD containers cannot load kernel modules, so depending on your Docker configuration you may need to have the needed extra kernel modules loaded by the host.

You can do so by setting a comma separate list of kernel modules that your container needs with:

lxc config set <container> linux.kernel_modules <modules>

We have also received some reports that creating a /.dockerenv file in your container can help Docker ignore some errors it’s getting due to running in a nested environment.

Container startup issues

If your container is not starting, or not behaving as you would expect, the first thing to do is to look at the console logs generated by the container, using the lxc console --show-log CONTAINERNAME command.

In this example, we will investigate a RHEL 7 system in which systemd can not start.

# lxc console --show-log systemd
Console log:

Failed to insert module 'autofs4'
Failed to insert module 'unix'
Failed to mount sysfs at /sys: Operation not permitted
Failed to mount proc at /proc: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.

The errors here say that /sys and /proc can not be mounted - which is correct in an unprivileged container. However, LXD does mount these filesystems automatically if it can.

The container requirements specify that every container must come with an empty /dev, /proc, and /sys folder, as well as /sbin/init existing. If those folders don’t exist, LXD will be unable to mount to them, and systemd will then try to. As this is an unprivileged container, systemd does not have the ability to do this, and it then freezes.

So you can see the environment before anything is changed, you can explicitly change the init in a container using the raw.lxc config param. This is equivalent to setting init=/bin/bash on the linux kernel commandline.

lxc config set systemd raw.lxc 'lxc.init.cmd = /bin/bash'

Here is what it looks like:

root@lxc-01:~# lxc config set systemd raw.lxc 'lxc.init.cmd = /bin/bash'
root@lxc-01:~# lxc start systemd
root@lxc-01:~# lxc console --show-log systemd

Console log:

[root@systemd /]#
root@lxc-01:~#

Now that the container has started, you can look in it and see that things are not running as well as expected.

root@lxc-01:~# lxc exec systemd bash
[root@systemd ~]# ls
[root@systemd ~]# mount
mount: failed to read mtab: No such file or directory
[root@systemd ~]# cd /
[root@systemd /]# ls /proc/
sys
[root@systemd /]# exit

Because LXD tries to auto-heal, it did create some of the folders when it was starting up. Shutting down and restarting the container will fix the problem, but the original cause is still there - the template does not contain the required files.

Networking issues

In a larger Production Environment, it is common to have multiple vlans and have LXD clients attached directly to those vlans. Be aware that if you are using netplan and systemd-networkd, you will encounter some bugs that could cause catastrophic issues

Do not use systemd-networkd with netplan and bridges based on vlans

At time of writing (2019-03-05), netplan can not assign a random MAC address to a bridge attached to a vlan. It always picks the same MAC address, which causes layer2 issues when you have more than one machine on the same network segment. It also has difficultly creating multiple bridges. Make sure you use network-manager instead. An example config is below, with a management address of 10.61.0.25, and VLAN102 being used for client traffic.

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      dhcp4: no
      accept-ra: no
      # This is the 'Management Address'
      addresses: [ 10.61.0.25/24 ]
      gateway4: 10.61.0.1
      nameservers:
        addresses: [ 1.1.1.1, 8.8.8.8 ]
    eth1:
      dhcp4: no
      accept-ra: no
      # A bogus IP address is required to ensure the link state is up
      addresses: [ 10.254.254.25/32 ]

  vlans:
    vlan102:
      accept-ra: no
      dhcp4: no
      id: 102
      link: eth1

  bridges:
    br102:
      accept-ra: no
      dhcp4: no
      interfaces: [ "vlan102" ]
      # A bogus IP address is required to ensure the link state is up
      addresses: [ 10.254.102.25/32 ]
      parameters:
        stp: false

Things to note

  • eth0 is the Management interface, with the default gateway.

  • vlan102 uses eth1.

  • br102 uses vlan102, and has a bogus /32 IP address assigned to it

The other important thing is to set stp: false, otherwise the bridge will sit in learning state for up to 10 seconds, which is longer than most DHCP requests last. As there is no possibility of cross-connecting and causing loops, this is safe to do.

Beware of ‘port security’

Many switches do not allow MAC address changes, and will either drop traffic with an incorrect MAC, or, disable the port totally. If you can ping a LXD instance from the host, but are not able to ping it from a different host, this could be the cause. The way to diagnose this is to run a tcpdump on the uplink (in this case, eth1), and you will see either ‘ARP Who has xx.xx.xx.xx tell yy.yy.yy.yy’, with you sending responses but them not getting acknowledged, or, ICMP packets going in and out successfully, but never being received by the other host.

Do not run privileged containers unless necessary

A privileged container can do things that effect the entire host - for example, it can use things in /sys to reset the network card, which will reset it for the entire host, causing network blips. Almost everything can be run in an unprivileged container, or - in cases of things that require unusual privileges, like wanting to mount NFS filesystems inside the container, you may need to use bind mounts.