News

Distrobuilder 2.0 has been released

18th of October 2021

Introduction

The distrobuilder team is proud to announce its initial release, distrobuilder 2.0!

As usual, this release fixes a variety of issues as Linux distributions evolve and change the way they're built. Additionally, it also improves the Windows ISO repack feature quite a bit and introduces support for directly importing an image into LXD

The main highlights are:

  • windows: Support for Windows 11 and Windows Server 2022 ISO images
  • windows: Addition of virtio GPU driver
  • gentoo: Support for custom variants
  • New --import-into-lxd command line option

The full list of commits is available below:

Detailed changelog
  • Makefile: Tweak generation of dist tarball
  • sources/plamolinux: Support new pkgtools8
  • main: Check systemd-sysctl path
  • systemd: Properly check for instance type
  • windows: Add Windows 11 support
  • sources/plamolinux: Fix a process related to pkgtools8
  • sources/plamolinux: more fix related to pkgtools8
  • Update gomod
  • main: Skip overlay on zfs
  • *: Replace Wrap with WithMessage
  • windows: Fix pongo variable name in balloon driver
  • *: Replace RunCommand with RsyncLocal
  • utils: Fix error handling in RsyncLocal
  • windows: Add driver viogpudo
  • Update mkdocs.yml
  • main: Add support for Windows Server 2022
  • *: Switch to errors without stack traces
  • Update go modules
  • sources/openwrt: Drop lxd-openwrt repo and SDK
  • main: Log image creation
  • sources/gentoo: Allow custom source variants
  • *: Record created filenames
  • main: Add --import-into-lxd flag
  • Update go mod
  • doc: Update packages.md
  • doc: Update building.md
  • main: Fix post-files actions for lxc
  • main: Check for snap when importing LXD image
  • shared/definition: s/-/_/
  • generators: Add definition to generator init
  • generators: Drop template rendering in cloud-init and dump
  • doc: Update renamend key values
  • doc/generators: Add missing source field
  • doc: Mention general pongo support
  • shared/util: Add Retry function
  • shared/net: Use Retry for downloads
  • sources: Use Retry for networking
  • Update go.mod
  • sources/debootstrap: Handle merged /usr
  • systemd: drop unused variable
  • systemd: double quote to prevent globbing and word splitting
  • systemd: generate zzz-lxc-service.conf in one go
  • main: remove NoNewPrivileges=no from systemd-generator
  • Revert "main: remove NoNewPrivileges=no from systemd-generator"
  • github: Bump Go versions
  • sources/oracle: Fix rpm/yum issue
  • Update gomod

Downloads

Distrobuilder 1.3 has been released

9th of August 2021

Introduction

The distrobuilder team is proud to announce its initial release, distrobuilder 1.3!

The main highlights are:
- New systemd generator to dynamically apply needed tweaks in containers
- Support for AlmaLinux, SpringdaleLinux and RockyLinux (using shared CentOS logic)
- Support for busybox images

The full list of commits is available below:

Detailed changelog
  • sources/centos: Support patch releases
  • sources/centos: Fix CentOS 8-Stream
  • sources/centos: Only print error on patch releases
  • sources/centos: Fix CentOS 8 Stream
  • sources/centos: Fix centos-stream-repos package on 8-Stream
  • sources: Support OpenWrt on ARM
  • main: Fix systemd capabilities
  • sources/funtoo: Handle missing releases
  • sources/funtoo: Improve error handling
  • managers/zypper: Add --replacefiles flag to install
  • main: Consider the case of boot.wim and install.wim
  • systemd: Also disable ProtectKernelTunables=no
  • Adding AlmaLinux to build code
  • Adds almalinux-http to valid downloaders list
  • sources/funtoo: Add missing break
  • main: Skip overlay on xfs
  • main: Run systemd fix before post-files actions
  • main: Add --disable-overlay flag
  • main: Add systemd generator
  • sources: Add generic HTTP downloader
  • shared/definition: Support rootfs-http
  • sources: Add Rocky Linux
  • shared/definition: Support rockylinux-http downloader
  • sources: Add Springdale Linux
  • shared/definition: Support springdalelinux-http downloader
  • main: Support source.URL template
  • systemd: Refactor generator
  • systemd-generator: Some small fixes
  • main: Fix typos in systemd-generator
  • main: Refactor networkd/resolved fixes
  • main: Simplify sed call
  • main: s/which/type/ in systemd generator
  • main: Make sure to return 0
  • main: Add settings for privileged containers
  • main: Fix cloud-init and NetworkManager detection
  • main: Add more distros to NetworkManager fix
  • main: Fix calls to fix_ro_paths
  • main: Restrict networkd/resolved fix to unprivileged containers
  • main: Fix path of ip command
  • main: Add /lib/systemd to searchable paths
  • systemd: Make sure override runs last
  • systemd: Fix Oracle networking
  • systemd: Make network-connection-activate work on Oracle
  • systemd: Always apply NM fix on centos
  • almalinux: clean up downloader code and add beta/rc support
  • main: Fix typo in systemd generator
  • main: Add fix for systemd-udev-trigger.service
  • systemd: Mask more units
  • main: Add fix for systemd-sysctl
  • sources/opensuse: Fix path to non-x86_64 tarballs
  • sources/opensuse: Fix URL to tarballs
  • main: Replace Sanity Checks with Quick Checks
  • sources: Rework downloader
  • sources: s/docker/docker-http/
  • systemd: Mask systemd-pstore.service
  • sources/centos: Support CentOS 8 on armhf
  • sources/oracle: Fix ISO paths and support aarch64
  • main: Mask ua-messaging.service
  • sources: Remove duplicate code
  • sources/almalinux: Update errors
  • sources/alpine: Update errors
  • sources/alt: Update errors
  • sources/apertis: Update errors
  • sources/archlinux: Update errors
  • sources/centos: Update errors
  • sources/debootstrap: Update errors
  • sources/docker: Update errors
  • sources/fedora: Update errors
  • sources/funtoo: Update errors
  • sources/gentoo: Update errors
  • sources/opensuse: Update errors
  • sources/openwrt: Update errors
  • source/oraclelinux: Update errors
  • sources/plamolinux: Update errors
  • sources/rhel-common: Update errors
  • sources/rocky: Update errors
  • sources/rootfs: Update errors
  • sources/sabayon: Update errors
  • sources/springdalelinux: Update errors
  • sources/ubuntu: Update errors
  • sources/voidlinux: Update errors
  • sources/centos: Add more GPG keys
  • sources/oracle: Run yum with --skip-broken
  • managers: Refactor
  • main: Use refactored managers
  • Drop apt-transport-https (transitional since 18.04)
  • Revert "sources/oracle: Run yum with --skip-broken"
  • sources/oracle: Fix iso script
  • Avoid systemd generator to break systemd-networkd connections
  • sources/oracle: Fix OL7 base URL
  • sources/centos: Add GPG keys early
  • sources/centos: Fix CentOS repo names
  • sources/centos: Check repo before disabling it
  • sources/centos: Include more gpg keys
  • openwrt: Add ability to make 21.02 RC images
  • systemd: Mask hwdb-update
  • Drop Ubuntu 14.04 (EOL)
  • Drop Ubuntu 16.04 (EOL)
  • sources/gentoo: Support arm64
  • sources/gentoo: Support source variant
  • systemd: Fix typo in unit name
  • generators: Drop upstart-tty
  • generators: Refactor generators
  • main: Use refactored generators
  • main: Check filter before loading generator
  • main: Clean up properly on timeout/interrupt
  • main: Write to stderr if GetLogger fails
  • main: Extend logging
  • Wrap errors
  • sources: Fix gentoo regexes
  • main: Add NoNewPrivileges=no to systemd-generator
  • sources: Fix gentoo source variant check
  • Update go modules
  • sources: Pass cache directory
  • sources: Replace TempDir with cacheDir
  • sources: Fix unmount for rhel based distros
  • sources: Add busybox
  • systemd: Add systemd-sysusers workaround
  • archlinux: Switch test to official mirror
  • sources/centos: Fix error message
  • sources/centos: Unmount raw file
  • sources/openwrt: Fix directory name
  • vm: Check existance of loop device before detaching it
  • vm: Fix error message
  • sources/busybox: Fix mkdir call
  • sources/busybox: Fix build
  • sources/busybox: Remove test logging message
  • sources/busybox: Rename directories to make more sense
  • Update gomod
  • doc: Update build instructions
  • doc: Update usage
  • main: Add logging
  • sources: Add logging
  • main: Wrap overlay errors
  • doc/index: Drop internal command
  • Hide completion sub-command
  • Makefile: Set GO111MODULE
  • image: Fix LXD compression
  • image: Add compression flag for lxc images
  • image: Update tests
  • shared: Add zstd compression
  • main: Add type and compression to help message
  • sources/alt: Update tarball url (add arch subdir)
  • sources/alt: Update checksum and gpg file paths
  • main: Remove duplicate defer statement
  • main: Disable overlay on FUSE filesystems
  • image: Fix unified LXD tarballs
  • sources/openwrt: Remove verification
  • main: Support ISO generation with mkisofs

Downloads

Distrobuilder 1.2 has been released

23rd of March 2021

Introduction

The distrobuilder team is proud to announce its initial release, distrobuilder 1.2!

This release's highlight is the introduction of Windows image repacking.

LXD is able to run Windows VMs; for that it needs a Windows ISO and a bunch of specific drivers from a separate ISO which need to be loaded during the Windows installation.

To make things easier, distrobuilder has added the repack-windows command which takes both a Windows ISO and an ISO containing the specific drivers, and bundles them together. With the resulting ISO, Windows can be installed easily.

With this release also come support for Go modules, and basic context logging. For a more detailed log output, the --debug flag can be used.

The full list of commits is available below:

Detailed changelog
  • sources: Fix Plamo 7.x
  • Cosmetic update to the README.me (index.md)
  • Updated instructions
  • sources: Fix CentOS 8-Stream rootfs
  • luet: Fix gofmt
  • Remove out-of-place statement
  • generators: Add virtio-fs to lxd-agent init scripts
  • generators: Add copy generator
  • generators: update doc
  • Added missing definitions in generators hosts, hostname
  • generators: Fix lxd-agent openRC scripts
  • sources/openwrt: Use fallback image if necessary
  • sources/docker: Permit to use private docker registry with credentials
  • distrobuilder: Add repack-windows command
  • distrobuilder: Call Sync() before unmounting overlay
  • windows: Add balloon driver
  • windows: Add netkvm driver
  • windows: Add vioinput driver
  • windows: Add viorng driver
  • windows: Add vioscsi driver
  • windows: Add vioserial driver
  • windows: Add viofs driver
  • Add Github Actions
  • Remove Travis integration
  • README: Point build status to GitHub Actions
  • github: Align workflow with LXD
  • generators: Refresh lxd-agent systemd units
  • repack-windows: Check for genisoimage
  • generators/lxd-agent: Fix systemd path
  • generators/lxd-agent: Fix path in unit file
  • windows: Add support for Windows Server 2016
  • windows: Determine correct Windows paths
  • support changing interpreter with shebang
  • sources/voidlinux: Fix checksum file names
  • Fix bad calls to RunScript
  • oracle: Symlink /bin to /usr/bin if missing
  • sources/plamo: Don't override PATH in scripts
  • windows: Optimize registry editing
  • shared/logger: Add logger
  • main: Add --debug flag
  • main: Add basic logging
  • Support go modules
  • Makefile: Move go get to update-gomod target
  • .github/workflows: Remove go dependency section
  • .github/workflows: Update Go modules before testing
  • main: Always unmount WIM file
  • main: Cleanup on unsuccessful mount
  • main: Fix NPE
  • windows: Change flag s/version/windows-version/
  • doc: Add Windows section
  • doc/building: Mention --vm flag
  • doc/packages: Clarify url key in packages
  • doc: Add filtering
  • Update gomod

Downloads

Distrobuilder 1.1 has been released

20th of August 2020

Introduction

The distrobuilder team is proud to announce its initial release, distrobuilder 1.1!

This release's highlight is the introduction of VM image building support following LXD's introduction of VM support.

As part of this, distrobuilder also now relies on overlayfs to store deltas while building images (rather than attempting to undo changes).

On top of those two changes, this release also fixes a variety of issues in the different distro source drivers as needed when URLs and files change.

The full list of commits is available below:

Detailed changelog
  • Advertise the (classic) snap package in the README
  • Fix link to examples dir
  • sources/oracle: Add support for Oracle Linux 8
  • Drop .snapcraf.yaml in favor of snapcraft-pkg-snap repo
  • sources/oracle: Fix Oracle Linux 8
  • managers/yum: Add repo handler
  • managers/dnf: Add repo handler
  • Remove references to "apt_sources"
  • sources: Fix OpenWrt
  • sources: Fix OpenWrt test
  • Fix syntax for custom-manager
  • sources: Fix type in OpenWrt build
  • distrobuilder: Extend pongo2 template usage
  • Add support "flags" in package sets
  • Don't attempt to re-install packages that were installed early
  • managers/apk: Add repo handler
  • sources/alpine: Fix edge builds
  • sources: Support CentOS Stream
  • sources/centos: Fix filename regexes
  • Fix example command in index.md
  • Update index.md - separate LXD and LXC examples
  • sources/void: Update location of checksums
  • sources/alpine: Fix regex
  • shared/definition: Add LXD target
  • shared/definition: Add VM field to target
  • shared/definition: Add GetTypes to filter
  • shared: Export ChrootMount struct
  • *: Pass custom mounts to SetupChroot()
  • *: Support VM filtering
  • distrobuilder: Use overlayfs
  • generators: Pass target to Run* functions
  • doc: Update docs
  • generators: Add lxd-agent generator
  • doc: Add lxd-agent generator
  • generators: Add fstab generator
  • generators: Fix lxd-agent function signatures
  • *: Add VM support
  • vm: Make VM size uint64
  • generators/lxd-agent: Fix systemd unit files
  • vm: Ensure loop device is unmounted before creating image
  • *: Use errors.Wrap() when possible
  • generators: Remove StoreFile and RestoreFiles
  • *: Replace syscall package with unix
  • main: Check VM dependencies
  • *: Use errors.Wrap() when possible
  • managers/pacman: Clean up properly
  • managers: Add preRefresh hook
  • managers/opkg: Use preRefresh hook
  • vm: Handle loop partitions inside containers
  • shared/chroot: Recursively bind /dev
  • main: Add more VM dependencies
  • image/lxd: Rename resulting VM image
  • shared/chroot: Make /dev/fuse read-only
  • shared: Close created files
  • generator/lxd-agent: Fix systemd units path
  • main: Fix build-lxd for VMs
  • chroot: Unmount /dev/fuse
  • shared: Differentiate between build-dir and others
  • chroot: Don't bind-mount /dev
  • chroot: Fix mode for special files in /dev
  • shared/definition: Fix early packages
  • main: Fix file generators
  • generators/lxd-agent: Add trans=virtio option
  • chroot: Remove obsolete code
  • main,shared: Fix undefined image target
  • chroot: Perform package refresh only when needed
  • generators/lxd-agent: Fix openRC service file
  • sources/opensuse: Fix openSUSE
  • sources/opensuse: Fix verification
  • data: Fix core16 initrd
  • generators/lxd-agent: Fix running order
  • lxd-agent: Restart on failure
  • generators/lxd-agent: Fixes ordering issue with lxd-agent
  • managers/apt: Handle repo keys
  • shared/chroot: Create /dev/shm
  • sources/openwrt: Fix snapshot release
  • shared/chroot: Fix /dev/shm file mode
  • sources/funtoo: Fix image and gpg URLs
  • shared/util: Fix checksum matching
  • test: Update checksum matching
  • shared/util: Fix checksum matching
  • test: Update checksum matching
  • cloud-init: Update for virtual machines
  • shared/definition: Add Mode, GID and UID to files
  • generators: Add file access handler
  • generators/dump: Allow changing mode, UID and GID
  • doc/generators: Update dump generator
  • doc/examples/schema: Add mode, gid and uid
  • generators/utils: Fix condition
  • shared: Handle multiple checksums
  • shared/util_test: Update checksum test
  • shared/net: Fix checksum check regression
  • shared/definition: Add Pongo option to files
  • generators/dump: Support pongo2 templates
  • generators/cloud-init: Support pongo2 templates
  • generators/template: Support pongo2 templates
  • doc: Document pongo2
  • Update for change to RunCommandSplit
  • travis: Move to bionic
  • shared/util: preserve xattrs in Pack/PackUpdate
  • sources/opensuse: Fix x86 tarballs
  • sources/centos: Fix CentOS 8 checksum file
  • shared/util: Fix locale when importing GPG keys
  • chroot: Resolve parent directory symlinks
  • shared/chroot: Fix parent directory symlink again
  • sources/oracle: Consider boot images only
  • lxd-agent: Update units
  • lxd-agent: Add udev rules
  • generators/lxd-agent: Add symlink workaround for openSUSE
  • lxd-agent: Fix symlink logic
  • doc/examples: Update Ubuntu example
  • managers/yum: Pass "--allowerasing" flag
  • Revert "managers/yum: Pass "--allowerasing" flag"
  • managers/yum: Use --allowerasing when available
  • sources/centos: Handle list of multiple images
  • sources: Fix apertis test
  • travis: Update Go versions
  • managers: Add luet pms
  • add arch deps
  • sources/apertis: Fix tests
  • doc/building: Document --vm flag
  • doc/generators: Add fstab

Downloads

Distrobuilder 1.0 has been released

21st of October 2019

Introduction

The distrobuilder team is proud to announce its initial release, distrobuilder 1.0!

Distrobuilder is a tool which produces root filesystems, LXC and LXD images.

It's the replacement of the LXC template scripts and has slowly been taking over the generation of the many pre-built images that LXC and LXD consume, finally taking over all of them a few months ago.

It's written in Go, shares some logic with LXD and directly generates artifacts suitable for consumption by LXC/LXD or ready to host on an image server.

Rather than relying on shell scripts, it's using a declarative input (YAML) and a number of sources, generators and managers to suit most Linux distributions and use cases.

It can be seen at work here building the many images that we publish daily.

Main components

Image details

This first section covers the usual suspects, the image name, distribution, release, description and architecture.

Example:

image:
  name: ubuntu-disco-x86_64
  distribution: ubuntu
  release: disco
  description: Ubuntu {{ image.release }}
  architecture: x86_64

Image sources

The image sources are distribution specific logic (in Go) to either retrieve a suitable minimal base image or produce one from scratch using commonly available tooling (like debootstrap).

Example:

source:
  downloader: openwrt-http
  url: https://downloads.openwrt.org
  keys:
    - 0x6768C55E79B032D77A28DA5F0F20257417E1CE16
    - 0x54CC74307A2C6DC9CE618269CD84BCED626471F1
    - 0x6D9278A33A9AB3146262DCECF93525A88B699029

Packages and package managers

Distrobuilder currently supports:

  • apk
  • apt
  • dnf
  • ego
  • equo
  • opkg
  • pacman
  • portage
  • xbps
  • yum
  • zypper

They perform package installation/removal, cleanup, updates and add/remove of package repositories.

Example:

packages:
    manager: yum
    update: true
    cleanup: true

    sets:
     - packages:
        - cronie
        - cronie-noanacron
        - curl
        - dhclient
        - initscripts
        - openssh-clients
        - passwd
        - policycoreutils
        - rootfiles
        - rsyslog
        - vim-minimal
       action: install

     - packages:
        - NetworkManager
       action: install
       releases:
        - 8

     - packages:
        - cloud-init
       action: install
       variants:
        - cloud

Files

Distrobuilder can generate common files like hostname/hosts, some amount of init system configuration files, cloud-init templates and more generating templating for LXD containers.

Example:

files:
 - path: /etc/hostname
   generator: hostname

 - path: /etc/hosts
   generator: hosts

 - path: /etc/resolvconf/resolv.conf.d/original
   generator: remove

 - path: /etc/resolvconf/resolv.conf.d/tail
   generator: remove

 - path: /etc/machine-id
   generator: remove

Actions

Actions are effectively a number of hook points at which a provided shell scripts can be run. Those hooks currently are:

  • post-unpack (after the source image was retrieved and unpacked)
  • post-files (after the files were generated)
  • post-updates (after the package updates were applied)
  • post-packages (after any additional package was installed/removed)

Example:

actions:
  - trigger: post-packages
    action: |-
      #!/bin/sh
      set -eux

      # Make sure the locale is built and functional
      echo en_US.UTF-8 UTF-8 >> /etc/locale.gen
      locale-gen en_US.UTF-8 UTF-8
      update-locale LANG=en_US.UTF-8

      # Cleanup underlying /run
      mount -o bind / /mnt
      rm -rf /mnt/run/*
      umount /mnt

      # Cleanup temporary shadow paths
      rm /etc/*-

  - trigger: post-packages
    action: |-
      #!/bin/sh
      set -eux

      # Enable cloud-init units
      systemctl enable cloud-init

    variants:
     - cloud

Architecture maps

One of those tiny internal details, but pretty much every Linux distribution has its own name for the various hardware architecture. Distrobuilder has maps for the most ones, translating between kernel architectures and the name used by the distribution.

This allows for consistent architecture naming in the output, to line up with the patterns used by LXC and LXD.

Example:

mappings:
  architecture_map: debian

Conditions

Packages, Files and Actions can all be set to only apply to specific image variants, distribution releases or OS architectures, making it possible to have a single YAML file for an entire Linux distribution. Some examples of this can be found in the previous examples.

Examples

All the production YAML files for our image server can be found here.

Those are all pretty featureful and will generally be used to generate multiple releases of the distribution for multiple architectures and often for two variants (standard and cloud-init enabled).

Supported distributions

Distrobuilder can build images for the following distributions:

  • Alpine Linux
  • Alt Linux
  • Apertis
  • ArchLinux
  • CentOS
  • Debian (and derivatives, Devuan, Kali, Mint and Ubuntu)
  • Fedora
  • Funtoo
  • Gentoo
  • OpenSUSE
  • OpenWRT
  • Oracle
  • Plamo
  • Sabayon
  • Ubuntu Core
  • Void Linux

On top of this, a generic docker-http source can also be used to use a Docker image as the base for a LXC/LXD image.

Adding a new distribution is usually pretty straightforward and just requires adding a new package manager wrapper and logic to fetch a minimal source image or to produce that image directly.

Architecture support

Distrobuilder supports and is actively used on:

  • aarch64
  • armv7l (with and without hard floating point)
  • i686
  • ppc64le
  • x86_64
  • s390x

Only native image building is supported at this point, no support for cross-building through qemu-user-static or similar tooling.

Security

A common issue with the older shell scripts in lxc-templates was the lack of validation of the downloaded packages and other artifacts instrumental to the image building process. This was often caused by lack of HTTPS servers to download those files from combined with difficulty of dealing with GPG for the very many distributions we had to support.

Distrobuilder changes that model by using pre-built minimal images downloaded over HTTPS or GPG validated (with support for key retrieval or in-line keyring), then either adding on top of that base image or using the base image to produce a whole new image using the distribution's native tooling.

How to get it

This new LXD release is already available for you to try on our demo service.

Installing it

Using the snap

A snap package is available with both stable and edge releases.

It can be installed with:

snap install distrobuilder --classic

Or to have the latest upstream snapshot:

snap install distrobuilder --classic --edge

From source

The release tarballs can be found on our download page. Those include a snapshot of all the Go dependencies needed to build distrobuilder.

Alternatively, the current upstream code can be built with:

go get github.com/lxc/distrobuilder/distrobuilder

Resources