Developer manual
Distro image builder

Design principles

  • Relatively simple design
    • bash script: components, configurations, recipes
    • Alpine Linux chroot install script
    • debootstrap and ansible for setting up the VM image
    • systemd-nspawn for building in a sandbox
  • Isolated builds
    • most build tasks executed in systemd-nspawn sandbox based on Alpine Linux
    • read-only view of the configurations
  • Lightweight
    • container sandbox much faster than full OS virtualization
  • Fast
    • building mostly in tmpfs (requires 8-16 GB of RAM depending on OS size)
    • if not enough memory is provided, builds on disk (still needs 2-4 GB of RAM)
  • CI/CD friendly
    • dependency tracking -> incremental rebuilds
    • cached task inputs and outputs
  • Few dependencies
    • basic core Linux userspace utilities / busybox + systemd
  • Can produce a VM for building itself
    • a bit slower, depending on the build target, up to 16GB of VM RAM needed

Basic structure

  • alpine-installer - derived from https://github.com/alpinelinux/alpine-chroot-install - installer for the 'builder' task
  • build.sh - the "brains", a build system for various components
    • configs/
      • assets/
      • shared/
        • *.sh - the scripts for building components (e.g. busybox or kernel), usually launches sandboxes for simple build tasks
        • *.conf - default INI style configuration files for components (defines e.g. component version, source url, checksum + other stuff)
        • componentname/* - other component related files needed to build the component
      • configurations/
        • configurationname/*.conf - INI style configuration files for components (overrides configs/shared/*.conf), the idea is to provide a shared special config for multiple recipes
        • configurationname/componentname/* - other component related files needed to build the component (overrides configs/shared/componentname/*)
      • recipes/
        • *.sh - "build plan" for a distro image
        • recipe/*.conf - INI style configuration files for components, overrides configs/shared/*.conf and configs/configurations/*.conf, special configs for a specific recipe
        • recipe/recipename/componentname/* - other component related files needed to build the component (overrides configs/shared/componentname/* and configs/configurations/configurationname/componentname/*)
graph TB
  subgraph "Configuration structure"
  configs --> assets
  configs --> shared
  shared --> sharedsh[:comp:.sh]
  shared --> sharedconfigs[:comp:.conf]
  shared --> shcomponent[:comp:]
  configs --> configurations --> configuration[:configuration:]
  configuration --> csh[:comp:.sh]
  configuration --> cconfigs[:comp:.conf]
  configuration --> ccomponent[:comp:]
  configs --> recipes
  recipes --> rsh[:recipe:.sh]
  recipes --> recipe[:recipe:]
  recipe --> resh[:comp:.sh]
  recipe --> reconfigs[:comp:.conf]
  recipe --> recomponent[:comp:]
end

Components

The component script describes how the component is built and installed. Each component.sh script is accompanied with a component.conf The script and the default configuration is located in configs/shared.

Fields

Each component.conf file must define

VERSION=<versionstring>

Each component.conf file may also define

SRCURL=<source url, if downloading files with 'dobuild pkg'>
SRCCHECKSUM=<sha1sum of the downloaded source file>
HOMEPAGE=<link to the upstream download page, check for updates manually>

Each component.sh file must define

function build() { ... }       -- builds the component

Each component.sh file may also define

function init() { ... }        -- define environments
  - DEPS="" -- space separated list of elements{param, params}
  - TARGET=<output of the component build script,
           located in packages/{build,out},
           by default depends on where the config override is found>

function fetch() { ... }       -- fetches the source file from SRCURL

function description() { ... } -- prints the component description

function reconfigure() { ... } -- functionality for reconfiguring the component

function install() { ... }     -- functionality for installing the component

graph TB
  subgraph "Component dependencies"
  start --> distova
  start --> bootdisk
  distova --> userstorage
  distova --> bootvmdk
  bootdisk_prepare --> image_application
  bootdisk --> initramfs
  bootdisk --> kernel
  bootdisk --> linux_firmware
  bootvmdk --> metadata
  bootvmdk --> bootdisk
  image_application --> image_custom
  image_application --> image_platform
  image_application --> preload_pkgs
  image_application --> preload_debs
  image_base --> preload_debs
  image_platform --> squashfstools
  image_platform --> image_base
  image_platform --> preload_pkgs
  image_platform --> vboxguest
  image_platform --> preload_debs
  initramfs --> metadata
  initramfs --> vbox
  kernel --> linux_firmware
  metadata --> bootdisk_prepare
  metadata --> image_application
  metadata --> kernel
  vbox --> vboxguest
  vbox --> kernel
end

graph TB
  subgraph "Not in use anymore"
  bootdisk_compress --> bootdisk
end

graph TB
  subgraph "Initramfs utitilies"
  initramfs --> busybox 
  initramfs --> column 
  initramfs --> dialog 
  initramfs --> e2fsprogs 
  initramfs --> htop 
  initramfs --> mc 
  initramfs --> nbdclient 
  initramfs --> ntfs3g 
  initramfs --> squashfstools 
end

The following are extracted from the sources with:

grep description -A1 *|grep echo|sed 's/^/ * /g;s/-    echo "/: */g;s/"$/*/g'|sort

Initramfs + utils

  • busybox: Compiles busybox (variant: $COMPONENTCONFIG). Busybox provides a wide set of core utilities used in the initramfs systems.
  • column: Compiles 'column'. The utility is used for pretty printing stuff in the initramfs.
  • dialog: Compiles 'dialog'. The utility is used for the initramfs menu system.
  • e2fsprogs: Compiles 'dumpe2fs', 'e2fsck', 'mke2fs', and 'resize2fs'. The utilities are used for manipulating ext4 systems in the initramfs.
  • gdisk: Compiles 'gdisk'. The utility is used to fix the partition table on USB sticks larger than the default image.
  • htop: Compiles 'htop'. The utility is available in the initramfs.
  • initramfs: Build the initramfs. Always contains basic Linux terminfo, build metadata file, configured kernel modules and initramfs configs and scripts
  • mc: Compiles 'mc' and 'mcedit'. The utilities are available in the initramfs.
  • nbdclient: Compiles 'nbd-client'. The utility is used to mount NBD shares (e.g. PXE boot) in the initramfs.
  • ntfs3g: Compiles 'ntfs-3g'. The utility is used to mount ntfs drives in the initramfs.
  • squashfstools: Compiles 'mksquashfs' and 'unsquashfs'. Could be used in a rescue initramfs. The 'mksquashfs' utility is being used for generating compressed spawn-container images.

General infrastructure

  • bootdisk_prepare: Prepares the bootdisk. This is the main disk for virtual machines ('userstorage' is the other). For USB media, this contains everything. The size of the image and the partitions varies depending on the size of the rootfs. The partition layout is a MBR+GPT hybrid. Note that the configuration is split in multiple .conf files and refers to the previous stages.
  • bootdisk: Builds a bootable disk image using syslinux as a bootloader, provided initramfs and kernel + amd/intel ucode blobs + other configured boot disk contents.
  • bootvmdk: Converts the RAW bootdisk image into a vmware vmdk file, version 4.
  • builder: Builds a relatively small squashfs compressed Alpine installation used for sandboxing further build stages.
  • distova: Generates a distributable OVA file from the bootvmdk + userstorage combo. Also fixes the OVA version number since vmware can't import files generated with the same vmware tools, yay.
  • metadata: Generates the metadata file listing build specific key-values. The metadata also considers git history and unstaged files so a rebuild is needed after toying around with the local repository.

Platform, application, and other images

  • image_application: Builds the 3rd stage target Debian system (variant: $COMPONENTCONFIG) based on Ansible rules & 'platform' image. Stores a deb cache in builddir/cache or uses a previous stage deb cache. Compresses the output with squashfs.
  • image_base: Builds the 1st stage target Debian system using debootstrap. The image contains opensshd & ansible for remote configuration and dbus/systemd parts for systemd-nspawn local administration. Stores a deb cache in builddir/cache. Compresses the output with squashfs.
  • image_custom: Builds a custom OS image. Compresses the output with squashfs.
  • image_platform: Builds the 2nd stage target Debian system (variant: $COMPONENTCONFIG) based on Ansible rules & 'base' image. Stores a deb cache in builddir/cache or uses a previous stage deb cache. Compresses the output with squashfs.
  • userstorage: Generates an empty disk with a single full size ext4 partition for user storage. The disk image is then converted to VMDK.

Kernel and modules

  • kernel: Compiles the Linux kernel (variant: $COMPONENTCONFIG). The resulting kernel and modules are stored in builddir/kernel and dstdir/kernelmodules.
  • linux_firmware: Builds a squashfs image of the redistributable firmware, based on kernel org sources and gentoo ebuild.
  • nvidia: Compiles the proprietary nvidia kernel modules to dstdir/kernelmodules.
  • vbox: Compiles the virtualbox guest utils kernel modules to dstdir/kernelmodules. Sadly the mainline versions are incompatible with the userspace guest utils.

Other

  • bootdisk_compress: Compresses the boot disk for release. The empty user partition mostly contains zeros so there are plenty of reasons to compress.
  • openjfx: Compiles Alpine/musl compatible OpenJFX runtime.
  • preload_debs: Preloads and caches list of deb files to speed up subsequent builds
  • preload_pkgs: Preloads and caches list of files to speed up subsequent builds.
  • vboxguest: Extracts the guest utils (both kernel and user space) from Virtualbox guest ISO distribution.

Configurations

Located in configs/configurations.

Shared sets of configurations files

  • buildvm: configuration for building a VM that can be used to run build.sh on any system
  • minimal: a definition of busybox{minimal} used by utuvm-usb && utuvm-vbox
  • rescue: a definition of initramfs, originally intented for rescue shells
  • unixcourse: a definition of lightweight builder image for the UNIX course
  • usb: utuvm-usb specific bootdisk, bootdisk_prepare, image_platform (platform applications), initramfs, and kernel
  • vm: utuvm-vm specific bootdisk, bootdisk_prepare, image_platform (platform applications), initramfs, and kernel

Recipes

Located in configs/recipes.

  • utuvm-builder: configuration for building a VM that can be used to run build.sh on any system
    • default target: distova
  • utuvm-usb: defines the USB platform (native x86-64 Linux)
    • default target: bootdisk_compress
  • utuvm-vbox: defines the virtual platform (virtualbox, vmware, qemu, ...)
    • default target: distova

Platforms

Hardware platforms

Located in configs/recipes, configs/shared/image_platform/, configs/configurations/{buildvm,vbox,usb}.

  • USB: recipe utuvm-usb, image_platform usb
  • VM: recipe utuvm-vbox, image_platform virtual, vbox (virtualbox kernel drivers)
  • Build VM: recipe utuvm-builder, image_platform buildvm

Application platforms

Located in configs/recipes, configs/shared/image_application/.

  • plain: does not contains any application specific stuff, only the platform image
  • java: contains the platform + Java application image
  • minimega: contains the platform + CPP & Web & mobile & declarative application images
  • latex: contains the platform + Java application image
  • mega: contains java + minimega

Deprecated application platforms

  • CPP: contains the platform + C/C++ application images
  • Webmob: contains the platform + Web & mobile application images

Ansible configuration

Base image

Doesn't actually use Ansible. Constructed with debootstrap. See configs/shared/image_base.conf for the package list.

Platform image

The "platform" image contain all common functionality for a VM or USB distro.

See configs/shared/image_platform/ (and also the overriding configs in configs/configurations/*). Ansible files can be searched with find configs/ -type f -name '*.yaml'|sort.

Contents of configs/shared/image_platform/:

Location Description
. contains the main ansible script: playbook-install-platform.yaml
library contains the ansible scripts
systemfiles system-wide (root) owned files - copied by playbook-conf-basesystem.yaml
systemfiles/common misc stuff copied to /etc/
systemfiles/usrlocal scripts copied to /usr/local/, e.g. backgrounds, system/helper scripts
userfiles files for /home/utu - copied by playbook-conf-user.yaml
userfiles/common bookmarks, xorg startup script, desktop entries
userfiles/manuals manuals (pdf)
userfiles/xfce_config xfce configuration

Application image

The high level functionality (e.g. Java SDK) belongs to the "application" image.

See configs/shared/image_application/ (and also the overriding configs in configs/configurations/*). Ansible files can be searched with find configs/ -type f -name '*.yaml'|sort.

Ansible library

library/playbook-clean-system.yaml
library/playbook-conf-basesystem.yaml
library/playbook-conf-user.yaml
library/playbook-install-seadrive.yaml
library/playbook-install-slim.yaml
library/playbook-postconf-usb.yaml
library/playbook-postconf-virtual.yaml
library/playbook-prepare-common-usb.yaml
library/playbook-prepare-common-virtual.yaml
library/playbook-prepare-common.yaml
playbook-conf.yaml
playbook-prepare-usb.yaml
playbook-prepare-virtual.yaml

System/common

systemfiles/common/20-wired.network
systemfiles/common/fixnet.service
systemfiles/common/locale.sh
systemfiles/common/pulsevolume.service
systemfiles/common/taustapun.jpg
systemfiles/common/taustasin.jpg

System/usrlocal

systemfiles/usrlocal/configurator.sh
systemfiles/usrlocal/fixnet.sh
systemfiles/usrlocal/gitk-browse.sh
systemfiles/usrlocal/install-qemu-additions.sh
systemfiles/usrlocal/install-vbox-guestadditions.sh
systemfiles/usrlocal/pulsevolume.sh
systemfiles/usrlocal/spawn-container

User/common

userfiles/common/bookmarks
userfiles/common/doxygen.desktop
userfiles/common/git.config
userfiles/common/gitk.desktop
userfiles/common/locale.conf
userfiles/common/xsessionrc

User/manuals

userfiles/manuals/git_guide_en.pdf
userfiles/manuals/Opiskelijan_opas.pdf
userfiles/manuals/pieni_git_opas_fi.pdf

User/xfce4

userfiles/xfce_config/keyboards.xml
userfiles/xfce_config/thunar.xml
userfiles/xfce_config/xfce4-desktop.xml
userfiles/xfce_config/xfce4-keyboard-shortcuts.xml
userfiles/xfce_config/xfce4-panel.xml
userfiles/xfce_config/xfce4-session.xml
userfiles/xfce_config/xfwm4.xml

Using

Lauching build.sh (note that if you have a system with X11 started up, a graphical user interface will be shown instead - use unset DISPLAY to hide it):

Screenshot1

Another help screen when a recipe has been selected: Screenshot1

Error message when a recipe and operation has been selected without specifying mandatory settings (here, BUILDVERSION):

Screenshot1

The deps command can be used to see what needs to be done for the (here, default) build target:

Screenshot1

In this screenshot, the system has been successfully built so nothing needs to be done, everything in printed in light green:

Screenshot1

This command demonstrates how we can also build just a subgoal instead of the default main goal:

Screenshot1