Tales from Beneath The GUI: A Minimal Container Image: ISC DHCP Server

ISC DHCP Server - From Scratch

Customizing a Docker software container is remarkably easy, but that ease imposes a penalty: Image size and layer dependency. This post describes how to create a container image with buildah, a Docker alternative that gives much more control than a simple Dockerfile can offer. The image contains only the service binary and the required libraries, dramatically reducing the container image size and reducing it to a single layer. This image is suitable for use as a systemd service on a container optimized OS like Fedora CoreOS.

The ISC DHCP server, running in the regular mode without backing servers like ldap, is a single process running from a single binary executable. A base image like fedora-minimal provides only package management tools, and those tools remain as a layer of the final image. Installing the dhcp-server package draws in an additional 10 packages and those too remain in the new image layer but are unused by the DHCP service. The dhcp-fedora image described in ISC DHCP - Fedora Base the base image is ~ 146MB and the service layer is an additional 27MB. The base image contains over 30,000 files that are entirely unused during operation.

If the service runs as a single process from a single binary, how small can a functional container image be?

Table of Contents

Modern Executable Files - Dynamic Linking and Shared Objects
Creating a Model File Tree
Building a Minimal Container Image
Summary
References

Modern Executable Files - Dynamic Linking and Shared Objects

Modern operating systems depend heavily on dynamic linking and shared objects. A shared object is a file of compiled code that provides a set of functions that are commonly used by many binaries. This reduces the size of those binaries because all those functions do not need to be duplicated in every statically linked binary. Every dynamically linked binary depends on a set of shared objects to resolve all of those dangling function bindings.

To build a working container image from scratch, it must at least contain the service binary and any shared libraries that it depends on. Those files must identified and retrieved. They must be arranged in the container image so that they can be located and linked when the server binary is invoked.

This two-step procedure is divided into two scripts, one to create a model of the file tree that will be placed into the image and a second to produce the image itself.

create-model-tree.sh
Given the name of a binary file that is provided by a package in the Fedora YUM repositories, create a model tree of the binary and libraries is requires.
minimal-dhcpd.sh
Given a model file tree for a DHCP container, populate the the container image filesystem and create the image.

Creating a Model File Tree

A container is (in the simplest form) a single process running from a single binary. In this example that’s the DHCP server running from /usr/sbin/dhcpd. When the container image is instantiated, the process is started with a command line that invokes that binary.

The container image must include the binary file and any files and resources that the binary operation depends on. Rather than installing packages on top of a base image, the files can be extracted directly from those packages and copied into a model file tree. This is directory tree that mirrors the final content of the the image file tree (or the file layout of a real OS containing only those files).

Note	This process is completely defined in `create-model-tree.sh` This document only highlights the specific steps.

This example uses RPM packages from the Fedora distribution, but the method applies to any Linux distro package system.

Examine the Binary

Since the runtime binary file is known the first step is to download the service package file and unpack that package for examination.

The script creates a workspace with subdirectories to contain all of the downloaded package files and to contain the unpacked file trees. Each package is unpacked into its own tree to keep the contents distinct from all of the others.

Identify the Service Package

The dnf command is used to search remote package repositories and the local system for packages and the files they provide.

dnf provides --quiet dhcpd

dhcp-server-12:4.4.3-14.P1.fc41.x86_64 : Provides the ISC DHCP server
Repo         : fedora
Matched From :
Filename     : /usr/sbin/dhcpd

This command lists the full package name of he current package for the current system and the full path to the file on the last line.

Pull the Service Package

Once the package has been identified, it can be retrieved and stored:

dnf --quiet download --arch ${arch} --destdir ${pkg_dir} ${full_name}

The parameters are provided by the script.

arch: The machine architecture of the new container
pkg_dir: Where to place the downloaded file
full_name: The package name determined in the previous step

Unpack the Service Package

RPM files are a specialized compressed file archive. An RPM must first be converted to a cpio archive before unpacking into the local filesystem.

rpm2cpio ${package_path} | cpio -idmu --quiet --directory ${unpack_dir}

package_path: The path to the service package including the file name
unpack_dir: The target for unpacking the package tree
This directory must have been created before unpacking. The script creates a separate root directory for each package so that the contents of one does not pollute or conflict with any others.

Identify Shared Libraries

Most binaries on Linux are dynamically linked. To run a dynamically linked binary the required libraries must be placed in the location where the dynamic linker expects them to be.

The ldd tool examines a dynamically linked binary and reports the libraries it expects to find.

ldd ${unpack_dir}/${binary_file}

	linux-vdso.so.1 (0x0000ffffb7a1c000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x0000ffffb7640000)
	liblber.so.2 => /lib64/liblber.so.2 (0x0000ffffb7610000)
	libldap.so.2 => /lib64/libldap.so.2 (0x0000ffffb7590000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x0000ffffb7480000)
	libc.so.6 => /lib64/libc.so.6 (0x0000ffffb72b0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffffb79e0000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000ffffb7270000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000ffffb7240000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x0000ffffb7210000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000ffffb71c0000)
	libcrypto.so.3 => /lib64/libcrypto.so.3 (0x0000ffffb6db0000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x0000ffffb6d80000)
	libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x0000ffffb6cf0000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x0000ffffb6c90000)
	libssl.so.3 => /lib64/libssl.so.3 (0x0000ffffb6ba0000)
	libcap.so.2 => /lib64/libcap.so.2 (0x0000ffffb6b50000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000ffffb6b10000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000ffffb6ab0000)
	libz.so.1 => /lib64/libz.so.1 (0x0000ffffb6a70000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x0000ffffb6a20000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x0000ffffb6950000)

unpack_dir: The root of the directory containing the unpacked file trees
binary_file: The absolute path to the binary in the unpacked tree
In this case: /usr/sbin/dhcpd

Each line of this output represents a required shared library. Most lines in this output contain three elements:

The name of the required library
The absolute path of the file containing the library
The memory location where the library is loaded

Only the absolute path is useful for our purposes.

There are two lines that are different from the others. Both relate to the operation of the dynamic linker.

The linux-vdso.so.1 is a virtual file that is provided by the kernel to to all processes in user space. This line can be ignored.

The other is the dynamic linking library, /lib/ld-linux-aarch64.so.1. It does not present a "file name" because only the path matters. This library implements the dynamic linking operations for the rest.

With a little processing this output results in a list of files with absolute pathnames. These can be used in the same way as the binary file name to identify the containing package.

Resolve the Shared Libraries

The next few steps must be done for each of the shared libraries indicated. Note that some of the packages provide more than one of these libraries, so it is beneficial, for each library, to check if the package has already been downloaded and unpacked before proceeding.

Identify a Shared Library Package

The library packages can be identified using the same dnf provide command as was used for the dhcp-server package, with one exception.

The Linux Filesystem Hierarchy Standard defines two possible locations for libraries. These are /lib and /usr/lib. 64-bit systems add two more, /lib64 and /usr/lib64. Most distributions now symlink the top level directories to those in /usr.

ls -l /lib*

lrwxrwxrwx. 1 root root 7 Jan  1  1970 /lib -> usr/lib
lrwxrwxrwx. 1 root root 9 Jan  1  1970 /lib64 -> usr/lib64

This means that the path given by ldd may not be the path that the package publishes for the file. Fortunately, the dnf provide command can take multiple paths and any that don’t resolve are ignored.

In this example libpath is /lib64/libkrb5.so.3

dnf --quiet provides ${libpath} /usr${libpath} 2>/dev/null | head -4

krb5-libs-1.21.3-3.fc41.aarch64 : The non-admin shared libraries used by Kerberos 5
Repo         : @System
Matched From :
Filename     : /usr/lib64/libkrb5.so.3

The full package name is the first word on the first line. This can be tokenized down to 4 components:

krb5-libs-1: The package name
1.21.3-3 : The major, minor, release and build numbers
fc41: Fedora version 41
aarch64: The machine architecture

Only the first element is needed to locate the package.

Note

This package name is an example of one variation that must be accounted for. Some package names end with a hyphenated number -1 or some other integer. I’m not sure what the value represents but it will interfere with package lookup. If the download with the full name fails to find a package, try it with the name minus that trailing string.

Retrieve a Shared Library Package

Downloading the library packages works in the same way as the dhcp-server package did. For this example the enviroment variables are:

package_name: krb5-libs
package_dir: The workspace for downloaded packages

dnf download ${package_name} --destdir ${package_dir}

Updating and loading repositories:
Repositories loaded.
Downloading Packages:
  krb5-libs-0:1.21.3-4.fc41.aarch64                           100% | 772.7 KiB/s | 763.4 KiB |  00m01s

The output indicates the actual package version retrieved. This command also accepts the --quiet option for scripting and parsing. If the package is already present it will indicate that and exit.

Unpack a Shared Library Package

Unpacking the library packages is done in the same way as it was for the dhcp-server package. Each package should be unpacked into a dedicated root directory to prevent the packages from overlaying each other.

rpm2cpio ${package_path} | cpio -idmu --quiet --directory ${unpack_dir}

package_path: The path to the service package including the file name
unpack_dir: The target for unpacking the package tree
This directory must have been created before unpacking. The script creates a separate root directory for each package so that the contents of one does not pollute or conflict with any others.

Populate the Model Tree

At this point all of the required packages are unpacked and all of the required files have been located by the package name and an absolute path from the root of the unpack tree. The model tree must be prepared for the the binary and library files.

mkdir ${model_root}
ln -s usr/lib ${model_root}/lib
ln -s usr/lib64 ${model_root}/lib64
mkdir -p ${model_root}/usr/lib
mkdir -p ${model_root}/usr/lib64
mkdir -p ${model_root}/usr/sbin

Most of the shared library files that ldd reported are actually symbolic links to a matching file with an additional version number.

For example, the libkrb5.so.3 library is a symlink to libkrb5.so.3.3.

(cd ${workdir} ; ls -l usr/lib64/libkrb5.so.*)

lrwxrwxrwx. 1 core core     14 Feb 11 00:00 usr/lib64/libkrb5.so.3 -> libkrb5.so.3.3
-rwxr-xr-x. 1 core core 873304 Feb 11 00:00 usr/lib64/libkrb5.so.3.3

It may be possible to copy the library to the short name but for rigor the script copies the file to the correct name and reproduces the symlink as it is created by the package.

The final result looks like this:

(cd ${model_root} ; ls -lgGR *)

lrwxrwxrwx. 1  7 Mar  4 15:23 lib -> usr/lib
lrwxrwxrwx. 1  9 Mar  4 15:23 lib64 -> usr/lib64

usr:
total 4
drwxr-xr-x. 2   35 Mar  4 15:23 lib
drwxr-xr-x. 2 4096 Mar  4 15:23 lib64

usr/lib:
total 816
-rwxr-xr-x. 1 832552 Mar  4 15:23 ld-linux-aarch64.so.1

usr/lib64:
total 12584
-rwxr-xr-x. 1 2301232 Mar  4 15:23 libc.so.6
lrwxrwxrwx. 1      14 Mar  4 15:23 libcap.so.2 -> libcap.so.2.70
-rwxr-xr-x. 1  200816 Mar  4 15:23 libcap.so.2.70
lrwxrwxrwx. 1      17 Mar  4 15:23 libcom_err.so.2 -> libcom_err.so.2.1
-rwxr-xr-x. 1   69296 Mar  4 15:23 libcom_err.so.2.1

... <lines elided>

lrwxrwxrwx. 1      21 Mar  4 15:23 libz.so.1 -> libz.so.1.3.1.zlib-ng
-rwxr-xr-x. 1  136752 Mar  4 15:23 libz.so.1.3.1.zlib-ng

usr/sbin:
total 2492
-rwxr-xr-x. 1 2548720 Mar  4 15:23 dhcpd

The model tree now contains the dhcpd binary and all of the required library files.

Building a Minimal Container Image

The idea of building a minimal container image is to decrease the amount of data that must be downloaded initially and downloaded again when the container image is updated and rebuilt (and the base image is updated underneath it). The ratio of size of the runtime required bits to the installation overhead is surprsingly large.

The other reason to minimize an image is that it decreases the attack surface of a container process by removing any files that aren’t critical to operation. Containers are not a security mechanism. If a cracker manages to exploit the running process and gain access to the container filesystem, the fewer resources the container gives them the better.

Note	This section only shows the highlights of this procedure. The procedure is fully described in the `minimal-dhcpd.sh` script.

Initialize a new container build

container_id=$(buildah from scratch)

The command above starts a container build procedure. It initializes a file space and metadata that will be manipulated in the steps that follow.

When building a container image using a distro base image, you get the access to the package management system and the distro repositories. When building from scratch you have to provide all of the image files and place them in a file tree that matches the expected structure for the application to run. Since the scratch image doesn’t have tools like mkdir, it’s not possible to use buildah run commands to manipulate the container file system.

The solution is to loopback mount the image filesystem onto the operating system and then use the OS tools to create the file tree. This is where buildah stands out.

`buildah unshare` for Rootless Containers

As Dan Walsh explains in a blog post on buildah unshare, the common build commands, run and copy, create a new namespace where the user appears to be UID 0 (root) and mount the image filesystem so that they can operate on the files in the image and then destroy that namespace before returning.

The common buildah commands do one thing at a time. Without a base image containing a shell, the run command isn’t useful. The copy command can import single files or the contents of a single directory into a single target directory, but it doesn’t offer recursive copies and the destination must already exist inside the container image.

The buildah unshare command creates a new namespace in the same way as the other commands, but it runs a shell inside that namespace that makes it possible for the caller to access the container filesystem without root access to the host system. For the purpose here this allows the user to loopback mount the container filesystem and copy the model file tree into it.

An example of buildah unshare

user@hostname:~/dhcpd-container$ buildah unshare
root@hosthame:~/dhcpd-container# id
uid=0(root) gid=0(root) groups=0(root)...
root@hostname:~/dhcpd-container# lsns
        NS TYPE   NPROCS   PID USER COMMAND
4026531834 time        3  4862 root buildah-in-a-user-namespace unshare
4026531835 cgroup      3  4862 root buildah-in-a-user-namespace unshare
4026531836 pid         3  4862 root buildah-in-a-user-namespace unshare
4026531838 uts         3  4862 root buildah-in-a-user-namespace unshare
4026531839 ipc         3  4862 root buildah-in-a-user-namespace unshare
4026531840 net         3  4862 root buildah-in-a-user-namespace unshare
4026532291 user        3  4862 root buildah-in-a-user-namespace unshare
4026532293 mnt         3  4862 root buildah-in-a-user-namespace unshare
root@hostname:~/dhcpd-container# env | grep BUILDAH
BUILDAH_ISOLATION=rootless
root@hostname:~/dhcpd-container# exit
user@hostname:~/dhcpd-container$

The fragment above shows what buildah unshare is doing.

All of the buildah commands can be run within the unshare namespace, but the only ones that require it for this procedure are the mount and unmount commands. The image build script can be run either way and will unshare for the copy steps if needed.

To make the container filesystem available, the unshare command takes the container id in Building a Minimal Container Image above.

Create a mock-root namespace for container filesystem access

buildah unshare ${container_id}

Rather than requiring the user to call buildah unshare before invoking the script, it checks to see if it’s already running in an unshare environment. If not, it calls itself again with unshare. Then it calls the copy_model_tree() function to mount the container filesystem and copy the model tree into it.

Re-call the script with unshare if needed.

# ...
if [ -z "${BUILDAH_ISOLATION}" ] ; then
    # Run the file copy in an unshare environement
    buildah unshare bash $0 -c ${container} -s ${SOURCE_ROOT}
else
    # Aldready in an unshare environment
    copy_model_tree ${SOURCE_ROOT} ${container}
fi
# ...

Copy the Model Tree

The critical step in creating a container is populating the filesystem for the image. For an image using a distro base, this is done with the distro package manager. Single files are added using the copy command.

For a minimal image, the file tree must be created and the files placed without access to tools inside the container base. The solution is to mount the container image filesystem onto the build host and copy the files in directly using the host tools.

The bash function below assumes that the process is already in an unshare environment. It mounts the container filesystem, copies the contents of a file tree into the image file tree recursively. It creates two directories required for the application configuration and data volumes. Finally it unmounts the container image and returns.

copy_model_tree function

function copy_model_tree() {
    local source_root=$1
    local container_id=$2

    # Access the container file space
    local mountpoint=$(buildah mount $container_id)

    # Create the model directory tree
    (cd ${source_root} ; find * -type d) | xargs -I{} mkdir -p ${mountpoint}/{}
    # Copy the model tree to the image filesystem.
    cp -r ${source_root}/* ${mountpoint}

    # Create volume mount points
    mkdir -p ${mountpoint}/etc/dhcp
    mkdir -p ${mountpoint}/var/lib/dhcpd

    # Release the container file space
    buildah unmount ${container_id}
}

The separate mkdir line insures that symlinks to directories in the model tree aren’t created in place of real directories.

Define Container Operation

The final container definition steps are identical to those for a distro based image.

Define container operation and metadata

# add a volume to include the configuration file
# Leave the files in the default locations
buildah config --volume /etc/dhcp/dhcpd.conf $container
buildah config --volume /var/lib/dhcpd $container

# open ports for listening
buildah config --port 68/udp --port 69/udp ${container}

# Define the startup command
buildah config --cmd "/usr/sbin/dhcpd -d --no-pid" $container

buildah config --author "${AUTHOR}" $container
buildah config --created-by "${BUILDER}" $container
buildah config --annotation description="ISC DHCPD 4.4.3" $container
buildah config --annotation license="MPL-2.0" $container

# Save the container to an image
buildah commit --squash $container dhcpd

This fragment defines the configuration volumes, opens the required ports and sets the image metadata before committing and naming the image within the local container namespace.

Review the new container image

podman image inspect localhost/dhcpd |
  jq '.[0] | {"Id": .Id, "Size": .Size, "Config": .Config }'
{
  "Id": "aacc40467b44590ece02a7c68c4e00ac6fcafaa08d7914452618f622cd65a445",
  "Size": 16260301,
  "Config": {
    "ExposedPorts": {
      "68/udp": {},
      "69/udp": {}
    },
    "Cmd": [
      "/usr/sbin/dhcpd",
      "-d",
      "--no-pid"
    ],
    "Volumes": {
      "/etc/dhcp/dhcpd.conf": {},
      "/var/lib/dhcpd": {}
    },
    "WorkingDir": "/",
    "Labels": {
      "io.buildah.version": "1.39.0"
    }
  }
}

You can always examine a container image this way to determine the run-time parameters. The full report is significantly bigger and more detailed.

Summary

As noted, this container image runs in exactly the same way as the Fedora based image. The real payoff is in the the size savings.

Compare scratch and distro based image size

podman images | grep dhcp
localhost/dhcpd                            latest      aacc40467b44  25 hours ago  16.3 MB
localhost/dhcpd-fedora                     latest      4581f80d82a6  2 days ago    172 MB

Tales from Beneath The GUI

Monday, March 10, 2025

A Minimal Container Image: ISC DHCP Server - From Scratch

A Minimal Container Image: ISC DHCP Server - From Scratch

Modern Executable Files - Dynamic Linking and Shared Objects

Creating a Model File Tree

Examine the Binary

Identify the Service Package

Pull the Service Package

Unpack the Service Package

Identify Shared Libraries

Resolve the Shared Libraries

Identify a Shared Library Package

Retrieve a Shared Library Package

Unpack a Shared Library Package

Populate the Model Tree

Building a Minimal Container Image

`buildah unshare` for Rootless Containers

Copy the Model Tree

Define Container Operation

Summary

References

No comments:

Post a Comment

Contributors

Monday, March 10, 2025

A Minimal Container Image: ISC DHCP Server - From Scratch

A Minimal Container Image: ISC DHCP Server - From Scratch

Modern Executable Files - Dynamic Linking and Shared Objects

Creating a Model File Tree

Examine the Binary

Identify the Service Package

Pull the Service Package

Unpack the Service Package

Identify Shared Libraries

Resolve the Shared Libraries

Identify a Shared Library Package

Retrieve a Shared Library Package

Unpack a Shared Library Package

Populate the Model Tree

Building a Minimal Container Image

buildah unshare for Rootless Containers

Copy the Model Tree

Define Container Operation

Summary

References

No comments:

Post a Comment

`buildah unshare` for Rootless Containers