Thursday, March 6, 2025

Deploying ISC DHCP Server in a systemd Container

ISC DHCP Server in a Container

This repository describes the operation of a software container to run the ISC DHCP service.

Note
ISC DHCP has been deprecated and end-of-lifed as of Dec 2022 in favor of ISC Kea

ISC DHCP has been the standard DHCP server since the release of version 1.0 in 1998. While ISC has deprecated the orginal DHCP as of Dec 2022 in favor of ISC Kea, the oringal DHCP server remains in use and is a standard package on most Linux distributions.

Conventional packages are tightly bound to the OS distribution and to the version of the OS installed on a particular host. Network services and the systems that host them are crititical infrastructure. The fear of downtime often results in aversion to regular updates.

The fear of update failures can be mitigated by providing for atomic rollback. This means that, in the event of an update induced failure, the service can be rolled back reliably to the previous version. Containerized services offer a way to accept frequent updates while also allowing rapid rollback to a previous version without the fear of conflict or system pollution that packaged software risks.

The dhcp container encapsulates the ISC DHCP daemon so that it can run on any Linux host that uses systemd for system initialization and service management.

Container Images

This repository contains scripts to create two different but functionally equivalent container images. Both are based on the Fedora dhcp-server package. This package provides the ISC DHCP server binary: dhcpd.

The dhcpd-fedora image uses the fedora-minimal base image. The dhcpd image is built from scratch.

This document describes the usage of these images to provide a DHCP service on a host capable of running software containers from systemd-containers. To learn how the container images were built, see:

Prerequisites

The system must have the BOOTP and DHCP ports (68/UDP and 69/UDP) free and must be capable of running systemd containers.

  • Static IP addresses
    The DHCP service itself requires that all IP addresses on the host are configured statically. The DHCP client will, obviously, conflict with the server for port bindings. This took me an embarrassingly long time to figure out when testing the container image.

  • containers-common package
    The host server must be able to run software containers and must have the containers-common package installed. This package provides the bindings that allow podman to generate systemd service unit files from a container service file. On Debian this is the golang-github-containers-common.

Configuration

The configuration of the DHCP daemon is the same as if it was running from an installed package. The Service configuration is contained in the /etc/dhcp/dhcpd.conf file and the lease database is in /var/lib/dhcpd/dhcp.leases.

When the service is packaged, the package creates and populates these with default values. A container can’t affect the OS outside its boundaries so the user must create those files before the service is started.

  • /etc/dhcp/dhcpd.conf
    The daemon is configured with the standard configuration file in the standard location. Describing the configuration is outside the scope of this document. See the excellent existing configuration documents for the ISC DHCP daemon.

  • /var/lib/dhcpd/dhcp.leases
    The lease file must exist when the daemon starts. Initially it is an empty file.

    sudo mkdir -p /var/lib/dhcpd ; sudo touch /var/lib/dhcpd.leases

Enable Containerized Services

OS support for Container and VM management are provided by the containers-common package. On Debian based systems this is called golang-github-containers-common

On recent Red Hat and Fedora releases it is installed by default along with podman. On Debian based systems you must install the package or one that depends on it, like the podman package.

sudo apt update ; sudo apt install golang-github-containers-common

Install DHCP container service

Containerized services are defined by a container unit file. This file follows a similar format to the other systemd unit files.

The container service system was originally known as quadlets but now is defined in the podman systemd specification.

The container file below pulls one of the public container images for the ISC DHCP server.

dhcpd.container
[Unit]
Description=ISC DCHP daemon Service Container
After=network-online.target

[Container]
Image=quay.io/markllama/dhcpd:latest
#Image=quay.io/markllama/dhcpd-fedora:latest

# This shouldn't be needed except for reading /etc/dhcp/dhcpd.conf
# Could move it to /opt/dhcp/dhcpd.conf
PodmanArgs=--privileged
Network=host

# Open listening ports
# bootps
PublishPort=67:67/udp
PublishPort=67:67/tcp

# bootpc
PublishPort=68:68/udp
PublishPort=68:68/tcp

# Mount the dhcp config dir into the container workingdir
Volume=/etc/dhcp/dhcpd.conf:/etc/dhcp/dhcpd.conf:ro,Z
Volume=/var/lib/dhcpd/:/var/lib/dhcpd/:rw,Z

[Install]
# Enable in multi-user boot
WantedBy=multi-user.target default.target

#  podman run --detach --name dhcpd \
#    --privileged  \
#    --network host \
#    --volume /etc/dhcp/dhcpd.conf:/etc/dhcp/dhcpd.conf:ro,Z \
#    --volume /var/lib/dhcpd/:/var/lib/dhcpd/:rw,Z \
#    quay.io/markllama/dhcpd

The two container images indicated in the file above are both created from the Fedora RPM dhcp-server. The dhcpd-fedora container image is based on the Fedora 41 base image. The dhcpd image is a minimal image create from scratch, containing only the dhcpd binary and the required shared libraries.

Limitations and Future Work

This is a proof-of-concept project. The container design currently does not support any of these features of the ISC DHCP server

  • No configuration of invocation parameters

  • No IPv6

  • No OMAPI

  • No external database

References

  • DHCP History
    The history of DHCP and of the ISC DHCP server

  • ISC DHCP
    A DHCP service available on all Linux distributions

  • Podman systemd unit file
    The container service unit file specification

  • systemd
    The modern Linux OS initialization and service management system

  • The containers-common package
    The package that provides the container bindings for systemd services

    • Debian golang-github-containers-common

    • Fedora containers-common

Monday, February 17, 2025

Deploying CoreDNS as a systemd Service

DNS Service - CoreDNS

CoreDNS was developed for use in Kubernetes as a light-weight name server for containerized services. For a small to medium sized network, CoreDNS is much simpler to configure and operate than any of the production quality alternatives.

This post is meant to demonstrate the use of systemd services running as software containers on Fedora CoreOS. However this technique is applicable to any host that has Podman version 4.4.0+ installed and is configurable by Ansible.

If you are curious about provisioning Fedora CoreOS for this demonstration, see the previous series of posts for guidance:


Running CoreDNS

The CoreDNS configuration consists of a single configuration file and a set of DNS zone files. The files can be contained in a single directory, commonly /opt/coredns. The program looks for a file called Corefile in the current working directory when it is invoked.

CoreDNS is meant to be run in a software container. CoreOS provides podman, a drop-in CLI replacement for Docker and the runc runtime.

DNS servers listen to UDP port 53 for queries. TCP port 53 is used for some operations such as zone transfers and secure DNS. By default it listens on all configured interfaces.

The configuration used in this example is meant for use inside a firewalled network. It does not serve queries to devices outside the local network. It is meant to provide a split-dns

The CoreDNS container image is always found here:

docker.io/coredns/coredns:latest

The coredns-server Playbook

The goal of the coredns-server playbook is to install and configure CoreDNS on a set of servers. The servers need to listen for and respond to DNS queries on port 53/UDP on one of a set of listed IPv4 addresses. The service runs in a software container and is managed as a systemd service.

The deployment steps can be grouped into four related sets of tasks:

  1. Switch to static resolver

  2. Configure network interface

  3. Deploy CoreDNS configuration

  4. Configure systemd service

For clarities sake these four are broken down into separate task files in the coredns-server role. These are detailed in corresponding sections below.

An Ansible playbook is defined in a .yaml formatted file. It is possible to contain the entire playbook in a single file, but it is usually helpful to have the playbook use a role. Roles are re-usable modules that

---
#
# The playbook creates a DNS server on the target hosts using CoreDNS
# It populates the zone files from files/zones
#
- name: CoreDNS Server
  hosts: dnsservers
  become: true

  vars_files:
    - dns_services.yaml

  roles:
    - coredns-server

The dns_services.yaml file specifies the parameters for the CoreDNS server. Among these are the locations and zones for the zone files. These reside in files/zones in the ansible directory. The zone files here are static and follow the RFC standards and will be familar to anyone who’s configured ISC Bind. They could be produced mechanically from other databases but that is outside the scope of this project.

Note
The dns_services.yaml file contains global variables that are not part of the playbook. They are stored at the top of the ansible tree along with the zone files in files/zones
---
#
# DNS services for example.com network
#
dns:
  nameservers:
    pi4-1:
      fqdn: ns1.example.org
      ipv4: 192.168.2.10
    pi4-2:
      fqdn: ns1.example.org
      ipv4: 192.168.2.11

  forwarders:
    - 192.168.2.1
    - 4.2.2.1     # Level3 caching DNS server IP address
    - 1.1.1.1	  # Cloudflare caching DNS server IP Address

  zones:
    - fqdn: example.org
      file: example.org.zone
    - fqdn: lab.example.org
      file: lab.example.org.zone

  search:
    - lan    # mDNS from Google Mesh DNS
    - example.org
    - lab.example.org

The coredns-server Role

This role encapsulates the process of installing a CoreDNS server on a host. The broad steps are described above.

coredns-server role tree
roles/coredns-server/
├── files
│   └── coredns.container
├── handlers
│   └── main.yaml
├── tasks
│   ├── config_files.yaml
│   ├── main.yaml
│   ├── network.yaml
│   ├── resolver.yaml
│   └── systemd_service.yaml
└── templates
    ├── Corefile.j2
    └── resolv.conf.j2

5 directories, 9 files

The task files are the primary driver of a playbook and role. The rest of the files provide resources that serve the tasks as they are run.

The task files are the primary driver of a playbook and role. The rest of the files provide resources that serve the tasks as they are run. The file main.yaml acts as the entry point for the tasks defined in the tasks/ subdirectory. The tasks are defined as if they were part of a playbook, as a YAML list. The main.yaml file refers to a set of smaller task files, grouping the tasks functionally.

---
#
# Coordinate creating a coredns service container
#
- name: Disable systemd-resolved and set static resolver file
  import_tasks: resolver.yaml

- name: Configure and set DNS Listener IP address
  import_tasks: network.yaml

- name: Place the Configration Files
  import_tasks: config_files.yaml

- name: Prepare Systemd Services
  import_tasks: systemd_service.yaml

Note that the first three sets of tasks are not special for CoreOS. They’re applicable to any DNS service. The final task list is the important one for this series.

Disable Dynamic DNS Resolver Service

Since 2020, with the release of Fedora 33, the the local DNS resolver is a daemon integrated with systemd. This daemon listens for local queries and is bound to port 53/UDP. The CoreDNS server needs to bind to the same port, so the systemd-resolved service must be stopped and disabled before coredns can start.

This set of tasks disables the systemd-resolved service and replaces the stock /etc/resolv.conf file with one configured for the target environment.

- name: Disable systemd-resolved - (avoid conflict with coredns)
  service:
    name: systemd-resolved
    state: stopped
    enabled: false

- name: Set static resolver file
  template:
    dest: /etc/resolv.conf
    src: resolv.conf.j2
    owner: root
    group: root
    mode: 0644
    backup: true
#
# Maintained by Ansible
#
nameserver 127.0.0.1
{% for nameserver in dns.forwarders %}
nameserver {{ nameserver }}
{% endfor %}
search {{ dns.search|join(' ') }}

The resolv.conf file directs DNS queries first to the local nameserver and then to the listed forwarders when the local server does not serve the requested domain.

Set DNS Listener IP Address

The DNS service requires two servers for each domain. The servers are identified by IP address because, well they provide the name services. This step ensures that each server host is listening on one of those two addresses.

This task set finds the default interface on this host and then creates a new connection that attaches to the physical one and answers the servers listener address. The connection type is macvlan and it allows this interface to be configured manually while allowing the main interface to use DHCP for the rest of the network information.

The critical step here is the second one. It creates a virtual interface dedicated to the DNS listener address.

- name: Record interface name(s)
  set_fact:
    default_interface_name: "{{ ansible_default_ipv4.interface }}"
  tags: network

- name: Create macvlan interface for DNS server
  nmcli:
    type: macvlan
    conn_name: coredns
    ifname: coredns
    macvlan:
      mode: 2
      parent: "{{ default_interface_name }}"
    method4: manual
    ip4:
      - "{{ dns.nameservers[ansible_hostname].ipv4 }}/{{ ansible_default_ipv4.prefix }}"
    autoconnect: true
    state: present
  tags: network
  register: macvlan

- name: Restart NetworkManager if needed
  systemd:
    name: NetworkManager
    state: restarted
  when: macvlan.changed is true
  tags: network

This results in three visible changes in the network setup. A new NetworkManager connection, a new ip link and address.

$ nmcli --fields connection.id,connection.type,macvlan.parent,macvlan.mode,ipv4.addresses c show coredns
connection.id:                          coredns
connection.type:                        macvlan
macvlan.parent:                         enabcm6e4ei0
macvlan.mode:                           2 (bridge)
ipv4.addresses:                         192.168.2.10/24

$ ip address show coredns
3: coredns@enabcm6e4ei0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 06:71:b3:d4:46:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.10/24 brd 192.168.2.255 scope global noprefixroute coredns
       valid_lft forever preferred_lft forever

Set CoreDNS Configuration

The system is now able to run a DNS server answering on one of the listner IP addresses specified in the vars/dns_servers.yaml data file.

The CoreDNS configuration consists of a single configuration file and a set of zone files. The entire configuration resides in a single directory tree /opt/coredns.

/opt/coredns
/opt/coredns/
├── Corefile
└── zones
    ├── example.org.zone
    └── lab.example.org.zone

2 directories, 3 files

The primary configuration file is the Corefile. It is placed at the root of the /opt/coredns/ tree. When the daemon starts it will use this as the current working directory. It reads the initial config from there.

The Corefile contains the root zone cache so that the server can forward queries for zones outside of this network. It then defines the zones as described in the dns_services.yaml file.

#
# A simple corefile for CoreDNS
#
.:53 {
  cache
  forward . {{ dns.forwarders|join(' ') }}
}

{% for zone in dns.zones %}
{{ zone.fqdn }}:53 {
  file zones/{{ zone.file }}
}
{% endfor %}

For this demonstration the zone files are static text files pulled from the files/zones sub-direcory of the Ansible file tree. They will be placed on the target machine in /opt/coredns/zones/. The Corefile contains the zone definitions and loads the files from there.

Add systemd Container Service

The final step is the significant one here. So far nothing has been particulary new.

As noted above, CoreDNS is meant to run as a container. Early in 2023 Podman integrated Quadlets, a utility to create systemd service unit files from a container spec and run software containers as first-class services. Podman is available on at least the Debian and Fedora derived distributions since the release of Podman 4.4. Podman is an OS integrated alternative to Docker. For the purposes of this document, the only important feature is the ability to run standard software containers as systemd services.

The whole point of this series was to get here: Creating a system service on Fedora CoreOS. It appears pretty anticlimactic. It’s rather like painting a room: All the real work is in the preparation. All that’s left to do now is to create one container spec file, reload the systemd daemon and enable/start the service.

- name: Set systemd container file
  copy:
    dest: /etc/containers/systemd/coredns.container
    src: coredns.container
    owner: root
    group: root
    mode: 644
  register: create_unit

- name: Reload Systemd Units
  systemd_service:
    daemon_reload: true
  notify: Restart CoreDNS Service
  #when: create_unit.changed is true

- name: Enable and Start CoreDNS container
  service:
    name: coredns.service
    state: started
    enabled: true

The container definition is a static file. The Podman components integrated into systemd services take this file and transform it into a systemd service unit file.

[Unit]
Description=CoreDNS Service Container
After=network-online.target

[Container]
Image=docker.io/coredns/coredns:latest

# Expect Corefile and zones/ within the working dir
PodmanArgs=--workdir=/root

PublishPort=53:53/udp
#PublishPort=953:953/udp
#PublishPort=53:53/tcp
#PublishPort=953:953/tcp

# Mount the coredns config dir into the container workingdir
Volume=/opt/coredns:/root

[Install]
# Enable in multi-user boot
WantedBy=multi-user.target default.target

# sudo podman run --detach --rm \
#       --name coredns \
#       --publish 53:53/udp \
#       --volume=/opt/coredns/:/root/ \
#       --workdir=/root \
#       coredns/coredns -conf /root/Corefile

This file is formatted like any other systemd unit file. Only the [Container] section is special to container service operation. That section specifies the location of the service container image and the run-time parameters. The sample above includes the corresponding command to make the mapping from CLI to configuration parameters.

This service starts after the network is active and is meant to be active for the multi-user target. It listens on port 53/udp. It could be configured for TCP and for SSL as well if the Corefile configuration calls for it. The container maps the system /opt/coredns directory to /root inside the container and instructs the container to set that as the working directory before starting the container. Without any arguments

Deployment

All the parts are in place now:

  • ✓ Disable systemd-resolved bound to port 53/udp

  • ✓ Configure the nameserver IP address

  • ✓ Place the CoreDNS configuration and zone files

  • ✓ Define a systemd service unit to manage the nameserver process

Confirm the changes to apply
ansible-playbook --check coredns-server-pb.yaml
Deploy the CoreDNS service
ansible-playbook coredns-server-pb.yaml

Operation

Over time the zones that are served will need to be updated. Make the needed changes to the Corefile zone files and then run the playbook with the zones tag.

Update the DNS configuration and content
ansible-playbook --tags zones coredns-server-pb.yaml

With this playbook, changes to the Corefile or zone files will trigger a restart of the coredns service. CoreDNS does include two plugins, reload and auto. The reload plugin tells the daemon to poll the Corefile periodically and to reload when it detects changes. the auto plugin does the same thing for zone files. These can be added later if needed, but the downtime associated with a service restart on a small network is neglegable.

To Do

In a larger network with servers geographically disbursed, they would also be set up as primary/secondary and would have zone transfers configured. In this example the network is localized, assumed to be a single site. Since both servers are present it is possible just to update them both at the same time and avoid the complexity of primary/secondary. Adding that would be a reasonable update.

The CoreDNS container path contains the latest tag and is embedded in the coredns.container systemd file. Ideally the CoreDNS version would be configurable by setting a variable in a file in /etc/sysconfig/coredns. It is not clear if this is possible yet using a Podman quadlet.

Summary

When this procedure is complete there will be two new DNS servers running CoreDNS. They will serve the configured zones and will forward any queries for other domains upstream for for resolution. The contents can be updated as needed by updating the zone files and new zones can be added by editing the dns_servers.yaml file and adding new zone files.

The DNS service can be managed on the hosts as a systemd service like any other. Restarts will automatically check and update the container image. If the host is running Fedora CoreOS it will update and reboot whenever an image update is made available. The OS and CoreDNS service software are decoupled so that there is no possibility of a dependency conflict between them. Both can be rolled back automically to the last known good version.

The CoreDNS version is allowed to update to the latest version on each restart. If the version must be rolled back, the last known good version can be found in the CoreDNS Releases on Github. Update the release tag in the coredns.container file and re-run the playbook to restore service using the required release.

The DHCP servers for the network will need to be configured with the new nameserver information, and any manually configured systems will also need to be updated.

References

Wednesday, January 29, 2025

Provisioning CoreOS - Intel and Raspberry Pi 4

Provisioning CoreOS

Installing CoreOS on a new system involves just booting a copy of the installation media with the configuration file (produced earlier) embedded. On most systems, the installation media is a USB stick and the target storage is an internal disk or SSD. The CoreOS image is copied bitwise to the destination and is then tuned according to the configuration file. On systems like the Raspberry Pi, that boot from an integrated SD card slot, this bootable media is the target device as well.

It is possible, on systems capable of network boot by DHCP/PXE, to boot in memory over a network and then to install to local disk. In that case the configuration file is retrieved by HTTP(S) from a local web site. This demonstration will only use bootable media.

Both of the procedures shown here are derived from the Fedora CoreOS Documentation, specifically the instructions for Bare Metal Intel and Raspberry Pi 4: EKD2 Combined Fedora CoreOS + EDK2 Firmware Disk. The bare metal procedure is very close to identical. For the Raspberry Pi 4 procedure the individual steps have been scripted to a single command, but that is not strictly necessary.

Bare Metal Installation - Intel

The Bare Metal installation is the most straightforward. It consists of just five steps:

  1. Download CoreOS ISO image

  2. Customize ISO image - Destination Disk and Ignition File

  3. Write ISO to USB Stick

  4. Boot from the USB stick

  5. Boot from the target disk

This example assumes that the Ignition file for the host is coreos-config.ign and the destination disk is /dev/sda. The invocation lets most of the CLI arguments default:

  • archtecture: x86_64

  • platform: metal

  • stream: stable

  • format: ISO

The network configuration is allowed to default to DHCP for any NICs that detect link.

# Download ISO if necessary. Write file name
IMAGE_FILE=$(coreos-installer download --format iso)
# Customize the ISO image for the target host
coreos-installer iso customize --dest-device /dev/sda --dest-ignition coreos-config.ign ${IMAGE_FILE}
# Write the ISO to bootable media
sudo dd if=${IMAGE_FILE} of=/dev/sda bs=1M status=progress
# Reset the local ISO file for next use
coreos-installer iso reset

Two things to note here about coreos-installer download

  1. It checks if the ISO file exists locally, and if so, checks that it is current before trying to download again.

  2. It writes the filename of the ISO file to stdout on exit. That value can be used in following script lines.

The final step of the list above restores the ISO image for next use. The image file is under 800MB, and so will fit on even very small USB sticks.

At this point booting from the USB will install CoreOS on the target host and disk. Insert the USB stick, boot and use the system boot options to boot from the USB. Observe the installation process from the console. When installation is complete, reboot and remove the USB stick.

Raspberry Pi 4 Installation

Currently only Raspberry Pi 4 are supported for Fedora CoreOS, and that only using a set of U-Boot or EFI files managed by third parties. Raspberry Pi 5 can run Fedora with a few minor tweaks, but CoreOS is still waiting for updated EFI and BMC files.

The Raspberry Pi boots from an integrated SD card reader. The CoreOS image is written to the SD card along with the Ignition file. The CoreOS image for aarch64 needs a bootloader, either U-Boot or EFI to boot correctly. This process installs a set of EFI binaries and auxiliary files into the boot partition to take the place of the typical firmware that other systems would have.

When the SD card is inserted and the system boots for the first time, the kernel and initrd are loaded into memory, including the Ignition file and the configuration is laid into the storage before mounting the disk filesystems and handing control to the init process.

The process of writing all the files to the SD card is described in Booting on Raspberry Pi 4 - EDK2: Combined Fedora CoreOS + EDK2 Firmware Disk. First the stock raw CoreOS aarch64 image is written to the SD card. Then the EFI partition is mounted and the EDK2 UEFI firmware is written. At that point the SD card is bootable on a Raspberry Pi 4.

The complete procedure for writing the SD Card is provide in the scripts sub-directory: prepare-pi.sh

Connect the SD card to the working system. Make sure that any auto-mounted partitions are unmounted before proceding. Determine the device path and provide the ignition file.

prepare-sd.sh
bash scripts/prepare-sd.sh <device path> <ignition file>

As with the Intel media, the final step is to install the SD Card in the Raspberry Pi 4 and power it on. Assuming that the Pi is connected to a network with DHCP and internet access, it will boot, complete the Ignition installation, install Ansible and reboot itself.

In both cases, at that point the new system is by SSH using the core user and it is ready to be managed by Ansible.

Finally Ready

With the OS installation complete it becomes possible to start addressing the goal of this series: Deploying containerized network services with Ansible. Keep an eye out for the next post where we’ll configure Ansible and demonstrate that we have connectivity and control of our target hosts.

References

  • coreos-installer
    Usage and arguments for the CoreOS installer binary. This can be run from a live ISO or on a second host to write to the boot media.

  • CoreOS on Bare Metal
    How to install CoreOS on Bare Metal. This includes variants for PXE, and Live ISO installations.

  • CoreOS on Raspberry Pi 4
    How to install CoreOS on Raspberry Pi 4 or 5. This includes instructions for installing EFI boot components that are not present in the Pi boot firmware.

  • Ignition
    Ignition is the engine that applies the provided configuration to a new CoreOS instance on first boot.

  • UEFI-Shell
    a UEFI Shell for built from EDK2 sources

  • Raspberry Pi 4 UEFI Firmware Images
    A build of the UEFI-Shell specifically for Rasberry Pi 4