Showing posts with label dns. Show all posts
Showing posts with label dns. Show all posts

Monday, February 17, 2025

Deploying CoreDNS as a systemd Service

DNS Service - CoreDNS

CoreDNS was developed for use in Kubernetes as a light-weight name server for containerized services. For a small to medium sized network, CoreDNS is much simpler to configure and operate than any of the production quality alternatives.

This post is meant to demonstrate the use of systemd services running as software containers on Fedora CoreOS. However this technique is applicable to any host that has Podman version 4.4.0+ installed and is configurable by Ansible.

If you are curious about provisioning Fedora CoreOS for this demonstration, see the previous series of posts for guidance:


Running CoreDNS

The CoreDNS configuration consists of a single configuration file and a set of DNS zone files. The files can be contained in a single directory, commonly /opt/coredns. The program looks for a file called Corefile in the current working directory when it is invoked.

CoreDNS is meant to be run in a software container. CoreOS provides podman, a drop-in CLI replacement for Docker and the runc runtime.

DNS servers listen to UDP port 53 for queries. TCP port 53 is used for some operations such as zone transfers and secure DNS. By default it listens on all configured interfaces.

The configuration used in this example is meant for use inside a firewalled network. It does not serve queries to devices outside the local network. It is meant to provide a split-dns

The CoreDNS container image is always found here:

docker.io/coredns/coredns:latest

The coredns-server Playbook

The goal of the coredns-server playbook is to install and configure CoreDNS on a set of servers. The servers need to listen for and respond to DNS queries on port 53/UDP on one of a set of listed IPv4 addresses. The service runs in a software container and is managed as a systemd service.

The deployment steps can be grouped into four related sets of tasks:

  1. Switch to static resolver

  2. Configure network interface

  3. Deploy CoreDNS configuration

  4. Configure systemd service

For clarities sake these four are broken down into separate task files in the coredns-server role. These are detailed in corresponding sections below.

An Ansible playbook is defined in a .yaml formatted file. It is possible to contain the entire playbook in a single file, but it is usually helpful to have the playbook use a role. Roles are re-usable modules that

---
#
# The playbook creates a DNS server on the target hosts using CoreDNS
# It populates the zone files from files/zones
#
- name: CoreDNS Server
  hosts: dnsservers
  become: true

  vars_files:
    - dns_services.yaml

  roles:
    - coredns-server

The dns_services.yaml file specifies the parameters for the CoreDNS server. Among these are the locations and zones for the zone files. These reside in files/zones in the ansible directory. The zone files here are static and follow the RFC standards and will be familar to anyone who’s configured ISC Bind. They could be produced mechanically from other databases but that is outside the scope of this project.

Note
The dns_services.yaml file contains global variables that are not part of the playbook. They are stored at the top of the ansible tree along with the zone files in files/zones
---
#
# DNS services for example.com network
#
dns:
  nameservers:
    pi4-1:
      fqdn: ns1.example.org
      ipv4: 192.168.2.10
    pi4-2:
      fqdn: ns1.example.org
      ipv4: 192.168.2.11

  forwarders:
    - 192.168.2.1
    - 4.2.2.1     # Level3 caching DNS server IP address
    - 1.1.1.1	  # Cloudflare caching DNS server IP Address

  zones:
    - fqdn: example.org
      file: example.org.zone
    - fqdn: lab.example.org
      file: lab.example.org.zone

  search:
    - lan    # mDNS from Google Mesh DNS
    - example.org
    - lab.example.org

The coredns-server Role

This role encapsulates the process of installing a CoreDNS server on a host. The broad steps are described above.

coredns-server role tree
roles/coredns-server/
├── files
│   └── coredns.container
├── handlers
│   └── main.yaml
├── tasks
│   ├── config_files.yaml
│   ├── main.yaml
│   ├── network.yaml
│   ├── resolver.yaml
│   └── systemd_service.yaml
└── templates
    ├── Corefile.j2
    └── resolv.conf.j2

5 directories, 9 files

The task files are the primary driver of a playbook and role. The rest of the files provide resources that serve the tasks as they are run.

The task files are the primary driver of a playbook and role. The rest of the files provide resources that serve the tasks as they are run. The file main.yaml acts as the entry point for the tasks defined in the tasks/ subdirectory. The tasks are defined as if they were part of a playbook, as a YAML list. The main.yaml file refers to a set of smaller task files, grouping the tasks functionally.

---
#
# Coordinate creating a coredns service container
#
- name: Disable systemd-resolved and set static resolver file
  import_tasks: resolver.yaml

- name: Configure and set DNS Listener IP address
  import_tasks: network.yaml

- name: Place the Configration Files
  import_tasks: config_files.yaml

- name: Prepare Systemd Services
  import_tasks: systemd_service.yaml

Note that the first three sets of tasks are not special for CoreOS. They’re applicable to any DNS service. The final task list is the important one for this series.

Disable Dynamic DNS Resolver Service

Since 2020, with the release of Fedora 33, the the local DNS resolver is a daemon integrated with systemd. This daemon listens for local queries and is bound to port 53/UDP. The CoreDNS server needs to bind to the same port, so the systemd-resolved service must be stopped and disabled before coredns can start.

This set of tasks disables the systemd-resolved service and replaces the stock /etc/resolv.conf file with one configured for the target environment.

- name: Disable systemd-resolved - (avoid conflict with coredns)
  service:
    name: systemd-resolved
    state: stopped
    enabled: false

- name: Set static resolver file
  template:
    dest: /etc/resolv.conf
    src: resolv.conf.j2
    owner: root
    group: root
    mode: 0644
    backup: true
#
# Maintained by Ansible
#
nameserver 127.0.0.1
{% for nameserver in dns.forwarders %}
nameserver {{ nameserver }}
{% endfor %}
search {{ dns.search|join(' ') }}

The resolv.conf file directs DNS queries first to the local nameserver and then to the listed forwarders when the local server does not serve the requested domain.

Set DNS Listener IP Address

The DNS service requires two servers for each domain. The servers are identified by IP address because, well they provide the name services. This step ensures that each server host is listening on one of those two addresses.

This task set finds the default interface on this host and then creates a new connection that attaches to the physical one and answers the servers listener address. The connection type is macvlan and it allows this interface to be configured manually while allowing the main interface to use DHCP for the rest of the network information.

The critical step here is the second one. It creates a virtual interface dedicated to the DNS listener address.

- name: Record interface name(s)
  set_fact:
    default_interface_name: "{{ ansible_default_ipv4.interface }}"
  tags: network

- name: Create macvlan interface for DNS server
  nmcli:
    type: macvlan
    conn_name: coredns
    ifname: coredns
    macvlan:
      mode: 2
      parent: "{{ default_interface_name }}"
    method4: manual
    ip4:
      - "{{ dns.nameservers[ansible_hostname].ipv4 }}/{{ ansible_default_ipv4.prefix }}"
    autoconnect: true
    state: present
  tags: network
  register: macvlan

- name: Restart NetworkManager if needed
  systemd:
    name: NetworkManager
    state: restarted
  when: macvlan.changed is true
  tags: network

This results in three visible changes in the network setup. A new NetworkManager connection, a new ip link and address.

$ nmcli --fields connection.id,connection.type,macvlan.parent,macvlan.mode,ipv4.addresses c show coredns
connection.id:                          coredns
connection.type:                        macvlan
macvlan.parent:                         enabcm6e4ei0
macvlan.mode:                           2 (bridge)
ipv4.addresses:                         192.168.2.10/24

$ ip address show coredns
3: coredns@enabcm6e4ei0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 06:71:b3:d4:46:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.10/24 brd 192.168.2.255 scope global noprefixroute coredns
       valid_lft forever preferred_lft forever

Set CoreDNS Configuration

The system is now able to run a DNS server answering on one of the listner IP addresses specified in the vars/dns_servers.yaml data file.

The CoreDNS configuration consists of a single configuration file and a set of zone files. The entire configuration resides in a single directory tree /opt/coredns.

/opt/coredns
/opt/coredns/
├── Corefile
└── zones
    ├── example.org.zone
    └── lab.example.org.zone

2 directories, 3 files

The primary configuration file is the Corefile. It is placed at the root of the /opt/coredns/ tree. When the daemon starts it will use this as the current working directory. It reads the initial config from there.

The Corefile contains the root zone cache so that the server can forward queries for zones outside of this network. It then defines the zones as described in the dns_services.yaml file.

#
# A simple corefile for CoreDNS
#
.:53 {
  cache
  forward . {{ dns.forwarders|join(' ') }}
}

{% for zone in dns.zones %}
{{ zone.fqdn }}:53 {
  file zones/{{ zone.file }}
}
{% endfor %}

For this demonstration the zone files are static text files pulled from the files/zones sub-direcory of the Ansible file tree. They will be placed on the target machine in /opt/coredns/zones/. The Corefile contains the zone definitions and loads the files from there.

Add systemd Container Service

The final step is the significant one here. So far nothing has been particulary new.

As noted above, CoreDNS is meant to run as a container. Early in 2023 Podman integrated Quadlets, a utility to create systemd service unit files from a container spec and run software containers as first-class services. Podman is available on at least the Debian and Fedora derived distributions since the release of Podman 4.4. Podman is an OS integrated alternative to Docker. For the purposes of this document, the only important feature is the ability to run standard software containers as systemd services.

The whole point of this series was to get here: Creating a system service on Fedora CoreOS. It appears pretty anticlimactic. It’s rather like painting a room: All the real work is in the preparation. All that’s left to do now is to create one container spec file, reload the systemd daemon and enable/start the service.

- name: Set systemd container file
  copy:
    dest: /etc/containers/systemd/coredns.container
    src: coredns.container
    owner: root
    group: root
    mode: 644
  register: create_unit

- name: Reload Systemd Units
  systemd_service:
    daemon_reload: true
  notify: Restart CoreDNS Service
  #when: create_unit.changed is true

- name: Enable and Start CoreDNS container
  service:
    name: coredns.service
    state: started
    enabled: true

The container definition is a static file. The Podman components integrated into systemd services take this file and transform it into a systemd service unit file.

[Unit]
Description=CoreDNS Service Container
After=network-online.target

[Container]
Image=docker.io/coredns/coredns:latest

# Expect Corefile and zones/ within the working dir
PodmanArgs=--workdir=/root

PublishPort=53:53/udp
#PublishPort=953:953/udp
#PublishPort=53:53/tcp
#PublishPort=953:953/tcp

# Mount the coredns config dir into the container workingdir
Volume=/opt/coredns:/root

[Install]
# Enable in multi-user boot
WantedBy=multi-user.target default.target

# sudo podman run --detach --rm \
#       --name coredns \
#       --publish 53:53/udp \
#       --volume=/opt/coredns/:/root/ \
#       --workdir=/root \
#       coredns/coredns -conf /root/Corefile

This file is formatted like any other systemd unit file. Only the [Container] section is special to container service operation. That section specifies the location of the service container image and the run-time parameters. The sample above includes the corresponding command to make the mapping from CLI to configuration parameters.

This service starts after the network is active and is meant to be active for the multi-user target. It listens on port 53/udp. It could be configured for TCP and for SSL as well if the Corefile configuration calls for it. The container maps the system /opt/coredns directory to /root inside the container and instructs the container to set that as the working directory before starting the container. Without any arguments

Deployment

All the parts are in place now:

  • ✓ Disable systemd-resolved bound to port 53/udp

  • ✓ Configure the nameserver IP address

  • ✓ Place the CoreDNS configuration and zone files

  • ✓ Define a systemd service unit to manage the nameserver process

Confirm the changes to apply
ansible-playbook --check coredns-server-pb.yaml
Deploy the CoreDNS service
ansible-playbook coredns-server-pb.yaml

Operation

Over time the zones that are served will need to be updated. Make the needed changes to the Corefile zone files and then run the playbook with the zones tag.

Update the DNS configuration and content
ansible-playbook --tags zones coredns-server-pb.yaml

With this playbook, changes to the Corefile or zone files will trigger a restart of the coredns service. CoreDNS does include two plugins, reload and auto. The reload plugin tells the daemon to poll the Corefile periodically and to reload when it detects changes. the auto plugin does the same thing for zone files. These can be added later if needed, but the downtime associated with a service restart on a small network is neglegable.

To Do

In a larger network with servers geographically disbursed, they would also be set up as primary/secondary and would have zone transfers configured. In this example the network is localized, assumed to be a single site. Since both servers are present it is possible just to update them both at the same time and avoid the complexity of primary/secondary. Adding that would be a reasonable update.

The CoreDNS container path contains the latest tag and is embedded in the coredns.container systemd file. Ideally the CoreDNS version would be configurable by setting a variable in a file in /etc/sysconfig/coredns. It is not clear if this is possible yet using a Podman quadlet.

Summary

When this procedure is complete there will be two new DNS servers running CoreDNS. They will serve the configured zones and will forward any queries for other domains upstream for for resolution. The contents can be updated as needed by updating the zone files and new zones can be added by editing the dns_servers.yaml file and adding new zone files.

The DNS service can be managed on the hosts as a systemd service like any other. Restarts will automatically check and update the container image. If the host is running Fedora CoreOS it will update and reboot whenever an image update is made available. The OS and CoreDNS service software are decoupled so that there is no possibility of a dependency conflict between them. Both can be rolled back automically to the last known good version.

The CoreDNS version is allowed to update to the latest version on each restart. If the version must be rolled back, the last known good version can be found in the CoreDNS Releases on Github. Update the release tag in the coredns.container file and re-run the playbook to restore service using the required release.

The DHCP servers for the network will need to be configured with the new nameserver information, and any manually configured systems will also need to be updated.

References

Friday, January 10, 2025

The Case for CoreOS - Network Infrastructure on an Immutable OS

The Lifetime of Silent Services

For small and medium sized organizations, a local network requires the creation of and management of local network services such as DNS, NTP, DHCP, monitoring and user access controls.  These are the ante needed to get in the game but when they work properly they become invisible. This is good, but it means they can be neglected from the standpoint of management and maintenance. As long as they work it's easy to ignore them until they do break. There is a tendency to treat maintenance is a risk rather than a benefit, the fear of service interruption and downtime leading to neglect and a sense that these services are somehow fragile and precious.

For these silent services, the neglect usually manifests when the admins discover that the OS has gone end-of-life or a bug is discovered in the current version of a service or there are 200 CVEs to apply because the last reboot was 700 days ago. The problem is that accumulated updates required and unfamiliarity with the services and the maintenance history makes admins gun-shy of updates. Time only makes the fear and the debt worse.

What are you afraid of, Really?

The modern alternate is the cliche "Fail Fast", which, when thrown about without comprehension, is correctly scorned.  I prefer to say "Find the scariest thing you have to do, and do it repeatedly until it stops being scary. Then find the next scariest thing.".

The real fear and risk is of downtime without a recovery plan.  In a corporate environment the tendency of management is to CYA by avoiding any downtime by avoiding any change. While this can provide the illusion of stability, it treats the infrastructure as a static monolith. It ignores the facts that failures and updates are inevitable and sets the operations teams up for failure. It restricts their ability to practice the very update and mitigation processes that would allow them to create a robust reliable service.

The real solution is to create a system where any change can be rolled back quickly, reliably and completely.  Fedora CoreOS provides that.

Git for Filesystems?

Fedora CoreOS is a distribution of Fedora Linux that is created specifically to run software containers.  Red Hat promotes it for cloud use and only supports it as a base for OpenShift.  It is a minimal distribution with no GUI only a simple installer that writes the initial state to a bootable storage device and a simple configuration file that is applied on first boot. This by itself is unremarkable. The feature that makes CoreOS significant is that the file and package systems are based on rpm-ostree. This is an integrated file and package management system. It presents to users as an XFS filesystem, but it is mounted read-only. The filesystem is immutable. To install packages you must use the rpm-ostree command to layer the package into a new image version and then reboot to the new image. Installing application packages is discouraged in favor of running services in containers.

Did you get that? The filesystem is read only. To see updated packages you have to reboot. Wait, there's more.

The Turtle or the Frog?

Most distributions provide updates through online package repositories. Admins must periodically poll the repository, pull down any new packages, and then overlay them into the running system. At that point it becomes extremely difficult to reliably roll back. If anything fails, the only recourse is to recover the system from backups, which is understandably an extreme and time-consuming process.  This leads to a "slow and steady" approach to updates. Updates are applied to a few test systems. If no problems are discovered they are rolled forward to a set of staging systems.  Finally the updates are deployed to production.

This is an expensive, time consuming system, suited only to large organizations with the resources to implement them. It's also error prone, as it is often difficult to adequately simulate the production operating conditions in a small test environment.  More commonly in smaller organizations, updates are shunted to backlog work and neglected in favor of feature requests or helpdesk issues until some outside event brings the problem to the attention of management, when it becomes an emergency.
To compound the problems, it is common to run package updates without rebooting the system. This can result in failures that don't appear until long after the actual change is applied. All together this makes IT management very averse to regular updates and reboots because they see these as introducing problems and risking downtime with long recovery periods.

Until recently (well ages in Internet Time) this "frog in the pot" approach was really the only option. The fact that it was impossible to reliably roll back changes rightly made management and operations averse to any change to a system that was "working". 

Double-Buffered Operating System

CoreOS updates are atomic. That is, updates are published as a unit.  The stable stream is updated approximately every two weeks. There are also test and "next" streams that update more often but aren't meant for regular use.  CoreOS runs a service called Zincati. This service polls the release streams for new images and will apply them and reboot when needed. Zincati can be tuned to create staged roll-outs, applying updates first to a set of canary systems before moving on to more critical systems. It can also be tuned to restrict reboots to specific days of the week and times of day.

By conventional standards, read-only systems that update automatically and require reboot every two weeks provides the opposite of stability and reliability. But the risks posed when this is implemented on a conventional Linux distribution are mitigated when presented using rpm-ostree, zincati and software containers.  The benefits of atomic rollback and application decoupling mean that it is possible to keep systems up to date and to respond instantly to any update-induced problems. In essence the operating system is double-buffered and the current system is preserved perfectly across updates. You don't have to worry about losing the working configuration because it's still there.

For The Best Services, Don't Install Any

On CoreOS you're discouraged from installing application or service software on the system.  CoreOS is designed to run software containers. The only major service component integrated into the OS is podman, while all of the network services run on Linux as systemd services.

In 2021, a project called Quadlets was created to allow containers to be managed as first-class services under systemd. In 2022 quadlets were merged into the systemd project and as of 2024 they are available on any systemd based Linux. This means that your system services no longer are tightly coupled to the OS updates.  They don't even need to be based on the same OS distribution.

Using Quadlets, deploying a network service is a matter of defining a systemd container spec, providing the service configuration files and enabling and starting the service. No service software needs to be installed or updated ever.  Updating the service software is a matter of updating the container image path and tag and restarting the systemd service.  Reverting is just as simple. It becomes possible to basically ignore the OS when updating system services and vice-versa.  The loose coupling means that changes to one are very unlikely to affect the other and that any change can be trivially and reliably reverted without affecting the other components.

Do it again! Do it again!

The simplicity and minimalism of using CoreOS with software containers enables one last element for providing stable reliable network services. CoreOS can be installed with a simple DHCP/PXE boot and, once installed, it can be configured with a small set of Ansible scripts. These aren't remarkable by themselves but the simplicity of and compartmentalization that the immutable OS are somewhat novel in the on-premise hardware environment.  These are usually thought of as features of cloud-based services, but are perfectly applicable for small and medium organizations with limited resources.

As a matter of practice I tend not to say I can do something until I can do it 100 times with the push of a single button. With some simple automation the infrastructure can be restored in a matter of minutes on the old hardware or new.  These services tend to be small and light-weight, so they can run on inexpensive redundant hardware.

So You Say, But How?

Well, I plan to show you.  This first post is a long pontification on some thoughts I've had over the last couple of years. I've put it into practice for my home network and at one employer.  It falls under a larger theme of adapting cloud networking practices for on-premise network services.  After all, Red Hat now only supports their CoreOS stream as the base for OpenShift, Red Hat's extended Kubernetes offering. Red Hat recommends the very practices I'm going to detail to maintain the underpinnings of their enterprise distributed application service. I suspect that part of the reason they don't support it for general use is that serious adoption would undercut their revenue stream from RHEL, and I can tell you from personal experience that matters to them a lot.

This isn't a perfect strategy for all purposes either.  Unless your application is extremely simple and has already been designed and implemented for containers it doesn't make sense to shoehorn it in.  Large distributed applications are better supported on a proper Kubernetes or OpenShift deployment, whether on-premise or on a cloud service. Heavy-weight monolithic services (I'm looking at you JBoss/Tomcat apps) aren't well suited to containers, despite the trend to push them in.

In following posts I mean to walk through the deployment of Fedora CoreOS, preparation for automated configuration management and the deployment of service containers. I'm not actually sure where this will end but I mean to see just how far I can push it.  Come along if it seems like your kind of fun.

Resources

  • Fedora Linux - An extremely popular and well managed Linux distribution
  • Fedora CoreOS - A spin of Fedora that is designed to run software containers
  • libostree - A checkpointed filesystem that allows atomic rollback of file changes
  • rpm-ostree - An extension of libostree that integrates RPM package management
  • butane - YAML schema to define OS configurations for CoreOS
  • ignition - JSON schema to define OS configurations for CoreOS
  • zincati - A service to control and tune updates from CoreOS image streams
  • Quadlets - Software containers as systemd services
  • Ansible - System configuration language and toolset
  • OpenShift - Red Hat's enterprise extended version of Kubernetes
  • Kubernetes - A computing cluster system for running applications in software containers