How to configure flatcar the Right Way.

Flatcar overview

According to the docs, flatcar linux is described “a container optimized OS that ships a minimal OS image, which includes only the tools needed to run containers. The OS is shipped through an immutable filesystem and includes automatic atomic updates.”

What this means in practice is a base operating system that you can more or less ignore. It’s immutable¹, so even if you wanted to install cool shit, you can’t. If you mess it up ? You just run flatcar-reset and reboot and it’s factory reset. It has systemd, a few binaries and not much else.

There are a few directories youcan write to however, and most tutorials I’ve seen involve hackery to write files to /etc, or /opt to make them persistent.

That’s too easy though, there exists a far more complicated way that involves systemd tools you’ve probably never heard of. It’s also the officially recommended way to go about this.

Butane

This is the language used to bake in the configuration of Flatcar upon install. A basic config looks something like this:

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      password_hash: "$6$dL2EUH0onVz7oYT6$0PTBgLeLo4f18JgyXWCym5SxtSGBmpNNKm86G89eZ8z1LLv.jocM1ybaAKTOCr6x8YcOmPonoVGqmmoYL67bl."
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIE9Q6a+1MhCyOdsW7ediXtCmdqHbOjyp52uk43DbQO5 dene@mba
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: node1

That configures the pre-existing user “core” with a password of “core”², and a ssh public key allowed to login as that user.

It also configures the hostname of the machine to be node1. Easy right ?

Full configuration

Let’s see a full configuration that I use for my nodes:

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      password_hash: "$6$dL2EUH0onVz7oYT6$0PTBgLeLo4f18JgyXWCym5SxtSGBmpNNKm86G89eZ8z1LLv.jocM1ybaAKTOCr6x8YcOmPonoVGqmmoYL67bl."
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIE9Q6a+1MhCyOdsW7ediXtCmdqHbOjyp52uk43DbQO5 dene@mba

storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: cn3
    - path: /etc/systemd/network/10-eno1.network
      mode: 0644
      contents:
        inline: |
          [Match]
          Name=eno1

          [Network]
          DHCP=ipv4
          VLAN=eno1.205
          LinkLocalAddressing=ipv6
          IPv6AcceptRA=true
          IPv6SendRA=false

          [DHCP]
          UseRoutes=false
    - path: /etc/systemd/network/20-eno1.205.netdev
      mode: 0644
      contents:
        inline: |
          [NetDev]
          Name=eno1.205
          Kind=vlan

          [VLAN]
          Id=205
    - path: /etc/systemd/network/30-eno1.205.network
      mode: 0644
      contents:
        inline: |
          [Match]
          Name=eno1.205

          [Network]
          DHCP=no
          LinkLocalAddressing=ipv6
          IPv6AcceptRA=true
          IPv6SendRA=false

          [Address]
          Address=205.233.128.107/24

          [Route]
          Gateway=205.233.128.1
          DNS=205.233.128.236
          DNS=1.1.1.1
    - path: /etc/extensions/tailscale-1.58.1-x86-64.raw
      contents:
        source: https://files.slush.ca/ts/tailscale-1.58.1-x86-64.raw
    - path: /etc/sysupdate.tailscale.d/tailscale.conf
      contents:
        inline: |
          [Transfer]
          Verify=false

          [Source]
          Type=url-file
          Path=https://files.slush.ca/ts/
          MatchPattern=tailscale-@v-%a.raw

          [Target]
          Type=regular-file
          Path=/etc/extensions
          InstancesMax=3

systemd:
  units:
    - name: tailscale-watcher.service
      enabled: true
      contents: |
        [Unit]
        Description=Tailscale restarter
        After=network.target
        StartLimitIntervalSec=10
        StartLimitBurst=5

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/systemctl restart tailscaled.service

        [Install]
        WantedBy=multi-user.target
    - name: tailscale-watcher.path
      enabled: true
      contents: |
        [Path]
        PathModified=/usr/sbin/tailscaled

        [Install]
        WantedBy=multi-user.target

    - name: systemd-sysupdate.timer
      enabled: true
    - name: systemd-sysupdate.service
      dropins:
        - name: tailscale.conf
          contents: |
            [Service]
            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C tailscale update
        - name: sysext.conf
          contents: |
            [Service]
            ExecStartPost=systemctl restart systemd-sysext
    - name: multi-user.target
      dropins:
        - name: 10-tailscale-path-watcher.conf
          contents: |
            [Unit]
            Upholds=tailscale-watcher.path
        - name: 20-tailscale-service-watcher.conf
          contents: |
            [Unit]
            Wants=tailscale-watcher.service

Starts off the same way, configures a password and a hostname and then waaaay-o. What?

Remember, there are no tools here so if you want to configure a network interface you need to drop the correct systemd configuration in by hand so let’s walk through that item by item.

Network Configuration

    - path: /etc/systemd/network/10-eno1.network
      mode: 0644
      contents:
        inline: |
          [Match]
          Name=eno1

          [Network]
          DHCP=ipv4
          VLAN=eno1.205
          LinkLocalAddressing=ipv6
          IPv6AcceptRA=true
          IPv6SendRA=false

          [DHCP]
          UseRoutes=false

This creates a file at the path /etc/systemd/network/10-eno1.network, which is the configuration for the physical device eno1. I use VLANs at home, in this case I attach a logical interface to eno1. I also use IPv4 and IPv6 SLAAC to get addresses. I then ignore the default route I get from DHCP. Don’t worry, we’ll set one up later on the VLAN interface.

    - path: /etc/systemd/network/20-eno1.205.netdev
      mode: 0644
      contents:
        inline: |
          [NetDev]
          Name=eno1.205
          Kind=vlan

          [VLAN]
          Id=205

This sets up the eno1.205 VLAN device. It’s attached to eno1 and unsurprisingly uses VLAN ID 205.

    - path: /etc/systemd/network/30-eno1.205.network
      mode: 0644
      contents:
        inline: |
          [Match]
          Name=eno1.205

          [Network]
          DHCP=no
          LinkLocalAddressing=ipv6
          IPv6AcceptRA=true
          IPv6SendRA=false

          [Address]
          Address=118.203.23.34/24

          [Route]
          Gateway=118.203.23.1
          DNS=1.1.1.1

This is where we set our VLAN interface’s network configuration. It has a static IP of 118.203.23.34, a default gateway of 118.203.23.1 and uses Cloudflare’s 1.1.1.1 DNS server.

All pretty straight forward so far, and despite the funky butane format, it’s what you would configure with $EDITOR to set the same things up.

Ignition

It would be easy enough to stop here, and just have a functional Flatcar Linux machine connected to a network. Flatcar doesn’t actually use butane though, we need to go through one more step. Butane is easy for you and I to read/write, but to use it to install we need to convert it to be machine readable, into a format named ignition.

Butane can be installed through Homebrew on OSX or run through podman/docker:

Assuming you’re using OSX, let’s convert this over.

brew install butane
butane myconfig

Notice the ugly wodge of json it puked out. That’s what Flatcar reads on install to bake the configuration it. In my case, I have one main butane configuration file, and I change the hostname and IP addresses as needed and then generate an Ignition file per node.

butane myconfig >node1.json

Keep all that in mind and let’s go further down the rabbit hole of systemd.

systemd

In the beginning, we mentioned Flatcar has systemd and not much else and you can see in the networking configuration we use systemd units to drive out network setup.

We also said there’s no way to install things.. and well, that’s a bit of a lie. Remember, the butane configuration is installed with the Flatcar. It’s there everytime you boot and when you run flatcar-reset it gets put back to how it was. Any changes you’ve made by hand will likely disappear. So how do we add necessary utilities, permanently, the Right Way ?

Enter systemd-sysext.

systemd-sysext

systemd-sysext is a way to add overlay images to an existing operating system, typically /usr and /opt. It takes a squashfs, or ext4/btrfs image and overlays in a read-only fashion at run time. We still have our immutable status, but now have a way to overlay the tools we need.

We’ll focus on squashfs, as it takes a simple directory and turns it into a systemd-sysext compatible image. Flatcar itself also needs a bit of special metadata to make it work. In this example, we’ll add Tailscale, one of the better VPN/PAM/remote access tools around and for me a necessary part of every machine I run.

Since we have no package manager we need to fetch the generic binaries appropriate to our desired architecture.. in most cases amd64 from tailscale and extract them.

curl -O https://pkgs.tailscale.com/stable/tailscale_1.58.1_amd64.tgz
tar zxvfp tailscale_1.58.1_amd64.tgz

And we need to make a folder structure to hold our squashfs image:

mkdir -p tailscale-1.58.1-amd64/usr/bin
mkdir -p tailscale-1.58.1-amd64/usr/sbin
mkdir -p tailscale-1.58.1-amd64/usr/lib/systemd/system
mkdir -p tailscale-1.58.1-amd64/usr/lib/systemd/system/multi-user.target.d
mkdir -p tailscale-1.58.1-amd64/usr/lib/extension-release.d

Copy the binaries from our download into the squashfs target:

mv tailscale_1.58.1_amd64/tailscale tailscale-1.58.1-amd64/usr/bin
mv tailscale_1.58.1_amd64/tailscaled tailscale-1.58.1-amd64/usr/sbin

Now we add in the metadata that Flatcar needs to use as a system image:

echo "ID=flatcar\nSYSEXT_LEVEL=1.0" >tailscale-1.58.1-amd64/usr/lib/extension-release.d/extension-release.tailscale-1.58.1-x86-64

Notice the x86-64 at the end, rather than amd64 which is what we’ve been using previously. This is because systemd refers internally to amd64 as x84-64 and gets confused if we don’t as well.

Now we just need to add our own systemd unit to run tailscale once the overlay has been applied:

echo "[Unit]
Description=Tailscale node agent
Documentation=https://tailscale.com/kb/
Wants=network-pre.target
After=network-pre.target systemd-resolved.service

[Service]
ExecStartPre=/usr/sbin/tailscaled --cleanup
ExecStart=/usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port=41641
ExecStopPost=/usr/sbin/tailscaled --cleanup

Restart=on-failure

RuntimeDirectory=tailscale
RuntimeDirectoryMode=0750
StateDirectory=tailscale
StateDirectoryMode=0700
CacheDirectory=tailscale
CacheDirectoryMode=0750
Type=notify

[Install]
WantedBy=multi-user.target
" >tailscale-${TS_VER}-${ARCH}/usr/lib/systemd/system/tailscaled.service

Add in a drop-in to enable it automatically (rather than run systemctl enable tailscaled.service).

echo "[Unit]
Upholds=tailscaled.service
" >tailscale-${TS_VER}-${ARCH}/usr/lib/systemd/system/multi-user.target.d/10-tailscale.conf

Now we’re ready to build our squashfs based systemd-sysext image. Easy eh ?

mksquashfs tailscale-1.58.1-amd64 tailscale-1.58.1-x86-64.raw

We use .raw here as that’s the convention from systemd.

You can inspect and validate the overlay on any machine that runs systemd:

core@cn1 ~ $ sudo systemd-dissect tailscale-1.58.1-x86-64.raw
      Name: tailscale-1.58.1-x86-64.raw
      Size: 24.7M

 Ext. Rel.: ID=flatcar
            SYSEXT_LEVEL=1.0

    Use As: ✗ bootable system for UEFI
            ✗ bootable system for container
            ✗ portable service
            ✓ extension for system
            ✗ extension for initrd
            ✓ extension for portable service

RW DESIGNATOR PARTITION UUID PARTITION LABEL FSTYPE   ARCHITECTURE VERITY GROWFS NODE                        PARTNO
ro root       -              -               squashfs -            -          no tailscale-1.58.2-x86-64.raw -

You’ll need to use sudo, and make sure you look for the check marks beside extension for system, and extension for portable service. Without those it won’t work properly.

Once you’ve installed a Flatcar machine with our massive butane config at the beginning of the article, you can see your systemd-sysext overlay:

core@cn1 ~ $ systemd-sysext
HIERARCHY EXTENSIONS              SINCE
/opt      none                    -
/usr      containerd-flatcar      Thu 2024-02-15 05:37:14 UTC
          docker-flatcar
          tailscale-1.58.1-x86-64

Mission accomplished! Now back to that immutable¹ thing again.. what if we want to update it ? systemd to the rescue again with a tool called systemd-sysupdate.

systemd-sysupdate

systemd-sysupdate is a tool to atomically update host OS, containers, or all manner of things on a linux machine. In this case we’ll use it to look out for updated tailscale systemd-sysext images. This will also cover the last portion of our butane config.

systemd:
  units:
    - name: tailscale-watcher.path
      enabled: true
      contents: |
        [Path]
        PathModified=/usr/sbin/tailscaled

        [Install]
        WantedBy=multi-user.target

This adds a systemd unit to watch the /usr/sbin/tailscaled binary, and if changed call a service with the same name.

    - name: tailscale-watcher.service
      enabled: true
      contents: |
        [Unit]
        Description=Tailscale restarter
        After=network.target
        StartLimitIntervalSec=10
        StartLimitBurst=5

        [Service]
        Type=oneshot
        ExecStart=/usr/bin/systemctl restart tailscaled.service

        [Install]
        WantedBy=multi-user.target

This is the service that actually calls reload, it must be named the same as the .path unit.

    - name: systemd-sysupdate.timer
      enabled: true

This just enabled a timer unit for systemd-sysupdate.

    - name: systemd-sysupdate.service
      dropins:
        - name: tailscale.conf
          contents: |
            [Service]
            ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C tailscale update
        - name: sysext.conf
          contents: |
            [Service]
            ExecStartPost=systemctl restart systemd-sysext

This unit just runs systemd-sysupdate against our component, tailscale.

    - name: multi-user.target
      dropins:
        - name: 10-tailscale-path-watcher.conf
          contents: |
            [Unit]
            Upholds=tailscale-watcher.path
        - name: 20-tailscale-service-watcher.conf
          contents: |
            [Unit]
            Wants=tailscale-watcher.service

These are just drop ins, they do the work of enabling our units rather than needing to run systemctl enable.

The big secret sauce here is this portion:

    - path: /etc/sysupdate.tailscale.d/tailscale.conf
      contents:
        inline: |
          [Transfer]
          Verify=false

          [Source]
          Type=url-file
          Path=https://files.slush.ca/ts/
          MatchPattern=tailscale-@v-%a.raw

          [Target]
          Type=regular-file
          Path=/etc/extensions
          InstancesMax=3

This tells systemd-sysupdate to check https://files.slush.ca/ts/ for files maching the pattern tailscale-@v-%a.raw. If it detects a new version (@v) for our architecture (%a) it will download the file for use by our systemd-sysupdate.service unit.

Assuming there’s a new version available, We can manually see this all working like so:

core@localhost ~ $ sudo /usr/lib/systemd/systemd-sysupdate -C tailscale
Discovering installed instances…
Discovering available instances…
⤵️ Acquiring manifest file https://files.slush.ca/ts/SHA256SUMS…
Pulling 'https://files.slush.ca/ts/SHA256SUMS'.
Downloading 374B for https://files.slush.ca/ts/SHA256SUMS.
Acquired 374B.
Download of https://files.slush.ca/ts/SHA256SUMS complete.
Operation completed successfully.
Exiting.
Determining installed update sets…
Determining available update sets…
  VERSION INSTALLED AVAILABLE ASSESSMENT
↻ 1.58.2                ✓     candidate
● 1.58.1      ✓         ✓     current

In this case there is an update, so let’s install it manually for fun:

core@localhost ~ $ sudo /usr/lib/systemd/systemd-sysupdate -C tailscale update
Discovering installed instances…
Discovering available instances…
⤵️ Acquiring manifest file https://files.slush.ca/ts/SHA256SUMS…
Pulling 'https://files.slush.ca/ts/SHA256SUMS'.
Downloading 374B for https://files.slush.ca/ts/SHA256SUMS.
Acquired 374B.
Download of https://files.slush.ca/ts/SHA256SUMS complete.
Operation completed successfully.
Exiting.
Determining installed update sets…
Determining available update sets…
Selected update '1.58.2' for install.
Making room for 1 updates…
Removed no instances.
⤵️ Acquiring https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw → /etc/extensions/tailscale-1.58.2-x86-64.raw...
Pulling 'https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw', saving as '/etc/extensions/.#sysupdatetailscale-1.58.2-x86-64.rawd2c6fa8e35acfc84'.
Downloading 24.7M for https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw.
Got 6% of https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw. 17s left at 1.2M/s.
Got 38% of https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw. 3s left at 4.2M/s.
Got 82% of https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw. 683ms left at 6.2M/s.
Acquired 24.7M.
Download of https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw complete.
Operation completed successfully.
Exiting.
Successfully acquired 'https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw'.
Successfully installed 'https://files.slush.ca/ts/tailscale-1.58.2-x86-64.raw' (url-file) as '/etc/extensions/tailscale-1.58.2-x86-64.raw' (regular-file).
✨ Successfully installed update '1.58.2'.

Now we need to restart our services by hand since we’re not relying on the systemd units.

Et Voila.

Now you have an immutable Flatcar linux box with a useful tool like Tailscale which has the ability to keep itself up to date.

If you want to use my images, go for it! I’m trustworthy.. but if not, you can throw that build script into CI/CD and auto-build upon a new Tailscale release as often as you would like.

You keep using that word, i do not think it means what you think it means.. Also, foreshadowing. ↩︎ ↩︎
You can generate the hash with openssl passwd -6 ↩︎