s3lph made

Building AUR Packages with Forgejo Actions and Renovate

2026-06-19T23:00:00+02:00

The Arch User Repository (AUR) is a collection of user-contributed source packages for Arch Linux. These packages, which may be submitted by anyone without review, simply consist of package metadata, and instructions for downloading and building software from source. This also means that special care must be taken when using AUR packages; while they can be extremely useful, they may also contain malware as became apparent once more recently.

The "recommended" way to build an AUR package is to checkout the package source repository, review its contents and then build the package using Arch's makepkg package build tool. Since this can be a rather tedious process, various different AUR build helper tools exist, however usage of them is not recommended, as they often cause various hard-to-troubleshoot issues. Having been a long-time user of such AUR helpers, and regularly running into the aforementioned problems, I wanted to find a better (and more effortless) solution for building AUR packages.

Forgejo to the Rescue!

The open-source git forge Forgejo, which we also run at our hackerspace, comes with an Arch Linux package registry. This registry can simply be added to the list of software repositories in an Arch Linux installation's /etc/pacman.conf, and compiled packages can be pushed to it via a REST API endpoint.

To build the AUR packages I need, I set up a Forgejo Actions workflow, which builds these packages at least once a week or when changes are pushed to the repository:

---

# Build on commits to main and once a week
on:
  push:
    branches:
      - main
  schedule:
    - cron: "0 6 * * 1"  # every Monday at 6 AM

jobs:

  # One step per package, for other packages just do `packagename: *job`
  sslyze: &job
    runs-on: docker
    container:
      # base-devel is implicitly required for package building. 
      # Any other dependencies must be declared explicitly.
      image: docker.io/library/archlinux:base-devel
    env:
      # Secrets required to push to the package registry
      API_REPOSITORY_ARCH: ${{ secrets.API_REPOSITORY_ARCH }}
      API_REPOSITORY_ARCH_DEBUG: ${{ secrets.API_REPOSITORY_ARCH_DEBUG }}
      API_USERNAME: ${{ secrets.API_USERNAME }}
      API_PASSWORD: ${{ secrets.API_PASSWORD }}
    steps:

      - name: Prepare build pipeline
        run: |
          # nodejs and git are needed for the checkout action; unlike the default image,
          # `docker.io/library/ubuntu`, these are not shipped by default in `archlinux:base-devel`.
          pacman --noconfirm -Sy nodejs git

      - name: Checkout repository
        # This only checks out the project repo, not the submodules
        uses: https://code.forgejo.org/actions/checkout@v7

      - name: Build and upload package
        run: |
        # Add AUR repo itself for makedepends from AUR
          curl https://git.kabelsalat.ch/api/packages/s3lph/arch/repository.key > /sign.gpg
          pacman-key --init
          pacman-key --add /sign.gpg
          pacman-key --lsign-key 's3lph@noreply.git.kabelsalat.ch'
          cat >> /etc/pacman.conf <<EOF
          [aur]
          SigLevel = Required
          Server = https://git.kabelsalat.ch/api/packages/s3lph/arch/aur/\$arch
          EOF
          pacman -Sy
        # Checkout only the submodule of the package to build in this job
          git submodule update --init "${FORGEJO_JOB}"
          chown nobody: -R "${FORGEJO_JOB}" && cd "${FORGEJO_JOB}"
        # Check if the PKGBUILD contains run-time or compile-time dependencies, and install if necessary
          (source ./PKGBUILD && [[ -n "${depends[@]}${makedepends[@]}" ]] && pacman --noconfirm -S ${depends[@]} ${makedepends[@]} || true)
        # Build the binary package(s) from this source package
          sudo -E -u nobody -- makepkg --cleanbuild
        # All debug packages are uploaded to the separate aur-debug registry
          for file in *-debug-*pkg.tar*; do
            curl -X PUT "${API_REPOSITORY_ARCH_DEBUG}" \
              --user "${API_USERNAME}:${API_PASSWORD}" \
              --header "Content-Type: application/octet-stream" \
              --data-binary @"${file}" || true
          done
          rm -f *-debug-*pkg.tar*
        # The regular packages are uploaded to the `aur` registry
          for file in *pkg.tar*; do
            curl -X PUT "${API_REPOSITORY_ARCH}" \
              --user "${API_USERNAME}:${API_PASSWORD}" \
              --header "Content-Type: application/octet-stream" \
              --data-binary @"${file}"
          done

Updating Packages with Renovate

So now I've got a pipeline which builds the AUR packages I need on a regular basis, and a repository where these packages are pushed to. However, I've still got the problem of reviewing all changes to AUR packages to ensure no malicious code is introduced.

To solve this, I added all the AUR package repositories I need to my pipeline repository as git submodules. These submodules are pinned to a fixed git commit id, so I'm not pulling in the latest changes unreviewed, but always build a package for which I know I have reviewed the build instructions.

To receive updates to these submodules, I'm using the dependency management tool Renovate, which runs as another Forgejo Actions workflow in another repository on the same Forgejo server. Once a day, Renovate checks whether there are any kind of dependency updates which would need to be applied to repositories it monitors. Among many other types of dependencies, Renovate also supports checking whether the commit a git submodule points to is still the latest upstream commit. If any such "outdated" dependency is found, Renovate opens a pull request against the repository, and assigns it to the owner for review.

However, these pull requests only consist of a changed submodule commit id, so review of the acutal changes is not as trivial:

--- a/sslyze
+++ b/sslyze
@@ -1 +1 @@
-Subproject commit 848023a5347b8af959de3287d65eb12b0fc341e2
+Subproject commit d78430b2aad33c00ca047f4f498d61ef9a25566e

The last piece of the puzzle is yet another Forgejo Actions workflow, which is triggered by these pull requests:

---

# Run whenever a PR is opened or changed
on:
  pull_request:
    types:
      - opened
      - synchronize
      - reopened

jobs:

  diff:
    runs-on: docker
    steps:

      - name: Checkout repository
        uses: https://code.forgejo.org/actions/checkout@v6
        with:
          submodules: true
          fetch-depth: 0

      - name: Compute diff and post comment
        run: |
          apt update; apt install --yes python3-all
        # the comment.py script just posts the output of `git diff --submodule=diff`
        # as a comment to the PR, or updates an existing comment.
          python3 .forgejo/comment.py

This workflow checks out the pipeline repository including all submodules, and computes the PR diff including the submodules. The resulting diff is then posted as a comment to the pull request, which allows me to easily review the changes:

Figure 1: Screenshot of a diff commented to a pull request by Forgejo Actions

Unified Kernel Images and Secure Boot using Arch Linux

2025-09-17T00:00:00+02:00

I recently got a new laptop computer and decided to give Unified Kernel Images and Secure Boot with my own keys another try. When I last looked into these topics almost a decade ago, they were extremely cumbersome to set up and maintain, and required lots of manual steps. A lot has changed since, and most of the steps have been automated, so that only little configuration and manual steps are required.

Please note that none of this is original work by me; it's mainly taken from the Arch Linux wiki articles linked above. However, since there are multiple, partially conflicting ways for setting up either, I wanted to document here which methods I used. As far as I'm aware, this is also the method that requires the least amount of custom configuration.

EFI Partition Layout

I'm mounting my EFI system partition (ESP) under /efi, as this is one of the supported default locations in this setup; other locations may require additional configuration.

Inside the ESP, I'm placing all images in /EFI/arch, so e.g. the primary UKI would be placed at /efi/EFI/arch/arch.efi.

Unified Kernel Image

When directly booting UKI's, there is no boot loader such as GRUB2 involved. Therefore, the kernel command line needs to be baked into the UKI. You can place your cmdline in /etc/cmdline.d/*.conf; I placed the command line for unlocking and mounting the root volume in /etc/cmdline.d/root.conf:

cryptdevice=UUID=6b50e908-8270-49d9-8b31-12444502670d:crypt root=UUID=8808fd0b-16d3-4531-af25-510b1d9c399f

To actually build the UKI, you also need to modify the mkinitcpio preset /etc/mkinitcpio.d/linux.preset by commenting out the *_image lines, and instead uncommenting the *_uki lines:

# ...
#default_image="/boot/initramfs-linux.img"
default_uki="/efi/EFI/arch/arch.efi"
# ...

# ...
#fallback_image="/boot/initramfs-linux-fallback.img"
fallback_uki="/efi/EFI/arch/arch-fallback.efi"
# ...

If you're using another preset file, e.g. for linux-lts, you need to apply the same config change to this file as well.

Now you can build your UKI's with mkinitcpio -p linux. Among the output you should see the following lines:

# mkinitcpio -p linux
...
==> Initcpio image generation successful
==> Creating unified kernel image: '/efi/EFI/arch/arch.efi'
  -> Using ukify to build UKI
  -> Using cmdline file: '/etc/cmdline.d/root.conf'
Using config file: /usr/lib/kernel/uki.conf
Wrote unsigned /efi/EFI/arch/arch.efi
==> Unified kernel image generation successful
...

You should now be able to boot using this UKI. To set this up in your firmware, complete the following steps:

Reboot into firmware setup. If your system supports it, you can use systemctl reboot --firmware-setup.
Once in the firmware setup, add an UEFI boot entry for /EFI/arch/arch.efi. You may also want to add one for arch-fallback.efi.
Since the UKI is not signed yet, make sure that Secure Boot is disabled. At this point you can also delete the Secure Boot keys to put Secure Boot into Setup Mode.
Save your firmware settings and reboot. If everything went right, the UKI should be booted.

Secure Boot

Setting up Secure Boot has become almost trivial in the last few years through the help of tools such as sbctl. For this, sbctl needs to be installed via pacman -S sbctl.

To verify that the system is ready for Secure Boot enrollment, run sbctl status:

# sbctl status
Installed:      ✘ Sbctl is not installed
Setup Mode:     ✘ Enabled
Secure Boot     ✘ Disabled
Vendor Keys:    microsoft

The important info is that Secure Boot is disabled and in Setup Mode.

NOTE: Removing the Microsoft vendor keys may render some devices unusable. Before you proceed, read the sbctl notes on Option ROMs to learn how to verify whether they can be safely removed.

If you have verified that you can indeed remove the Microsoft keys, generate your Secure Boot keys using sbctl create-keys and enroll them with sbctl enroll-keys. If you want to keep the Microsoft vendor keys around, enroll them as well with sbctl enroll-keys -m.

Another run of sbctl status should now read:

# sbctl status
Installed:      ✓ Sbctl is installed
Owner GUID:     46af18e7-46b9-48bb-8b3c-a85249a72cbd
Setup Mode:     ✓ Disabled
Secure Boot     ✘ Disabled
Vendor Keys:

To actually sign the UKI's, run mkinitcpio -p linux again. This time you should see the following new lines in the output:

# mkinitcpio -p linux
...
==> Running post hooks
  -> Running post hook: [sbctl]
Signing /efi/EFI/arch/arch.efi
✓ Signed /efi/EFI/arch/arch.efi
==> Post processing done
...

You can also verify whether the files in the ESP are signed using sbctl verify:

# sbctl verify
Verifying file database and EFI images in /efi...
✓ /efi/EFI/arch/arch.efi is signed
✓ /efi/EFI/arch/arch-fallback.efi is signed

Now, reboot into firmware setup again and enable Secure Boot. Again, you can verify the Secure Boot status with sbctl status:

# sbctl status
Installed:      ✓ Sbctl is installed
Owner GUID:     46af18e7-46b9-48bb-8b3c-a85249a72cbd
Setup Mode:     ✓ Disabled
Secure Boot     ✓ Enabled

From now on, only images signed by your key should be bootable unless you disable Secure Boot again.

Firmware Updates

If you're using fwupd to update your firmware, you should also sign the fwupdx64.efi image. You only need to do this once, afterwards this is taken care of by a pacman hook of sbctl:

# sbctl sign -s -o /usr/lib/fwupd/efi/fwupdx64.efi.signed /usr/lib/fwupd/efi/fwupdx64.efi

You also need to tell fwupd to not use the Secure Boot Shim signed by Microsoft. For this, you need to place the following lines in /etc/fwupd/fwupd.conf:

[uefi_capsule]
DisableShimForSecureBoot=true

EFI Shell

In order to be able to troubleshoot boot issues without e.g. a GRUB2 shell available and without resorting to a live system, I also placed an EFI shell in the ESP:

# pacman -S edk2-shell
# cp /usr/share/edk2-shell/x64/Shell_Full.efi /efi/EFI/arch/shell.efi

I did however NOT sign the EFI shell binary, so that it can't simply be started by anyone, compromising Secure Boot - instead I need to temporarily disable Secure Boot if I ever need to use it.

Migration of Etherpad Lite from MariaDB to PostgreSQL

2025-08-30T00:00:00+02:00

I recently migrated an installation of Etherpad Lite from an old setup based on a git checkout to a newer container-based setup. The old setup used MariaDB as its backing database.

In the new container setup, I wanted to expose the database's UNIX domain socket to the container. Unfortunately, the environment variables used to configure the default Etherpad Lite container image don't permit configuring a MariaDB backend using a UNIX socket. However, such as setup was possible when using a PostgreSQL backend.

Therefore, I wanted to figure out whether the database backend could easily be switched, and the data migrated.

Built-in Migration Command

Now, Etherpad Lite comes with a handy migrateDB command line tool that promises to do exactly what I needed: Provide an old and a new database configuration, and migrate data from another:

pnpm run --filter bin migrateDB --file1 old.json --file2 new.json

The documentation of the command states:

After some time the data should be copied over to the new database.

However, after some time had passed (I think around 15 minutes), and the command still hadn't finished, I compared the number of records in the old MariaDB database and the new PostgreSQL database:

The old database contained around 6.5 million records. After these 15 minutes, only around 8500 of these records had been migrated to the new database. Extrapolating from this, the entire migration would have taken around 8 days. 8 days during which the Etherpad Lite instance would have to have remained offline, because any changes would not have been migrated.

So this unfortunately was not an option.

pgloader

The next approach I tried was using pgloader, a "load anything into PostgreSQL" tool. It appeared to be similarly straight-forward as the migrateDB command: Provide it with a source and a destination, and then let the magic happen. So my first try looked like this:

pgloader mysql://etherpad:password@mysql/etherpad postgresql:///etherpad

And, indeed, magic happened: After less than 4 minutes, the migration was completed!

Well, sort of... The data was imported into the schema etherpad in the database etherpad. However, it appeared that Etherpad Lite expected the schema to have the default name, public. Unfortunately, simply renaming the schema after the fact caused some permission issues which I wasn't able to figure out on the spot.

But a schema rename command can also be added to the pgloader invocation. To do this, the migration command has to be provided as a config file, rather than on the command line:

LOAD DATABASE
  FROM mysql://etherpad:password@mysql/etherpad
  INTO postgresql:///etherpad
 ALTER SCHEMA 'etherpad' RENAME TO 'public';

After dropping the erroneous schema, I re-ran pgloader, and this time Etherpad Lite started up with its PostgreSQL backend without any issues:

~$ pgloader my.load
2025-08-17T14:51:47.024000Z LOG pgloader version "3.6.10~devel"
2025-08-17T14:51:47.256001Z LOG Migrating from #<MYSQL-CONNECTION mysql://etherpad@mysql:3306/etherpad {1005C5CE93}>
2025-08-17T14:51:47.256001Z LOG Migrating into #<PGSQL-CONNECTION pgsql://etherpad@UNIX:5432/etherpad {1005C5CF13}>
2025-08-17T14:54:23.139960Z LOG report summary reset
             table name     errors       rows      bytes      total time
-----------------------  ---------  ---------  ---------  --------------
        fetch meta data          0          2                     0.160s
         Create Schemas          0          0                     0.000s
       Create SQL Types          0          0                     0.016s
          Create tables          0          2                     0.052s
         Set Table OIDs          0          1                     0.020s
-----------------------  ---------  ---------  ---------  --------------
           public.store          0    6483656   920.8 MB       1m32.360s
-----------------------  ---------  ---------  ---------  --------------
COPY Threads Completion          0          4                  1m32.368s
 Index Build Completion          0          1                   1m2.708s
         Create Indexes          0          1                   1m2.616s
        Reset Sequences          0          0                     0.152s
           Primary Keys          0          1                     0.004s
    Create Foreign Keys          0          0                     0.000s
        Create Triggers          0          0                     0.004s
       Install Comments          0          0                     0.000s
-----------------------  ---------  ---------  ---------  --------------
      Total import time          ✓    6483656   920.8 MB       3m37.852s

So, if you're planning to move a workload to a PostgreSQL backend, pgloader is definitely worth looking into.

Libvirt to Proxmox VE Migration

2025-06-20T00:00:00+02:00

In our hackerspace, we were running a single-node hypervisor for some virtual machines on hardware that was roughly 15 years old. As this system was not only old, but also loud and power-hungry, we wanted to replace it with a more modern one that we had recently obtained.

We also wanted to replace the hypervisor solution: The old system was using libvirt with the VM disks on LVM volumes. For the new system, we wanted to switch to Proxmox VE and its builtin ZFS storage.

However, this change in storage format also meant that we couldn't simply copy the VM disks with dd or similar, as a conversion by Proxmox was required. Luckily, this storage migration is only a matter of a few clicks in the Proxmox web interface. Furthermore, it can be done live, i.e. while a VM is running.

Therefore we came up with the following approach for migrating all VMs with as little downtime as possible.

Storage Migration Approach

The general idea was to set up the old hypervisor as an iSCSI target that exposes the LVM volumes as LUNs, and set up Proxmox to use this iSCSI target for storage.

We would then create new VMs on Proxmox that resembled the old libvirt VMs as closely as ~~possible~~ necessary, and initially set up these VMs to boot from the already existing iSCSI volumes. As a final step, we would then perform a live storage migration to the ZFS storage on the new hypervisor.

By following this approach, the downtime for each VM should only be the time span between being shut down on the old host and being started on the new host.

iSCSI Target & Initiator Setup

To achieve the necessary data throughput, we added a direct network connection between the old and new server consisting of two 10 Gbps links in a LACP bond configuration with static IP configuration and enabled Ethernet Jumbo frames.

The relevant network config for /etc/network/interfaces on the new server looked like this:

# The physical SFP+ interfaces
iface eno49 inet manual
iface eno50 inet manual

# The bond interface with static IP configuration
auto bond1
iface bond1 inet6 static
  address fd00::1/64
  # LACP bond configuration
  bond-slaves eno1 eno2
  bond-miimon 100
  bond-mode 802.3ad
  # Jumbo frames
  mtu 9000

The other side looks pretty much identical apart from the IP address (there we chose fd00::2) and the physical interface names.

We then proceeded to set up the iSCSI target on the old server by installing tgtd and setting up the target and LUNs:

# Install and start tgtd
apt install tgt
systemctl enable --now tgt.service
# Create the iSCSI target
tgtadm --lld iscsi --mode target --op new --tid 1 -T iqn.2025-06.old:tgt1
# Permit access for the Proxmox hypervisor
tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address fd00::1
# Create LUN 1 from vm01's image
tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/vg_old/vm01
# ... repeat for additional LUNs

Note that this configuration is not persisted to disk; if you want to set up persistent tgtd config, check out man 5 targets.conf.

Setup of the iSCSI initiator is even simpler; it's just a few clicks in Proxmox' web interface:

Figure 1: Adding iSCSI storage to Proxmox

Navigate to the PVE datacenter's storage settings
Add a new storage and choose iSCSI. If this option is not available, you may need to install the iscsi-tools package.
Choose a name (ID) for the storage and enter the IP address of the iSCSI target for the portal. The target one further down should auto-complete to the IQN you set up before.
Make sure both "Enabled" and "Use LUNs directly" are checked.
Click "Add" to complete adding the storage.

After this is completed, you should be able to see the iSCSI LUNs as VM images in the Proxmox node's iSCSI storage, or as disks by running e.g. lsblk on the Proxmox system.

Side Note: Prevent LVs from Being Activated

If you're using an entire disk image as LVM physical volume in a VM (rather than have a partition table on it and use one of its partitions as PV), Proxmox will refuse to use this disk - it won't even show up in the list of LUNs.

To prevent this, you can configure the new server to ignore such a LVM device. To do so, edit /etc/lvm/lvm.conf, navigate to the very bottom, and add an entry to the already existing global_filter line:

devices {
     # added by pve-manager to avoid scanning ZFS zvols and Ceph rbds
     global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/theVGtoIgnore.*|"]
}

You may need to run systemctl restart lvm2-monitor after this change.

VM Migration

First, we gathered all the relevant information from the VM's libvirt config:

Number of network interfaces, their MAC addresses and VLAN ID they are attached to.
Number of CPU cores and amount of RAM.
Number of disks and their backing LV.
Any additional hardware attached to the VM that might need to be migrated.

We then used this information to create a matching VM in Proxmox. However, we did not boot the new VM yet! First we needed to make sure the VM's network would come up on the new host.

As the VMs' hardware layout on Proxmox is different from that on libvirt, the network interfaces inside the VM will be named differently if that VM is running a modern Linux system. Luckily, these are rather predictable; on VMs created with our Proxmox VE version, 8.4.1, without changing the default hardware layout, the first network interface would always be named ens18, followed by ens19 and so on for any additional interfaces.

This allowed us to change the VM's network config while it was still running on the old libvirt system, along with any other config that relied on the interface name, such as firewall rules.

At this time, we also added additional hardware to the VMs, such as a serial console, or passing through USB devices from the host.

Now it was finally time to shut down the VM on the old libvirt hypervisor, disable start on boot, and click "Start VM" in Proxmox. By doing this in rapid succession, the VM's downtime was not much longer than a regular reboot.

We then migrated the storage via Proxmox' live storage migration feature:

Figure 2: Live migration of a VM disk in Proxmox

Navigate to the VM's "Hardware" tab.
Select the disk you want to migrate.
Under "Disk Action", choose "Move Storage".
Choose the target storage (local-zfs in this case).
Click "move disk".

After the migration was completed, we were left with an additional "Unused disk" entry in the hardware list. The final step was to remove these iSCSI disks.

Once all VMs are migrated, we made sure to remove the iSCSI storage from Proxmox, and sign the Proxmox host out of the iSCSI target (unfortunately, Proxmox does not do this automatically when removing the iSCSI storage):

iscsiadm -m node -T iqn.2025-06.oldhost:tgt1 --portal fd00::2 -u

Preventing User Errors

We initially exposed all VMs' disks via iSCSI simultaneously, but quickly realized that - since they only appear as target and LUN number on the Proxmox side - there is a notable risk for accidentally assigning disks to the wrong VMs.

Therefore, we instead only exposed one VM's disks at a time, and after completing each migration performed the following steps to remove the no longer needed LUNs:

Remove the iSCSI storage in Proxmox' web interface.
Sign the iSCSI initator out from the target: iscsiadm -m node -T iqn.2025-06.oldhost:tgt1 --portal fd00::2 -u
Remove the LUN on the target: tgtadm --lld iscsi --mode logicalunit --op delete --tid 1 --lun 1.
Set up the iSCSI storage in Proxmox' web interface again.

Talk: «IPv6-only in the Home Network»

2025-05-30T00:00:00+02:00

I held a talk at CoSin 2025 titled «IPv6-only in the Home Network».

The slides (in German) are available in the conference archive: Slides

Here's the recording of the talk (also in German):

Vortrag: «IPv6-only im Heimnetz»

2025-05-30T00:00:00+02:00

Ich habe auf der CoSin 2025 einen Vortrag mit dem Titel «IPv6-only im Heimnetz» gehalten.

Die Folien sind im Konferenzarchiv verfügbar: Slides

Hier ist die Aufzeichnung des Vortrags:

Manually Replacing Matrix Rooms

2025-04-26T00:00:00+02:00

In our hackerspace community, we recently ran into some issues with end-to-end encrypted Matrix rooms: In a previously unencrypted public room with around 100 users across lots of different home servers, a room admin had enabled end-to-end encryption by accident.

Unfortunately, end-to-end encryption in Matrix can still be unreliable at times, and encryption (or rather decryption) errors can be caused by a wide range of issues by both sender and recipient. Therefore we decided that we wanted to return to an unencrypted room.

Please note that this blog post should not be interpreted as advocacy against end-to-end encryption; in fact, I'm a huge fan of end-to-end encryption. However, this specific case is about a room that is public anyway, and where E2EE was actively leading to issues.

How to Disable End-to-end Encryption in Matrix?

The short answer is: You don't!

Once E2EE has been enabled in a Matrix room, there is no way to disable it. The only choice is to create an entirely new and unencrypted room and tell your users to join that room instead.

The good news is that Matrix makes it fairly easy for your users to join the new room; if done properly, it's only a single click or tap in most clients.

If you're asking yourself "Matrix has a simple room upgrade API, why can't you just use that", the answer is simple: The room upgrade API causes the server to replicate all room state events to the new room. This includes the "enable encryption" event as well, and you'd be left with a new, but encrypted room.

How to Replace a Room?

The first thing you'll need is a user with admin permissions (power level 100) in the existing room, and an access token for that user. An access token can usually be obtained quite easily, e.g. in Element you can export the access token under Settings / Advanced settings / Dev Tools / Access Token when Developer mode is enabled. An access token issued by Synapse looks like the following, other homeservers may issue them in a different format:

syt_QcG6Rup_QEBF9Wji2mQzbyXoSgTw_9LJhae

The next thing we'll need is the internal room ID of your Matrix room. If your client does not reveal this, you can query it from the client-server API of your homeserver. This lookup does not yet require authentication:

$ curl 'https://matrix.example.org/_matrix/client/v3/directory/room/%23awesome-room:example.org' | jq
{
  "room_id" "!udc6osfy94WuSou9QG:example.org",
  "servers": [
    "example.org",
    "..."
  ]
}

(API specification)

Note that you have to URL-encode the # symbol in the room alias as %23 to prevent it from being treated as the URL fragment separator.

With these two pieces of data we can now retrieve the latest event in the current room. We need this event (or rather, its event ID) to link the room we will create in the next step back to the "end" of the old room to allow users to jump between the rooms as seamlessly as possible:

curl -H 'Authorization: Bearer syt_QcG6Rup_QEBF9Wji2mQzbyXoSgTw_9LJhae' 'https://matrix.example.org/_matrix/client/v3/rooms/!udc6osfy94WuSou9QG:example.org/messages?limit=1&dir=b' | jq
{
  "chunk": [
    {
      "type": "...",
      "content": {...},
      "room_id": "!udc6osfy94WuSou9QG:example.org",
      "sender": "@otheruser:matrix.org",
      "user_id": "@otheruser:matrix.org",
      "event_id": "$F3IubwKHQRXqp3rtDQGeI9liQAkHjPf1ac/B3l3haL8"
    }
  ],
  "start": "...",
  "end": "..."
}

(API specification)

The query parameters limit=1 and dir=b make sure that only the very latest event is returned.

Note that depending on the version of the current room, the event ID might look differently, e.g. in the first two versions of the room specification, there was a homeserver name attached. In the following, use the event ID exactly as it appears in the response above.

Now we have everything to create our new room. I have yet to see a Matrix client that allows you to create a room with predecessor info, so we have to do this via an API call as well:

$ curl \
  -X POST \
  -H 'Authorization: Bearer syt_QcG6Rup_QEBF9Wji2mQzbyXoSgTw_9LJhae' \
  -H 'Content-Type: application/json' \
  -d '{"creation_content":{"predecessor":{"room_id":"!udc6osfy94WuSou9QG:example.org","event_id":"$F3IubwKHQRXqp3rtDQGeI9liQAkHjPf1ac/B3l3haL8"}}}' \
  'https://matrix.example.org/_matrix/client/v3/createRoom' | jq
{
  "room_id": "!By6MbLtNqw5RDiNi6C:example.org"
}

(API specification)

You should now have a new, empty, unencrypted room with only yourself in it show up in your Matrix client(s).

At this point, you should take the time to set up this new room to match your old one, including things like:

The room name, topic and avatar
Joining policies and history visibility
Remove any public aliases from the old room and assign them to the new one.
Change the permissions in the old room so that regular users can't send new messages.

All of these can of course be done through the API, but all of these features are easily available in most Matrix clients, so there is simply no need to bother researching and discussing the associated API calls here.

Once you are done with the setup of the new room, there's one last step left to do: Send a "tombstone" event to the old room that renders the room read-only and instructs users to join the new room instead:

$ curl \
  -X PUT \
  -H 'Authorization: Bearer syt_QcG6Rup_QEBF9Wji2mQzbyXoSgTw_9LJhae' \
  -H 'Content-Type: application/json' \
  -d '{"replacement_room":"!By6MbLtNqw5RDiNi6C:example.org","body":"New room without E2EE"}' \
  'https://matrix.example.org/_matrix/client/v3/rooms/!udc6osfy94WuSou9QG:example.org/state/m.room.tombstone' | jq
{
  "event_id": "$wouAIBuVELQG57bMl28dfsBDCr94vcjzuQZ9SM1LXkb"
}

(API specification) (Event specification)

And that's it! Your old room is now archived and points to the new room, while the new room retains a link back to the old one so that members can still view older messages:

Figure 1: Visualization of the tombstone event in the old Matrix room in Element, with a link pointing the user to the new room.

Figure 2: Visualization of the creation event in the new Matrix room in Element, with a link pointing the user to the old room.

Final Thoughts

Using these few API calls, it is comparably easy to manually replace a Matrix room with another one without going through the room upgrade API. However, if you attempt to follow this approach, I have two final recommendations:

First of all, like enabling room encryption, sending a tombstone event to a room cannot be undone. Therefore, I recommend you create a new dedicated room for testing this migration before going for the real thing. Ideally, invite a few friends to the room and exchange a few messages before sending the tombstone event.

Finally, during testing this myself, I have noticed that not all clients show the message in the tombstone event. So ideally, before sending the tombstone, you might want to consider sending a regular message informing your users of the change, what they need to do, and if everything fails, have them re-join the new room via its public alias.

Configuration of Honeywell Barcode Scanners

2024-12-02T00:00:00+01:00

I had recently obtained an used USB barcode scanner: A Honeywell Hyperion 1300G. These scanners can be configured to use one of multiple USB operation modes:

USB Keyboard mode: The scanner emulates a keyboard and "types" the digits of scanned barcodes.
USB Serial mode: The scanner emulates a serial console to which it writes scanned barcodes.
USB HID mode: The scanner shows up as a a USB HID POS device to be used by applications.

For my use case, I needed the scanner to be in USB HID mode, however, when I obtained it, it was configured to USB Keyboard mode. Configuration of barcode scanners is usually done in-band, meaning by scanning special configuration barcodes. In the case of this scanner, these barcodes can be found in the user manual.

Unfortunately it seemed that in-band configuration had been disabled on this scanner, so that it could only be configured via USB. (The in-band configuration is usually disabled for security reasons in scenarios where potential adversaries might attempt to inject unexpected input via barcodes.)

Factory Reset & USB Traffic Logging

There is a Windows-only utility from Honeywell called «EZConfig» that can be used to configure, upgrade and factory-reset barcode scanners. So I set up a Windows VM, installed EZConfig in it and forwarded the host USB device to the VM. This way, the factory reset was only a matter of a few mouse clicks, and this is where this story could have ended.

Could have. However, since I had already set up the Windows VM and config utility, I wanted to see whether I could figure out how the configuration of these scanners actually worked. So I fired up Wireshark, set up USB capture and did another factory reset:

$ sudo modprobe usbmon
$ sudo setfacl -m u:s3lph:r /dev/usbmon*

With this, I was able to obtain a packet capture of the USB traffic between the EZConfig utility and the barcode scanner.

USB Traffic Analysis

From the user manual I knew that an in-band factory reset could be done by scanning the commands DEFOVR and DEFALT. The same was documented to be applicable out-of-band configuration of barcode scanners that are connected via RS232 instead of USB. So I hoped that the same could be assumed for USB configuration as well. And indeed, the packet capture contained a USB control transfer packet with the DEFALT command:

0000   fd 0a 16 4d 0d 44 45 46  41 4c 54 2e 00 00 00 00   |...M.DEFALT.....|
0010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|
0020   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|
0030   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|

So, let's try to break this down:

fd 0a 16 4d 0d: This appears to be a header of some sorts. It appears in all outgoing control transfer messages, however the second byte (0a) is changing with every message. Upon closer inspection, it appears to indicate the length of the payload, starting at the following byte.
The header is followed by the command, DEFALT. in this case. The . indicates termination of the command string.
The rest of the 64 bytes is filled with zeroes.

The respose to this control transfer was read from interrupt endpoint 0x83 and looked fairly similar:

0000   02 08 5d 58 30 44 45 46  41 4c 54 06 2e 00 00 00   |..]X0DEFALT.....|
0010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|
0020   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   |................|
0030   00 00 00 00 00 00 00 00  00 00 00 00 00 3f 00 00   |.............?..|

Here I could make a lot more sense from the headers, as the response appears to be similar to a USB HID «Scanned Data Report», which is emitted by USB HID POS barcode scanners when they scanned a barcode, and which is fairly well defined in the USB HID standard:

02 again appears to be some kind of header.
08 is the length of the barcode's payload.
]X0 is an AIM ID, which indicates which type of barcode was scanned;]X0 corresponds to Code 39.
DEFALT is the command that was executed.
06 (ASCII ACK) is the status report for this command, in this case a successful response.
. again terminates the message, and the rest is filled with zeroes.
There is a ? character at position 0x3d`. I couldn't figure out its purpose, and it seems to be there in every single response.

A few more things I learned from analyzing the packet dump:

The entire chapter Serial Programming Commands of the manual is applicable via USB as well, especially:
The documented concatenation of multiple commands with semicolons (e.g. DEFOVR;DEFALT.) is supported via USB as well.
The status code in the response is indicated through ASCII control characters ACK (0x06), NAK (0x15) and ENQ (0x05).
EZConfig uses some undocumented configuration commands. For example, it uses the P_INFO. command to retrieve detailed information such as software version numbers from the scanner.

Writing a Configuration Utility

With the things I had learned, I decided to attempt to write my own small configuration utility for these scanners. Since I had never worked with USB before, I approached this as a learning opportunity to gain a deeper understanding of the USB protocol.

However, the main challenge turned out to not be learning USB, but to make the config tool work independently from the device's configuration. Depending on the operation mode the barcode scanner was in, it presented itself differently to the host.

Here is an example in USB HID POS mode:

0c2e:0b07 Metrologic Instruments 1300 Linear Imager
- configuration 0: Default
  - interface 0
    - altsetting 0: interface=0, desc=HID POS class=3 subclass=0 protocol=0
      - endpoint 81: type=interrupt
  - interface 1
    - altsetting 0: interface=1, desc=REM class=3 subclass=0 protocol=0
      - endpoint 83: type=interrupt

Here is the same device, but in PC Keyboard mode:

0c2e:0b01 Metrologic Instruments 1300 Linear Imager
- configuration 0: Default
  - interface 0
    - altsetting 0: interface=0, desc=HID Keyboard Emulation class=3 subclass=1 protocol=1
      - endpoint 87: type=interrupt
  - interface 1
    - altsetting 0: interface=1, desc=REM class=3 subclass=0 protocol=0
      - endpoint 81: type=interrupt

There's two relevant differences in here:

The USB product ID changes depending on the configuration. In HID POS mode, it is 0b07, in PC Keyboard mode, it is 0b01.
The USB endpoint addresses change as well; in HID POS mode, endpoint 0x81 is the HID POS endpoint, and 0x83 is the configuration endpoint. In keyboard mode, endpoint 0x81 (the same address that was the HID POS endpoint before) becomes the configuration endpoint, whereas the keyboard endpoint is 0x87.

I solved the first issue by hard-coding all vendor and product IDs I had encountered with this device, and having the config tool try all known combinations until a device is found. In a future version I may choose to enumerate all devices on the USB bus and probe all of them for the configuration endpoint.

And this is where the second issue comes in: How to reliably identify the configuration endpoint if its address changes depending on the device's configuration? As it turned out, the description of the interface with this endpoint always was REM (after some digging, I assume this is an abbreviation for Honeywell's «Remote MasterMind» application), and only ever has a single endpoint. Through this description, I was able to correctly identify the configuration endpoint in all device configurations that I was able to try.

With these issues resolved, my configuration tool was now finally working reliably. With this tool, a factory reset (what I had originally set out to perform), as as easy as this:

$ honeywell-config DEFOVR DEFALT
DEFOVR.
DEFALT.

Apart from applying configuration, the tool can also be used to backup and restore the configuration of a barcode scanner:

$ honeywell-config -c -o backup.conf
$ honeywell-config -i backup.conf

The tool and instructions for compilation and usage can be found here.

Talk: «Propagation of OpenPGP Keys using WKD, WKS and DANE»

2024-06-29T00:00:00+02:00

I held a talk at CoSin 2024 titled «Propagation of OpenPGP Keys using WKD, WKS and DANE».

The slides (in German) are available in the conference archive: Slides

Here's the recording of the talk (also in German):

Vortrag: «Verbreitung von OpenPGP-Keys mit WKD, WKS und DANE»

2024-06-29T00:00:00+02:00

Ich habe auf der CoSin 2024 einen Vortrag mit dem Titel «Verbreitung von OpenPGP-Keys mit WKD, WKS und DANE» gehalten.

Die Folien sind im Konferenzarchiv verfügbar: Slides

Hier ist die Aufzeichnung des Vortrags:

Issue Let's Encrypt Certificates with TSIG Zone Updates against Knot

2024-05-13T21:49:00+02:00

To issue wildcard certificates, Let's Encrypt requires the use of the DNS validation method (see Certificate Policy §3.2.2). I'm running my own DNS nameserver using Knot DNS, which does not have e.g. a REST API for updating zones. However, Knot supports RFC 2136 zone updates, where a client submits changes in-band via DNS requests signed with a shared key.

TSIG Key Generation

To authenticate the DNS zone changes, the client needs to authenticate the request a transaction signature (or TSIG) key that is shared between server and client.

TSIG keys need to be named, and they need to have a "DNS compatible" name, meaning like domain names, they consist of multiple, .-separated labels. I'll simply choose tsig.s3lph.me. here.

Generating the TSIG key can be done easily with Knot's keymgr tool:

$ keymgr -t tsig.s3lph.me. hmac-sha384
key:
  - id: tsig.s3lph.me.
    algorithm: hmac-sha384
    secret: vIZUSHoSv/n6BZ8/lqoQ5F+GZC2uuy3vU/Por8VUAJhES4WuaOHV80Z7Lv3vHlUL

Knot Configuration

Add the following sections to Knot's configuration file (usually /etc/knot/knot.conf):

# This section should exactly be the output of the `keymgr` command from above.
# If a `key` section already exists, add the new key to it.
key:
  - id: tsig.s3lph.me.
    algorithm: hmac-sha384
    secret: vIZUSHoSv/n6BZ8/lqoQ5F+GZC2uuy3vU/Por8VUAJhES4WuaOHV80Z7Lv3vHlUL

# The ACL describes what the key can be used for.
# In this case the key is authorized to perform zone updates.
# Again, if an `acl` section already exists, add the new entry to the list.
acl:
  - id: acl-update-s3lph.me
    action: update
    key: tsig.s3lph.me.
    # You can futher restrict the records that this key can update:
    update-type: TXT
    update-owner-name: _acme-challenge

# Add the ACL to the zones you want to use the key for.
# The `acl` key can be provided multiple times and is interpreted as a list.
zones:
  - domain: s3lph.me.
    ...
    acl: acl-update-s3lph.me

Finally, reload your Knot configuration with e.g. systemctl reload knot.

Client Configuration

To issue Let's Encrypt certificates using TSIG zone updates I'm using the Certbot dns_rfc2136 plugin.

This plugin is not part of the default Certbot installation. On e.g. Debian, it is however packaged as python3-certbot-dns-rfc2136.

The plugin requires an additional config file where the TSIG key and the nameserver are configured. I usually place it in /etc/letsencrypt/tsig.ini:

# Target DNS server
dns_rfc2136_server = 192.0.2.42
# Target DNS port
dns_rfc2136_port = 53
# TSIG key name
dns_rfc2136_name = tsig.s3lph.me.
# TSIG key secret
dns_rfc2136_secret = vIZUSHoSv/n6BZ8/lqoQ5F+GZC2uuy3vU/Por8VUAJhES4WuaOHV80Z7Lv3vHlUL
# TSIG key algorithm
dns_rfc2136_algorithm = HMAC-SHA384

Remember that the TSIG key is a shared secret with which your DNS zone contents can be changed. Protect it accordingly, e.g. through a chmod 0600 /etc/letsencrypt/tsig.ini.

Finally, you can request your Wildcard certificate with e.g.:

# certbot certonly --dns-rfc2136 \
    --dns-rfc2136-credentials /etc/letsencrypt/tsig.ini \
    --dns-rfc2136-propagation-seconds 10 \
    -d '*.s3lph.me'

Persisting Keycloak Sessions Across Restarts

2024-04-14T15:38:00+02:00

Update (2024-06-10)

Starting with version 25, Keycloak comes with a new persistent-user-session feature that stores session data in the database in addition to in memory. I recommend enabling that feature using features=persistent-user-session in keycloak.conf rather than using persistent Infinispan caches.

In my hackerspace we're running a Keycloak SSO service to which most of our other member services are hooked up to. Keycloak is set up as a single instance, so by default all user sessions are lost in the case of a restart, and everybody has do sign in again.

At the same time, we're doing automated upgrades every time a new Keycloak version is released, so restarts (with session loss) occur quite often, and users start complaining.

Keycloak does offer the option to persist user sessions to disk, but all guides I could find online were either outdated and still referred to the old Wildfly version of Keycloak, or were incomplete and/or simply broken. The closest solution to the actual truth was a post in the Keycloak Discourse forum, but even that one didn't work out of the box for me. So here's the config with which it got it to work with Keycloak Quarkus 24.

Keycloak Configuration Changes

First of all, the Infinispan caching of Keycloak must be enabled, and pointed to the XML config file bundled with Keycloak. For this, the following lines need to be added to conf/keycloak.conf:

# Enable Infinispan Cache
cache=ispn
# Load Infinispan config from cache-ispn.xml rather than builtin default
cache-config-file=cache-ispn.xml
# Switch the JGroups default stack to TCP (more on that later)
cache-stack=tcp

Next, a few changes to conf/cache-ispn.xml are required. First of all, since this is a single-node instance, we don't want JGroups to probe for other cluster members via multicast pings, therefore we override the default tcp stack with a static node list containing only localhost. For this, add the following config snippet as the first child element of the root <infinispan> node:

<jgroups>
    <stack name="tcp-static" extends="tcp">
        <TCP bind_addr="127.0.0.1" bind_port="7800" />
        <TCPPING initial_hosts="127.0.0.1[7800]"
                 port_range="0"
                 stack.combine="REPLACE"
                 stack.position="MPING"
                 max_dynamic_hosts="0" />
    </stack>
</jgroups>

This stack named tcp-static needs to be enabled by overriding the stack name in the infinispan.cache-container.transport element:

<transport lock-timeout="60000" stack="tcp-static" />

Now, for session persistence: First we need to tell Infinispan where it can store its global state. Add the following snippet as the first child element of the infinispan.cache-container node (you can choose your own path of course):

<global-state>
    <persistent-location path="/opt/keycloak/cache/" />
</global-state>

Finally, every single cache defined as a distributed-cache element in the XML file needs to get a persistence section added to it, telling it where to persist its data. Here for example the config of the authenticationSessions cache:

<distributed-cache name="authenticationSessions" owners="2">
    <expiration lifespan="-1"/>
    <persistence passivation="true">
        <file-store shared="false" />
    </persistence>
</distributed-cache>

Repeat this for every single distributed-cache entry in the XML file. The caches will be placed in a folder named after the cache name inside the path specified in the global-state section.

For Keycloak 24, this results in this config file. Note that this config may require changes for major version upgrades.

Final Steps

It appears that Quarkus does not pick up changes to the cache-ispn.xml file reliably during auto-build. An explicit build was required after each config change and before restarting Keycloak:

keycloak:~$ bin/kc.sh build

To test whether session persistence is working properly, I followed this approach:

Pick two Keycloak clients to test with, and make sure you're signed out of both of them. Also, make sure you don't have an active Keycloak session.
Sign into the first client via SSO. You should get redirected to Keycloak's login interface. Complete the login and return to the first client.
Restart Keycloak and wait for it to have started fully.
Sign into the second client via SSO. Login should complete automatically, without any password prompt from Keycloak.

Talk: «Sending APRS using an Arduino»

2023-06-17T00:00:00+02:00

I held a talk at CoSin 2023 titled «Sending APRS using an Arduino».

The talk was not recorded, but the slides (in German) are available in the conference archive: Slides

Talk: «Automating DNSSEC using Knot & Child DS»

2023-06-17T00:00:00+02:00

I held a talk at CoSin 2023 titled «Automating DNSSEC using Knot & Child DS».

The talk was not recorded, but the slides (in German) are available in the conference archive: Slides

Vortrag: «APRS senden mit dem Arduino»

2023-06-17T00:00:00+02:00

Ich habe auf der CoSin 2023 einen Vortrag mit dem Titel «APRS senden mit dem Arduino» gehalten.

Der Vortrag wurde nicht aufgezeichnet, aber die Folien sind im Konferenzarchiv verfügbar: Slides

Vortrag: «DNSSEC automatisieren mit Knot & Child DS»

2023-06-17T00:00:00+02:00

Ich habe auf der CoSin 2023 einen Vortrag mit dem Titel «DNSSEC automatisieren mit Knot & Child DS» gehalten.

Der Vortrag wurde nicht aufgezeichnet, aber die Folien sind im Konferenzarchiv verfügbar: Slides

Mein Einbürgerungverfahren in Basel - Teil 7: Fazit

2023-03-31T01:30:00+02:00

Achtung: Die Informationen in diesem Artikel beziehen sich auf das Verfahren im Kanton Basel-Stadt, die Verfahren in anderen Kantonen weichen z.T. stark ab.

Persönliche Eindrücke

Im Grossen und Ganzen habe ich das Einbürgerungsverfahren als deutlich «harmloser» empfunden, als ich es erwartet hatte. Insbesondere beim Gespräch mit der Einbürgerungskommission hatte ich zum einen schwierigere Wissensfragen, und zum anderen mehr Fragen zum persönlichen Leben erwartet. Diese Erwartung war vor allem von diversen Berichten geprägt, die in den letzten Jahren durch die Medien gegangen sind, wie z.B. das international bekannt gewordene Einbürgerungsverfahren von Fulda Yilmaz aus Buchs AG.

Allerdings bin ich mir auch bewusst, dass ich die Einbürgerung quasi «auf einfach durchgespielt» habe: Da ich Deutsch als Muttersprache spreche, musste ich keinen Sprachtest ablegen, und als weisser Mann blieben mir auch Begegnungen mit den Behörden erspart, von denen man in anderen Berichten liest, und die z.T. offen diskriminierend sind.

Als ich angefangen habe, mich über das Einbürgerungsverfahren zu informieren, waren anfangs noch sehr viele Punkte unklar - insbesondere, welche Dokumente einzureichen sind. Das lag zwar unter anderem daran, dass z.B. die Checkliste der einzureichenden Dokumente nicht online verfügbar war (siehe Teil 2), aber vor allem auch dran, dass je nach Staatsbürgerschaft und Aufenthaltssituation unterschiedliche Unterlagen benötigt werden. Die Leute in den Behörden waren allerdings allesamt sehr hilfsbereit, und konnten alle Fragen beantworten.

Timeline

12.07.2021: Beratung vom Migrationsamt via E-Mail
17.09.2021: Einreichung des Einbürgerungsgesuchs auf dem Postweg
24.09.2021: Eingangsbestätigung vom kantonalen Migrationsamt
24.02.2022: Einladung zum Erhebungsgespräch
01.03.2022: Telefonisches Erhebungsgespräch
08.03.2022: Übergabe des Gesuchs vom Kanton an die Bürgergemeinde Basel
03.06.2022 – 17.06.2022: «Einbürgerungskurs» bei der Bürgergemeinde
29.07.2022: Einladung zum Einbürgerungsgespräch
26.08.2022: Einbürgerungsgespräch
27.08.2022: Schriftlicher Einbürgerungsbescheid der Bürgergemeinde Basel
26.10.2022: Übergabe des Gesuchs vom der Bürgergemeinde Basel an den Bund
31.01.2023: Erwerb des Bürgerrechts
17.02.2023: Schriftliche Information des Migrationsamts über den Erwerb des Bürgerrechts

Kostenaufstellung

Kostenpunkt	Betrag (CHF)
Bestellung von diversen Dokumenten	~ 200
Verfahrenskosten Kanton Basel-Stadt	600
Verfahrenskosten Bürgergemeinde Basel	950
Verfahrenskosten Bund	100
Einbürgerungskurs	120
---	---
Total	1'970

Mein Einbürgerungverfahren in Basel - Teil 6: Abschluss des Verfahrens

2023-03-31T00:30:00+02:00

Gegen Ende Februar habe ich vom Migrationsamt des Kantons Basel-Stadt einen doch eher unzeremoniellen Brief bekommen, der mich darüber informierte, dass mir am 31. Januar 2023 das Bürgerrecht der Stadt Basel, des Kantons Basel-Stadt und der Schweiz verliehen wurde.

Damit ist das Einbürgerungsverfahren an sich nun abgeschlossen; als «frischgebackener» Schweizerbürger bin ich nun unter anderem stimm- und wahlberechtigt. An den kantonalen Abstimmungen vom 12. März konnte ich aber leider noch nicht teilnehmen.

Der Brief vom Migrationsamt enthielt auch Informationen über das weitere Vorgehen: Ich könne nun eine Identitätskarte und Schweizerpass beantragen, und zudem würde ich demnächst noch eine Einladung der Bürgergemeinde Basel zur Überreichung des Bürgerbriefs erhalten.

Bestellung von ID und Pass

Den Termin zur Beantragung von ID («Identitätskarte») und/oder Pass kann unter http://www.schweizerpass.ch/ online gebucht werden. Und ja, das http://www. ist wichtig! Der Link ist nur mit www-Subdomain und ohne HTTPS erreichbar. Dabei handelt es sich allerdings nur um eine Weiterleitung auf die Website des fedpol, und diese kann auch HTTPS.

Nach der Eingabe der benötigten Daten erhielt ich dann ein E-Mail vom kantonalen Passamt mit der Aufforderung, einen Termin zur persönlichen Vorsprache und Aufnahme der biometrischen Daten (Passfoto und Fingerabdrücke) auszuwählen. Den frühesten Termin gab es allerdings erst Mitte März. Das E-Mail informierte mich auch direkt darüber, dass ich zum Termin lediglich meinen bisherigen Ausländerausweis mitbringen müsse.

Bereits zwei Tage nach dem Termin bekam ich ID und Pass in der Post.

Die Sache mit der Wehrpflicht

Am gleichen Tag bekam ich auch Post von der kantonalen Militärbehörde, mit der Information, dass ich durch die Einbürgerung militärdienstpflichtig geworden bin. Da ich aber bereits über 25 Jahre alt bin, muss ich stattdessen Zivilschutzdienst leisten, sowie eine einkommensbemessene (durch den Zivilschutzdienst reduzierte) Wehrpflichtersatzabgabe zahlen. Dazu habe ich gleich noch die Information bekommen, dass ich bereits im August zur Zivilschutz-Rekrutierung aufgeboten werde.

Migration from LDAP to OIDC: Matrix

2023-01-27T01:25:00+01:00

In my hackerspace we operate various services for our members. Up until this month, most of these services used to do user authentication against a LDAP server. For a multitude of reasons, we replaced the LDAP server with an OpenID Connect SSO using a Keycloak server as the OIDC Identity Provider.

In this series of articles I summarize the efforts required to migrate each of the services from LDAP to OIDC authentication. This article covers the setup and migration in the Matrix homeserver Synapse.

OIDC Setup in Synapse

Adding an OIDC provider in Synapse is accomplished by adding its definition to the homeserver.yaml configuration file:

oidc_providers:
  - idp_id: "oidc"
    idp_name: "Human Readable Name"
    issuer: "https://sso.example.org/realms/example/"
    client_id: "oidc-client-id-matrix"
    client_secret: "oidc-client-secret-matrix"
    scopes: ["openid", "profile"]
    allow_existing_users: true
    user_mapping_provider:
      config:
        localpart_template: "{{ user.preferred_username }}"
        confirm_localpart: true

The crucial setting here is allow_existing_users: true; this is what enables migration from LDAP with OIDC. Synapse stores all users in the same database, and providers such as LDAP or OIDC are primarily used for authentication and initial account creation. This makes the migration in Synapse pretty straight-forward: as long as the LDAP provider and OIDC provider maps the user to the same Matrix ID localpart (the s3lph in @s3lph:kabelsalat.ch), they map to the same user and can be used interchangeably.

How the localpart is derived from the OIDC id_token is configured in the localpart_template setting. This setting takes a Jinja2 template where the token claims are available in the user object. So the example above uses the preferred_username token claim.

Even though I have not tested this, I'm fairly confident that a migration should work even when the localpart_template does not match the existing users' localparts. However, then you'd have to fill the user_external_ids database table with the mappings between OIDC subject identifiers (the sub token claim) and the full Matrix ID (@localpart:example.org). This table is already populated each time a user signs in via OIDC:

synapse=# select * from user_external_ids;
 auth_provider |             external_id              |       user_id
---------------+--------------------------------------+----------------------
 oidc          | ee5ff004-07c7-4c8f-b609-298aa1b5cd88 | @s3lph:kabelsalat.ch

You can also allow new users on your homeserver to choose a different username when they sign in via OIDC for the first time. This is controlled by the confirm_localpart setting in the example above. This too is controlled by entries in the user_external_ids table.

This architecture of only using LDAP and OIDC as an authentication providers and decoupling them from the actual users in this way, alongside with the allow_existing_users OIDC provider setting made Synapse the easiest service to migrate. If you can even call it "migration", that is - you simply add the OIDC provider and later remove the LDAP provider and you're done.

Limitations

Unfortunately SSO handling in Matrix heavily depends on the specific clients your users are using, with greatly varying behaviors:

Some clients don't support SSO at all. Check the client feature matrix to see which clients support SSO login.
Some clients only support SSO if the default password authentication flow, m.login.password, is disabled. However, if you need password login for some local users, disabling it may not be an option for you.
However, most clients I tested properly implement the m.login.sso authentication flow and give the user the option to sign in via SSO.

You can disable the m.login.password flow if you don't need it by adding the following to your homeserver.yaml:

password_config:
  enabled: false

Migration from LDAP to OIDC: Nextcloud

2023-01-26T22:00:00+01:00

In this series of articles I summarize the efforts required to migrate each of the services from LDAP to OIDC authentication. This article covers the setup and migration in Nextcloud.

OIDC Setup in Nextcloud

While LDAP integration is part of the Nextcloud core, OIDC integration must be installed as an additional app, user_oidc. Once this app is installed and enabled (occ app:enable user_oidc), you'll find a new tab in Nextcloud's admin settings called "OpenID Connect", where you can add your Identity Provider.

When adding a new IdP, you need to pay attention to choose an user ID claim that matches the user IDs from LDAP: If you used the uid or sAMAccountName attribute in the LDAP backend, chances are you can simply use the preferred_username claim in the OIDC backend and call it a day. (Please test whether these indeed match for all users!) However, if you used e.g. the objectGUID LDAP attribute, you need to make sure that the same attribute is available as an OIDC token claim.

Finally, remove the Use unique user id check mark, otherwise the OIDC backend generates its own user IDs anyway (by using a hash over the OIDC username and the name of the IdP).

Migration of Users From LDAP to OIDC

Users in Nextcloud are not managed in one central database table. Instead, each user backend maintains its own set of users and mapping to Nextcloud user IDs, usually each using its own database table, such as:

oc_users: Users created directly in Nextcloud.
oc_ldap_user_mapping: Users maintained by the user_ldap backend and the mapping from LDAP DNs to Nextcloud usernames.
- Groups and group memberships synced from LDAP are stored in oc_ldap_group_mapping and oc_ldap_group_members.
oc_user_oidc: Users maintained by the user_oidc backend.

This is why, as soon as the user_ldap app is disabled, all users and groups that were synced from LDAP disappear immediately. Don't worry - the users aren't gone. If you re-enable the user_ldap app, they automagically reappear. But the LDAP user mappings need to be migrated to the oc_user_oidc table so that the users are still around when the LDAP backend is disabled. For this we first need to understand the table schemas:

MariaDB [nextcloud]> select * from oc_ldap_user_mapping;
+--------------------------------------+---------------+----------------+------------------------------------------------------------------+
| ldap_dn                              | owncloud_name | directory_uuid | ldap_dn_hash                                                     |
+--------------------------------------+---------------+----------------+------------------------------------------------------------------+
| uid=s3lph,cn=users,dc=example,dc=org | s3lph         | s3lph          | faea045b8b1be34c7e02fa0771f3a2d6b7cbad4297a68c3c3fc1c39ff3cd2988 |
| uid=s4lph,cn=users,dc=example,dc=org | s4lph         | s4lph          | 14df2aecf103280c8fd693b77959f7fdab4c530fce9ebc35575d9b763fd1c5e6 |
| uid=s5lph,cn=users,dc=example,dc=org | s5lph         | s5lph          | a5938499eb735d4cdb2aebe641a155aa930bb0962003e595ce960eb2a956d10d |
+--------------------------------------+---------------+----------------+------------------------------------------------------------------+

MariaDB [nextcloud]> select * from oc_user_oidc;
+----+-----------+--------------+
| id | user_id   | display_name |
+----+-----------+--------------+
|  2 | s3lph     | s3lph        |
+----+-----------+--------------+

In the oc_ldap_user_mapping table, we have a mapping of DNs (ldap_dn) to usernames in Nextcloud (owncloud_name). The key component here is the owncloud_name column. This is the Nextcloud internal name, and also the name that must be provided by the OIDC backend (i.e. it must be present as a claim in the id_token, and that claim must be chosen as the "User ID mapping" attribute in the OIDC provider settings in Nextcloud. The oc_user_oidc table, on the other hand, does not store any mappings between OIDC and Nextcloud IDs. Instead it only contains the user ID and display name as provided by the OIDC provider.

The user migration from LDAP to OIDC can then be performed with a single SQL statement:

INSERT IGNORE INTO oc_user_oidc (user_id, display_name) SELECT owncloud_name, owncloud_name FROM oc_ldap_user_mapping;

Though you may want to keep the display name in mind. With our Nextcloud instance, we used the username as display name as well, so we simply used the same value for both columns. I don't think the LDAP display names are stored in the database, so if you have different display names, this simple insert statement may not work as well for you.

Limitations

Synchronization of user groups from your OIDC IdP to Nextcloud is not possible yet, but there's an open pull request that implements this feature. This means that until this is merged and released, group memberships that have previously been synced from LDAP won't be updated automatically anymore. Furthermore, if/when you disable the user_ldap app, any group that was synced from LDAP will disappear, and has to be recreated.

Another limitation is that OIDC users are only created in Nextcloud once they sign in for the first time. This means that you e.g. can't share files with users that have never signed in before. If your IdP has an automatable impersonation feature, you could work around this issue with a workflow such as:

Make sure the option "Check Bearer token on API and WebDav requests" is enabled in Nextcloud's OIDC settings.
Use the IdP's impersonation API to obtain an id_token for the new user.
Use this id_token to perform a simple request to Nextcloud, e.g. a WebDAV PROPFIND.
Nextcloud automatically creates the new user.

Tips and Tricks

Nextcloud does not automatically redirect to the IdP for login - users always have to click the "Login with SSO" button. If you want your users to be redirected to the SSO automatically when they try to log in, you have to configure this in your web server. For example, in Apache 2:

RewriteEngine on
RewriteCond %{QUERY_STRING} !local=1
RewriteCond %{REQUEST_METHOD} GET
RewriteRule ^/login$ /apps/user_oidc/login/1 [L,R,QSD]

Replace the number at the end of the URL (/1) with the ID of the OIDC provider in Nextcloud. You can obtain this ID with occ user_oidc:provider. The RewriteConds in this config allow you to append ?local=1 to disable the redirect, e.g. if you need to sign in as a local admin user.

Migration from LDAP to OIDC: Forgejo

2023-01-26T10:18:00+01:00

In this series of articles I summarize the efforts required to migrate each of the services from LDAP to OIDC authentication. This article covers the setup and migration in Forgejo.

OIDC Setup in Forgejo

Forgejo is a fork of the Gitea Git hosting platform that has emerged recently as a result of the takeover of the Gitea project by for-profit Gitea Ltd. Since both projects are still mostly identical at the time of writing, this section should apply 1:1 to Gitea as well.

To get started, sign into Forgejo as an admin user and create a new authentication source in the settings:

Choose OAuth2 as the authentication type and OpenID Connect as OAuth2 provider.
Fill in the Client ID and Client Secret as well as the auto-discovery URL.
Use at least openid profile email as the scope, and expand as necessary.

Once the authentication source is created and enabled, users can start signing in via OIDC. If a user has already signed in before using their LDAP password, they will be prompted for said password in order to "link" their OIDC user to the existing LDAP user. Whether users actually do this or not does not affect the migration.

Migration of Users From LDAP to OIDC

To understand the next migration steps, we first need to get an overview of the different types of names used to identify users in Forgejo:

The username is the name users are known as within Forgejo. You can consider this name to be a primary key (though not exactly true at the database schema level).
The display name is the name shown in the web UI.
The authentication sign-in name is the name provided by the authentication source.

This decoupling of the username and authentication name simplifies the migration a lot. With an LDAP authentication source, the authentication name is configurable - usually you'd choose an LDAP attribute such as uid or sAMAccoutName or (if you are really concerned about all-time uniqueness) objectGUID.

For the OIDC authentication source, the authentication name is not configurable - Forgejo insists on using the OIDC Subject Identifier (the sub key in the id_token). Keycloak generates an UUID for the subject identifier. Fortunately, Forgejo is smart enough to also consider the preferred_username token claim, and maps it to the Forgejo username - and prompts an already existing user for linking the account, as mentioned above.

However, since we wanted to disable LDAP authentication, this account linking would not have worked, so we needed to change the mapping for all users. For this, you need to obtain the subject ids and usernames of all users in Keycloak - this is most easily done by asking Keycloak's database (using PostgreSQL in this case):

$ sudo -u postgres psql keycloak
keycloak=# SELECT id, username FROM user_entity;
                  id                  | username
--------------------------------------+----------
 7ac0e93b-8725-4efa-95dc-0e20c6dbfa0a | s3lph
 8933480e-146f-4e7c-ac9c-9c8adfd69069 | s4lph
 d398d274-7d9a-45b3-96e7-96b8811afa72 | s5lph

If there's only a small number of users in your Forgejo instance, you could now edit every single one in the admin settings, and change its authentication source to the OIDC provider and the login name to this subject identifier. The more scalable approach, however, would be to transform this list into a series of UPDATE statements for Forgejo's database (in our case, MariaDB is used here):

BEGIN;
UPDATE user SET login_name = '7ac0e93b-8725-4efa-95dc-0e20c6dbfa0a', login_type = 6, login_source = 4 where lower_name = 's3lph';
UPDATE user SET login_name = '8933480e-146f-4e7c-ac9c-9c8adfd69069', login_type = 6, login_source = 4 where lower_name = 's4lph';
UPDATE user SET login_name = 'd398d274-7d9a-45b3-96e7-96b8811afa72', login_type = 6, login_source = 4 where lower_name = 's5lph';
COMMIT;

Note the additional columns login_type and login_source we need to modify:

login_type is the type of login source being used. This is 6 for all OAuth sources. The list of type identifiers can be found in the Forgejo source code.
login_source is the numeric id of the OIDC login source you created in Forgejo. You can see this ID in the list of authentication sources in the Forgejo admin settings.

Finally, if you configured Forgejo to import SSH public keys from LDAP, you need to remove the mapping from SSH keys to the login source they came from. Otherwise, users with such keys won't be able to open their SSH key settings anymore:

UPDATE public_key SET login_source_id = 0 WHERE login_source_id = 1;

Replace the login source ID in the WHERE clause with the ID of your LDAP authentiation source (again, to be found in the Forgejo admin settings).

This concludes the OIDC migration of Forgejo, and the LDAP authentication source can now be safely disabled or removed.

Tips and Tricks

If user self-registration is disabled on your Forgejo instance, you should add the following to your app.ini so that new OIDC users can sign into Forgejo without admin intervention:

[oauth2_client]
ENABLE_AUTO_REGISTRATION=true

Forgejo does not automatically redirect to the IdP for login - users always have to click the "Login with OpenID" button. If you want your users to be redirected to the SSO automatically when they try to log in, you have to configure this in your reverse proxy. For example, in Apache 2:

RewriteEngine on
RewriteCond %{QUERY_STRING} !local=1
RewriteCond %{REQUEST_METHOD} GET
RewriteRule ^/user/login$ /user/oauth2/oidc [L,R,QSD]

Replace the last URL component (/oidc) with the name you gave to your OIDC authentication source in Forgejo. The RewriteConds in this config allow you to append ?local=1 to disable the redirect, e.g. if you need to sign in as a local admin user.

How much ActivityPub can a Static Site Generator implement?

2022-11-18T01:00:00+01:00

There already are multiple blogging solutions which are part of the Fediverse. Among them are dedicated Fediverse blogs such as Plume or WriteFreely, but there also are plugins which retrofit existing CMS, e.g. for Wordpress or Drupal.

ActivityPub, the network protocol behind the Fediverse can only be fully implemented by means of an active server component: Among other things, incoming messages delivered to inboxes have to be processed. Sometimes they need to be forwarded, and outgoing messages need to be signed.

Figure 1: The basic mechanisms behind ActivityPub: Users publish resources in their outbox, and retrieve received resources from their inbox.

Nevertheless I wanted to figure out, which parts of the ActivityPub protocol can be implemented in a purely static website, and how well other servers in the Fediverse interact with it. My goal was to attach this blog to the Fediverse. The blog is generated using the static site generator software Pelican.

Metadata Endpoints

One prerequisite for implementing ActivityPub are multiple static metadata endpoints, with which a server signals whether it supports ActivityPub, and under which HTTP endpoints the various ActivityPub resources are reachable:

/.well-known/nodeinfo simply links to the "real" nodeinfo endpoint:

{
  "links": [
    {
      "href": "https://s3lph.me/activitypub/nodeinfo",
      "rel": "http://nodeinfo.diaspora.software/ns/schema/2.0"
    }
  ]
}

Behind the linked path (which can be chosen freely), the global metadata of this ActivityPub instance is published:

{
  "version": "2.0",
  "software": {
    "name": "pelican-activitypub",
    "version": "0.1"
  },
  "protocols": [
    "activitypub"
  ],
  "services": {
    "inbound": [],
    "outbound": [
      "atom1.0",
      "rss2.0"
    ]
  },
  "openRegistrations": false,
  "usage": {
    "users": {
      "total": 1
    },
    "localPosts": 27
  },
  "metadata": {
    "nodeName": "s3lph made"
  }
}

This JSON document describes the server, however we still need to discover the individual users of this instance. This is where the webfinger endpoint comes in. Usually it is found at /.well-known/webfinger, but some pieces of software insist on first resolving this path using the /.well-known/host-meta endpoint:

<?xml version="1.0" encoding="UTF-8"?>
<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
  <Link rel="lrdd" template="https://s3lph.me/.well-known/webfinger?resource={uri}" type="application/xrd+xml" />
</XRD>

Now we already stumbled over the fist obstacle preventing a purely static implementation: The name of the user to be resolved is passed as an URL parameter, which needs to be handled by a HTTP server.

There are two possible workarounds:

If there only is a single user on the server, the parameter can be ignored. Instead, always the same static response is returned, no matter the URL parameter.
If there are multiple users, we can generate a static webfinger endpoint per user. This requires the webserver to be configured to redirect the regular webfinger endpoint to these user specific endpoints.

For example, using the Apache webserver, this can be achieved like this:

RewriteEngine on
RewriteRule ^/.well-known/webfinger?resource=acct:([^@]+)@s3lph.me$ /.well-known/_webfinger/$1 [L]

The webfinger endpoint links to the actual user resources:

{
  "subject": "acct:s3lph@s3lph.me",
  "aliases": [
    "https://s3lph.me/author/s3lph.html",
    "https://s3lph.me/activitypub/users/s3lph"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://s3lph.me/author/s3lph.html"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://s3lph.me/activitypub/users/s3lph"
    }
  ]
}

In this case, the author @s3lph@s3lph.me is resolved to two alias URLs: The author's feed in the blog, as well as the ActivityPub Person resource.

ActivityPub: Persons and Articles

Now that we can resolve usernames such as @s3lph@s3lph.me to ActivityPub URLs such as https://s3lph.me/activitypub/users/s3lph, we can take a closer look at the Person resource behind this URL:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "schema": "http://schema.org#",
      "toot": "http://joinmastodon.org/ns#",
      "PropertyValue": "schema:PropertyValue",
      "value": "schema:value",
      "alsoKnownAs": {
        "@id": "as:alsoKnownAs",
        "@type": "@id"
      },
      "movedTo": {
        "@id": "as:movedTo",
        "@type": "@id"
      },
      "discoverable": "toot:discoverable"
    }
  ],
  "type": "Person",
  "id": "https://s3lph.me/activitypub/users/s3lph",
  "preferredUsername": "s3lph",
  "url": "https://s3lph.me/author/s3lph.html",
  "name": "s3lph made",
  "summary": "This is an EXPERIMENTAL implementation for a read-only ActivityPub feed of my blog.",
  "icon": {
    "type": "Image",
    "mediaType": "image/png",
    "url": "https://s3lph.me/favicon.ico"
  },
  "image": {},
  "tag": [],
  "attachment": [
    {
      "type": "PropertyValue",
      "name": "Web",
      "value": "<a href=\"https://s3lph.me\">s3lph.me</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Mastodon",
      "value": "<a rel=\"me\" href=\"https://chaos.social/@s3lph\">@s3lph@chaos.social</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Matrix",
      "value": "<a rel=\"me\" href=\"https://mto.kabelsalat.ch/#/@s3lph:kabelsalat.ch\">@s3lph:kabelsalat.ch</a>"
    }
  ],
  "movedTo": "https://chaos.social/users/s3lph",
  "alsoKnownAs": [
    "https://chaos.social/users/s3lph"
  ],
  "inbox": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "outbox": "https://s3lph.me/activitypub/collections/outbox/s3lph",
  "following": "https://s3lph.me/activitypub/collections/following/s3lph",
  "followers": "https://s3lph.me/activitypub/collections/followers/s3lph",
  "discoverable": true,
  "manuallyApprovesFollowers": true,
  "published": "2020-02-05T01:36:00+01:00",
  "updated": "2022-11-12T14:05:23Z",
  "endpoints": {
    "sharedInbox": "https://s3lph.me/activitypub/collections/inbox/s3lph"
  },
}

This resource is a bit bigger than before, so let's look at it piece by piece. At first the schema of this JSON document is described:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "schema": "http://schema.org#",
      "toot": "http://joinmastodon.org/ns#",
      "PropertyValue": "schema:PropertyValue",
      "value": "schema:value",
      "alsoKnownAs": {
        "@id": "as:alsoKnownAs",
        "@type": "@id"
      },
      "movedTo": {
        "@id": "as:movedTo",
        "@type": "@id"
      },
      "discoverable": "toot:discoverable"
    }
  ],

Next the basic information about the person are described. These are used by ActivityPub-Clients to display the user's profile page:

  "type": "Person",
  "id": "https://s3lph.me/activitypub/users/s3lph",
  "preferredUsername": "s3lph",
  "url": "https://s3lph.me/author/s3lph.html",
  "name": "s3lph made",
  "summary": "This is an EXPERIMENTAL implementation for a read-only ActivityPub feed of my blog...",
  "icon": {
    "type": "Image",
    "mediaType": "image/png",
    "url": "https://s3lph.me/favicon.ico"
  },
  "image": {},

Some additional metadata can be provided:

  "tag": [],
  "attachment": [
    {
      "type": "PropertyValue",
      "name": "Web",
      "value": "<a href=\"https://s3lph.me\">s3lph.me</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Mastodon",
      "value": "<a rel=\"me\" href=\"https://chaos.social/@s3lph\">@s3lph@chaos.social</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Matrix",
      "value": "<a rel=\"me\" href=\"https://mto.kabelsalat.ch/#/@s3lph:kabelsalat.ch\">@s3lph:kabelsalat.ch</a>"
    }
  ],
  "movedTo": "https://chaos.social/users/s3lph",
  "alsoKnownAs": [
    "https://chaos.social/users/s3lph"
  ],

"PropertyValue" attachments are rendered as tables on the user's profile page by most ActivityPub clients. tags list attributes such as hashtags and mentions of other users within the profile's description, so that they can be easily indexed by other servers in the Fediverse.

movedTo and alsoKnownAs are used to indicate that the account has moved, and the new account should be followed instead.

With all of this, we get a user profile that can be viewed on Fediverse client applications, such as the Mastodon app "Tusky":

Figure 2: The profile of @s3lph@s3lph.me, as shown in the Mastodon app Tusky

Finally, the URLs to linked ActivityPub resources have to be provided:

  "inbox": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "outbox": "https://s3lph.me/activitypub/collections/outbox/s3lph",
  "following": "https://s3lph.me/activitypub/collections/following/s3lph",
  "followers": "https://s3lph.me/activitypub/collections/followers/s3lph",

I've already mentioned the relevance of the inbox and outbox before. Since this implementation does not process inboxes, there only is an empty colllection (list of ActivityPub resources) behind the inbox URL. The same goes for the "following" and "followers" collections:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams"
  ],
  "type": "OrderedCollection",
  "id": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "totalItems": 0,
  "orderedItems": []
}

The outbox is a collection as well, containing the articles published by this user. One of these articles (actually this one) looks like this:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams"
  ],
  "type": "Article",
  "id": "https://s3lph.me/activitypub/posts/activitypub-static-site",
  "published": "2022-11-17T02:00:00+01:00",
  "inReplyTo": null,
  "url": "https://s3lph.me/activitypub-static-site-de.html",
  "attributedTo": "https://s3lph.me/activitypub/users/s3lph",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://s3lph.me/activitypub/collections/followers/s3lph",
    "https://chaos.social/users/s3lph"
  ],
  "name": "How much ActivityPub can a Static Site Generator implement?",
  "nameMap": {
    "en": "How much ActivityPub can a Static Site Generator implement?",
    "de": "Wie viel ActivityPub kann ein Static Site Generator?"
  },
  "content": "<p>There already are multiple blogging solutions...",
  "contentMap": {
    "en": "<p>There already are multiple blogging solutions...",
    "de": "<p>Es gibt bereits verschiedene Blog-Lösungen...",
  },
  "summary": null,
  "attachment": [],
  "tag": [
    {
      "type": "Hashtag",
      "name": "#activitypub",
      "href": "https://s3lph.me/activitypub/tags/activitypub"
    },
    {
      "type": "Hashtag",
      "name": "#pelican",
      "href": "https://s3lph.me/activitypub/tags/pelican"
    },
    {
      "type": "Mention",
      "href": "https://chaos.social/users/s3lph",
      "name": "@s3lph@chaos.social"
    }
  ]
}

The beginning of an article is more or less equal to that of a person, so I won't be repeating it here.

Next the relationships to other resources are described, e.g. who wrote the article, and to whom it is addressed. The special URL https://www.w3.org/ns/activitystreams#Public describes that the article is public, and should e.g. be listed in global timelines:

  "inReplyTo": null,
  "url": "https://s3lph.me/activitypub-static-site-de.html",
  "attributedTo": "https://s3lph.me/activitypub/users/s3lph",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://s3lph.me/activitypub/collections/followers/s3lph",
    "https://chaos.social/users/s3lph"
  ],

Title and content of the article can be multilingual. However, the contentMap und nameMap attributes are only supported by a few ActivityPub servers or clients. Most others simply always show the untranslated default content:

  "name": "How much ActivityPub can a Static Site Generator implement?",
  "nameMap": {
    "en": "How much ActivityPub can a Static Site Generator implement?",
    "de": "Wie viel ActivityPub kann ein Static Site Generator?"
  },
  "content": "<p>There already are multiple blogging solutions...",
  "contentMap": {
    "en": "<p>There already are multiple blogging solutions...",
    "de": "<p>Es gibt bereits verschiedene Blog-Lösungen...",
  },
  "summary": null,

Finally, same as with persons, additional data like tags and mentions of other users are listed in a machine readable form:

  "attachment": [],
  "tag": [
    {
      "type": "Hashtag",
      "name": "#activitypub",
      "href": "https://s3lph.me/activitypub/tags/activitypub"
    },
    {
      "type": "Hashtag",
      "name": "#pelican",
      "href": "https://s3lph.me/activitypub/tags/pelican"
    },
    {
      "type": "Mention",
      "href": "https://chaos.social/users/s3lph",
      "name": "@s3lph@chaos.social"
    }
  ]

Sow now we have modelled authors and their articles as ActivityPub resources. This is also everything which can be reasonably implemented as a purely static site.

The code for generating these ActivityPub resources is available as a Pelican plugin. However, before you go ahead and install this on your own Pelican blog, I'd advise you to finish reading this article.

Compatibility to Fediverse Services

For testing my implementation, I set up three different Fediverse instances to interact with my blog: Mastodon, Pleroma and Misskey.

All three services are able to retrieve and show the profile @s3lph@s3lph.me, and the number of blog articles is shown. However, the articles themselves are not shown. As it turns out, articles from other instances are usually not retrieved automatically.

The articles would only be shown if they were POSTed from their origin instance to the target instance. Alternatively, all three test instances were able to show the articles by pasting the article's URL into the search bar. Afterwards these articles were also shown in the timeline of their author.

Unfortunately, this is almost everything that can be achieved with a purely static implementation. Especially the following important functions are not available:

Following: If Alice wants to follow the Fediverse account of Bob, she has to send a follow request to Bob. Bob (or his instance) has to confirm this request before it becomes effective.
Replies: If Alice wants to reply to a message of Bob, her response is only visible on Alice's instance at first. The response will also be posted to the inbox of Bob, and it would be in his instance's responsibility to forward the reply to all other involved instances. This crucial step is not possible with a static ActivityPub endpoint.
Deletion: If an article ha been cached by another instance, it is usually kept in the cache until it's explicitly deleted by its origin instance. For this, a cryptographically signed deletion request would have to be submitted to the instance's inbox.
Updates: The same limitations that apply to deletions apply to updates as well. In fact, this list item will never show up on instances that already have an earlier version of this article in their cache.

What does work are likes and boosts, tough they do not increment the counters under an article.

Additionally, it turns out that e.g. Mastodon only shows a link preview to the article, since the Article object type is not fully supported; the Note type is used for short messages. However, both Pleroma and Misskey show the full article natively.

Conclusion

The question "How much ActivityPub can a Static Site Generator implement?" can thus be best answered as "quite a lot, but not enough to be relevant in practice." Even for this blog, the purely static ActivityPub implementation will most likely not have any relevance, however the articles will presumably keep being published on the Fediverse in this limited form.

PS: If you're reading this article on the Fediverse and want to write a reply, please (additionally) address it to @s3lph@chaos.social, otherwise I won't see it.

Wie viel ActivityPub kann ein Static Site Generator?

2022-11-18T01:00:00+01:00

Es gibt bereits verschiedene Blog-Lösungen, die Teil des Fediverse sind: Darunter sind sowohl dedizierte Fediverse-Blogs wie Plume und WriteFreely, aber es gibt auch Plugins, die bestehende CMS nachrüsten, z.B. für Wordpress oder für Drupal.

ActivityPub, das Netzwerk-Protokoll hinter dem Fediverse, lässt sich nur mit einer aktiven Serverkomponente vollständig umsetzen: Unter anderem muss auf eingehende Nachrichten in den Inboxen reagiert werden, teilweise müssen diese auch an andere Server weitergeleitet werden, und ausgehende Nachrichten müssen zeitnah signiert werden.

Abbildung 1: Die grundsätzlichen Mechanismen hinter ActivityPub: User veröffentlichen Ressourcen in ihrer eigenen Outbox, und rufen empfangene Ressourcen aus ihrer Inbox ab.

Trotzdem wollte ich herausfinden, welche Teile des ActivityPub-Protokolls mit einer rein statischen Website implementiert werden können, und wie gut andere Server im Fediverse damit umgehen können. Mein Ziel war, meinen Blog hier ans Fediverse zu hängen. Der Blog wird mit der Static Site Generator-Software Pelican erzeugt.

Metadaten-Endpunkte

Eine Grundvoraussetzung, um ActivityPub zu implementieren, sind diverse statische Metadaten-Endpunkte, mit denen der Server signalisiert, dass er ActivityPub unterstützt, und unter welchen HTTP-Endpunkten die verschiedenen ActivityPub-Ressourcen zu finden sind:

/.well-known/nodeinfo verlinkt einfach nur auf den "echten" nodeinfo-Endpunkt:

{
  "links": [
    {
      "href": "https://s3lph.me/activitypub/nodeinfo",
      "rel": "http://nodeinfo.diaspora.software/ns/schema/2.0"
    }
  ]
}

Unter dem verlinkten Pfad (der frei gewählt werden kann), werden die globalen Metadaten der ActivityPub-Instanz publiziert:

{
  "version": "2.0",
  "software": {
    "name": "pelican-activitypub",
    "version": "0.1"
  },
  "protocols": [
    "activitypub"
  ],
  "services": {
    "inbound": [],
    "outbound": [
      "atom1.0",
      "rss2.0"
    ]
  },
  "openRegistrations": false,
  "usage": {
    "users": {
      "total": 1
    },
    "localPosts": 27
  },
  "metadata": {
    "nodeName": "s3lph made"
  }
}

Mit diesem JSON-Dokument wird der Server an sich beschrieben, nun müssen aber noch die einzelnen User der Instanz gefunden werden. Hierzu dient der webfinger-Endpunkt. Dieser ist zwar üblicherweise unter /.well-known/webfinger zu finden, aber einzelne Softwarelösungen bestehen trotzdem darauf, den Pfad erst unter einem anderen Endpunkt nachzuschlagen, und zwar unter /.well-known/host-meta:

<?xml version="1.0" encoding="UTF-8"?>
<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
  <Link rel="lrdd" template="https://s3lph.me/.well-known/webfinger?resource={uri}" type="application/xrd+xml" />
</XRD>

Hier sehen wir schon das erste Hindernis, das eine rein statische Implementation verhindert: Der Name des aufzulösenden Users wird als URL-Parameter übergeben, der durch einen HTTP-Server behandelt werden muss.

Hierzu gibt es zwei mögliche Lösungen:

Wenn es auf dem Server nur einen einzelnen User gibt, kann der Parameter eigentlich ignoriert werden. Stattdessen wird einfach immer die gleiche, statische Antwort zurückgegeben.
Wenn es mehrere User gibt, kann für jeden User ein statischer webfinger-Endpunkt generiert werden. Dies setzt aber voraus, dass der Webserver konfiguriert wird, den eigentlichen webfinger-Endpunkt entsprechend umgeleitet werd.

Dies lässt sich z.B. im Apache-Webserver so umsetzen:

RewriteEngine on
RewriteRule ^/.well-known/webfinger?resource=acct:([^@]+)@s3lph.me$ /.well-known/_webfinger/$1 [L]

Der webfinger-Endpunkt verlinkt nun weiter auf die tatsächlichen User-Ressourcen:

{
  "subject": "acct:s3lph@s3lph.me",
  "aliases": [
    "https://s3lph.me/author/s3lph.html",
    "https://s3lph.me/activitypub/users/s3lph"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://s3lph.me/author/s3lph.html"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://s3lph.me/activitypub/users/s3lph"
    }
  ]
}

In diesem Fall wird der Autor, in diesem Fall @s3lph@s3lph.me auf zwei Alias-URLs aufgelöst: Der Autoren-Feed im Blog, sowie die Person ActivityPub-Ressource.

ActivityPub: Personen und Artikel

Jetzt, wo wir einen Username wie @s3lph@s3lph.me zu einer ActivityPub-URL wie https://s3lph.me/activitypub/users/s3lph auflösen können, können wir die Personen-Ressource unter dieser URL genauer anschauen:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "schema": "http://schema.org#",
      "toot": "http://joinmastodon.org/ns#",
      "PropertyValue": "schema:PropertyValue",
      "value": "schema:value",
      "alsoKnownAs": {
        "@id": "as:alsoKnownAs",
        "@type": "@id"
      },
      "movedTo": {
        "@id": "as:movedTo",
        "@type": "@id"
      },
      "discoverable": "toot:discoverable"
    }
  ],
  "type": "Person",
  "id": "https://s3lph.me/activitypub/users/s3lph",
  "preferredUsername": "s3lph",
  "url": "https://s3lph.me/author/s3lph.html",
  "name": "s3lph made",
  "summary": "This is an EXPERIMENTAL implementation for a read-only ActivityPub feed of my blog.",
  "icon": {
    "type": "Image",
    "mediaType": "image/png",
    "url": "https://s3lph.me/favicon.ico"
  },
  "image": {},
  "tag": [],
  "attachment": [
    {
      "type": "PropertyValue",
      "name": "Web",
      "value": "<a href=\"https://s3lph.me\">s3lph.me</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Mastodon",
      "value": "<a rel=\"me\" href=\"https://chaos.social/@s3lph\">@s3lph@chaos.social</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Matrix",
      "value": "<a rel=\"me\" href=\"https://mto.kabelsalat.ch/#/@s3lph:kabelsalat.ch\">@s3lph:kabelsalat.ch</a>"
    }
  ],
  "movedTo": "https://chaos.social/users/s3lph",
  "alsoKnownAs": [
    "https://chaos.social/users/s3lph"
  ],
  "inbox": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "outbox": "https://s3lph.me/activitypub/collections/outbox/s3lph",
  "following": "https://s3lph.me/activitypub/collections/following/s3lph",
  "followers": "https://s3lph.me/activitypub/collections/followers/s3lph",
  "discoverable": true,
  "manuallyApprovesFollowers": true,
  "published": "2020-02-05T01:36:00+01:00",
  "updated": "2022-11-12T14:05:23Z",
  "endpoints": {
    "sharedInbox": "https://s3lph.me/activitypub/collections/inbox/s3lph"
  },
}

Diese Ressource ist schon ein gutes Stück grösser, daher schauen wir das am besten Stück für Stück an. Zuerst wird das Schema des JSON-Dokuments beschrieben:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "schema": "http://schema.org#",
      "toot": "http://joinmastodon.org/ns#",
      "PropertyValue": "schema:PropertyValue",
      "value": "schema:value",
      "alsoKnownAs": {
        "@id": "as:alsoKnownAs",
        "@type": "@id"
      },
      "movedTo": {
        "@id": "as:movedTo",
        "@type": "@id"
      },
      "discoverable": "toot:discoverable"
    }
  ],

Als nächstes werden die grundlegenden Informationen über die Person beschrieben. Diese werden von ActivityPub-Clients benutzt, um die User-Seite darzustellen:

  "type": "Person",
  "id": "https://s3lph.me/activitypub/users/s3lph",
  "preferredUsername": "s3lph",
  "url": "https://s3lph.me/author/s3lph.html",
  "name": "s3lph made",
  "summary": "This is an EXPERIMENTAL implementation for a read-only ActivityPub feed of my blog...",
  "icon": {
    "type": "Image",
    "mediaType": "image/png",
    "url": "https://s3lph.me/favicon.ico"
  },
  "image": {},

Zusätzlich können noch einige weitere Metadaten angegeben werden:

  "tag": [],
  "attachment": [
    {
      "type": "PropertyValue",
      "name": "Web",
      "value": "<a href=\"https://s3lph.me\">s3lph.me</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Mastodon",
      "value": "<a rel=\"me\" href=\"https://chaos.social/@s3lph\">@s3lph@chaos.social</a>"
    },
    {
      "type": "PropertyValue",
      "name": "Matrix",
      "value": "<a rel=\"me\" href=\"https://mto.kabelsalat.ch/#/@s3lph:kabelsalat.ch\">@s3lph:kabelsalat.ch</a>"
    }
  ],
  "movedTo": "https://chaos.social/users/s3lph",
  "alsoKnownAs": [
    "https://chaos.social/users/s3lph"
  ],

Attachments vom Typ "PropertyValue" werden z.B. von den meisten ActivityPub-Clients als Tabelle im Profil des Users dargestellt. Unter tags werden z.B. Hashtags oder Erwähnungen von anderen Usern im Profil des Users aufgelistet, damit diese von anderen Servern im Fediverse indexiert werden.

Mit movedTo und alsoKnownAs wird angegeben, dass der Account umgezogen ist, und andere User dem neuen Profil folgen sollen.

Damit kommt ein Profil zustande, das mit Fediverse-Clients aufgerufen werden kann. So zum Beispiel in der Mastodon-App Tusky:

Abbildung 2: Das Profil von @s3lph@s3lph.me, dargestellt in der Mastodon-App Tusky.

Schlussendlich müssen noch die URLs zu den verknüpften ActivityPub-Ressourcen angegeben werden:

  "inbox": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "outbox": "https://s3lph.me/activitypub/collections/outbox/s3lph",
  "following": "https://s3lph.me/activitypub/collections/following/s3lph",
  "followers": "https://s3lph.me/activitypub/collections/followers/s3lph",

Die Bedeutung von Inbox und Outbox wurden vorhin schon kurz erwähnt. Da in dieser Implementation keine Inboxen verarbeitet werden, steckt hinter der Inbox einfach nur eine leere "Collection" (Liste aus ActivityPub-Ressourcen). Das gleiche gilt für die "following"- und "followers"-Collections:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams"
  ],
  "type": "OrderedCollection",
  "id": "https://s3lph.me/activitypub/collections/inbox/s3lph",
  "totalItems": 0,
  "orderedItems": []
}

Die Outbox ist ebenfalls eine Collection, die die von diesem User veröffentlichten Artikel enthält. Ein solcher (dieser) Artikel sieht z.B. so aus:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams"
  ],
  "type": "Article",
  "id": "https://s3lph.me/activitypub/posts/activitypub-static-site",
  "published": "2022-11-17T02:00:00+01:00",
  "inReplyTo": null,
  "url": "https://s3lph.me/activitypub-static-site-de.html",
  "attributedTo": "https://s3lph.me/activitypub/users/s3lph",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://s3lph.me/activitypub/collections/followers/s3lph",
    "https://chaos.social/users/s3lph"
  ],
  "name": "How much ActivityPub can a Static Site Generator implement?",
  "nameMap": {
    "en": "How much ActivityPub can a Static Site Generator implement?",
    "de": "Wie viel ActivityPub kann ein Static Site Generator?"
  },
  "content": "<p>There already are multiple blogging solutions...",
  "contentMap": {
    "en": "<p>There already are multiple blogging solutions...",
    "de": "<p>Es gibt bereits verschiedene Blog-Lösungen...",
  },
  "summary": null,
  "attachment": [],
  "tag": [
    {
      "type": "Hashtag",
      "name": "#activitypub",
      "href": "https://s3lph.me/activitypub/tags/activitypub"
    },
    {
      "type": "Hashtag",
      "name": "#pelican",
      "href": "https://s3lph.me/activitypub/tags/pelican"
    },
    {
      "type": "Mention",
      "href": "https://chaos.social/users/s3lph",
      "name": "@s3lph@chaos.social"
    }
  ]
}

Der Anfang eines Artikels sieht mehr oder weniger gleich aus wie bei einer Person, daher wiederhole ich das hier nicht nochmal.

Als nächstes wird die Beziehung zu anderen Ressourcen angegeben, z.B. wer den Artikel verfasst hat, und an wen der Artikel adressiert ist. Die spezielle URL https://www.w3.org/ns/activitystreams#Public beschreibt, dass der Artikel öffentlich ist und z.B. in globalen Timelines aufgeführt werden soll:

  "inReplyTo": null,
  "url": "https://s3lph.me/activitypub-static-site-de.html",
  "attributedTo": "https://s3lph.me/activitypub/users/s3lph",
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "cc": [
    "https://s3lph.me/activitypub/collections/followers/s3lph",
    "https://chaos.social/users/s3lph"
  ],

Titel und Inhalt des Artikels können mehrsprachig angegeben werden. Allerdings werden contentMap und nameMap nur von wenigen ActivityPub-Servern implementiert. Die meisten Server zeigen einfach immer den unübersetzten Standard-Inhalt an:

  "name": "How much ActivityPub can a Static Site Generator implement?",
  "nameMap": {
    "en": "How much ActivityPub can a Static Site Generator implement?",
    "de": "Wie viel ActivityPub kann ein Static Site Generator?"
  },
  "content": "<p>There already are multiple blogging solutions...",
  "contentMap": {
    "en": "<p>There already are multiple blogging solutions...",
    "de": "<p>Es gibt bereits verschiedene Blog-Lösungen...",
  },
  "summary": null,

Schlussendlich werden - genau wie bei Personen auch - weitere Daten wie Tags oder Erwähnungen anderer Personen in maschinenlesbar aufgeführt:

  "attachment": [],
  "tag": [
    {
      "type": "Hashtag",
      "name": "#activitypub",
      "href": "https://s3lph.me/activitypub/tags/activitypub"
    },
    {
      "type": "Hashtag",
      "name": "#pelican",
      "href": "https://s3lph.me/activitypub/tags/pelican"
    },
    {
      "type": "Mention",
      "href": "https://chaos.social/users/s3lph",
      "name": "@s3lph@chaos.social"
    }
  ]

Damit haben wir Autoren und deren Artikel vollständig als ActivityPub-Ressourcen abgebildet. Damit ist auch alles implementiert, was sich sinnvoll als rein statische Seite implementieren lässt.

Der Code, um diese ActivityPub-Ressourcen zu erzeugen, ist als Pelican-Plugin verfügbar. Bevor irgendjemand das Plugin in das eigene Pelican einbaut, würde ich aber dazu raten, diesen Artikel bis zum Ende zu lesen.

Kompatibilität mit Fediverse-Diensten

Zum Testen habe drei verschiedenen Fediverse-Testinstanzen aufgesetzt, um mit meinem Blog zu interagieren: Mastodon, Pleroma und Misskey.

Mit allen drei Diensten kann das Profil @s3lph@s3lph.me aufgerufen werden. Auch die Anzahl an Blogartikeln wird korrekt dargestellt, allerdings werden die Artikel selbst nicht angezeigt. Wie sich herausstellt, werden Artikel von anderen Instanzen üblicherweise nicht automatisch geladen.

Die Artikel würden nur dann angezeigt, wenn sie von ihrer Ursprungsinstanz in die Inbox der Zielinstanz gePOSTed werden. Alternativ war es aber bei allen drei Testinstanzen möglich, die URL des Artikels in der Suche einzugeben und so aufzurufen. Danach wird der jeweilige Artikel auch in der Timeline des Autors angezeigt.

Leider ist dies eigentlich auch schon alles, dass mit einer rein statischen Implementation wirklich funktioniert. Insbesondere die folgenden - doch recht zentralen - Funktionen sind nicht verfügbar:

Folgen: Wenn Alice im Fediverse dem Account vom Bob folgen möchte, sendet sie eine Follow Request in die Inbox von Bob. Bob (resp. Bob's Instanz) muss diese Anfrage allerdings erst bestätigen.
Antworten: Wenn Alice auf eine Nachricht von Bob antwortet, ist diese Antwort zunächst nur auf der Instanz von Alice sichtbar. Die Antwort wird auch in die Inbox von Bob zugestellt, und Bobs Instanz wäre dafür zuständig, die Antwort an alle anderen involvierten Instanzen weiterzuleiten. Dieser Schritt bleibt im statisch generierten Fall aber aus.
Löschen: Wenn ein Artikel im Cache einer anderen Instanz ist, wird er dort üblicherweise behalten, bis er von der Ursprungsinstanz explizit gelöscht wird. Hierzu muss eine signierte Löschanfrage an die Inbox der anderen Instanz gesendet werden.
Aktualisieren: Für Änderungen an Artikeln gelten die gleichen Einschränkungen wie für das Löschen von Artikeln. So wird dieses Listenelement niemals auf Instanzen auftauchen, die bereits eine ältere Version des Artikels in ihrem Cache haben.

Was hingegen funktioniert sind Likes und Boosts, allerdings werden diese nicht in den Zählern unter den Artikeln reflektiert.

Zudem stellt z.B. Mastodon nur einen Link zum Artikel dar, da die hier verwendeten Article-Objekte nicht vollständig unterstützt werden; für Kurznachrichten wird der Objecttyp Note verwendet. Sowohl Pleroma als auch Misskey können den Artikel aber vollständig nativ darstelle.

Fazit

Die Frage «Wie viel ActivityPub kann ein Static Site Generator?» lässt sich zusammenfassend wohl am besten beantworten mit «viel, aber nicht genug, um praktische Relevanz zu haben.» Auch für diesen Blog wird eine rein statische ActivityPub-Implementation wahrscheinlich keine wirkliche Relevanz haben, aber die Artikel werden voraussichtlich weiterhin in dieser eingeschränkten Form im Fediverse verfügbar sein.

PS: Falls du diesen Artikel im Fediverse liest, und eine Antwort schreiben willst, adressiere deine Antwort bitte (zusätzlich) an @s3lph@chaos.social, sonst bekomme ich davon nichts mit.

Automated Generation of POI Maps from Open Data

2022-10-27T02:00:00+02:00

The website of the Chaos Computer Club features a map of Germany and its surrounding countries, where hackerspaces who consider themselves as part of the CCC are marked.

Up until now, this map (shown in Figure 1) had been created and updated manually in a time consuming process. Not only did this cause the map to be almost always outdated, but the process was also error-prone, with many hackerspaces being misplaced.

Figure 1: The old, manually created map of hackerspaces, last updated 2018.

The desire to automate the creation of this map had been voiced multiple times over the last few years, but all previous attempts have never been finished. Since I was one of the last persons asking for the map to be updated, I went ahead and started my own attempt at generating this map automatically.

My plan was to generate the map entirely from Open Data sources (apart from the actual hackerspace locations, where the "single source of truth" is a password-protected MediaWiki server). I also planned to generate the map background, rather than using a preexisting map, in order to avoid dealing with map projection issues.

Country Borders from Wikidata

In order to generate the map's background layer, the borders of Germany's federal states as well as surrounding countries were retrieved from Wikidata. Geographic regions in Wikidata are (or can be) equipped with the «geoshape» property, describing the outline of the region as a GeoJSON file.

For the federal states, the SPARQL query to retrieve these geoshape files is fairly simple:

SELECT DISTINCT ?item ?map WHERE {
  # ?item is instance of "Federal State of Germany"
  ?item wdt:P31 wd:Q1221156.
  # ?item has geoshape ?map
  ?item wdt:P3896 ?map.
}

Unfortunately, the same can't be said for the query to retrieve the geoshapes of all European countries, due to data being somewhat inconsistent:

When multiple properties of the same type are defined for a subject, they may carry different ranks. If one of these properties has a preferred rank, Wikidata will consider it «truthy» and not return any other properties of the same type.

Some countries had multiple geoshape properties, differing in resolution («zoom level»), with the most accurate geoshape being considered preferred. However, in order to properly render a map, all countries borders needed to have the same resolution and come from the same data source, so that the borders of two neighbouring countries would coincide. To achieve this, the query needed to be crafted so that even non-preferred geoshapes were returned.
The type (preferred «instance of» property) of some countries was "instance of country", while others were considered "instance of sovereign state". Both requirements needed to be included in the query.
While some countries are labeled as "part of Europe", some others are labeled as e.g. "part of Central Europe", which would then again be "part of Europe". The "part of Europe" constraint thus needed to be declared transitively.
Some countries were labeled neither "part of Europe" nor "on continent Europe" at all. However, coincidentally, all the countries where these relationships were not defined, were labeled as "part of the European Economic Area".

Finally, taking all these inconsistencies into consideration, the query to retrieve the borders of all European countries looked like this:

SELECT DISTINCT ?item ?map WHERE {
  # ?item is instance of country or sovereign state (see filter below)
  ?item wdt:P31 ?stateclass.
  # ?item is transitively (+) part of Europe (Contintent) or EEA (see filter below)
  ?item wdt:P361+ ?euroclass.
  # ?item has geoshape ?map (including all non-deprecated results)
  ?item p:P3896 ?st.  # ?item has geoshape statement ?st
  ?st ps:P3896 ?map;  # ?st has geoshape value ?map
  MINUS { ?st wikibase:rank wikibase:DeprecatedRank }  # Exclude results where the ?st rank is "deprecated"
  # ?stateclass is "Country" or "Sovereign State"
  FILTER (?stateclass = wd:Q6256 || ?stateclass = wd:Q3624078).
  # ?euroclass is "Europe (Continent)" or "European Economic Area"
  FILTER (?euroclass = wd:Q46 || ?euroclass = wd:Q8932).
}

This query returned all the countries required for the map. It also included some of these countries' overseas territories, but those were then filtered at a later stage by comparing them to the chosen viewport of the generated map.

Hackerspace Names and Addresses from Semantic MediaWiki

There is a CCC-internal MediaWiki instance which makes heavy use of the Semantic MediaWiki (SMW) extension. Among lots of other things, every hackerspace which considers itself part of CCC is listed there, along with their properties (e.g. street addresses).

SMW makes these properties machine-readable and queryable through different interfaces. For example, queries can be formulated directly within MediaWiki pages to automatically generate and update tables or lists based on these properties. Another interface is a collection of HTTP API endpoints, making SMW queryable from the outside.

Among these API endpoints, I ended up using the Semantic Search JSON API to retrieve a list of all hackerspaces along with their addresses, display names (there can be multiple) and website URLs:

GET /index.php?title=Spezial:Semantische_Suche&x=[[Category:Erfa-Kreise]][[Chaostreff-Active::wahr]]/?Chaostreff-City/?Chaostreff-Nickname/?Chaostreff-Physical-Address/...&format=json HTTP/1.1

{
    "printrequests": [...],
    "results": {
        "Chaos Computer Club Basel": {
            "printouts": {
                "Chaostreff-City": [
                    "Basel"
                ],
                "Chaostreff-Physical-Address": [
                    "Birsfelderstrasse"
                ],
                "Chaostreff-Physical-Housenumber": [
                    "6"
                ],
                "Chaostreff-Physical-Postcode": [
                    "4132"
                ],
                "Chaostreff-Physical-City": [
                    "Muttenz"
                ],
                "Chaostreff-Country": [
                    "Schweiz"
                ],
                "Public-Web": [
                    "https://www.ccc-basel.ch/"
                ],
                "Chaostreff-Longname": [
                    "Chaos Computer Club Basel"
                ],
                "Chaostreff-Nickname": [
                    "CCC Basel"
                ],
                "Chaostreff-Realname": [
                    "Chaos Computer Club Basel"
                ]
            },
            "fulltext": "Chaos Computer Club Basel",
            "fullurl": "https://doku.ccc.de/Chaos_Computer_Club_Basel",
            "namespace": 0,
            "exists": "1",
            "displaytitle": ""
        },
        ...
    },
    "serializer": "SMW\\Serializers\\QueryResultSerializer",
    "version": 2,
    "rows": 16
}

Of course the request params would be URL-encoded, in this example they are unencoded for better readability.

Now that we have the full list of hackerspaces, there is one more step required to display them on a map: Only the street addresses are entered in this wiki. In order to display them on a map, the addresses need to be geocoded, i.e. translated into coordinates. One such geocoding service is Nominatim. It uses OpenStreetMap to look up addresses and return WGS84 coordinates.

However, not all the street addresses returned by SMW are resolvable: Some hackerspaces encode additional information (e.g. the building level) in unstandardized, non-machine-readable formats. Others temporarily don't have a street address at all, e.g. because they are moving between places. In order to handle these special cases, less accurate addresses are queried, if a more accurate address fails to resolve. For example, in order to resolve the location of CCC Basel, the following address formats would be considered:

Birsfelderstrasse 6, 4132 Muttenz, Schweiz
4132 Muttenz, Schweiz
Muttenz, Schweiz

Automatic Label Layouting

Finally, we have all the data to render the map. In a first step, both the country borders and the hackerspace locations are transformed from their polar 3D longitude/latitude representation to a 2D Cartesian projection suitable for display on a map.

However, the map should also display labels for some of the hackerspaces (so-called Erfas), and these labels should neither overlap with each other, nor should they obstruct other hackerspaces marked on the map. On the old map, these labels had been laid out manually to avoid overlap. For the new map, this too should be automated.

For each label to be placed, the text is rendered using the chosen font, and measured to compute its bounding box. From this bounding box, a discrete set of candidates for placement of this label is generated. Figure 2 depicts a set of bounding box candidates for CCC Berlin, placed around the marker of CCC Berlin.

Figure 2: Bounding boxes of all label position candidates around CCC Berlin.

Weights are assigned to each candidate based on multiple factors. The exact weights are magic numbers obtained through trial and error, but it mainly boils down to these factors:

Prefer candidates closer to their marker
Prefer candidates furthest away from other markers and (already placed) labels
Prefer candidates to the left and right over those at the top or bottom
Prefer candidates to the right over those to the left (unless close to the right-hand side of the map)
Penalize candidates overlapping with other elements with a huge weight

The actual label placement happens in three steps:

If a label has at least one candidate that doesn't overlap with any other label candidate or marker, this candidate is chosen. Since this step eliminates the other candidates of the label, it may render candidates of other labels usable again, if they were were obstructed before.
If there are still some unplaceable labels, they are placed where they cause the least overlap.
All weights are recomputed and already placed labels are optimized based on these weights. This will reduce possible overlap and spread labels further apart, making the map better readable.

This algorithm may not be perfect, but it seems to find a locally-optimized solution which at least to the human eye does not look too bad.

Conclusion

After all the labels are laid out, an SVG image is generated and written to a file. A CSS stylesheet is embedded in the SVG file so it can be used standalone, either for directly displaying it in a web browser, or for further processing. The markers and labels in the SVG image are amended with clickable links, leading the user to each hackerspace's website.

Additionally, a PNG image is generated, alongside with a HTML imagemap file (with the same links as in the SVG) so that the map can be used interactively in web browsers with limited or no SVG support.

Figure 3: The new, fully automatically created map, as it is now also available on ccc.de.

The new map (as seen in Figure 3) has now replaced the old one on ccc.de. The entire script to generate the map can be found in my Gitea repo. However, in order to actually use this script as-is, you need to know the Basic Auth credentials for the CCC MediaWiki instance. If you just want to check it out, you can use the example data in the repo, as described in the README.

Automatische Erzeugung von POI-Karten aus Offenen Daten

2022-10-27T02:00:00+02:00

Die Website des Chaos Computer Clubs enthält eine Karte von Deutschland und den umliegenden Ländern, auf der Hackerspaces verzeichnet sind, die sich als Teil des CCC betrachten.

Bisher wurde diese Karte (siehe Abbildung 1) händisch erzeugt und aktualisiert, was jeweils sehr zeitaufwändig war. Das hat nicht nur dazu geführt, dass die Karte oft nicht mehr aktuell war, sondern der Prozess war auch fehleranfällig, und die einzelnen Hackerspaces waren oft nicht ganz korrekt platziert.

Abbildung 1: Die alte, händisch erzeugte Karte der Hackerspaces.

Der Wunsch, die Erzeugung dieser Karte zu automatisieren, wurde in den letzten Jahren mehrfach geäussert, aber bisherige Ansätze wurden nie fertiggestellt. Da ich eine der letzten Personen war, die mal wieder darum gebeten hatten, die Karte zu aktualisiert, habe ich einfach kurzerhand selber versucht, die Erzeugung der Karte zu automatisieren.

Mein Plan war, die gesamte Karte aus offenen Daten zu erzeugen (ausgenommen von den Adressen der Hackerspaces, die aus einem passwortgeschützten MediaWiki kommen). Auch den Hintergrund der Karte wollte ich dynamisch erzeugen, anstatt eine bestehende Karte zu verwenden, um mich nicht mit abweichenden Kartenprojektionen herumschlagen zu müssen.

Ländergrenzen aus Wikidata

Um den Hintergrund der Karte zu erzeugen, wurden die Grenzverläufe der Bundesländer Deutschlands sowie der umliegenden Länder von Wikidata heruntergeladen. Geografische Regionen werden in Wikidata mit dem Attribut «geoshape» versehen (resp. können damit versehen werden), welches den Grenzverlauf der Region als GeoJSON-Datei beschreibt.

Die geoshape-Dateien für die Bundesländer können via SPARQL recht einfach abgerufen werden:

SELECT DISTINCT ?item ?map WHERE {
  # ?item ist vom Typ "Bundesland von Deutschland"
  ?item wdt:P31 wd:Q1221156.
  # ?item hat die geoshape ?map
  ?item wdt:P3896 ?map.
}

Leider ist es nicht mehr ganz so einfach, die Grenzverläufe aller europäischen Länder abzurufen, da die Daten nicht sonderlich konsistent sind:

Wenn ein Object in Wikidata mehrere Attribute vom gleichen Typ besitzt, können diesen unterschiedliche «Ränge» («ranks») zugewiesen werden. Falls eins dieser Attribute den Rang «bevorzugt» («preferred») trägt, wird es von Wikidata als «die Wahrheit» betrachtet, und andere Attribute vom gleichen Typ werden nicht zurückgegeben.

Einige Länder haben mehrere geoshapes, die sich in ihrer Auflösung («zoom level») unterschieden, wobei der am höchsten aufgelöste Grenzverlauf bevorzugt wird. Um allerdings aus den Grenzverläufen sinnvoll eine Karte zu erzeugen, sollten alle Verläufe die gleiche Auflösung haben und idealerweise aus der gleichen Quelle stammen, damit die Grenzen von benachbarten Ländern genau übereinander liegen. Um an diese zusätzlichen Auflösungsstufen zu kommen, muss die SPARQL-Abfrage angepasst werden, um auch nicht bevorzugte Attribute zurückzugeben.
Der Typ (bevorzugtes «instance of»-Attribute) ist bei manchen Ländern «country», bei anderen Ländern «sovereign state». Um alle Länder abzudecken, müssen beide Werte berücksichtigt werden.
Manche Länder sind als Teil von Europa («part of Europe») gekennzeichnet, während andere z.B. als Teil on Zentraleuropa («part of Central Europe») gekennzeichnet sind, welches wiederum «part of Europe» ist. Der «part of Europe»-Filter muss daher transitiv angegeben werden.
Manche Länder waren weder «part of Europe» noch «on continent Europe». Zufälligerweise waren aber alle solche Länder auch Teil des Europäischen Wirtschaftsraums («part of the European Economic Area»).

Wenn man diese Inkonsistenzen alle in Betracht zieht, kommt man auf folgende SPARQL-Query, um die Grenzverläufe aller Länder in Europa abzurufen:

SELECT DISTINCT ?item ?map WHERE {
  # ?item ist vom Typ «country» oder «sovereign state» (siehe FILTER weiter unten)
  ?item wdt:P31 ?stateclass.
  # ?item ist transitiv (+) Teil von «Europe (Contintent)» oder «EEA» (siehe FILTER weiter unten)
  ?item wdt:P361+ ?euroclass.
  # ?item hat die geoshape ?map (inklusive aller Resultate, die nicht den Rang «veraltet» («deprecated») haben)
  ?item p:P3896 ?st.  # ?item hat ein geoshape-Statement ?st
  ?st ps:P3896 ?map;  # Das statement ?st hat die geoshape ?map
  MINUS { ?st wikibase:rank wikibase:DeprecatedRank }  # Alle Resultate ignorieren, deren Rang «deprecated» ist
  # ?stateclass ist "Country" oder "Sovereign State"
  FILTER (?stateclass = wd:Q6256 || ?stateclass = wd:Q3624078).
  # ?euroclass ist "Europe (Continent)" oder "European Economic Area"
  FILTER (?euroclass = wd:Q46 || ?euroclass = wd:Q8932).
}

Mit dieser Query kommen alle Länder zusammen, die für die Karte benötigt werden. Zwar sind dort bei einigen Ländern noch die Übersee-Territorien mit dabei, diese werden aber später entfernt, wenn die Kartenränder festgelegt werden.

Hackerspace-Namen und -Adressen von Semantic MediaWiki

Das CCC-interne MediaWiki macht starken Gebrauch von der Erweiterung Semantic MediaWiki (SMW). Unter vielen anderen Daten ist dort jeder Hackerspace aufgeführt, der sich als Teil des CCC sieht, inklusive seiner Attribute (wie z.B. Postanschriften).

SMW macht diese Attribute maschinenlesbar und ermöglicht die Abfrage dieser Attribute über verschiedenen Schnittstellen. Zum Beispiel können so direkt in MediaWiki-Seiten Abfragen eingebunden werden, die automatisch Tabellen oder Listen aus diesen Daten generieren und aktualisieren. Zudem gibt es noch verschiedene HTTP-Schnittstellen, durch welche SMW auch durch die Aussenwelt abgefragt werden kann.

Um die Liste aller Hackerspaces inklusive ihrer Adressen, Namen (wovon es mehrere geben kann) und Website-URLs abzufragen, habe ich die Semantic Search JSON-API verwendet:

GET /index.php?title=Spezial:Semantische_Suche&x=[[Category:Erfa-Kreise]][[Chaostreff-Active::wahr]]/?Chaostreff-City/?Chaostreff-Nickname/?Chaostreff-Physical-Address/...&format=json HTTP/1.1

{
    "printrequests": [...],
    "results": {
        "Chaos Computer Club Basel": {
            "printouts": {
                "Chaostreff-City": [
                    "Basel"
                ],
                "Chaostreff-Physical-Address": [
                    "Birsfelderstrasse"
                ],
                "Chaostreff-Physical-Housenumber": [
                    "6"
                ],
                "Chaostreff-Physical-Postcode": [
                    "4132"
                ],
                "Chaostreff-Physical-City": [
                    "Muttenz"
                ],
                "Chaostreff-Country": [
                    "Schweiz"
                ],
                "Public-Web": [
                    "https://www.ccc-basel.ch/"
                ],
                "Chaostreff-Longname": [
                    "Chaos Computer Club Basel"
                ],
                "Chaostreff-Nickname": [
                    "CCC Basel"
                ],
                "Chaostreff-Realname": [
                    "Chaos Computer Club Basel"
                ]
            },
            "fulltext": "Chaos Computer Club Basel",
            "fullurl": "https://doku.ccc.de/Chaos_Computer_Club_Basel",
            "namespace": 0,
            "exists": "1",
            "displaytitle": ""
        },
        ...
    },
    "serializer": "SMW\\Serializers\\QueryResultSerializer",
    "version": 2,
    "rows": 16
}

(Üblicherweise würden die HTTP-Request-Parameter URL-encoded werden, in diesem Beispiel sind sie zwecks Lesbarkeit aber nicht encoded.)

Jetzt haben wir die gesamte Liste an Hackerspaces, aber bevor diese auf einer Karte dargestellt werden können, ist noch ein weiterer Schritt nötig: Das Wiki enthält nur Adressen. Um die Hackerspaces auf einer Karte darzustellen, müssen diese erst geocoded werden, d.h. in Geokoordinaten übersetzt werden. Ein solcher Geocoding-Dienst ist Nominatim. Nominatim verwendet OpenStreetMap, um Adressen nachzuschlagen und in WGS84-Koordinaten zu übersetzten.

Leider sind nicht alle der Adressen, die vom SMW zurückkommen, auflösbar: Manche Hackerspaces tragen in den Adress-Attributen zusätzliche Informationen (wie z.B. das Stockwerk) ein, die weder standardisiert, noch maschinenlesbar sind. Andere Spaces haben vorübergehend gar keine Adresse, z.B. weil sich gerade am umziehen sind. Um diese Fälle zu behandeln, werden zusätzliche, weniger genaue Adressen abgefragt, wenn eine genauere Adresse nicht auflösbar ist. So werden z.B. für den CCC Basel die folgenden Adressformate in Erwägung gezogen:

Birsfelderstrasse 6, 4132 Muttenz, Schweiz
4132 Muttenz, Schweiz
Muttenz, Schweiz

Automatische Anordnung von Beschriftungen

Nun haben wir endlich alle Daten, um die Karte zu erzeugen. Zunächst werden sowohl die Grenzverläufe als auch die Hackerspace-Koordinaten von ihrer 3-dimensionalen polaren Darstellung aus Latitude und Longitude in eine 2-dimensionale kartesische Projektion umgerechnet, die für eine Kartendarstellung geeignet ist.

Einzelne der Hackerspaces (die sogenannten Erfas) sollen allerdings auch auf der Karte beschriftet werden. Diese Beschriftungen sollen weder mit den Markern auf der Karte, noch mit sich selbst überlappen. Auf der alten Karte wurde die Beschriftungen noch händisch angeordnet, um Überlappungen zu vermeiden.

Für jede Beschriftung, die auf der Karte platziert werden soll, wird zunächst der Text einmal mit der gewählten Schriftart gerendert und ausgemessen, um den Rahmen um den Text zu bestimmen. Aus diesem Rahmen wird ein diskretes Set an Platzierungs-Kandidaten erzeugt. Abbildung zeigt die Kandidaten, die für den CCC Berlin generiert wurden, und rund um dessen Marker platziert wurden.

Abbildung 2: Rahmen aller möglichen Beschriftungs-Positionen des CCC Berlin.

Jedem Kandidaten wird anhand von verschiedenen Faktoren eine Gewichtung zugeordnet. Die genaue Gewichtungen der einzelnen Faktoren sind magic numbers, die durch Ausprobieren zustandegekommen sind. Grundsätzlich gelten aber folgende Regeln:

Bevorzuge Kandidaten, die nahe an ihrem Marker stehen.
Bevorzuge Kandidaten, die weit weg von anderen Markern und anderen, bereits platzierten Kandidaten stehen.
Bevorzuge Kandidaten, die links oder rechts von ihrem Marker stehen, über solche, die oberhalb oder unterhalb ihres Markers stehen.
Bevorzuge Kandidaten, die rechts von ihrem Marker stehen über solche, die auf der linken Seite stehen. Nahe am rechten Rand der Karte wird diese Regel umgekehrt.
Bestrafe Kandidaten, die mit anderen Objekten überlappen, mit einem hohen Gewicht.

Die Platzierung der Beschriftungen findet in drei Schritten statt:

Wen es für eine Beschriftung mindestens einen Kandidaten gibt, der mit keinem anderen Objekt oder Kandidaten überlappt, wird dieser Kandidat als Platzierung festgelegt. Dieser Schritt eliminiert die anderen Kandidaten für diese Beschriftung, daher werden für die Beschriftung anderer Marker eventuell wieder weitere Kandidaten «frei», die vorher mit den eliminierten Kandidaten überlappt waren.
Wenn es immer noch Beschriftungen gibt, die nicht überlappungsfrei platziert werden können, werden diese dort platziert, wo die Überlappung minimal ist.
Alle Gewichte werden (jetzt, wo die allermeisten Kandidaten eliminiert wurden) neu berechnet, und die bereits platzierten Beschriftungen werden anhand der neuen Gewichte optimiert. Dadurch werden allfällige Überlappungen reduziert, und auch nicht überlappende Beschriftungen werden weiter voneinander weggeschoben, um die Lesbarkeit der Karte zu erhöhen.

Dieser Algorithmus ist nicht perfekt, aber er scheint lokale Optima finden zu können, die zumindest für das menschliche Auge akzeptabel aussehen.

Fazit

Nachdem alle Beschriftungen platziert wurden, wird eine SVG-Vektorgrafik erzeugt und in eine Datei geschrieben. In dem SVG wird ein CSS-Stylesheet eingebettet, damit es alleinstehend verwendet werden kann, z.B um direkt in einem Webbrowser dargestellt zu werden, oder um weiterverarbeitet werden zu können. Die Marker und Beschriftungen werden zusätzlich noch mit klickbaren Links ergänzt, die auf die Websites der einzelnen Hackerspaces weiterleiten.

Aus dem SVG wird auch noch ein PNG-Bild erzeugt, zusammen mit einer HTML Image Map (mit den gleichen Links wie im SVG), sodass die Karte auch in solchen Browsern interaktiv verwendet werden können, die nur reduziert oder gar nicht mit SVG-Dateien umgehen können.

Abbildung 3: Die neue, vollständig automatisch erzeugte Karte, wie sie nun auch auf ccc.de verfügbar ist.

Die so erzeugte neue Karte (siehe Abbildung 3) hat nun auch auf ccc.de die bisher händisch erzeugte Karte ersetzt. Das ganze Skript zur Generierung der Karte ist in meinem Gitea-Repo verfügbar. Um das Skript verwenden zu können, musst du allerdings die Basic Auth-Zugangsdaten für das CCC MediaWiki kennen. Wenn du das Skript nur kurz ausprobieren willst, kannst du aber auch auf die Beispieldaten im Repo zurückgreifen, wie im README beschrieben.

Talk: «DNSSEC (in the year 2022)»

2022-09-10T00:00:00+02:00

I held a talk at CCC Basel titled «DNSSEC (in the year 2022) - An introduction to DNSSEC and its development in the recent years».

Here's the recording of the talk (in German):

Vortrag: «DNSSEC (im Jahr 2022)»

2022-09-10T00:00:00+02:00

Ich habe im CCC Basel einen Vortrag mit dem Titel «DNSSEC (im Jahr 2022) - Eine Einführung in DNSSEC und dessen Entwicklung in den letzten Jahren» gehalten.

Hier ist die Aufzeichnung des Vortrags:

Mein Einbürgerungverfahren in Basel - Teil 5: Das Einbürgerungsgespräch

2022-08-26T17:00:00+02:00

Gestern früh hatte ich das Gespräch mit der Einbürgerungskommission der Bürgergemeinde Basel. Das Gespräch dauert gemäss Einladung ca. 20 Minuten, war bei mir aber schon nach ca. 15 Minuten vorbei.

Das Gespräch

Abbildung 1: Der Bürgerratssaal im Basler Stadthaus, in dem das Einbürgerungsgesprach stattfindet. Quelle: Bürgergemeinde der Stadt Basel, CC-BY-SA-4.0.

Das Gespräch findet gegenüber der gesamten, 7-köpfigen Einbürgerungskommission im Bürgerratssaal (siehe Abbildung 1) statt. Nach der Begrüssung durch die Vorsitzende wurde das gesamte Gespräch allerdings von einer einzelnen Person geleitet.

Die Einladung war so formuliert, dass das Gespräch «teilweise Testcharakter» habe. Daher war meine Erwartung, dass sich das Gespräch z.B. aus 10 Minuten «Mündlichprüfung» und 10 Minuten Fragen zu meinem Leben und meiner Integration zusammensetzt.

Die persönlichen Fragen hielten sich aber sehr in Grenzen, nach einer Frage zur Vereinbarkeit von Studium und Beruf und einer Frage zu meiner Vereinsaktivität bestand das ganze restliche Gespräch aus einer mündlichen Staatsbürgerkunde-Prüfung.

Unter anderem wurden folgende Fragen gestellt:

Welche Sehenswürdigkeiten würde ich einem Besucher in Basel empfehlen?
Welche Gremien treffen sich im Basler Rathaus?
Welche Gerichte gibt es in Basel?
Was kann ich machen, wenn auch das kantonale Appellationsgericht mir nicht Recht gibt?
Welche Gerichte gibt es auf Bundesebene?
Wer wählt die Bundesrichter?
Was ist die vereinigte Bundesversammlung?
Wer vertritt Basel-Stadt im Ständerat?
Nenne 3 Nationalräte aus Basel-Stadt
Welche Positionen werden noch von der vereinigten Bundesversammlung gewählt?
Wer ist der Bundespräsident?
Wer wird nächstes Jahr Bundespräsident? (Wusste ich nicht)
Wie heisst die Exekutive der Bürgergemeinde?
Durch wen wird der Bürgerrat gewählt?
Welche Aufgaben hat die Bürgergemeinde?
Wer war Christoph Merian?
In Basel gibt es Zünfte und Ehrengesellschaften, welche gehören zu welcher Seite vom Rhein?
Die kleinbasler Ehrengesellschaften organisieren eine jährliche Veranstaltung. Wie heisst diese und wann findet sie statt?
Was passiert, wenn der Termin auf einen Sonntag fällt? (Wusste ich nicht)
Welche Flüsse gibt es in Basel?
Welche Seen teilt sich die Schweiz mit den angrenzenden Ländern?
Nenne 3 Berge.
Welche politischen Rechte hat man als Schweizer Bürger?
Wann ist der nächste Abstimmungstermin?
Über welche Vorlagen wird am nächsten Abstimmungstermin abgestimmt?
Was ist der Unterschied zwischen einem Referendum und einer Volksinitiative?
Wieviele Unterschriften werden für das Zustandekommen einer Volksinitiative benötigt?

Schlussendlich wurde wieder zurück an die Vorsitzende der Kommission übergeben, die mich nur noch darüber informierte, dass ich den Entscheid der Einbürgerungskommission via Post erhalten werde, und dann verabschiedete.

Das Ergebnis

Bereits heute, einen Tag nach dem Gespräch, hatte ich einen Brief von der Bürgergemeinde Basel im Briefkasten. Die Einbürgerungskommission hat bereits einen positiven Entscheid gefällt, und an den Bürgerrat (die Exekutive der Bürgergemeinde) weitergereicht.

Das Lange Warten, 3.0

Nun ist wieder warten angesagt. Trotz dem so schnellen positiven Entscheid der Einbürgerungskommission dauert es nun noch ca. 8-10 Monate, bis das Einbürgerungsverfahren abgeschlossen ist.

Nach dem Entscheid der Bürgergemeinde geht mein Einbürgerungsgesuch zurück an den Kanton und von dort aus weiter an den Bund, genauer ans Staatssekretariat für Migration. Nach dem Bundesentscheid geht es dann wieder zurück an den Kanton zum kantonalen Einbürgerungsentscheid.

Der ganze Ablauf ist übrigens im Leitfaden Einbürgerung des Kantons Basel-Stadt auf Seite 33 abgebildet.

Mein Einbürgerungverfahren in Basel - Teil 4: «Einbürgerungskurs»

2022-06-18T03:42:00+02:00

Achtung: Die Informationen in diesem Artikel beziehen sich auf das Verfahren im Kanton Basel-Stadt, die Verfahren in anderen Kantonen weichen z.T. stark ab.

Im Rahmen des Einbürgerungsverfahrens muss man einen «Einbürgerungstest» absolvieren, in dem Kenntnisse über die politischen, historischen und geografischen Gegebenheiten der Schweiz geprüft werden. Wie ich bereits in Teil 1 erwähnt hatte, wird dieser Test in Basel nicht wie in manchen anderen Kantonen schriftlich durchgeführt, sondern findet mündlich im Rahmen des Einbürgerungsgesprächs bei der Bürgergemeinde statt.

Zur Vorbereitung auf diesen Test bietet die Bürgergemeinde einen optionalen Kurs an. Genau genommen sind es sogar zwei verschiedene Kurse: Zum Einen gibt es den Kurs «Fit für Basel», der aus 5 Lektionen à 1h45 besteht, inklusive einem Stadtrundgang. Zum anderen gibt es den «Kompaktkurs Einbürgerung», der aus nur 3 Lektionen besteht. Die Kurse werden je in zwei verschiedenen Varianten angeboten: Vor Ort im Stadthaus, oder online via Zoom. Ich hatte mich dazu entschieden, den Kompaktkurs via Zoom zu buchen.

Abbildung 1: Das Basler Stadthaus, Sitz der Bürgergemeinde Basel. Quelle: Bürgergemeinde der Stadt Basel, CC-BY-SA-4.0.

Erste Lektion: Die Schweiz

Im ersten Kurstermin ging es um die Schweiz als Bundesstaat und diverser Kennzahlen auf Bundesebene. Es wurde gleich vom Anfang an, und auch während dem Kurs immer wieder, darauf hingewiesen, dass der Kurs ein gutes Stück mehr ins Detail ginge, als wir für das Einbürgerungsgespräch wissen müssten. Alles, was für die Einbürgerung relevant sei, erhielten wir jeweils im Anhang der Zoom-Einladungen.

Die relevanten Fakten dieser Lektion waren:

Bevölkerungszahl und Fläche der Schweiz
Nennung aller Landessprachen und Nachbarländer
Nennung von 4 Flüssen und 4 Seen
Die drei Gewalten und deren wichtigsten Organe auf Bundesebene
- National- und Ständerat inkl. Anzahl Mitglieder
- Nennung aller Bundesräte und welchen Departementen diese vorstehen
- Nennung der Bundesgerichte und deren Standorte
- Wer diese Organe wählt

Aus meiner Sicht ist das erforderliche Wissen auf Bundesebene damit recht überschaubar, allerdings bin ich auch in der Schweiz (wenn auch in einem anderen Kanton) zur Schule gegangen, und habe auch vor meiner Auswanderung in de Schweiz in einem demokratischen Staat mit Gewaltenteilung gelebt, daher waren diese Konzepte nicht wirklich neu für mich.

Ich hatte von dem Kurs monotone, vorlesungsartige Lektionen erwartet. Tatsächlich war der Kurs aber eine recht kleine Runde (ca. 15 Personen) und recht locker und vor allem interaktiv gestaltet. Neben den häufigen Fragen in die Runde, ob das schon jemand beantworten könne, warf der Kursleiter auch immer wieder ein paar humorvolle Kommentare mit ein. So kommentierte er z.B. das Bundesratsfoto mit «Das ist nicht etwa eine missglückte Photoshop-Arbeit von mir, das ist das offizielle Bundesratsfoto».

Besonders beeindruckt hat mich, das bei den Slides zu Bevölkerungswachstum und Migration die Hintergründe zu den Schwankungen erläutert wurden, z.B. warum die Bevölkerungszahl seit 1900 während zwei Phasen abgenommen hatte (einmal die Spanische Grippe, einmal eine Wirtschaftsrezession, während der Gastarbeiter nicht länger beschäftigt wurden und in ihre Heimat zurückkehrten).

Auch wurde z.B. beim Vergleich des Ausländeranteils der Schweizer Bevölkerung mit anderen Ländern darauf hingewiesen, dass diese Vergleiche mit Vorsicht zu geniessen seien, und dass es eher sinnvoll wäre, die Migrationsströme zu vergleichen, da sich die Rechtslage zum Erhalt der jeweiligen Staatsbürgerschaften zwischen den Ländern teils stark unterscheide und manche Ländern z.B. im Inland geborenen Kindern ausländischer Eltern bei der Geburt automatisch die jeweilige Staatsbürgerschaft verleihen.

Auch historische Ereignisse wurden entsprechend eingeordnet. So wurde z.B. erklärt, dass das Zweikammernsystem aus National- und Ständerat eine Kompromisslösung ist, die als Konsequenz des Sonderbundskrieges von 1847 entstanden ist, und von den damals noch jungen Vereinigten Staaten von Amerika abgekupfert wurde. Neben solchen hinstorischen Einordnungen gab es allerdings auch Fakten, die ich persönlich eher in die Kategorie «unnützes Wissen» einordnen würde. So z.B., dass es seit 1848 (Grundüngs des Schweizer Bundesstaates) genau ein einziges Bundesgerichtsurteil gab, welches auf rätoromanisch verfasst wurde.

Zweite Lektion: Der Kanton Basel-Stadt

Der zweite Kurstermin startete gleich schon mit 35 Minuten Verspätung - die Bürgergemeinde hatte einen falschen Zoom-Link an die Teilnehmenden geschickt, und dann alle Teilnehmenden angerufen, um die richtigen Zugangsdaten telefonisch mitzuteilen. Zwar war die Zoom-Einladung an sich fehlerhaft, aber auch dieses Mal waren wieder Dokumente mit allen Information, die wir für das Einbürgerungsgespräch wissen sollten, angehängt:

Anzahl Kantone in der Schweiz
Bevölkerungszahl und Fläche des Kantons Basel-Stadt
Die drei Gemeinden in Basel-Stadt, Nachbargemeinden und -länder
Flüsse in Basel-Stadt, höchstgelegener Punkt im Kanton
Die drei Gewalten und ihre wichtigsten Organe im Kanton Basel-Stadt:
- Grosser Rat und Anzahl Mitglieder
- Regierungsrat inkl. Nennung aller Mitglieder und deren Departemente
- Alle Basler Abgeordneten im National- (5) und Ständerat (1)
- Die kantonalen Gerichte und ihre Standorte
Historische Daten:
- 1291: Gründung der Eidgenossenschaft
- 1501: Beitritt Basels zur Eidgenossenschaft
- 1833: Trennung in Basel-Stadt und Basel-Landschaft
- 1971: Einführung des Frauenwahl- und stimmrechts auf Bundesebene

Auch dieses Mal wurde wieder deutlich mehr ins Detail und in die Geschichte gegangen, als wir für das Einbürgerungsgespräch wissen müssen. So sind wir z.B. die Geschichte der alten Eidgenossenschaft mit u.a. den Beitrittsdaten der "wichtigsten" Kantone durchgegangen. Auch sind wir auf einer unbeschrifteten Schweiz-Karte alle 26 Kantone mitsamt Hauptorten durchgegangen, was doch einige Zeit gedauert hatte.

Neben einer doch sehr fakten- und zahlenlastigen Lektion gab es doch noch einige historische Anekdoten zu hören. So zum Beispiel die Geschichte der Justitia-Statue auf dem Basler Rathaus. Diese war nämlich ursprünglich eine Marienstatue, die im Rahmen der Reformation aus dem Basler Münster entfernt worden war. Als der Entscheid gefällt wurde, eine Justitia-Statue aufzustellen, wurde kurzerhand einfach diese Marienstatue genommen, und Jesuskind und Zepter durch Schwert und Waage ersetzt. Daher trägt die «Basler Justita» auch eine Krone statt Augenbinde, und ist in rot und blau gewandet. (Nein, die Farben stehen nicht wie sonst in Basel für den FCB!)

Dritte Lektion: Die Stadt Basel

Im dritten Kurstermin ging es primär um die Stadt Basel. Der Kurs begann mit einer Erklärung, dass die politischen Organisationsstrukturen in Basel ein Ausnahmefall seien: Denn Basel verfügt nicht über eine eigenständige Stadtregierung. So gibt es keine Einwohnergemeindeversammlung, und keine Bürgermeister*innen. Diese Rollen werden von den kantonalen Gremien wahrgenommen, also vom Grossen Rat und dem Regierungsrat. Die anderen beiden Gemeinden im Kanton Basel-Stadt haben jedoch diese Organe. Andersherum ist jedoch die Bürgergemeinde der Stadt Basel eine der wenigen Bürgergemeinden mit einem Parlament auf Gemeindeebene.

Neben dieser «Kuriosität» gab es auch dieses Mal wieder eine lange Liste an Informationen, die wir für das Einbürgerungsgespräch wissen sollten, darunter:

Die Bürgergemeinde Basel
- Tagungsort und Grösse des Bürgergemeinderats (Legislative)
- Grösse des Bürgerrats (Exekutive), aber ohne die Mitglieder nennen zu können
- Aufgaben der Bürgergemeinde
Brauchtum
- Kenntnis von Basler Bräuchen (Vogel Gryff, Fasnacht)
- Nennung einiger Zünfte und der drei Ehrengesellschaften
Geschichte der Stadt Basel
- Besiedelung durch Kelten und Römer
- Das Basler Erdbeben 1356
- Gründung der Universität und Erlangung des Messerechts
- Bau der Mittleren Rheinbrücke
Wirtschaft in Basel (Pharma, Messen, Tourismus)
Kenntnis der politischen Rechte (Initiativen und Referenden)

Diese Informationen sind wir - wie auch in den vorigen beiden Kursterminen - ziemlich detailliert durchgegangen, mitsamt historischen Anekdoten und Einordnung in das heutige Stadtbild.

Alles in allem kann ich sagen, dass der Kurs sich mehr als gelohnt hat. Nicht nur wurde uns mitgeteilt, welche Informationen wir für das Einbürgerungsgespräch wissen müssen, sondern es wurden auch sehr interessante Einblicke in die Geschichte der Schweiz und von Basel gegeben, die für mich zumindest teilweise neu waren. Allerdings frage ich mich, da ich ja den «Kompaktkurs» besucht hatte, wieviel mehr ins Detail wohl der «grosse» Kurs mit 5 Lektionen gegangen wäre.

Das Warten Geht Weiter

Im letzten Beitrag hatte ich schon angekündigt, dass wieder Warten angesagt sei. Der Kurs war nur ein kleiner Unterbruch dieser mehrmonatigen Frist, bis die Bürgergemeinde mich zum Einbürgerungsgespräch einlädt.

Talk: «Post Mortem»

2022-05-14T00:00:00+02:00

I held a talk at CCC Basel titled «Post Mortem: Lessons Learned from the Death of a Sysadmin».

Here's the recording of the talk (in German):

Vortrag: «Post Mortem»

2022-05-14T00:00:00+02:00

Ich habe im CCC Basel einen Vortrag namens «Post Mortem: Lessons Learned aus dem Todesfall eines Serveradmins» gehalten.

Hier ist die Aufzeichnung des Vortrags:

Mein Einbürgerungverfahren in Basel - Teil 3: Telefonisches Erhebungsgespräch

2022-03-02T03:42:00+01:00

Achtung: Die Informationen in diesem Artikel beziehen sich auf das Verfahren im Kanton Basel-Stadt, die Verfahren in anderen Kantonen weichen z.T. stark ab.

Letzte Woche hatte ich von Migrationsamt Basel-Stadt einen Brief erhalten, mit der Bitte, mich telefonisch beim Migrationsamt zu melden, da noch ein paar offene Fragen bestünden.

Telefonisches Erhebungsgespräch

Gestern rief ich bei der Sachbearbeiterin meines Einbürgerungsgesuchs an. Dabei erfuhr ich auch, dass es sich bei diesem Telefonat um das angekündigte «Erhebungsgespräch» handelte.

Die meisten Fragen, die mir im Lauf des Gesprächs gestellt wurden, bezogen sich auf meine Angaben, die ich auf den eingereichten Formularen gemacht hatte. Darunter waren aber auch einige Fragen, die mir eher seltsam und übertrieben vorkamen. Zum Beispiel musste ich angeben, in welchem Stockwerk meine Wohnung liegt, und wie viele Zimmer sie hat. Auch nach den Berufen und Staatsangehörigen meiner Familienangehörigen wurde gefragt.

Zusätzlich wurde ich noch darum gebeten, einen Lebenslauf einzusenden. Dies hatte mich ebenfalls etwas verwirrt, da ich bereits in den Gesuchsformularen deutlich mehr Angaben machen musste, als in einem normalen Lebenslauf üblich sind. Auf meine Rückfrage hin wurde mir jedoch versichert, dass tatsächlich ein normaler Lebenslauf gefragt war, wie er bei einer Stellenbewerbung üblich ist.

Befragung von Referenzpersonen

Am Nachmittag nach dem Erhebungsgespräch erzählte mir dann noch mein Chef, den ich auf dem Einbürgerungsgesuch als Referenzperson angegeben hatte, dass er kurz nach dem Gespräch vom Migrationsamt angerufen wurde. Ihm wurden ebenfalls einige Fragen gestellt, darunter ob:

er es befürwortet, dass ich eingebürgert werde
ich gut in der Schweiz integriert bin
ich Kontakt zu Schweizer Bürger*innen pflege
ich mit lokalem Brauchtum vertraut bin

Das Lange Warten, 2.0

Nun ist wieder warten angesagt. Der nächste Schritt in meinem Einbürgerungsverfahren, das Einbürgerungsgespräch bei der Bürgergemeinde Basel, wird anscheinend erst in mindestens 6 Monaten stattfinden. Davor werde ich noch einen Einbürgerungskurs besuchen, der ebenfalls von der Bürgergemeinde veranstaltet wird.

Restoring a Commodore CBM 3016

2022-02-09T00:00:00+01:00

We recently got our hands on a Commodore CBM 3016, built between 1978 and 1980. Unfortunately, as we attempted to power it up, we were greeted with a screen full of random characters:

Figure 1: CBM 3016 screen filled with random characters.

A short web search told us that a common cause for this issue are bad ROM modules.

Understanding the CBM Memory Layout and ROM

The MOS 6502 CPU used in the PET / CBM series has a 8 bit data bus and a 16 bit address bus. Schematics found online reveal that the uppermost 4 of those 16 bits are used to determine which device to talk to: The entire lower half of the address space ($0000:$7fff are mapped to DRAM, though on our model 3016 with 16KiB RAM, only the area $0000:$3fff can actually be used. Next follows the video buffer at $8000:$8fff, and after that follow different 4KiB ROM chips at $9000:9fff through $f000:$ffff. Again, only part of this is used; the address space $9000:$bfff maps to empty sockets which can be populated with software ROMs by the user.

The actual operating system of the CBM is stored on 4 different ROMs:

$c000:$cfff: BASIC editor/interpreter/runtime, part 1
$d000:$dfff: BASIC editor/interpreter/runtime, part 2
$e000:$e7ff: IO processing routines. This is actually a 2KiB ROM, the upper half is mapped to actual hardware IO (e.g. the keyboard).
$f000:$ffff: The KERNAL ("kernel" with a typo that kept on living).

There also is an additional ROM, which stores the character set that can be displayed on the screen, but this ROM is not attached to the MOS 6502 address bus. Instead it is used directly by the screen rendering part of the PCB.

We removed these ROMs and dumped each of them using the MiniPRO TL866A programmer:

minipro -yp 27C32A@DIP24 -r basic1.bin
minipro -yp 27C32A@DIP24 -r basic2.bin
minipro -yp 27C32A@DIP24 -r kernal.bin
minipro -yp M2716@DIP24 -r io.bin
minipro -yp M2716@DIP24 -r chars.bin

Multiple versions of the KERNAL+BASIC system have been spotted in the wild, and we did not know which version was used by our CBM, so we didn't yet know what these dumps were actually supposed to look like. Fortunately, we managed to find dumps of most of these versions online, and only one of those matched our dumps: The BASIC 2.0 version.

At least most of our dumps matched. The IO ROM and the first BASIC ROM did not even closely resemble the dumps we had found online. Additionally, when we attempted to dump the ROMs for a second time, these two ROMs yielded totally different data. After repeating this for a few times, we concluded that these two ROMs were broken completely.

Creating New ROMs

We managed to find some pinout-compatible EPROMs we wanted to use as a replacement. However, after erasing them with our UV eraser, it turned out that our programmer was not capable of producing the 25V required for programming these chips. We fiddled around with an external PSU, and did achieve some success (as in: we managed to write some random bits), but we never managed to complete a full write sequence.

We started looking for alternatives and found multiple rails of AT27C040 PLCC-packaged one-time-programmable ROMs. These PROMs feature a whole 512 KiB of storage. Our plan was to use the lowermost 2 resp. 4 KiB and after programming short the excess address pins to GND. I designed a small adapter PCB from PLCC32 to DIP24, suitable for replacing both a 2K and a 4K ROM.

Figure 2: Schematic of the PLCC32 to DIP24 adapter PCB designed for using an AT27C040 ROM as a replacement for the CBM's broken ROMs.

The solder bridge in this schematic is used to choose between the 2K and 4K version; with 4K the OE (Output Enable) pin of the ROM is always pulled low, for the 2K version it is connected to A11 (the 12th address bit): when this bit is HIGH, the output is disabled and the pins are kept floating as to not disturb the communcation between the CPU and the IO hardware mapped to the upper half of the $e000:$efff address space.

Though, in the end we used the ugly but easy solution of soldering some short wires between the ROM and a DIP socket, rather than manufacturing a full-blown PCB. And indeed, once we had put our replacement ROMs into their sockets, we did at least achieve some progress: The screen was still showing random characters on startup, but after a few seconds the entire screen was cleared, and then nothing happened anymore. So it appeared the CPU was doing something, but apparently not the right thing.

Figure 3: The replacement ROMs placed in their sockets.

Hunting Down CPU Misbehavior

In order to figure out what the CPU was doing, I attached a logic analyzer to the address bus pins and recorded every address on the bus between startup and the screen blanking. One thing that became apparent immediately was that all communication on the address bus stopped after the screen was blanked. As the MOS 6502 CPU has to at least fetch an instruction every 1-6 cycles, it appeared as if the CPU had halted, which under normal conditions is only possible when it's executing a HCF-style illegal opcode.

So I took a closer look at the instructions the CPU was loading right before freezing. I used the dxa65 disassembler to turn the ROM dumps into 6502 assembly mnemonics in order to understand which instructions were located at which addresses. And indeed, the last instruction executed before freezing, read from address $f4e0, was $d2, an illegal opcode that halts the CPU. The disassembly revealed that this was actually not supposed to be an instruction, but a zeropage address passed as an argument to a previous instruction. Going a few cycles backwards, I found something that seemed promising: A rts (return from subroutine) instruction was loaded, followed by two accesses in the stack page at $01fe:$01ff (popping the return address from the stack), and then the next instruction was loaded from $f4d8. The odd part here is that a rts should only ever jump back to an address preceded by a corresponding jsr (jump to subroutine) instruction, which was absent here.

Figure 4: Left: A logic analyzer attached to the address bus pins and the Φ₂ system clock. Right: Screenshot of the logic analyzer software, showing the decoded memory addresses at the top.

So clearly the CPU was returning to a wrong address. At this point I started to suspect faulty RAM. To find out more I searched even further back until I found the jsr instruction — at address $fcd5. So, according to the specification of the jsr instruction, the real return address pushed to the stack had to be $fcd7 (address of the jsr + 2). When we compare this to the address actually popped from the stack, $f6d7 (actual return address - 1), we can see that these addresses differ in exactly one bit, which confirmed my suspicion of faulty RAM:

$fcd7 = 1111 1100 1101 0111
$f4d7 = 1111 0100 1101 0111
             ^

Finding the Faulty RAM

In order to figure out which part of the RAM was broken, we first need to understand the DRAM chips' physical layout on the PCB.

The CBM 3016 series' DRAM consists of 16 MOSTEK MK4108 modules, each in a DIP16 package, organized in 2 groups of 8. Reading the datasheet of the MK4108/4116 (p.140ff), reveals that each one stores 8K single bits. Probing around on the PCB with a multimeter revealed the following memory layout:

               D0    D1    D2    D3    D4    D5    D6    D7
               __    __    __    __    __    __    __    __
              |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
              |  |  |  |  |  |  |\/|  |  |  |  |  |  |  |  |
$0000:$1fff   |  |  |  |  |  |  |/\|  |  |  |  |  |  |  |  |
              |__|  |__|  |__|  |__|  |__|  |__|  |__|  |__|

               __    __    __    __    __    __    __    __
              |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
              |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
$2000:$3fff   |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
              |__|  |__|  |__|  |__|  |__|  |__|  |__|  |__|


                            VVVV  FRONT VVVV

Since the flipped bit was in the 4th position (counting from the LSB of the upper byte), and the stack (where the wrong return address was read from) is at $0100:$01ff, the faulty module should be the D3 module in the upper row, marked with an X. To validate this, we swapped the module with the D3 module in the front row - luckily these DIP modules are socketed. Lo and behold, the computer started and we were presented with the BASIC interpreter. Note that instead of the usual ~15K memory, only 7K are reported as free and usable:

Figure 5: The CBM 3016 showing the BASIC interpreter. 7K of memory are reported as free, instead of the expected 15K.

Now that the system had at least some working memory, I decided to write a small memtest program. I ended up writing this directly in 6502 assembly, written to a ROM plugged into the $f000:$ffff socket normally occupied by the KERNAL ROM, due to multiple reasons:

The KERNAL actively uses the memory region $0000:$2000, so loading the memtest program from another location after the KERNAL has been loaded would corrupt this memory region.
An entire memory page is occupied by the stack, but this page should be tested as well. Writing in assembly allowed me to avoid implicitly using the stack, or any other implicit memory allocations.

The memtest program only uses 7 bytes of zeropage memory to keep its state, the entire remainder of the memory is tested. For more efficient testing (and to not waste tons of single-write PROMs) I also ended up wrapping a MOS 6502 emulator with the CBM 3016's memory map and simulated memory failures. The memtest program can be found on Gitlab.

Flashing the memtest16k image to a PROM and booting it on our CBM 3016 revealed memory errors for every single address in the upper half of the DRAM address space, $2000:$3fff. Most of these were indeed caused by the faulty memory module we had already identified earlier, however, there have also been some additional reports caused by a different module, as they had happened at a different bit.

Figure 6: Memtest running on the CBM 3016 and reporting errors in the entire upper half of the DRAM address space. Each entry consists of an address, the byte written to said address and the wrong byte that was read back from RAM.

Replacing Defective RAM Modules

We assumed it would be quite hard to find compatible memory modules. However, in our hackerspace I found a Commodore C64 which had already been scavenged for spare parts, so there were no hard feelings in taking its memory as well. The C64 has 8 memory modules, each holding 64Kib. These modules, Micron MT4264-15 were mostly pin-compatible to the MK4108/4116; they have an additional address pin, but fewer power pins (they run on 5V only, rather than the MK4108's triplet of -5V, +5V and +12V). Making an adapter socket was trivial, the hardest part was desoldering the DIP24 package from the C64 PCB; instead of using DIP sockets like in the PET/CBM series, the C64 has its RAM soldered directly to the PCB.

Figure 7: Left: The pin mapping adapter schematic for using MT4264 chips in a socket designed for MK4108/4116 DRAM. Right: The replacement memory module on its adapter socket.

Once we had replaced the one RAM module that was entirely broken, we started another memtest run, and no errors were reported at all. This came as a surprise, since based on the results from previous memtests we assumed two modules had to be faulty. We then reinserted the KERNAL ROM, and finally the CBM 3016 started up with its full memory size available:

Figure 8: The CBM 3016 BASIC interpreter screen, now reporting 15K of free memory.

Layer 2 TFTP Relay for TFTP Recovery with IP Conflicts

2022-01-09T00:00:00+01:00

Today I had to deal with an embedded device which refused to boot up due to a misconfiguration. Normally, this device, a Mitel DECT RFP 32 IP, obtains a DHCP lease on boot and fetches the firmware to run via TFTP. The firmware is downloaded at every reboot, since the device can not store it locally.

It can however store some boot options in flash, such as static IP and TFTP configuration. And this is where the problem started: Someone had accidentally configured 192.168.42.2 for the device's own IP AND the TFTP server IP. Meaning the device would boot up, assign itself a statically configured IPv4 address and then attempt to fetch the boot image via TFTP from exactly the same IP. Said TFTP traffic of course never left the device.

Flash Chip Short-Circuit

As a first measure, together with a friend, we took the device apart and checked whether any of the chips on there was a flash storage chip, and indeed we found one: An 8K I²C flash chip. Hoping that this is where this configuration was stored, we shorted the I²C bus' SDA to GND in order to disable communication with said chip.

Unfortunately this didn't do anything - apparently the boot configuration is not stored on this chip. And unfortunately we didn't find another storage chip on the PCB. So I decided to try another approach.

At least probing around the board we found a serial console, on which the device wrote log output during the boot process, which did help a lot.

ARP Spoofing

I first attempted ARP spoofing to convince the device to send its TFTP requests to the outside world:

$ sudo pacman -S dsniff
$ sudo arpspoof -i enp2s0 -c both 192.168.42.2

And indeed, amidst all the unsolicited ARP replies TFTP requests started popping up! So, at least one direction was working now. I did have to assign the same 192.168.42.2 to my notebook computer running the TFTP server. After doing so, the TFTP server started responding.

However, the TFTP response did not make it out onto the network, as the destination IPv4 was assigned to the notebook itself. I tried to mess around with ebtables/arptables a bit, but didn't find anything that worked as intended.

Layer 2 Relay

So I decided to attempt to get it to work by having the TFTP server listen on localhost only, and write a small piece of software which would relay the network traffic between localhost and the LAN.

I originally planned to simply open two raw sockets (i.e. sending and receiving full Ethernet frames), one bound to the loopback interface, and one bound to the LAN interface. This piece software would then simply relay the Ethernet frames between the two interfaces, but replace MAC and IP addresses as needed, and recalculate the IPv4 header checksum along the way:

Figure 1: Visualization of the first layer 2 relay approach. The frames are passed almost as-is, only the MAC addresses and IP addresses are replaced, and the IPv4 header checksum is recomputed.

However, for some reason this did not work out quite as intended. It turned out that the frames sent out to the loopback interface also ended up being caught by the relay software and be sent out to the LAN, somehow disrupting TFTP traffic. The workaround I ended up with was to use a regular UDP datagram socket on the loopback interface instead. With this approach, frames coming in on the LAN interface would be stripped of their Ethernet, IP and UDP headers and the payload forwarded to the loopback UDP socket. For responses from the TFTP server, on the other hand, the relay had to construct these headers and send a full Ethernet frame out to the LAN:

Figure 2: Visualization of the layer 2 / UDP relay approach. This time, there is a UDP datagram socket on the loopback side, and protocol overhead is removed and added in the relay.

And it worked! At least at first: The first client - server - client roundtrip went through. However, when the client attempted to acknowledge this transaction, things started to fall apart. Closer inspection of the network traffic revealed that the UDP payload sizes didn't match. The TFTP ACK was much longer than it should have been, and the excess bytes looked oddly familiar.

As it turns out, the DECT RFP is reusing the same buffer for outgoing TFTP messages over and over again, but when sending those messages out to the network, they are not limited to the actual message length; instead the full buffer is sent. The length fields in the UDP and IP headers were correctly set to only include the actual message though. The remainder of the buffer was just noise at the end of the Ethernet frame.

Figure 3: Visualization of the TFTP buffer reuse issue. Note how the IPv4 and UDP headers terminate at the end of the ACK message, but the Ethernet frame still contains the entire buffer.

This caused problems in the relay software, since I was deducing the IP and UDP length fields from the total length of the received Ethernet frame. Reading the actual payload size from the UDP header instead of just assuming its size resolved the problem.

And with that, it was finally working: The DECT RFP successfully pulled its firmware image and booted. From this point, we were able to perform a factory reset, which switched it back to DHCP.

Conclusion

It turns out you can make two hosts on the same network talk to each other even when they have the same IP address. You just need to take a dive down to the Ethernet layer and take routing decisions away from your Operating System. The relay software I wrote for this purpose can be found on Gitlab. It still needs to be used in conjunction with arpspoof.

Though, in hindsight, it would probably have been possible to get the TFTP responses out onto the network simply by messing with my notebook computer's ARP table. However, I didn't think of this at the time and haven't had an opportunity to try since. (Not gonna intentionally brick that device again just for testing purposes.)

Chainload GRUB 2 from GRUB 2

2021-11-27T00:00:00+01:00

I recently needed to set up a PC to dual boot Windows and Linux. Legacy BIOS, no UEFI. Unfortunately Windows likes to do Windows things and override the first-stage GRUB bootloader in the MBR with its own, and thereafter booting Windows only.

As this PC needs to be used by different persons with different technological skillsets, and in order to make repairing the boot process easier even if e.g. no Linux live boot medium were available, I came up with the following solution which loads GRUB via PXE to repair the "real" bootloader.

This solution is quite easy for end users. In the case of this specific PC, PXE boot can be triggered by keeping F8 pressed during boot. The instructions to do so fit on a small post-it note which I attached to the PC's screen.

Set Up DHCP and TFTP

You'll need:

A configurable DHCP server
A TFTP server
A device with GRUB 2 already installed

All of those can of course be on the same device. Sometimes the first two even have to be, since a lot of consumer-grade network cards can only load data via TFTP from the same device that's running the TFTP server. Basically they ignore and don't request DHCP option 66 (server-name), and only handle DHCP option 67 (bootfile-name) and attempt to load it via TFTP from the address that offered the DHCP lease.

First we need to set up the TFTP server. I'll use dnsmasq for both TFTP and DHCP. TFTP configuration in dnsmasq is as easy as the following two lines in /etc/dnsmasq.conf:

enable-tftp
tftp-root=/srv/tftp

We then need to configure a static DHCP lease with additional DHCP options. Again in dnsmasq.conf:

# "pool" is the pool name
dhcp-range=pool,192.168.0.10,192.168.0.254,24h
# "target" is the target's host name. "tftp-grub" is a "tag" used in the next two lines.
dhcp-host=01:23:45:67:89:0a,set:tftp-grub,192.168.0.10,target
# Set TFTP server and bootfile path
dhcp-option=tag:tftp-grub,option:tftp-server,192.168.0.1
dhcp-option=tag:tftp-grub,option:bootfile-name,/grub/i386-pc/core.0"

Create a GRUB 2 Netboot Environment

On the DHCP server (can also be done on a different host and then copied over), run the following command:

grub-mknetdir --net-directory /srv/tftp --subdir grub -d /usr/lib/grub/i386-pc/

The main difference between a system's /boot/grub and this netboot environment is the grub/i386-pc/core.0 file. This is GRUB's first-stage loader, the same thing that normally gets written to the MBR, but here it's in a file instead, which can be loaded via TFTP.

If you now tell your target machine to boot from PXE, you should get a GRUB console.

Chainload GRUB 2 from GRUB 2

I searched around a long time to figure out how this is done, and most of the results i got said to "just load the second GRUB's config file into the first GRUB instance". This may work in most cases, but will eventually break in this case, since the second GRUB's config file may attempt to load additional modules from it's own prefix (/boot/grub/i386-pc), which will be incompatible with the PXE-loaded GRUB if they're not the same version.

Long story short, unlike GRUB 1, GRUB 2's second-stage loader (the core.img file) cannot be loaded on its own. Only the first-stage loader (from MBR or core.0) can be chainloaded. And since we're trying to cover the case where the first-stage loader has been overwritten, we need to get it from somewhere else.

In this case I decided to set up a small loopback image in the target system's root partition which contained a copy of the system's GRUB installation.

To set up this loopback image, run the following commands on the target system:

dd if=/dev/zero of=/grubloop bs=100M count=1
parted /grubloop mklabel msdos
# Offset from the start is needed so that there's enough space for
# the first-stage loader.
parted /grubloop mkpart primary ext4 1048576B 100%
parted /grubloop set 1 boot on
losetup -fo 1048576 /grubloop
# Loop device number may be different
mkfs.ext4 /dev/loop0
losetup -d /dev/loop0
mkdir /altboot
mount -o offset=1048576 /grubloop /altboot

Also enter this into your /etc/fstab:

/grubloop   /altboot    ext4    loop,offset=1048576 0   0

Finally install GRUB to /grubloop resp. /altboot:

grub-install --boot-directory=/altboot --target=i386-pc --force /grubloop
grub-mkconfig -o /altboot/grub/grub.cfg

This image, which now contains the same boot config as your /boot/grub, can now be booted like you'd boot any hard disk. To chainload this GRUB image from the PXE grub, create the file /srv/tftp/grub/grub.cfg on your TFTP server and add the following:

insmod biosdisk
insmod part_msdos
insmod ext2
insmod chain

# Adapt to your real Linux root partition
set root=(h0,msdos4)
chainloader /grubloop +1
boot

I'm intentionally hardcoding the root partition here, instead of using GRUB's search module, since I want the PXE boot to be fast and simple. The search module, on the other hand, loads a LOT of other modules via TFTP, which may become a performance issue.

When you now boot your target device via PXE, it will load the first GRUB instance from TFTP, which will then chainload the second GRUB instance from the loopback image, which will then boot into Linux.

Automatically Re-Install GRUB

I'm doing all of the above not only to boot via PXE, but also to automatically re-install GRUB to the disk's MBR. I achieved this with the following systemd unit, which is executed on every boot of the Linux system:

[Unit]
Description=Reinstall GRUB 2 to MBR and loopback altboot

[Service]
Type=oneshot
ExecStart=/usr/sbin/grub-install --boot-directory=/altboot --target=i386-pc --force /grubloop
ExecStart=/usr/sbin/grub-install --boot-directory=/boot --target=i386-pc /dev/disk/by-id/<DISK_ID>
ExecStart=/usr/sbin/grub-mkconfig -o /altboot/grub/grub.cfg
ExecStart=/usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg

[Install]
WantedBy=multi-user.target

Maybe a bit overkill, but it gets the job done and is easy to use for end users.

DNSSEC Automation with Knot - Automated KSK Rollover

2021-11-16T00:00:00+01:00

UPDATE: Due to how KSK algorithm rollovers work, cds-cdnskey-publish: double-ds should not be used. The configuration example below has been updated to use cds-cdnskey-publish: always instead.

In the previous post I've shown how to bootstrap DNSSEC in a fully automated fashion using the Knot DNS server and a CDS-enabled domain registry. (Again, when I'm saying CDS, this is meant to include CDNSKEY as well.)

Now I'm going to take this a step further and show how to have all the tasks required for a KSK rollover performed automatically by Knot.

The KSK Rollover Process

We start out with an already established trust relationship between a zone and its parent. Same as before, I'll use my s3lph.ch. domain as an example.

Figure 1: Visualization of the DNSSEC chain of trust before beginning the KSK rollover process, as rendered by the online tool DNSViz. Again, I've added the CDS nodes afterwards. The trust relationship between the zone and its parent is established, and there is one active KSK and one active ZSK.

When the current KSK is about to expire, Knot will automatically generate a new KSK and publish it in the zone. It is signed by the current KSK:

Figure 2: Visualization of the DNSSEC chain of trust after publishing the new key. The new KSK (42001) is signed by the current key (46451).

Once Knot can assume that all other nameservers in the zone have replicated the new DNSKEY set (after the zone TTL has expired), a corresponding CDS record is published. The DNSKEY and CDS records are signed by both the old and the new key.

The exact behavior regarding the creation of CDS records depends on the config option cds-cdnskey-publish: If you chose double-ds (which I did here, please check the update at the top of the page), then records for both the old and the new KSK will be published simultaneously during the rollover process. If you chose rollover or always, only a CDS record for the new key is published.

Figure 3: Visualization of the DNSSEC chain of trust after publishing the new CDS record. The CDS records and the ZSK are now signed by both keys.

Knot will wait for the parent registry to publish the new DS record. Now we have reached a "symmetry" in the rollover process where both keys are equally trusted and used for ZSK signing:

Figure 4: Visualization of the DNSSEC chain of trust after the new DS record has been published. The ZSK is still signed by both KSKs, both of which are now trusted.

Knot will periodically check for existence of the DS record. Once it has been found, the CDS records are updated to only refer to the new key:

Figure 5: Visualization of the DNSSEC chain of trust after Knot has removed the old CDS record. Both keys are still being used for ZSK signing, however, only the new CDS record is published.

A short while after (again after having waited until all other nameservers have had time to replicate the changes) Knot will retire the old key, meaning it's no longer being used for ZSK signing. The zone gets resigned using only the new KSK:

Figure 6: Visualization of the DNSSEC chain of trust after the old key has been retired. Now only the new key (42001) is used for ZSK signing.

Sometime afterwards the parent registry will scrape for CDS records again and discover the new CDS RRset, which now only contains the new CDS record. This will cause the old DS record to be removed from the parent zone, only the new DS record remains:

Figure 7: Visualization of the DNSSEC chain of trust after trust for the old key has been removed.

Finally, some time afterwards, Knot will remove the old key from the zone, and only the new chain of trust remains:

Figure 8: Visualization of the DNSSEC chain of trust after the old key has been removed.

Set Up Knot for Fully Automated KSK Rollovers

Let's have a look at /etc/knot/knot.conf again, once more reduced to the relevant bits:

zone:
  - domain: s3lph.ch.
    dnssec-signing: on
    dnssec-policy: dnssec-s3lph.ch.

policy:

  - id: dnssec-s3lph.ch.
    # These options are explained in the previous blogpost. 
    algorithm: ecdsap384sha384
    nsec3: on
    cds-cdnskey-publish: always
    # ZSKs are valid for 30 days by default, KSKs are valid indefinitely
    # This instructs knot to initiate a KSK rollover every 180 days.
    ksk-lifetime: 180d
    # This will add an additional backoff time to account for cache ttl.
    # Choose this appropriately to your zone's TTL.  Defaults to 1h.
    propagation-delay: 1h
    # This tells Knot how to check for new DS records after the key has
    # been published.
    ksk-submission: submission-s3lph.ch.

submission:

 - id: submission-s3lph.ch.
   # How often to check for new DS records.
   check-interval: 1h
   # Nameservers to check for new DS records, must be configure in the
   # "remote" section below. Using an external, DNSSEC-validating
   # resolver makes sure the old DS records aren't cached anymore.
   parent: remote-1.1.1.1

remote:

- id: remote-1.1.1.1
  address: 2606:4700:4700::1111
  address: 1.1.1.1

This is all that's required to get fully automated KSK rollover running. A bit more config than for the last blogpost, but still manageable. Again the config options are documented in detail in the Knot Configuration Reference.

Conclusion

I don't think KSK rollovers have ever been this easy; this is basically a "fire and forget" operation and everything's taken care off afterwards.

You can even make use of these features if your domain registry is not CDS-enabled. In this case there's one manual step involved though: You have to manually submit the new DS record to your registry, usually through your registrar's config interface. Everything else, including the auto-detection of the new DS record in the parent zone will still work automatically. You'll of course have to rely on your monitoring to know when a KSK rollover is due.

You're monitoring your DNS, right? ... Right?

DNSSEC Automation with Knot - Automated Bootstrapping

2021-11-14T00:00:00+01:00

Knot DNS is an authoritative DNS name server developed by CZ.NIC. One of the features that sets it apart from BIND and other nameserver software is its support for fully automated DNSSEC key management. In this blogpost I'll show how to achieve fully automated DNSSEC bootstrapping, assuming that the parent zone implements trust management via CDS/CDNSKEY records.

RFC 8078: CDS and CDNSKEY

Usually, in order to bootstrap a DNSSEC trust relationship, a series of steps is necessary:

A KSK and ZSK or CSK must be created for the child zone (e.g. s3lph.ch.).
The public keys must be published in the child zone as DNSKEY records.
The zone's records must be signed with the ZSK, with the exception of the DNSKEY records, which must be signed by the KSK.
A DS record matching the KSK must be created.
The DS record must be published in the parent zone (e.g. ch.).

Most of these steps can be automated and contained within a single piece of software such as Knot, but the last step usually requires a human to sign into their domain registrar's web management interface and configure the DS record manually.

RFC 8087 describes a mechanism to automate this last step. In order to achieve this, two new DNS RR types were introduced: CDS and CDNSKEY. They serve as a way for the child zone to tell the parent zone "I want the parent zone to serve DS records for these KSKs". Both types serve the same purpose, the only difference is that the CDNSKEY records are copies of the DNSKEY records for which a DS record should be created in the parent zone, whereas the CDS records are already-prepared DS records for the parent zone to copy. This can not only be used for bootstrapping DNSSEC upon introduction, but also for maintaining the chain of trust when rolling over to a new KSK.

The procedure to bootstrap or update a trust relationship using CDS/CDNSKEY records boils down to:

The CDNSKEY and/or CDS records are published in the child zone.
The parent zone scans all its child zones for CDNSKEY/CDS changes on a regular basis (e.g. once a day). Usually this is done by the registry operating the zone.
DS records are computed from the CDNSKEY/CDS records and inserted into the parent zone.

Usually this last step is bound to a set of policies defined by the registry. For example, the CDS policies of SWITCH, the registry for the ch. and li. TLDs, require among other things:

The set of CDS records must be valid and consistent across all of the zone's nameservers
The new DS records must not break the chain of trust
the CDS records are signed with an already trusted KSK

Note: From hereon, every mention of CDS will imply CDS and/or CDNSKEY unless otherwise noted.

DNSSEC Bootstrap Example

Let's look at an example and have DNSSEC automatically bootstrapped for the s3lph.ch. zone. At first, immediately after setting up Knot to enable DNSSEC, keys are generated, the zone is signed and the keys and CDS records are published in the zone:

; KSK and ZSK
s3lph.ch.   300 IN  DNSKEY  257 3 14 zjmE...AnD5 ; KSK ; key id = 46451
s3lph.ch.   300 IN  DNSKEY  256 3 14 fSZj...d+gU ; ZSK ; key id = 19881
; ZSK signature, signed with KSK
s3lph.ch.   300 IN  RRSIG   DNSKEY ... 46451 ...
; CDS referencing the KSK
s3lph.ch.     0 IN  CDS     46451 14 2 C129...DDC0
; CDS signature, signed with KSK
s3lph.ch.     0 IN  RRSIG   CDS ... 46451 ...
; AAAA record, signed with ZSK
s3lph.ch.   300 IN  AAAA    2a01:4f8:1c1c:1ae7::1
s3lph.ch.   300 IN  RRSIG   AAAA ... 19881

However, the ch. zone does not yet know about DNSSEC in the s3lph.ch. zone:

s3lph.ch.   3600    IN  NS  ns1.s3lph.me.
s3lph.ch.   3600    IN  NS  ns2.s3lph.me.
S8BEAUIIVUO5RD59BCLAL4MK7MDKBIOC.ch. 900 IN NSEC ... NS DS RRSIG
S8BEAUIIVUO5RD59BCLAL4MK7MDKBIOC.ch. 900 IN RRSIG NSEC3 ...

You can also see the chain of trust visualized in Figure 1. The foundation for this image has been created using the the online tool DNSViz resp. it's eponymous backend tool. I've added the CDS node manually afterwards, since DNSViz does not (yet) support CDS.

Figure 1: Visualization of the DNSSEC chain of trust before CDS bootstrapping. Note the black arrow between ch. and s3lph.ch. indicating insecure delegation, as well as the NSEC3 record indicating the absence of a DS record.

One day later, we can see that SWITCH has successfully picked up the CDS record:

Figure 2: SWITCH's CDS Status Check tool reporting "pending" status after the first successful validation.

However, SWITCH requires the CDS RRset to remain unchanged for 3 days when bootstrapping DNSSEC on a zone, so they haven't published the matching DS record yet.

Let's fast-forward to 3 days later: There now is a DS record in the ch. zone:

s3lph.ch.   3600    IN  NS      ns1.s3lph.me.
s3lph.ch.   3600    IN  NS      ns2.s3lph.me.
s3lph.ch.   3600    IN  DS      46451 14 2 C129...DDC0
s3lph.ch.   3600    IN  RRSIG   DS ...

Figure 3: Visualization of the DNSSEC chain of trust after CDS bootstrapping, as rendered by DNSViz. Again, the CDS record was added afterwards. Note the arrow between ch. and s3lph.ch. has turned blue, indicating secure delegation, as well as the new DS record in the ch. zone.

And there we have it: The DNSSEC trust relationship between ch. and s3lph.ch. has been established without any manual interaction.

Now let's look into how to set up Knot for this.

Set Up Knot for Fully Automated DNSSEC Key Management

Let's have a look at my /etc/knot/knot.conf. I've boiled it down to the relevant pieces:

zone:
  - domain: s3lph.ch.
    # This option tells Knot to enable automatic zone signing ...
    dnssec-signing: on
    # ... and this tells Knot HOW to sign the zone.
    dnssec-policy: dnssec-s3lph.ch.

policy:
  # This is the Knot DNSSEC signing policy for s3lph.ch.
  - id: dnssec-s3lph.ch.
    # SWITCH doesn't yet support ed25519/ed448, so let's use some other
    # recommended algorithm, ECDSA using the P-384 curve, plus SHA384.
    algorithm: ecdsap384sha384
    # Automatically create NSEC3PARAMs as well, and respond with NSEC3
    # instead of regular NSEC.
    nsec3: on
    # This is the magic bit to turn on CDS and CDNSKEY record generation.
    # "always" means that a CDS record is always published, not only during KSK rollovers.
    # Before, I had used "double-ds" here, which publishes a CDS RRset for both the
    # old and the new key during rollover, which unfortunately breaks when rolling over to
    # a different KSK algorithm (see update at the top of this page).
    cds-cdnskey-publish: always

And that's already it; those few lines are enough to have Knot automatically sign zones and generate CDS and CDNSKEY records on the fly.

There are other options available for the cds-cdnskey-publish option, you can look up the list in the Knot Configuration Reference.

Conclusion

If your registry supports CDS or CDNSKEY, this is a really simple method to bootstrap DNSSEC without any manual intervention. Even more, this still works if the registrar through which you bought your domain doesn't offer a way to configure DS records.

This procedure can also be used to make KSK rollovers fully automatic, which I'll be covering in the next blogpost.

Mein Einbürgerungverfahren in Basel - Teil 2: Vorbereitung und Gesuchsstellung

2021-09-20T23:16:00+02:00

Achtung: Die Informationen in diesem Artikel beziehen sich auf das Verfahren im Kanton Basel-Stadt, die Verfahren in anderen Kantonen weichen z.T. stark ab.

Normalerweise muss man sich vor der Einreichung des Einbürgerungsgesuchs vom Migrationsamt beraten lassen. Pandemiebedingt ist die Beratung momentan aber nur telefonisch oder via E-Mail möglich.

Beratung via E-Mail

Einen Grossteil der Informationen zur Einbürgerung lässt sich online finden. Eines der wichtigsten Dokumente - die Checkliste, welche Dokumente überhaupt einzureichen sind - wird aber nur bei der Beratung ausgehändigt.

In meinem Fall hatte sich die Beratung auf einen kurzen E-Mail-Austausch beschränkt: Ich habe meine Situation geschildert und ein paar Fragen zu den Online-Informationen gestellt, und hatte schon am nächsten Tag eine Antwort in der Inbox, mit der Checkliste im Anhang. Soweit, so gut. Allerdings fände ich es durchaus begrüssenswert, wenn die Checkliste online verfügbar wäre, allein nur schon deswegen, damit man den Aufwand für das Zusammentragen der Dokumente im Voraus abschätzen kann.

Die Checkliste ist sehr umfangreich. Neben der Checkliste selbst sind unter anderem folgende Dokumente einzureichen:

Das Gesuchsformular, in dem man Angaben zur Person, Wohnsitzfristen etc. macht
Kopien von Ausweisen und Aufenthaltstiteln
Wohnsitzbestätigungen, Betreibungsregisterauszüge und Steuerausweise von ehemaligen Wohnkantonen (in meinem Fall der Kanton Aargau)
Ein Registerauszug vom Zivilstandsamt, mehr dazu weiter unten
Nachweise über Berufstätigkeit, Ausbildung o.Ä.
Ein Sprachnachweis, wobei in meinem Fall die Selbstdeklaration genügt, deutsch als Muttersprache zu sprechen
Ein Formular über die Erfüllung der Einbürgerungsvoraussetzungen

Zusammenstellung der Dokumente

Die meisten Dokumente sind schnell zusammengetragen: Wohnsitzbestätigungen und Betreibungsregisterauszüge lassen sich bei den meisten Gemeinden online bestellen. Ausweiskopien, Arbeitgeberbestätigung und Ausbildungsnachweis sind ebenfalls schnell parat. Der Steuerausweis kann gemäss Information auf der Checkliste erst nach Erhalt des Zivilstands-Registersauszugs bestellt werden.

Das Gesuchsformular und Formular über die Einbürgerungsvoraussetzungen sind ziemlich umfangreich, man muss unter anderem Angaben machen zu:

Ehemaligen Wohnsitze innerhalb und ausserhalb der Schweiz
Die Ausbildungs- und Schullaufbahn zurück bis zum Kindergarten
Art der finanziellen Absicherung (Berufstätigkeit, Rente, etc.)
Vorstrafen, Gerichtsverfahren und/oder unbeglichen Schulden
Teilnahme am sozialen Leben (z.B. Vereine, Hobbies)
Art der Kontakte zu Schweizer Bürger*innen

Zudem muss man 4 Referenzpersonen angeben, die - mit Ausnahme von Vorgesetzten oder Lehrpersonen - das schweizer Bürgerrecht haben müssen.

Besuch auf dem Zivilstandsamt

Der Registerauszug des Zivilstandsamts muss auch trotz Pandemie persönlich vor Ort bestellt werden.

Mein erster Besuch auf dem Zivilstandsamt ging ziemlich schnell und war ziemlich informativ: Um den Auszug bestellen zu können, muss ich erst noch ein paar Dokumente einreichen... Das meiste davon hatte ich schon parat, allerdings musste ich noch eine aktuelle Geburtsurkunde von meinem Geburtsort in Deutschland bestellen. Die Sachbearbeiterin gab mir noch eine Checkliste mit, und damit war das Gespräch auch schon erledigt. Zudem hatte sie mir noch den Tipp gegeben, mit dem Bestellen von anderen Dokumenten wie Wohnsitzbestätigungen abzuwarten, da die Eintragung im Zivilstandsregister und anschliessende Erstellung des Auszugs einige Wochen dauern könne.

Nach einigen Tagen hatte ich die Geburtsurkunde im Briefkasten, und bin zum zweiten Mal aufs Zivilstandsamt, dieses mal ausgestattet mit allen benötigten Dokumenten und den Gebühren in bar.

Einreichung des Gesuchs

5 Wochen nach der Bestellung auf dem Zivilstandsamt wurde ich nochmal via E-Mail eingeladen, um die erfassten Daten nochmal zu bestätigen und den Registerauszug abzuholen. Danach musste ich noch den Steuerausweis beim Finanzdepartement des Kantons bestellen, dieser war trotz Bestellung auf dem Postweg aber bereits 2 Tage später im Briefkasten.

Nun hatte ich endlich alle Unterlagen zusammengetragen, und habe das Einbürgerungsgesuch via Post ans Migrationsamt geschickt. Seit der ersten Beratung hat es fast 10 Wochen gedauert, um die benötigten Dokumente zu besorgen. Aber das lange Warten beginnt erst jetzt...

Mein Einbürgerungverfahren in Basel - Teil 1: Intro

2021-09-20T13:15:00+02:00

Letzte Woche habe ich mein Einbürgerungsgesuch beim Migrationsamt in Basel eingereicht. Damit beginnt für mich das langwierige (1.5-2 Jahre) Einbürgerungsverfahen, über das ich hier im Blog berichten werde, da ich bisher kaum ausführliche Erfahrungsberichte im Internet finden konnte, und quasi gar keine aus Basel.

Hintergrund

Ich lebe seit 10 Jahren in der Schweiz, bin aber in Deutschland geboren und deutscher Staatsbürger. In Basel lebe ich seit zwei Jahren, vorher habe ich im Kanton Aargau (zwischen Basel und Zürich) gelebt.

Als Ausländer mit einer Niederlassungsbewilligung habe ich weitestgehend die gleichen Rechte wie Schweizer Staatsbürger, darf aber z.B. nicht an Wahlen und Abstimmungen teilnehmen. Mit der Einbürgerung entfallen diese Restriktionen, dafür kommen aber auch neue Verpflichtungen dazu.

Der Theorie-Teil

Das Bürgerrecht in der Schweiz gliedert sich in drei Teile:

Das Staatsbürgerrecht der Schweiz
Das Kantonsbürgerrecht eines Kantons
Das Gemeindebürgerrecht einer Gemeinde

Den Kantons- und Gemeindebürgerrechten kommt heutzutage nur noch wenig praktische Bedeutung zu, trotzdem können alle drei Rechte nur gemeinsam erlangt werden. Entsprechend aufwändig ist das Einbürgerungsverfahren, da es sowohl vom Kanton, der Gemeinde und dem Bund bearbeitet wird, und jede Ebene eigene Voraussetzungen für die Erteilung des ihres jeweiligen Bürgerrechts festlegt.

Dementsprechend bezieht sich mein Einbürgerungsgesuch auf das

Schweizer Staatsbürgerrecht
Kantonsbürgerrecht des Kantons Basel-Stadt
Gemeindebürgerrecht der Stadt Basel

Die eben erwähnten Anforderungen sind zum Beispiel:

Für das Staatsbürgerrecht muss man min. 10 Jahre in der Schweiz gelebt haben und min. eine der Landessprachen sprechen. Zudem muss man darlegen, dass man finanziell abgesichert ist, keine Sozialhilfe bezieht und keine Steuerschulden hat.
Der Kanton Basel-Stadt verlangt den Wohnsitz im Kanton während der letzten 2 Jahre, und spezifiziert, dass man deutsch sprechen können muss.
Die Bürgergemeinde der Stadt Basel prüft, ob die Gesuchsstellenden in der Schweiz und im Kanton integriert sind, und führt ein entsprechendes Einbürgerungsgespräch.

Im Gegensatz zu diversen anderen Kantonen gibt es in Basel-Stadt keinen schriftlichen Einbürgerungstest, stattdessen werden entsprechende Kenntnisse im Einbürgerungsgespräch der Bürgergemeinde abgefragt.

Extracting 3D Models From CesiumJS - Part 2: Terrain Map Parsing

2021-05-09T00:00:00+02:00

CesiumJS is a open source JavaScript framework for rendering 2D and 3D maps - everything from a local area to whole planets - in a web browser using WebGL. In the past few weeks I've been working on obtaining 3D model data in a situation where the only easily available way of accessing the data is through a CesiumJS based viewer. As far as I know, Cesium deals with two different kinds of 3D data: On one side, there's 3D models used for small-scale objects like buildings or trees, on the other side there's terrain maps.

The quantized-mesh File Format

In the previous article, I wrote about how to scrape quantized-mesh-formatted terrain map tiles from a CesiumJS web app. At the end of the previous article, we were left with a bunch of .terrain files.

Luckily, the quantized-mesh format is documented fairly well in the specification:

The file starts with an 88-byte header consisting of:

The center of the tile in earth-centered Cartesian coordinates.
The height range covered by this tile, as distance from the earth's center. You'll see later why this is required.
The parameters (center and radius) of the tile's bounding sphere.
Coordinates of a «horizon occlusion point», used to simplify rendering: The point is chosen so that if it lies below the horizon, the entire tile is behind the horizon, and thus invisible, and does not need to be rendered. Cesium wrote an entire blogpost about this.

After the header follows the vertex data consisting of:

The number of vertices.
An array of horizontal coordinates, 0 means westmost, 32767 eastmost, everything in between is linearly interpolated.
An array of vertical coordinates, 0 means southmost, 32767 northmost.
An array of heights. Here the height range mentioned earlier comes in: 0 encodes the lower height bound, and 32767 encodes the upper bound.

The vertex data (the contents of the three arrays) follows a rather odd encoding with two specialties:

Each vertex' data is not encoded as absolute values, but as deltas to the data of the preceding vertex. Only the first vertex' data is encoded with absolute values.
The individual numbers (unsigned 16-bit integers) are not encoded as two's complement, as is common in today's computing, but as something Cesium calls "zig-zag encoding": The sequence of encoded numbers alternatingly encode positive and negative numbers, as depicted in Figure 1.

Luckily, the quantized-mesh specification provides pseudocode for decoding this data.

Figure 1: quantized-mesh zig-zag encoding visualized on a number line. The unencoded, "real" numbers are shown in black on top, the zig-zag encoded data in green on the bottom.

The specification states that this encoding is chosen «in order to make small integers, regardless of their sign, use a small number of bits».

I'm not entirely sure why they want to achieve this (after all, each number takes up the same amount of space, no matter how many bits are zero), but I suspect this is done to potentially further reduce the resulting size if the file is compressed, esp. when using HTTP's gzip transport compression.

Following the vertex data is the index data: Similar to a lot other file formats dealing with 3D data, a quantized-mesh file ultimately encodes a set of polygons, or to be more precise, triangles. The index data describes a series of triangles trough:

The number of triangles.
An array of vertex indices, with three indices per triangle, one for each corner of a triangle.

A vertex index is simply the index of a vertex in the aforementioned vertex data. This indirection is used because most vertices are part of multiple triangles, so some space can be saved. In addition, the index data is again encoded somewhat specially (but a different kind of special than the vertex data). I did not look too deep into understanding what exactly is done and why it is done, but again I assume this is to make the file more compressible.

Depending on how many vertices the file contains, the indices are encoded as either 16-bit or 32-bit integers, and padding is inserted before the index data to ensure memory alignment.

Finally, the file contains four more vertex index lists, describing which vertices are part of each of the four edges (western, southern, eastern, northern) of the terrain tile.

Optionally, there can be some extension data added to the end of the file, such as normal vectors for lighting computation when rendering the terrain, but this is beyond the scope of this post.

To sum it up, Figure 2 shows an illustration of the file format and which parts encode which features of a 3D terrain.

Figure 2: Visualization of the quantized-mesh file format with an example terrain file. The left-hand side depicts the sections of a quantized-mesh terrain file, the right-hand side shows a 3D terrain rendering with certain features highlighted. The arrows indicate which section describes which type of feature.

Parsing The quantized-mesh Files

I wrote a simple converter script that parses a quantized-mesh file and creates an ASCII STL file containing the terrain. The STL coordinates are in meters, using Cartesian coordinates with the origin in the Earth's center. To be more precise, the coordinates are in EPSG:4978.

You can find the script on Gitlab.

The next article in this series will cover the scraping of 3D building models.

Mailinglist Archive URL Redirection from Pipermail to Hyperkitty

2021-03-17T01:00:00+01:00

I recently migrated a mailing list setup running on Mailman 2 to a new Mailman 3-based setup on a new server. The migration itself is pretty straight forward, especially thanks to the official upgrade guide. The migration from Pipermail (the default archiver used in Mailman 2) to Hyperkitty (Mailman 3 archiver) is easy as well, with Hyperkitty providing a command to import the per-list mbox files maintained by Mailman 2.

However there is one thing that will break: Hyperlinks to the old setup can not all be trivially translated to the new URLs. This is why I came up with the following solution to provide a low-effort redirection service that will get most users to the correct new destination.

URLs to Redirect

Let's first have a look at what kind of URLs are used in the old setup so we can figure out where to redirect them.

Before we start, note that all list-specific Mailman 3 URLs include the full address of the list, not just the local part. You need to pay special attention here if you're operating mailing lists for multiple domains.

/cgi-bin/mailman/listinfo shows a list of all publicly visible lists. This can simply be redirected to the Mailman 3 root URL, i.e. /.
/cgi-bin/mailman/listinfo/test is the welcome page of the test mailing list. It shows the list info and the subscription interface. This can trivially be redirected to the Mailman 3 list welcome page, /postorius/lists/test.example.org/.

I chose to simply redirect every "verb" (the listinfo part) to the welcome page, as most, if not all, other verbs are only to be used by list admins, and they should ideally already know the new location.

/pipermail is the archive landing page listing all public mailing list archives. This can simply be redirected to /hyperkitty.
/pipermail/test is the archive overview of the test mailing list. Redirect to /hyperkitty/list/test@example.com.
There is a bunch of per-month overviews, such as /pipermail/test/2021-March/thread.html for a threaded overview of the posts from March 2021, or /pipermail/test/2021-March/author.html for an overview grouped by author. As Hyperkitty only has a monthly overview by thread, I redirect all of these overviews to the same Hyperkitty monthly overview page, e.g. /hyperkitty/list/test@example.org/2021/3

The timestamp in the URL is definitely human readable, however this is the first obstacle that requires a bit more effort. Of course this could be solved with 12 individual rewrites, each for one month, I chose to incorporate the redirection for this in my solution for the next item.

Finally, there are links to individual messages, such as /pipermail/test/2021-March/001337.html. These turned out to be the greatest challenge: Pipermail uses sequential numbers for addressing individual messages, while Hyperkitty uses a hash of the message's Message-Id header, e.g. /hyperkitty/list/test@example.org/message/7HPT35IASDHNIW6MGBDEDR4LENSIR5F4. Unfortunately, there exists no trivial mapping between these two methods, so I wrote a small PHP script to perform the redirection.

Redirecting Individual Message URLs

As mentioned before, Pipermail uses sequential numbers to identify single messages, while Hyperkitty bases its IDs on the Message-Id header. Unfortunately, while generating the archives, Pipermail removes all information that would be required to match the sequential ID to a Message-Id:

The HTML files know about the sequential ID, but not the Message-Id.
The mbox files and textfile archives contain the Message-Id, but not the sequential ID assigned by Pipermail.

And even if they did, I didn't want to keep the old archives around anyway, and only provide redirection based on the URLs. So how can we achieve this?

As it turns out, when importing the mbox file into Hyperkitty, the order in which the messages are inserted into the database backend of Hyperkitty is the same order in which Pipermail assigns the IDs. Of course this only works flawlessly iff all messages in the mbox file were successfully processed by both Pipermail and Hyperkitty. Otherwise a skew is introduced. However, I only encountered this issue a few times, so I chose to simply correct this manually.

I came up with a little PHP script that takes the list name and the sequential ID as arguments, and looks up the (hopefully) matching message ID in the Hyperkitty database. The SQL query used for this simply fetches a single message ID hash from the row that contains the message with the specified offset:

SELECT e.message_id_hash
FROM hyperkitty_email e
LEFT JOIN hyperkitty_mailinglist m
ON e.mailinglist_id = m.id
WHERE m.name = :listid
LIMIT :postid, 1

The script takes some additional parameters for manual offset correction. It additionally takes care of Pipermails 2021-March timestamps by parsing them with the appropriate srtptime formatter. You can find the script along with a sample Apache rewrite configuration on Gitlab.

Apache Rewrite Rules

To perform all the static redirects explained above, as well as invoke the PHP script, I configured Apache 2 somewhat like this:

# Exempt the conversion script from proxying to mailman3-web
ProxyPass /pipermail2hyperkitty.php !
ProxyPass / unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/


RewriteEngine on

#
# List info pages get rewritten directly through mod_rewrite
#

# Global list overview page
RewriteRule ^/cgi-bin/mailman/listinfo/?$ /

# Individual list overview pages
# some lists may need special handling, e.g. due to renaming or a non-default domain
RewriteRule ^/cgi-bin/mailman/[^/]+/oldlist(/.*)?$ /postorius/lists/newlist.example.net
# Catch-all for all other lists
RewriteRule ^/cgi-bin/mailman/[^/]+/([^/]+)(/.*)?$ /postorius/lists/$1.example.org

#
# All archive URLs get handed over to pipermail2hyperkitty.php
#

# Special handling for some lists, e.g. renamed lists or non-default domains
RewriteRule "^/pipermail/oldlist/([^/]+)/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=newlist@examle.net&ym=$1&postid=$2 [L,PT,QSD]
RewriteRule "^/pipermail/oldlist/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=newlist@example.net&ym=$1 [L,PT,QSD]
RewriteRule "^/pipermail/oldlist((\.|/).*)?$" /pipermail2hyperkitty.php?listid=newlist@example.net [L,PT,QSD]
# The newlist@example.net archive was merged with the oldlist archives, need to know the db offset (= number of posts in oldlist)
RewriteRule "^/pipermail/newlist/([^/]+)/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=newlist@examle.net&ym=$1&postid=$2&idoffset=1337 [L,PT,QSD]
# The messages 23 through 42 are missing from the test mailinglist archives, skew needs to be compensated
RewriteRule "^/pipermail/test/([^/]+)/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=test@example.org&ym=$1&postid=$2&missing=23_42 [L,PT,QSD]

# Catch-all for all other lists
# Everything containing a list name, timestamp and sequential message ID -> to the single message
RewriteRule "^/pipermail/([^/]+)/([^/]+)/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=$1@example.org&ym=$2&postid=$3 [L,PT,QSD]
# Everything containing a list name and timestamp, but no sequential message ID -> to the month's overview page
RewriteRule "^/pipermail/([^/]+)/([^./]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=$1@example.org&ym=$2 [L,PT,QSD]
# Everything containing a list name, but no timestamp or sequential message ID -> to the list's overview page
RewriteRule "^/pipermail/([^/]+)((\.|/).*)?$" /pipermail2hyperkitty.php?listid=$1@example.org [L,PT,QSD]
# Everything else -> to the global archive overview
RewriteRule "^/pipermail((\.|/).*)?$" /hyperkitty [L,PT,QSD]

Conclusion

This is by no means a perfect, bulletproof, or even good solution. And it does not intend to be. I only wrote this script to ease the migration phase for users, and it performs that job well enough.

Migration of IRC Services from Hybserv2 to Anope

2020-12-23T23:00:00+01:00

I recently inherited a small IRC network, consisting of a single server and a rather small but still active community. The software on this server was quite outdated, running a version of ircd-hybrid from 8 years ago, and for IRC services a comparably old version of the now-unmaintained hybserv2.

I decided to not even attempt to get this stuff up to date, but rather replace the server entirely with recent and maintained software. The choice fell on InspIRCd 3 and Anope 2. Switching to a new IRC server is not really that much of a challenge, especially in a single-server network. There is some minor tweaking required, such as making user and channel modes compatible with each other (e.g. InspIRCd 3 dropped core support for the halfop (+h) channel mode, which can hovever be reintroduced with the "customprefix" module). The far greater challenge were the IRC services, as there are some major incompatibilities:

Hybserv2 stores password hashes using the original crypt algorithm based on DES. It would support MD5 instead of DES as well, but that would have to have been enabled beforehand. Anope on the other hand, doesn't support DES. They offer various variants of crypt-style hashes as well, but the "least modern" of those is MD5.
Both Hybserv and Anope store their persistent data in plain text files (Anope does support other methods, such as database backends, but recommends the text file), however the formats of these files are incompatible and neither format is (well) documented.

Password Hash Migration

As it turns out, Anope has built-in functionality for password hash migration: When more than one password hashing module (enc_*) is loaded, only the first one is used for hashing new passwords. The other modules are only used for verifying already existing hashes. On top of that, when a user successfully authenticates against NickServ, and the hash is not in the default format (i.e. the first module), Anope will automatically re-hash the password entered by the user using the default method, and write it to disk.

Using this method, the old DES hashes are replaced by more modern ones as each user logs in, until all hashes have been converted (or the NickServ registrations of inactive users have expired). So the only thing left to do is to write an Anope module to verify DES crypt password hashes. This is pretty straight forward, as all that's needed is a thin wrapper around crypt that conforms to the Anope module API. My implementation can be found on Gitlab along with the rest of the sources. To build the module, and including it in your Anope installation, the following steps are required:

Clone the Anope repo from Github: git clone https://github.com/anope/anope
Checkout the exact version of Anope you intend to run. In my case, this was version 2.0.9, so I'd do git checkout 2.0.9. This step is important, because Anope refuses to load modules built for another version.
Place the enc_des.cpp file in modules/encryption/ in the Anope repo and follow the Anope build instructions:
1. ./Config
2. cd build
3. make. Actually, since we only need the DES module, we can run make enc_des.so instead, which will complete a lot faster.
Link against libcrypt. Since I didn't want to deal with all the CMake stuff, I decided to just patch the .so file afterwards: patchelf --add-needed libcrypt.so.1 modules/enc_des.so. Replace the actual version of libcrypt.so by whatever is available on your target system.
Put the built module into your Anope installation's modules directory. If you're using the Anope Debian package, this would be /usr/lib/anope/modules/enc_des.so.

The final step left to do is to configure Anope to load the module:

// Used for new passwords and converting old passwords
module { name = "enc_bcrypt" }
// Used for verification only, passwords are automatically rehashed with bcrypt
module { name = "enc_des" }

I chose to use bcrypt as the new hashing method. You can choose whichever method you like, however you must make sure that enc_des is listed after the chosen module, otherwise all passwords would be rehashed to DES crypt.

And that's about it for password hashes. You should then occasionally check whether there are still any DES hashes left, and once they're all gone, unload and remove the enc_des module.

Understanding Hybserv's Database

Hybserv's "database" consists of multiple files: one for nicks, one for channels, and so on. In this specific case, only NickServ and ChanServ were used, so there was no need to migrate the others.

As mentioned before, there is no documentation of what the content in these files looks like, so we'll have to figure this out on our own.

A user in the nick.db file looks like this:

s3lph 14600 1602803962 1608435182
->PASS EelyBKlmXzO5r
->EMAIL s3lph@example.com
->LASTUH ~s3lph bnc.example.com
->LASTQMSG :So long and thanks for all the fish
->HOST *~s3lph@bnc.example.com
->HOST *~s3lph@127.0.0.*
->TS 1608435182
->LASTSERVER irc.example.com

And this is a channel in the chan.db file:

#example 4 1602804235 1603929600
->FNDR s3lph 1602804235
->PASS wahF8nhZXc4D
->TOPIC :This is an example
->ALVL -1 5 8 5 5 8 10 10 10 8 15 20 25 40 50
->ACCESS s3lph 50 1602804235 1603929600 *As Founder*
->ACCESS s4lph 10 1602803962 1608435256 s3lph

Each item is started with a line consisting of 4 fields: the name, some number and two numbers that look like UNIX timestamps. Looking at Hybserv's source code, we learn that the first number is a set of flags which control the services' behavior in regard to the nick or channel. The source code also tells us that the timestamps are the registration time and the time the resource was last "seen".

The following lines are mostly straight-forward. They all start with ->, which seems to indicate continuation of the object. They feature "somewhat human readable" keys and one or more values separated by spaces. For NickServ registrations, there are values such as the DES password hash, the email address, or which hostmasks the user has successfully identified from before. For ChanServ registrations, we see values such as the channel topic or the founder.

The last few lines were not entirely obvious, so again a look at the source code provided clarification. The ALVL chain of numbers turns out to be a definition of which permissions (e.g. voice, halfop, chanop, founder) are granted starting from which "access level". This also clarifies the format of the ACCESS lines: After the nick comes the assigned access level, then two timestamps (the time access was granted and the time it was last used) and finally the nick of the user granting the access. In the example above, s3lph would be the channel founder, and s4lph would be a halfop.

Understanding Anope's Database

Anope uses a similar approach to storing data. Again, data is stored in plain text, however all data is stored in one file, anope.db, instead of multiple files. The format is actually quite similar, but does feature some crucial differences (empty lines only inserted for readability):

OBJECT NickCore
DATA display s3lph
DATA pass des:EelyBKlmXzO5r
DATA email 
DATA language 
DATA access s3lph@127.0.0.* 
DATA memomax 0
DATA HIDE_EMAIL 1
DATA HIDE_MASK 1
DATA NS_PRIVATE 1
DATA AUTOOP 1
DATA NS_SECURE 1
END

OBJECT NickAlias
DATA nick s3lph
DATA last_quit So long and thanks for all the fish
DATA last_realname Unknown
DATA last_usermask s3lph@bnc.example.com
DATA last_realhost s3lph@bnc.example.com
DATA time_registered 1602803962
DATA last_seen 1608435182
DATA nc s3lph
END


OBJECT ChannelInfo
DATA name #example
DATA founder s3lph
DATA description 
DATA time_registered 1602804235
DATA last_used 1603929600
DATA last_topic This is an example
DATA last_topic_setter s3lph
DATA last_topic_time 1608173683
DATA bantype 2
DATA levels ACCESS_CHANGE 10 ACCESS_LIST 3 AKICK 10 ASSIGN 10001 AUTOHALFOP 4 AUTOOP 5 AUTOOWNER 9999 AUTOPROTECT 10 AUTOVOICE 3 BADWORDS 10 BAN 4 FANTASIA 3 FOUNDER 10000 GETKEY 5 HALFOP 5 HALFOPME 4 INFO 9999 INVITE 5 KICK 4 MEMO 10 MODE 9999 NOKICK 1 OP 5 OPME 5 OWNER 10001 OWNERME 9999 PROTECT 9999 PROTECTME 10 SAY 5 SET 9999 SIGNKICK 9999 TOPIC 5 UNBAN 4 VOICE 4 VOICEME 3 
DATA banexpire 0
DATA memomax 0
DATA PEACE 1
DATA SECUREFOUNDER 1
DATA CS_SECURE 1
DATA SIGNKICK 1
DATA KEEPTOPIC 1
END

OBJECT ChanAccess
DATA provider access/xop
DATA ci #example
DATA mask s4lph
DATA creator s3lph
DATA last_seen 1608435256
DATA created 1602803962
DATA data HOP
END

Due to the similarities, I'll only highlight the main differences:

Objects are typed (since all are kept in the same file) and are terminated by an explicit END rather than by the start of the next object.
There is no single number containing a lot of flags; instead each flag is represented by its own line (e.g. HIDE_EMAIL, NS_SECURE, ...)
NickServ passwords are prefixed with the used hashing module. I chose to use "des" as the prefix for my enc_des module.
There are no more ChanServ passwords. Instead, users are automatically granted their permissions once they identify through NickServ.
Attributes that can occur multiple times in a Hybserv object are extracted into their own type. For example, channel ACCESS attributes are now represented by ChanAccess objects.
ChanAccess objects don't have a numeric access level, but a "data" attribute. This can be one of VOP (voice), HOP (halfop), SOP (chanop) or QOP (founder).

Textfile Database Conversion

With the information gathered above, the conversion from one format to another can be automated. My conversion script can be found on Gitlab. However, you should not use it without careful review as there may be some subtle differences between my setup and yours. Especially the numeric access levels will most likely need to be adapted on both the Hybserv as well as on the Anope side. The conversion can be done by following these steps:

STOP ANOPE. Otherwise Anope will overwrite your changes the next time it flushes changes to disk.
Find your anope.db file. When using the Debian package, it's located under /var/lib/anope/db/anope.db.
MAKE A BACKUP of the anope.db file. The following two commands will append the converted data to anope.db. If something goes wrong, you may lose existing Anope registrations.
./hybserv2anope.py nickserv .../hybserv/nick.db anope.db
./hybserv2anope.py chanserv .../hybserv/chan.db anope.db
REVIEW anope.db. Make sure that everything looks as expected.
Start Anope and observe the log for errors.

Conclusion

The two migrations above (password hashes and database files) should allow you to migrate from Hybserv2 to Anope without requiring any action whatsoever from your users. If everything went fine, you should be able to log into NickServ and obtain ChanOp on your channels.

When you look into your anope.db, you will see that your DES password hash was replaced by a bcrypt hash (or whatever method you chose). If it's not visible immediately, you can either wait some time, /msg OperServ UPDATE or restart Anope so that all changes are immediately flushed to disk. Once there are no more DES hashes left, unload the enc_des module.

Reminders with VoIP Phones, Asterisk & Crontab

2020-09-11T00:00:00+02:00

Update (2020-10-30)

As it turns out, this solution causes people to get used to the ringing phone, causing not only the reminder to be ignored, but actual incoming calls as well. We have thus stopped using this solution.

It is considered best practice to regularly and often ventilate closed rooms due to SARS-CoV-2. But how can we make sure we remember to open the windows every now and then? We wanted to build some kind of visual or acoustic mechanism that would tell us when to open the windows.

I came up with the idea to abuse our already existing telephone system as a reminder system. This system consists of a central Asterisk server, and multiple SIP phones. The idea was to have Asterisk dial all the SIP phones, and set the reminder text as caller ID. The "secret" behind this is Asterisk's console calling feature. And here is how it's done:

First, we need to enable the chan_alsa module in Asterisk's /etc/asterisk/modules.conf:

[modules]
load => chan_alsa.so
; Comment out existing noload directives
;noload => chan_alsa.so

Afterwards restart Asterisk.

In a default installation of Asterisk on Debian, this should be all that's required for console calling to work. In our case, Asterisk is running in a VM and the default audio device is a virtual sound card not used otherwise, so there was no need for additional configuration.

Next, we can configure the dialplan in /etc/asterisk/extensions.conf for our reminders:

[reminders]
exten => ventilate,1,NoOp()
same => n,Answer()
; Set the caller ID name to the reminder text
same => n,Set(CALLERID(name)=Lueften!)
; Dial all the phones
same => n,Dial(PJSIP/phone1&PJSIP/phone2)

To apply this change, we have to reload the dialplan:

# asterisk -rx 'dialplan reload'

We can then test our reminder by telling Asterisk to originate a call from its console context:

# asterisk -rx 'originate console/dsp extension ventilate@reminders'

Now your phones should start ringing, and display your reminder text as the caller ID.

Finally, in order to schedule the reminders, just put them into the crontab of the Asterisk server. In our case, I configured it to ring every full hour on Tuesdays between 8 and 11 PM:

0 20-23 * * 2 root /usr/sbin/asterisk -rx 'originate console/dsp extension ventilate@reminders'

Cisco 7900 series IP Phone Logo Converter

2020-07-14T00:00:00+02:00

Last week we discovered a box full of old Cisco 7900 series IP phones hidden deep in a pile of boxes in our hackerspace. Of course we tried to get them up and running and figure out how to configure them.

After a bit of research and reading manuals, we learned that the phones can be configured via TFTP using a binary config file format. When we learned that the phones can display a custom 88x27 monochrome image instead of the default Cisco logo, we made it our top priority to get the image customization working.

What we learned was that the images had to be served alongside the configuration via TFTP in a proprietary format. The tool to convert images into this format comes bundled with every firmware release for the phone. However, we quickly encountered some limitations:

The converter, called bmp2logo.exe, was a proprietary, Windows only piece of software. Luckily, the tool ran in Wine without any problems (as long as you don't mess with the input file header).
The tool required the input file to be in a very specific format, namely a 88x27 pixel, monochrome, 1 bit per pixel BMP file.
While the image is drawn as dark pixels on a light background, the input file had to be white on black. Effectively, the colors were inverted.

We also learned that each image had to be tagged with a serial number. The phone uses this serial number to decide whether a new image has to be loaded from the TFTP server. If the serial number is the same as the number of the previous image, the new image is not loaded. So the serial number needs to be incremented by at least 1 for each new image.

To get around the limitations of the converter tool, I attempted to figure out the file format and write a free and cross-platform converter without these limitations.

Reverse Engineering the File Format

So let's have a look at a hexdump of one of those image files converted with bmp2logo.exe:

We know that the image consists of 88 x 27 = 2376 pixels. And since the input to bmp2logo.exe has to be a 1-bit-per-pixel uncompressed image, let's just assume that the same holds true for the output. This would give us a payload size of 297 bytes, so with a file size of 304 bytes, there should be a 11 byte header.

The first two bytes were always the same, no matter what image was converted. So it should be fairly safe to assume that they are a magic number. The next two bytes were always different for different serial numbers or different images. Also, if the serial number was changed by one, these two bytes would also change in a fairly consistent manner, so this appeared to be some some sort of checksum. Let's just ignore that for now and put it aside for later.

The next three bytes were all zeros, followed by a byte representing the serial number. Or so i thought at first, until I passed a serial number of -1 to bmp2logo.exe, and got ff ff ff ff as these four bytes, so this pretty clearly is a 32-bit representation of this number. The next two bytes were pretty obvious as well, they are the height and width of the image, 27 and 88 respectively.

The last byte of the header seems to represent the number of bytes that comprise a single row. I'm not entirely sure about this, but it made the most sense, especially if we assume that there are other phones out there which may support grayscale images, and need a greater color depth and consequently more bits per row.

Now, on to the last part: The checksum. I was a bit lost here, so I just tried various ways of compressing data into 16 bits: Adding them, xor'ing them, swapping bytes around before adding them, always wrapping the results to 16 bits. In the end it was a typo that brought me to the solution: Instead of sum = (sum + swapped) & 0xffff, I wrote sum = (sum + swapped) % 0xffff. So instead of performing addition mod 65536, i was performing addition mod 65535 by accident. The result of this turned out to be just the binary inverse of the intended checksum, so let's add a final sum ^= 0xffff before returning the result and call it a day.

File Format

With all that information, and verifying that the payload did indeed match the uncompressed monochrome image, we finally got our file format:

+----+----+----+----+----+----+----+----+
| 10h 60h |  CHKSUM |       SERIAL      |
+----+----+----+----+----+----+----+----+
| H  | W  | RW |                        |
+----+----+----+                        |
|                                       |
:               PAYLOAD                 :
:                                       :
|                                       |
|                                       |
+----+----+----+----+----+----+----+----+

To summarize, the file consists of an 11 byte header, followed by the payload. The header consists of:

The magic number 0x10 0x60
A 16 bit checksum
The 32 bit serial number in big endian
H, the height of the image in pixels, 1 byte. Since the image is 27px high, this is 27, or 0x1b
W, the width of the image in pixels, 1 byte. Since the image is 88px wide, this is 88, or 0x58
RW, the number of bytes in a single row, 1 byte. Since the image is 88px wide and each byte holds 8 pixels at once, this is 11, or 0x0b

Following the header is the image payload, with each pixel expressed as a single bit, in row-major order starting in the top left corner.

A Python implementation of the checksum algorithm could like this:

sum = 0
# Iterate the file starting at byte 4 (directly after the checksum)
# Iterate two bytes at a time (a and b), stick them together in
# reverse order, and add to the total sum mod 0xffff.
for a, b in zip(data[4::2], data[5:2]):
    sum = (sum + ((b << 8) | a)) % 0xffff
# Flip all bits in the result
sum ^= 0xffff

Implementation

Now that we know how such an image file is composed, we can write a tool to generate these files. And here's the result:

You can find the converter script on Gitlab. It can be used in exactly the same way as the original bmp2logo.exe converter:

./ciscologo.py <serial> <infile.something> <outfile.dat>
Copy <outfile.dat> to your TFTP server
Update the phone configuration to point to the new file
Recompile the configuration file and copy it to your TFTP server
Reboot your phone

Remember to increment the serial number every time you change the image, or the phone won't pick up the new file.

Scaling and Slicing PDF Documents with pdfjam and mutool

2020-06-22T00:00:00+02:00

I recently encountered the following challenge: I had a PDF document consisting of multiple A4 pages, each of which I needed to print scaled up to A2. However, I only had an A4 printer available, and the printer driver was not able to perform the scaling and/or slicing on its own.

After a long search, and lots of disappointments (since neither of Libre Office, Inkscape and Evince could do what I wanted, or I simply couldn't find the feature), I came up with the following solution:

$ pdfjam -o scaled.pdf --a2paper input.pdf
$ mutool poster -x 2 -y 2 scaled.pdf sliced.pdf

The first command takes the input file, scales it up to A2 size, and writes it to an intermediate file. The second command slices each page from the intermediate file into 2 slices both vertically and horizontally, totaling in 4 slices per page, which again results in A4-sized slices the printer can print.

Automated Debian package building with Gitlab CI and Reprepro

2020-06-21T00:00:00+02:00

I previously deployed some ad-hoc services using ugly and difficult to maintain solutions, such as binaries manually extracted from container images and copied into /usr/local/bin. Of course, this is a lot of manual work for each upgrade of the service, and if you upgrade often, a lot of repetitive manual work. Or, in other words, the perfect kind of work to be automated.

So I decided to package the software in question for Debian, which is the Linux distribution I run these services on, and to automate the process of building the packages and publishing them to a repository. There's three different "categories" of software I wanted to have in my repository:

Binaries published as build artifacts in releases of their upstream repository. An example for this is Gitea. The binaries can be automatically downloaded and put into a package.
Software released as container images only. An example for this is the Drone CI Server. This is a bit more tricky, as the binaries must be extracted from the image before it can be put into a package.
Software already released as Debian packages, but not available through a repository. An example for this is a project of my own, the iCalendar Timeseries Server, where the CI pipeline automatically builds Debian packages and adds them as build artifacts to releases. Here, the already-built package just needs to be fetched and added to the repository.

For creating and maintaining the repository, I'm using Reprepro, as it is extremely simple to use, and generally is rather uncomplex and lightweight. It does however come with some limitations; I especially encountered the problem that Reprepro will only keep the latest version of a package in the repo index, so older versions won't be available to clients.

For automating the packaging and repository build process, I'm using Gitlab CI. The pipeline is running each night, building the latest stable version for each package, adding the packages to the in-container Reprepro repository, and then synchronize the repository to a web server of mine.

The source repository with the package build scripts and pipeline configuration is available on Gitlab. Though, before using this as a template, be advised that the packages built by this pipeline are not exceptionally high-quality. They are also usually built to fit my personal needs, so don't expect ready-to-use packages.

Recording IPTV Using ffmpeg

2020-05-20T00:00:00+02:00

I don't usually watch TV. But from time to time there is something interesting on the programme, such as debates on local politics. Unfortunately, those usually run at a time of day where I'm not able (or more likely not willing) to tune in an pay attention to an hour of political discourse. So I want to record them and watch later instead.

My ISP provides its IPTV programme as MPEG-TS streams via multicast UDP. And they even link a M3U playlist of all stations on their website, so you can basically watch TV with any client whatsover, as long as it speaks IGMP and understands MPEG-TS video streams. This makes recording very easy, as this is supported by a lot of multimedia processing software, including ffmpeg.

The playlist consists of a list of TV stations, each of which is represented by its own multicast group and an UDP port. So let's just take the first station and see what ffmpeg finds in there:

Input #0, mpegts, from 'udp://239.77.0.77:5000':
  Duration: N/A, start: 41892.675600, bitrate: N/A
  Program 9038 
    Metadata:
      service_name    : SRF 1 HD
      service_provider: Schweizer Radio und Fernsehen
    Stream #0:0[0x50]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
    Stream #0:1[0x51](deu): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s (clean effects)
    Stream #0:2[0x52](eng): Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, fltp, 192 kb/s (clean effects)
    Stream #0:3[0x5b](deu): Audio: ac3 ([6][0][0][0] / 0x0006), 48000 Hz, 5.1(side), fltp, 448 kb/s (clean effects)
    Stream #0:4[0x6e](deu,deu): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006)
    Stream #0:5[0x70]: Unknown: none ([5][0][0][0] / 0x0005)
    Stream #0:6[0x72]: Unknown: none ([12][0][0][0] / 0x000C)

We can see that the MPEG-TS stream contains multiple indiviual streams, which are listed in the output above. Now, I don't know what's up with 0:5 and 0:6, or why ffmpeg doesn't understand them. Anyway, I only need the video and one audio channel. Let's just pick the first two, and record an one hour TV show:

ffmpeg -f mpegts -i udp://239.77.0.77:5000 -map 0:0 -map 0:1 -c copy -t 3600 recording.mkv

To break it down:

-f mpegts tells ffmpeg that the input is an MPEG transport stream.
-i udp://239.77.0.77:5000 tells ffmpeg to join the specified multicast group and receive the MPEG-TS stream on UDP port 5000.
-map 0:0 -map 0:1 only extracts the streams 0:0 (the H.264 video stream) and 0:1 (the German audio channel in MP2)
-c copy causes the input stream to be demuxed only, and the selected streams to be written to the output without CPU-intensive decoding and reencoding.
-t 3600 terminates the stream after one hour, when the show is over.
recording.mkv is the output filename. The container format (here MKV) is deduced from the filename.

So the whole ffmpeg command takes the original stream as input, demultiplexes it to get the individual media streams, then throws out all but one video and one audio stream, multiplexes them into a Matroska container, which is then written to disk.

Extracting 3D Models From CesiumJS - Part 1: Terrain Map Scraping

2020-05-09T00:00:00+02:00

Addressing Terrain Tiles

To get started with the terrain map, I needed to figure out how to obtain terrain data for a certain geographical region. Luckily, this is fairly well documented. CesiumJS uses its own solution called quantized-mesh.

quantized-mesh supports different "zoom levels", for which the whole globe is divided into more and more single "tiles". At level 0, there are only two tiles: the first tile covers the western hemisphere, the second tile covers the eastern hemisphere. With each increase in zoom level, each tile is split into 4, each new tile containing a quadrant of the previous tile. Each tile can then be identified by its zoom level, a x coordinate and a y coordinate. x starts at 0, representing -180° longitude, and when incrementing x, you go eastward until you reach the tile ending at +180° longitude, at x=2^(z+1). y=0 starts at the south pole at -90° latitude, going north, and reaches +90° latitude at y=2^z. No +1 in the exponent here, since for full coverage, x needs to cover the full 360° longitude, while y only needs to cover half as much for a total of 180° latitude.

Using these three variables, z, x and y, the quantized-mesh specification defines an URL template for addressing an individual tile via HTTP:

http://example.com/tiles/<z>/<x>/<y>.terrain

This can be a little hard to imagine, so I attempted to visualize the first two zoom levels in Figure 1.

Figure 1: Tiles at zoom level 0 and 1: At level 0, there are 2 tiles, each tile covering a hemisphere. At level 1, there are 8 tiles in total. Each tile from level 0 was divided into 4 tiles

If you're familiar with OpenStreetMap, you may recognize this way of dividing the globe into tiles and addressing individual tiles. An OpenStreetMap tile URL looks like this: https://tile.openstreetmap.org/14/8537/5725.png. This similarity is not accidental; in fact, the quantized-mesh tiling schema was designed to follow the Tile Map Service standard's tiling schema.

All of the above assumes our data source uses a WGS84 projection and the TMS tiling schema. quantized-mesh supports other configurations as well, where there is only a single tile at level 0, or with the x and y coordinates swapped. You can find more information in the documentation.

Mapping Geographical Regions to Terrain Tiles

Now that we know how quantized-mesh tiles are addressed, let's find out which tiles we actually need. In my use case, I wanted to obtain all tiles in a bounding box defined by lower and upper latitudes and longitudes. Converting x and y to coordinates is quite easy:

lat = -90 + y * 180 / (2**z)
lon = -180 + x * 180 / (2**z)

So to get the ranges for x and y, we solve those equations for x and y and add proper rounding:

x_min = floor( lon_min * (2**z)/180 + 2**z     )
x_max = ceil(  lon_max * (2**z)/180 + 2**z     )
y_min = floor( lat_min * (2**z)/180 + 2**(z-1) )
y_max = ceil(  lat_max * (2**z)/180 + 2**(z-1) )

The resulting ranges, with x_max and y_max being exclusive upper bounds, are then formulated as

tiles_x = range(x_min, x_max)
tiles_y = range(y_min, y_max)

So once we know at which zoom level our tiles are available, we can compute which tiles to download in order to fully cover our region.

Scraping Terrain Tiles

Now all that's left to to is to figure out how to actually download the individual tiles. Your web browser's inspector can be a great help here. Open the Cesium-based application in your web browser, and open the inspector's network tab. As you move around in the map, you will most likely see a lot of requests, which can be grouped into the following categories:

Image tiles, addressed in the same way as the terrain tiles. I didn't need these for my use case, but you should be able to scrape them the same way I'm scraping the terrain tiles.
3D models, with file extensions of b3dm, glb and cmpt. I'll take a look at some of those in later articles.
The terrain tiles, with a file extension of terrain. These are the ones we want to obtain.

As you zoom in and out of the map, you'll see that the zoom levels in the requests to image and terrain tiles will increase or decrease. Since I wanted the most detailed tiles, I just continued with the maximal zoom level that was available in the application i was working with.

Now just right click on one of the terrain requests, and navigate to Copy / Copy as cURL. You'll get something like this:

curl 'https://maps.example.com/terrain/tiles/12/12345/12345.terrain' \
  -H 'Origin: https://maps.example.com' \
  -H 'Accept-Encoding: gzip, deflate, br' \
  -H 'User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; ...)' \
  -H 'Accept: application/vnd.quantized-mesh,application/octet-stream;q=0.9' \
  -H 'Referer: https://maps.example.com/' \
  --compressed

At least the Accept header is usually required, and the quantized-mesh specification recommends setting it, especially since it's used to tell the server which extensions to the quantized-mesh standard are supported by the client, if any. Some other headers, especially the Origin, Referer and User-Agent may be required as a "soft form of access control", depending on the server's configuration. I found it worked best to just keep the entire curl request as-is, and only modify the z, x and y parameters in the URL.

Knowing the supported values for z and our ranges for x and y, we can now easily script the download of the individual files. A small hint regarding the filenames: Use all three parameters, z, x and y in the output filename, otherwise you end up overwriting the same files over and over again.

Once the download is done, we are left with a bunch of binary .terrain files. The next article will cover how to parse them.

Mounting a 2.5" Drive Inside an APU2

2020-03-07T15:15:00+01:00

At home, I'm using a PC Engines APU2 as a firewall and WiFi access point. Since this is the only device in my home constantly running and constantly connected to the internet, I decided to use it as an additional backup site for my servers.

However, for this to become viable, I needed to add around 200 GB of storage. I considered the following options:

Add more mSATA storage
Attach external (USB?) storage
Attach a 2.5" drive internally

I quickly discarded the second option, since this approach would take up more space, which could become challenging especially since I wall-mounted the device. As for the other two options, I had a few spare 2.5" drives lying around, so I figured I'd try to use those first before buying new storage.

The Problem

Space inside the APU2 case is tight. There is some 9-10mm space between the highest parts of the system board (pin headers, capacitors, mPCIe and mSATA cards) and the case cover, at least for most parts of the board's footprint. Sure, I could just try to glue the drive to the case ceiling, but that would be both ugly and extremely cumbersome to handle. Also, I really wanted to prevent the disk from ever touching anything on the board, since most disk casings are made from metal, and the things it could touch on the board are stuffs like pin headers.

The Solution

So I decided to come up with a 3D-printed mount. It consists of 3 parts: two side parts, which are put between the board and the case walls, and a center part, on which the disk is screwed in place, held in place by the two side parts. Effectively, a "bridge" over the system board.

The first attempt already turned out pretty well, but a few problems became apparent:

I took a wrong measurement at one point and had to move some cutouts so they would properly align with the pin headers they were meant for.
The plugs of SATA cables (at least the ones I had lying around, as well as the PC Engines satacab1) extended below the "base line" of the disk they are plugged into. This was solved by adding another hole into the 3D-printed part.
Due to the limited space, this design only works for drives 7mm high. The much more common 9.5mm high drives won't fit, or at least put some stress on the printed part, and possible the system board.

So I ended up using a 250GB SSD (the only 7mm drive I had lying around). However, when the PC Engines-specific SATA cable ("satacab1", required because there is no SATA power connector on the APU2 board) arrived, another problem popped up:

The cable is both quite short and rather rigid, so if you use it to connect the drive to the SATA connector on the APU2 board, you end up putting some stress on both the connector and the printed parts. This can be easily mitigated by putting a short SATA extension cable in between the satacab1 and the board's connector.

The Result

I've published the resulting design on Thingiverse. Alternatively, you can download the files directly from here. The design is licensed under the CC BY-SA 4.0 license.

The results of my print can be seen in the following photos. I made this print out of PLA using an Ultimaker 3.

Binary Calculator Implementation in Apache mod_rewrite

2020-02-13T02:00:00+01:00

Many text replacement dialects utilizing regular expressions become turing complete when combined with conditional loops or recursion. Apache's mod_rewrite is no exception here; you can write turing-complete programs using RewriteRules only. A few weeks ago I decided to try my hand at this and wrote a binary calculator (well, only an adder, to be precise) implemented in an Apache2 config file.

mod_rewrite Basics

Apache's mod_rewrite allows for conditionally rewriting request URLs. A RewriteRule directive looks like this:

RewriteRule "pattern" "replacement" [flags]

The pattern is matched against the URL of the request, and if it matches, the matching region is replaced by the replacement expression. The flags control the exact behavior of the rule. We only make use of three flags in this implementation:

L aka last terminates the rewrite engine, no more rules are processed for this request. L is used to "exit the program".
R aka redirect causes the web server to respond with a 302 Found, rather than processing the rewritten URL internally. This is used to return the result to the client in the Location header.
N aka next causes the rewrite engine to start over from the beginning, operating on the result of the last iteration. The combination of N and L is what gives us conditional recursion.

The Calculator

The calculator works as explained in the following example, which adds the binary representations of 11 and 7, and should ideally result in 18.

The client sends a request such as GET /1011+111 HTTP/1.1
The server first adds an equality sign to the end: /1011+111=.
The server iteratively performs the addition bit-by-bit (# represents the carry bit):
1. /1011+111= → /101+11=#0
2. /101+11=#0 → /10+1=#10
3. /10+1=#10 → /1+=#010
4. /1+=#010 → /1+0=#010 (add zero padding)
5. /1+0=#010 → /+=#0010
The server cleans up the request, removing the operators and resolving the last carry bit, if present:
- /+=#0010 → /10010
The server responds with a redirect with a Location: /10010 header.

And here's the whole mod_rewrite Config:

RewriteEngine on
# Termination condition: no more digits left, also strips leading zeros
RewriteRule "^/\+=0*([01]+)$"                    "/$1"           [L,R]
# Termination condition with carry flag
RewriteRule "^/\+=#0*([01]+)"                    "/1$1"          [L,R]
# Add = if absent
RewriteRule "^/([01]+)\+([01]+)$"              "/$1+$2="       
# Pad first number with zeros if too short
RewriteRule "^/\+([01]+)=(#?[01]*)"            "/0+$1=$2"      
# Pad second number with zeros if too short
RewriteRule "^/([01]+)\+=(#?[01]*)"            "/$1+0=$2"      
# 0+0 nocarry
RewriteRule "^/([01]*)0\+([01]*)0=([01]*)$"    "/$1+$2=0$3"    [N]
# 0+0 carry
RewriteRule "^/([01]*)0\+([01]*)0=#([01]*)$"   "/$1+$2=1$3"    [N]
# 1+0 nocarry
RewriteRule "^/([01]*)1\+([01]*)0=([01]*)$"    "/$1+$2=1$3"    [N]
# 1+0 carry
RewriteRule "^/([01]*)1\+([01]*)0=#([01]*)$"   "/$1+$2=#0$3"   [N]
# 0+1 nocarry
RewriteRule "^/([01]*)0\+([01]*)1=([01]*)$"    "/$1+$2=1$3"    [N]
# 0+1 carry
RewriteRule "^/([01]*)0\+([01]*)1=#([01]*)$"   "/$1+$2=#0$3"   [N]
# 1+1 nocarry
RewriteRule "^/([01]*)1\+([01]*)1=([01]*)$"    "/$1+$2=#0$3"   [N]
# 1+1 carry
RewriteRule "^/([01]*)1\+([01]*)1=#([01]*)$"   "/$1+$2=#1$3"   [N]

Showing Webcal Calendar Events in Grafana

2020-02-12T02:00:00+01:00

I'm running a Grafana at home, with a dashboard giving me an overview over my day. It contains information like public transport departures or the guest WiFi's password. But the most important part is a list of my upcoming appointments. Now, iCalendar files served via HTTP is not something Grafana understands out of the box. To work around this, I wrote a small service that scrapes the calendar endpoints and exposes the events as metrics in a Prometheus-compatible API.

How it works

Consider the following iCalendar file, served at an HTTP endpoint:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ACME//NONSGML Rocket Powered Anvil//EN
BEGIN:VEVENT
UID:20190603T032500CEST-foo
SUMMARY:Foo
DESCRIPTION:An example event
DTSTART;TZID=Europe/Zurich;VALUE=DATE-TIME:20190603T032500
DTEND;TZID=Europe/Zurich;VALUE=DATE-TIME:20190603T040000
END:VEVENT
BEGIN:VEVENT
UID:20190603T032500CEST-bar
SUMMARY:Bar
DESCRIPTION:Another example event
DTSTART;TZID=Europe/Zurich;VALUE=DATE-TIME:20190603T032500
DTEND;TZID=Europe/Zurich;VALUE=DATE-TIME:20190603T040000
END:VEVENT
END:VCALENDAR

The service retrieves this calendar from the endpoint, parses it and extracts a list of events together with metadata from it. It then serves the data in a Prometheus-compatible time series API. Clients can request all upcoming events using the following call:

GET /api/v1/query?query=events

To which the service returns the time series of events:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "event",
          "calendar": "0",
          "uid": "20190603T032500CEST-foo",
          "summary": "Foo",
          "description": "An example event"
        },
        "value": [
          1560043497,
          1
        ]
      },
      {
        "metric": {
          "__name__": "event",
          "calendar": "1",
          "uid": "20190603T032500CEST-bar",
          "summary": "Bar",
          "description": "Another example event"
        },
        "value": [
          1560043497,
          1
        ]
      }
    ]
  }
}

Caveats

Since a Prometheus label can't be used multiple times, event categories can't be easily mapped to them. Thus, event categories are currently not exported. If someone has an idea how to model categories in the output, while keeping it easy to query and manage, feel free to contact me.

Grafana uses a hardcoded 1+1 query to test Prometheus data sources, so the API currently has a special check for that and returns 2, as expected by Grafana.

Interested?

The project, which I named iCalendar Timeseries Server, can be found on Gitlab. Each release comes with Python Wheel and Debian packages.

Automatically Rotating Guest WiFi Passwords With hostapd

2020-02-06T02:28:00+01:00

I like to have control over who gets on my networks and who doesn't.

To obtain this level of control in my home network, I'm running a separate WiFi for guests, which among other things separates guest devices from my private infrastructure.

Authorization in hostapd

The most simple way of configuring WPA2-PSK authorization in hostapd is a static passphrase:

# /etc/hostapd/hostapd.conf
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_passphrase=Nobody expects the Spanish Inquisition!

So far, so good - but once a person knows this passphrase, they can get on my WiFi all the time, and they could share the passphrase with other people. This way, I lose control over who gets on my networks.

hostapd also supports device-specific passphrases, configured in a separate file:

# /etc/hostapd/hostapd.conf
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_psk_file=/etc/hostapd/hostapd.wpa_psk

Now, how should this file look like? The hostapd "documentation" is a bit shady in this regard, and only mentions (PSK,MAC address) pairs; the exact format is not mentioned. However, multiple sources on the internet seem to agree on this format:

# /etc/hostapd/hostapd.wpa_psk
ma:ca:dd:re:ss:00 The Passphrase For Device A
ma:ca:dd:re:ss:01 The Passphrase For Device B

And, most important, some sources also mention that the MAC address 00:00:00:00:00:00 can be used as a wildcard, so the associated passphrase works for all devices. This alone does not give us any advantage over the hardcoded passphrase. However, having the passphrase in a separate file makes automated rotation extremely easy. By doing this, I have a fairly good control over who can access my guest WiFi when the passphrase is rotated frequently through a cronjob.

To take things a step further, we can decouple the passphrases rotation rate from how long a passphrase remains valid. As it turns out, the wildcard MAC address can be used multiple times, and all wildcard passphrases are accepted. This allows us to do the following:

Generate a new passphrase once a day
Add the new passphrase as a wildcard entry to the wpa_psk file
Remove all but the seven newest entries from the file
Reload hostapd

So, this gives us a new passphrase every day, and each passphrase remains valid for a week.

Giving the Passphrase to Guests

I'm using qrencode to generate a QR code with the latest passphrase, and display the result, together with its plaintext form, in a Grafana HTML panel:

qrencode \
  -t PNG --size=6 --output=/var/www/html/wifi-guest.png \
  "WIFI:S:${SSID};T:WPA2;P:${PASSPHRASE};;"

cat > /var/www/html/wifi-guest.html <<EOF
  <!-- Timestamp for browser cache circumvention -->
  <img src="/wifi-guest.png?$(date +%s)" />
  <br/><br/><br/>
  <h3><tt>${SSID}</tt></h3>
  <h1><tt>${PASSPHRASE}</tt></h1>
EOF

And the result looks like this:

Bringing Swiss Public Transport Departures to Grafana

2020-02-05T02:26:00+01:00

Update (2020-11-27)

The API endpoint used here has been deprecated, and a new endpoint is available. The updated script can be found on Gitlab.

The Swiss Railways (SBB) provide a collection of static data sets and dynamic APIs at opentransportdata.swiss. One endpoint provides a list of departures or arrivals for a given train, bus or tram station.

In this blogpost, I'm showing you how I'm using this API to get a list of upcoming departures for the station next to my home, and how do get this list into Grafana.

The API

The XML API is documented in the "API Cookbook". A request looks like this:

POST /trias HTTP/1.1
Host: https://api.opentransportdata.swiss
Authorization: TOKEN
Content-Type: text/xml

<?xml version="1.0" encoding="UTF-8"?>
<Trias
    version="1.1"
    xmlns="http://www.vdv.de/trias"
    xmlns:siri="http://www.siri.org.uk/siri"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceRequest>
    <siri:RequestTimestamp>  NOW  </siri:RequestTimestamp>
    <siri:RequestorRef>EPSa</siri:RequestorRef>
    <RequestPayload>
      <StopEventRequest>
        <Location>
          <LocationRef>
            <StopPointRef>  BPUIC  </StopPointRef>
          </LocationRef>
          <DepArrTime>  NOW  </DepArrTime>
        </Location>
        <Params>
          <NumberOfResults>  N_RESULTS  </NumberOfResults>
          <StopEventType>departure</StopEventType>
          <IncludePreviousCalls>false</IncludePreviousCalls>
          <IncludeOnwardCalls>false</IncludeOnwardCalls>
          <IncludeRealtimeData>true</IncludeRealtimeData>
        </Params>
      </StopEventRequest>
    </RequestPayload>
  </ServiceRequest>
</Trias>

This request is fairly minimal; it is limited to a single station, and without further information such as previous and following stops. You only need to fill in the following arguments to make this work for your station of choice:

TOKEN: API Token, need to register an account.
NOW (2x): Current time in ISO-8601 form.
BPUIC: Numeric ID of the station ("Betriebspunkt"), can be looked up in the DiDok dataset.
N_RESULTS: Maximal number of results to return.

Prometheus Ingestion

XML is a bit... let's say, uncomfortable to handle in Bash scripts, so I resorted to using the xsltproc tool to transform the API response into something easily iterable; the XSLT document i came up with looks like this and generates CSV content:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:trias="http://www.vdv.de/trias">
  <xsl:output method="text" />
  <xsl:template match="/">
    <xsl:for-each select="//trias:StopEvent">
      <xsl:value-of select="trias:Service/trias:PublishedLineName/trias:Text"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:Service/trias:DestinationText/trias:Text"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:ThisCall/trias:CallAtStop/trias:ServiceDeparture/trias:TimetabledTime"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:ThisCall/trias:CallAtStop/trias:ServiceDeparture/trias:EstimatedTime"/>
      <xsl:text>&#x0A;</xsl:text>
    </xsl:for-each>
  </xsl:template>
</xsl:stylesheet>

Each line in the result represents a stop at the station, with the following fields:

Number of the train or bus line
Name of the destination
Scheduled departure time
Estimated/actual departure time

This format is quite easy to handle in Bash; let's parse the ISO-8601 timestamps, compute the delay for each stop and then emit the results in Prometheus collector format:

# TYPE sbb_station_departure gauge
# HELP sbb_station_departure Departures from a train or bus station
# TYPE sbb_station_delay gauge
# HELP sbb_station_delay Departure delay
sbb_station_departure{line="26",planned="1580875380000",destination="Erstfeld"} 1580875380000
sbb_station_delay{line="26",planned="1580875380000",destination="Erstfeld"} 0
sbb_station_departure{line="36",planned="1580875980000",destination="Zürich HB"} 1580875980000
sbb_station_delay{line="36",planned="1580875980000",destination="Zürich HB"} 0

I'm using the Textfile Collector feature of the Prometheus Node Exporter to ingest this document into Prometheus.

Display in Grafana

I'm showing this data in a table panel in Grafana, using two queries: one for the scheduled departure, one for the delay. Here, you can filter the departures by destination, if not already done in your script:

min without (__name__) (sbb_station_departure{destination=~".*Zürich.*"})

and

min without (__name__) (sbb_station_delay{destination=~".*Zürich.*"})

And finally, after some styling, the result looks like this:

The code can be found on Gitlab.

SBB-Abfahrtszeiten in Grafana anzeigen

2020-02-05T02:26:00+01:00

Update (2020-11-27)

Der hier verwendete API-Endpunkt ist veraltet und wurde durch einen neuen Endpunkt ersetzt. Das aktualisierte Skript liegt auf Gitlab.

Die SBB bieten unter opentransportdata.swiss eine Sammlung an statischen Datensätzen und dynamischen APIs an. Ein Endpunkt bietet eine Liste an Zug-, Bus- oder Tramabfahrten oder -Ankünften für gewisse Haltestellen an.

In diesem Blogpost erkläre ich, wie ich diese API verwende, um eine Liste an kommenden Abfahrten ab meinem "Heimatbahnhof" zu erhalten, und wie diese Liste in Grafana landet.

Die API

Die XML-API ist im "API Cookbook" dokumentiert. Eine Anfrage an die API sieht etwas so aus:

POST /trias HTTP/1.1
Host: https://api.opentransportdata.swiss
Authorization: TOKEN
Content-Type: text/xml

<?xml version="1.0" encoding="UTF-8"?>
<Trias
    version="1.1"
    xmlns="http://www.vdv.de/trias"
    xmlns:siri="http://www.siri.org.uk/siri"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceRequest>
    <siri:RequestTimestamp>  NOW  </siri:RequestTimestamp>
    <siri:RequestorRef>EPSa</siri:RequestorRef>
    <RequestPayload>
      <StopEventRequest>
        <Location>
          <LocationRef>
            <StopPointRef>  BPUIC  </StopPointRef>
          </LocationRef>
          <DepArrTime>  NOW  </DepArrTime>
        </Location>
        <Params>
          <NumberOfResults>  N_RESULTS  </NumberOfResults>
          <StopEventType>departure</StopEventType>
          <IncludePreviousCalls>false</IncludePreviousCalls>
          <IncludeOnwardCalls>false</IncludeOnwardCalls>
          <IncludeRealtimeData>true</IncludeRealtimeData>
        </Params>
      </StopEventRequest>
    </RequestPayload>
  </ServiceRequest>
</Trias>

Diese Anfrage ist sehr minimalistisch; sie beschränkt sich auf eine einzelne Haltestelle, und ignoriert weitere Informationen wie Zwischenhalte. Hier muss man einfach nur die folgenden Argumente einfügen, um die API für die "eigene" Haltestelle zu verwenden:

TOKEN: Das API Token, man muss einen Benutzeraccount anlegen.
NOW (2x): Aktueller Zeitpunkt in ISO-8601-Form.
BPUIC: Numerische ID der Haltestelle ("Betriebspunkt"), kann im DiDok-Datensatz nachgeschlagen werden.
N_RESULTS: Maximale Anzahl an zurückgegebenen Ergebnissen.

Daten ins Prometheus bekommen

XML ist ein bisschen... mühsam in Bash-Scripten zu verwenden, daher habe ich hier das xsltproc-Tool zur Hilfe genommen, um die Antwort der API in ein handlicheres Format zu übersetzen; das XSLT-Dokument, das hierfür entstanden ist, sieht so aus und generiert eine Ausgabe in CSV-Form:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:trias="http://www.vdv.de/trias">
  <xsl:output method="text" />
  <xsl:template match="/">
    <xsl:for-each select="//trias:StopEvent">
      <xsl:value-of select="trias:Service/trias:PublishedLineName/trias:Text"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:Service/trias:DestinationText/trias:Text"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:ThisCall/trias:CallAtStop/trias:ServiceDeparture/trias:TimetabledTime"/>
      <xsl:text>;</xsl:text>
      <xsl:value-of select="trias:ThisCall/trias:CallAtStop/trias:ServiceDeparture/trias:EstimatedTime"/>
      <xsl:text>&#x0A;</xsl:text>
    </xsl:for-each>
  </xsl:template>
</xsl:stylesheet>

Jede Zeile der Ausgabe repräsentiert einen Halt an der Haltestelle, mit den folgenden Datenfeldern:

Nummer der Bus- oder Zuglinie
Name des Fahrtziels
Fahrplanmässige Abfahrtszeit
Tatsächliche/geschätzte Abfahrtszeit

Dieses Format lässt sich in Bash deutlich einfacher verarbeiten; für jeden Halt werden die ISO-8601-Timestamps geparst, daraus die Verspätung berechnet, und das Ergebnis im Prometheus-Collector-Format ausgegeben:

# TYPE sbb_station_departure gauge
# HELP sbb_station_departure Departures from a train or bus station
# TYPE sbb_station_delay gauge
# HELP sbb_station_delay Departure delay
sbb_station_departure{line="26",planned="1580875380000",destination="Erstfeld"} 1580875380000
sbb_station_delay{line="26",planned="1580875380000",destination="Erstfeld"} 0
sbb_station_departure{line="36",planned="1580875980000",destination="Zürich HB"} 1580875980000
sbb_station_delay{line="36",planned="1580875980000",destination="Zürich HB"} 0

Um diese Daten in Prometheus zu importieren, verwende ich das Textfile Collector-Feature des Prometheus Node Exporter.

Anzeige in Grafana

Die Abfahrtsdaten lassen sich in Grafana in einer Tabelle darstellen. Hierzu werden zwei Prometheus-Queries benötigt; eine für die planmässige Abfahrt, eine für die Verspätung. Wenn man nach bestimmten Fahrtzielen filtern möchte, lässt sich dies zum Beispiel hier tun:

min without (__name__) (sbb_station_departure{destination=~".*Zürich.*"})

und

min without (__name__) (sbb_station_delay{destination=~".*Zürich.*"})

Nach ein bisschen Styling sieht das ganze dann so aus:

Der Code hierzu ist auf Gitlab zu finden.

Monitoring Freifunk Nodes With Prometheus

2020-02-05T01:36:00+01:00

Update (2020-03-07)

We now collect the number of connected clients as well as whether the node is online.

We recently installed a Freifunk node from Freifunk Dreiländereck (FF3L) in our hackerspace. While changing network configuration during the testing phase, the node went offline without us noticing. Since we're using Prometheus for monitoring our space's infrastructure, I went ahead and hacked together a solution which I want to present here:

Idea

Instead of monitoring the node directly, I decided to tap into the status information already collected by the Freifunk community. They are publishing some interesting statistics, however we only really cared about one information: Whether FF3L currently considers the node to be online and reachable.

FF3L publishes these status information at the following endpoint:

https://map.freifunk-3laendereck.net/data/nodes.json

Many (if not all?) Freifunk communities provide such an endpoint, though with some you may have to search for a while to find it.

Implementation

The nodes endpoint yields the information of all nodes at once. Unfortunately, I didn't find a way to reduce the request to specific nodes; if someone knows more about this, don't hesitate to tell me. (As far as I can tell there are multiple implementations of this endpoint, some of which appear to support filters.)

This is how a single node object from the API response looks like:

{
  "nodeinfo": {
    "software": {
      "firmware": {
        "base": "gluon-v2019.1",
        "release": "v2019.1.0+001"
      },
      ...
    },
    "network": {
      ...
    },
    "location": {
      "latitude": ...,
      "longitude": ...
    },
    "system": {
      "role": "node",
      "site_code": "ff3l",
      "domain_code": "3land"
    },
    "node_id": "...",
    "hostname": "...",
    ...
  },
  "flags": {
    "online": true
  },
  "statistics": {
    "uptime": 626614.08,
    "clients": 2,
    ...
  },
  "lastseen": "...",
  "firstseen": "..."
}

We were especially interest in the .flags.online and .statistics.clients fields; our implementation extracts nothing but these two field. The fields .nodeinfo.node_id and .nodeinfo.hostname are suitable for filtering for your own nodes.

Using a bit of "curl|jq magic", we can create a shell script for parsing the data and converting it into a format understood by Prometheus. We added the script as a Textfile Collector to an existing Prometheus Node Exporter instance. The output then looks like this:

# HELP freifunk_node_online 1 if the Freifunk node is online, 0 otherwise
# TYPE freifunk_node_online gauge
# HELP ff3l_node_clients Number of clients connected to the node
# TYPE ff3l_node_clients gauge
freifunk_node_online{node="<node0_id>",hostname="<node0_hostname>"} 1
freifunk_node_clients{node="<node0_id>",hostname="<node0_hostname>"} 2
freifunk_node_online{node="<node1_id>",hostname="<node1_hostname>"} 0
freifunk_node_clients{node="<node1_id>",hostname="<node1_hostname>"} 0

The resulting script can be found on Gitlab.

Freifunkknoten mit Prometheus überwachen

2020-02-05T01:36:00+01:00

Aktualisiert 2020-03-07: Wir erfassen inzwischen, zusätzlich zur Verfügbarkeit der Node, wieviele aktuell verbunden sind.

Wir haben in unserem Hackerspace neulich einen Freifunk-Knoten von Freifunk Dreiländereck (FF3L) in Betrieb genommen. Dieser ist uns in der Probephase mehrfach bei Änderungen der Netzwerkkonfiguration offline gegangen, ohne das wir es gemerkt hatten. Da wir in unserem Space Prometheus zur Überwachung unserer Infrastruktur einsetzen, habe ich kurzerhand eine entsprechende Lösung gebastelt, die ich hier kurz vorstellen will:

Idee

Anstatt den Knoten direkt zu überwachen, habe ich mich dazu entschieden, die von der Freifunk-Community gesammelten Statusinformationen anzuzapfen. Dort fallen einige interessante Statistiken heraus, aber für uns war ein einziger Wert relevant: Ist der Freifunkknoten laut FF3L momentan online und erreichbar?

Die Statusinformationen lassen sich bei FF3L über den folgenden Endpoint abfragen:

https://map.freifunk-3laendereck.net/data/nodes.json

Viele (vielleicht sogar alle?) Freifunk-Communities haben einen solchen Endpoint, bei manchen muss man eventuell aber etwas suchen, bevor man die API findet.

Umsetzung

Der nodes-Endpoint liefert auf einen Schlag die Informationen zu allen Knoten. Ich habe leider keinen Weg gefunden, die Anfrage direkt auf einzelne Knoten zu reduzieren; wenn hier jemand mehr weiss, würde ich mich über Infos dazu freuen. (Soweit ich das sehen kann, gibt es verschiedene Implementationen, manche scheinen Filter zu unterstützen.)

Für einen einzelnen Knoten sieht diese API-Response etwa so aus:

{
  "nodeinfo": {
    "software": {
      "firmware": {
        "base": "gluon-v2019.1",
        "release": "v2019.1.0+001"
      },
      ...
    },
    "network": {
      ...
    },
    "location": {
      "latitude": ...,
      "longitude": ...
    },
    "system": {
      "role": "node",
      "site_code": "ff3l",
      "domain_code": "3land"
    },
    "node_id": "...",
    "hostname": "...",
    ...
  },
  "flags": {
    "online": true
  },
  "statistics": {
    "uptime": 626614.08,
    "clients": 2,
    ...
  },
  "lastseen": "...",
  "firstseen": "..."
}

Für uns waren hier vor allem die Felder .flags.online und .statistics.clients interessant; mehr lesen wir in unserer Implementation gar nicht aus. Die Felder .nodeinfo.node_id und .nodeinfo.hostname eignen sich, um nach den eigenen Knoten zu filtern.

Mit ein bisschen "curl|jq-Magie" lässt sich das ganze recht einfach in einem Shellscript parsen und in ein Format bringen, mit dem Prometheus etwas anfangen kann. Wir haben das ganze als Textfile Collector an einen Prometheus Node Exporter angebunden. Der Output sieht dann etwa so aus:

# HELP freifunk_node_online 1 if the Freifunk node is online, 0 otherwise
# TYPE freifunk_node_online gauge
# HELP ff3l_node_clients Number of clients connected to the node
# TYPE ff3l_node_clients gauge
freifunk_node_online{node="<node0_id>",hostname="<node0_hostname>"} 1
freifunk_node_clients{node="<node0_id>",hostname="<node0_hostname>"} 2
freifunk_node_online{node="<node1_id>",hostname="<node1_hostname>"} 0
freifunk_node_clients{node="<node1_id>",hostname="<node1_hostname>"} 0

Das Ergebnis gibt es als fertiges Script auf Gitlab.