Blog Articles

Migration of IRC Services from Hybserv2 to Anope


I recently inherited a small IRC network, consisting of a single server and a rather small but still active community. The software on this server was quite outdated, running a version of ircd-hybrid from 8 years ago, and for IRC services a comparably old version of the now-unmaintained hybserv2.

I decided to not even attempt to get this stuff up to date, but rather replace the server entirely with recent and maintained software. The choice fell on InspIRCd 3 and Anope 2. Switching to a new IRC server is not really that much of a challenge, especially in a single-server network. There is some minor tweaking required, such as making user and channel modes compatible with each other (e.g. InspIRCd 3 dropped core support for the halfop (+h) channel mode, which can hovever be reintroduced with the "customprefix" module). The far greater challenge were the IRC services, as there are some major incompatibilities:

  1. Hybserv2 stores password hashes using the original crypt algorithm based on DES. It would support MD5 instead of DES as well, but that would have to have been enabled beforehand. Anope on the other hand, doesn't support DES. They offer various variants of crypt-style hashes as well, but the "least modern" of those is MD5.
  2. Both Hybserv and Anope store their persistent data in plain text files (Anope does support other methods, such as database backends, but recommends the text file), however the formats of these files are incompatible and neither format is (well) documented.

Password Hash Migration

As it turns out, Anope has built-in functionality for password hash migration: When more than one password hashing module (enc_*) is loaded, only the first one is used for hashing new passwords. The other modules are only used for verifying already existing hashes. On top of that, when a user successfully authenticates against NickServ, and the hash is not in the default format (i.e. the first module), Anope will automatically re-hash the password entered by the user using the default method, and write it to disk.

Using this method, the old DES hashes are replaced by more modern ones as each user logs in, until all hashes have been converted (or the NickServ registrations of inactive users have expired). So the only thing left to do is to write an Anope module to verify DES crypt password hashes. This is pretty straight forward, as all that's needed is a thin wrapper around crypt that conforms to the Anope module API. My implementation can be found on Gitlab along with the rest of the sources. To build the module, and including it in your Anope installation, the following steps are required:

  1. Clone the Anope repo from Github: git clone
  2. Checkout the exact version of Anope you intend to run. In my case, this was version 2.0.9, so I'd do git checkout 2.0.9. This step is important, because Anope refuses to load modules built for another version.
  3. Place the enc_des.cpp file in modules/encryption/ in the Anope repo and follow the Anope build instructions:
    1. ./Config
    2. cd build
    3. make. Actually, since we only need the DES module, we can run make instead, which will complete a lot faster.
  4. Link against libcrypt. Since I didn't want to deal with all the CMake stuff, I decided to just patch the .so file afterwards: patchelf --add-needed modules/ Replace the actual version of by whatever is available on your target system.
  5. Put the built module into your Anope installation's modules directory. If you're using the Anope Debian package, this would be /usr/lib/anope/modules/

The final step left to do is to configure Anope to load the module:

// Used for new passwords and converting old passwords
module { name = "enc_bcrypt" }
// Used for verification only, passwords are automatically rehashed with bcrypt
module { name = "enc_des" }

I chose to use bcrypt as the new hashing method. You can choose whichever method you like, however you must make sure that enc_des is listed after the chosen module, otherwise all passwords would be rehashed to DES crypt.

And that's about it for password hashes. You should then occasionally check whether there are still any DES hashes left, and once they're all gone, unload and remove the enc_des module.

Understanding Hybserv's Database

Hybserv's "database" consists of multiple files: one for nicks, one for channels, and so on. In this specific case, only NickServ and ChanServ were used, so there was no need to migrate the others.

As mentioned before, there is no documentation of what the content in these files looks like, so we'll have to figure this out on our own.

A user in the nick.db file looks like this:

s3lph 14600 1602803962 1608435182
->PASS EelyBKlmXzO5r
->LASTUH ~s3lph
->LASTQMSG :So long and thanks for all the fish
->HOST *
->HOST *~s3lph@127.0.0.*
->TS 1608435182

And this is a channel in the chan.db file:

#example 4 1602804235 1603929600
->FNDR s3lph 1602804235
->PASS wahF8nhZXc4D
->TOPIC :This is an example
->ALVL -1 5 8 5 5 8 10 10 10 8 15 20 25 40 50
->ACCESS s3lph 50 1602804235 1603929600 *As Founder*
->ACCESS s4lph 10 1602803962 1608435256 s3lph

Each item is started with a line consisting of 4 fields: the name, some number and two numbers that look like UNIX timestamps. Looking at Hybserv's source code, we learn that the first number is a set of flags which control the services' behavior in regard to the nick or channel. The source code also tells us that the timestamps are the registration time and the time the resource was last "seen".

The following lines are mostly straight-forward. They all start with ->, which seems to indicate continuation of the object. They feature "somewhat human readable" keys and one or more values separated by spaces. For NickServ registrations, there are values such as the DES password hash, the email address, or which hostmasks the user has successfully identified from before. For ChanServ registrations, we see values such as the channel topic or the founder.

The last few lines were not entirely obvious, so again a look at the source code provided clarification. The ALVL chain of numbers turns out to be a definition of which permissions (e.g. voice, halfop, chanop, founder) are granted starting from which "access level". This also clarifies the format of the ACCESS lines: After the nick comes the assigned access level, then two timestamps (the time access was granted and the time it was last used) and finally the nick of the user granting the access. In the example above, s3lph would be the channel founder, and s4lph would be a halfop.

Understanding Anope's Database

Anope uses a similar approach to storing data. Again, data is stored in plain text, however all data is stored in one file, anope.db, instead of multiple files. The format is actually quite similar, but does feature some crucial differences (empty lines only inserted for readability):

DATA display s3lph
DATA pass des:EelyBKlmXzO5r
DATA email 
DATA language 
DATA access s3lph@127.0.0.* 
DATA memomax 0

OBJECT NickAlias
DATA nick s3lph
DATA last_quit So long and thanks for all the fish
DATA last_realname Unknown
DATA last_usermask
DATA last_realhost
DATA time_registered 1602803962
DATA last_seen 1608435182
DATA nc s3lph

OBJECT ChannelInfo
DATA name #example
DATA founder s3lph
DATA description 
DATA time_registered 1602804235
DATA last_used 1603929600
DATA last_topic This is an example
DATA last_topic_setter s3lph
DATA last_topic_time 1608173683
DATA bantype 2
DATA banexpire 0
DATA memomax 0

OBJECT ChanAccess
DATA provider access/xop
DATA ci #example
DATA mask s4lph
DATA creator s3lph
DATA last_seen 1608435256
DATA created 1602803962

Due to the similarities, I'll only highlight the main differences:

  • Objects are typed (since all are kept in the same file) and are terminated by an explicit END rather than by the start of the next object.
  • There is no single number containing a lot of flags; instead each flag is represented by its own line (e.g. HIDE_EMAIL, NS_SECURE, ...)
  • NickServ passwords are prefixed with the used hashing module. I chose to use "des" as the prefix for my enc_des module.
  • There are no more ChanServ passwords. Instead, users are automatically granted their permissions once they identify through NickServ.
  • Attributes that can occur multiple times in a Hybserv object are extracted into their own type. For example, channel ACCESS attributes are now represented by ChanAccess objects.
  • ChanAccess objects don't have a numeric access level, but a "data" attribute. This can be one of VOP (voice), HOP (halfop), SOP (chanop) or QOP (founder).

Textfile Database Conversion

With the information gathered above, the conversion from one format to another can be automated. My conversion script can be found on Gitlab. However, you should not use it without careful review as there may be some subtle differences between my setup and yours. Especially the numeric access levels will most likely need to be adapted on both the Hybserv as well as on the Anope side. The conversion can be done by following these steps:

  1. STOP ANOPE. Otherwise Anope will overwrite your changes the next time it flushes changes to disk.
  2. Find your anope.db file. When using the Debian package, it's located under /var/lib/anope/db/anope.db.
  3. MAKE A BACKUP of the anope.db file. The following two commands will append the converted data to anope.db. If something goes wrong, you may lose existing Anope registrations.
  4. ./ nickserv .../hybserv/nick.db anope.db
  5. ./ chanserv .../hybserv/chan.db anope.db
  6. REVIEW anope.db. Make sure that everything looks as expected.
  7. Start Anope and observe the log for errors.


The two migrations above (password hashes and database files) should allow you to migrate from Hybserv2 to Anope without requiring any action whatsoever from your users. If everything went fine, you should be able to log into NickServ and obtain ChanOp on your channels.

When you look into your anope.db, you will see that your DES password hash was replaced by a bcrypt hash (or whatever method you chose). If it's not visible immediately, you can either wait some time, /msg OperServ UPDATE or restart Anope so that all changes are immediately flushed to disk. Once there are no more DES hashes left, unload the enc_des module.

Reminders with VoIP Phones, Asterisk & Crontab


Update: As it turns out, this solution causes people to get used to the ringing phone, causing not only the reminder to be ignored, but actual incoming calls as well. We have thus stopped using this solution.

It is considered best practice to regularly and often ventilate closed rooms due to SARS-CoV-2. But how can we make sure we remember to open the windows every now and then? We wanted to build some kind of visual or acoustic mechanism that would tell us when to open the windows.

I came up with the idea to abuse our already existing telephone system as a reminder system. This system consists of a central Asterisk server, and multiple SIP phones. The idea was to have Asterisk dial all the SIP phones, and set the reminder text as caller ID. The "secret" behind this is Asterisk's console calling feature. And here is how it's done:

First, we need to enable the chan_alsa module in Asterisk's /etc/asterisk/modules.conf:

load =>
; Comment out existing noload directives
;noload =>

Afterwards restart Asterisk.

In a default installation of Asterisk on Debian, this should be all that's required for console calling to work. In our case, Asterisk is running in a VM and the default audio device is a virtual sound card not used otherwise, so there was no need for additional configuration.

Next, we can configure the dialplan in /etc/asterisk/extensions.conf for our reminders:

exten => ventilate,1,NoOp()
same => n,Answer()
; Set the caller ID name to the reminder text
same => n,Set(CALLERID(name)=Lueften!)
; Dial all the phones
same => n,Dial(PJSIP/phone1&PJSIP/phone2)

To apply this change, we have to reload the dialplan:

# asterisk -rx 'dialplan reload'

We can then test our reminder by telling Asterisk to originate a call from its console context:

# asterisk -rx 'originate console/dsp extension ventilate@reminders'

Now your phones should start ringing, and display your reminder text as the caller ID.

Finally, in order to schedule the reminders, just put them into the crontab of the Asterisk server. In our case, I configured it to ring every full hour on Tuesdays between 8 and 11 PM:

0 20-23 * * 2 root /usr/sbin/asterisk -rx 'originate console/dsp extension ventilate@reminders'

Cisco 7900 series IP Phone Logo Converter


Last week we discovered a box full of old Cisco 7900 series IP phones hidden deep in a pile of boxes in our hackerspace. Of course we tried to get them up and running and figure out how to configure them.

After a bit of research and reading manuals, we learned that the phones can be configured via TFTP using a binary config file format. When we learned that the phones can display a custom 88x27 monochrome image instead of the default Cisco logo, we made it our top priority to get the image customization working.

What we learned was that the images had to be served alongside the configuration via TFTP in a proprietary format. The tool to convert images into this format comes bundled with every firmware release for the phone. However, we quickly encountered some limitations:

  • The converter, called bmp2logo.exe, was a proprietary, Windows only piece of software. Luckily, the tool ran in Wine without any problems (as long as you don't mess with the input file header).
  • The tool required the input file to be in a very specific format, namely a 88x27 pixel, monochrome, 1 bit per pixel BMP file.
  • While the image is drawn as dark pixels on a light background, the input file had to be white on black. Effectively, the colors were inverted.

We also learned that each image had to be tagged with a serial number. The phone uses this serial number to decide whether a new image has to be loaded from the TFTP server. If the serial number is the same as the number of the previous image, the new image is not loaded. So the serial number needs to be incremented by at least 1 for each new image.

To get around the limitations of the converter tool, I attempted to figure out the file format and write a free and cross-platform converter without these limitations.

Reverse Engineering the File Format

So let's have a look at a hexdump of one of those image files converted with bmp2logo.exe:

Hexdump of a converted image file

We know that the image consists of 88 x 27 = 2376 pixels. And since the input to bmp2logo.exe has to be a 1-bit-per-pixel uncompressed image, let's just assume that the same holds true for the output. This would give us a payload size of 297 bytes, so with a file size of 304 bytes, there should be a 11 byte header.

The first two bytes were always the same, no matter what image was converted. So it should be fairly safe to assume that they are a magic number. The next two bytes were always different for different serial numbers or different images. Also, if the serial number was changed by one, these two bytes would also change in a fairly consistent manner, so this appeared to be some some sort of checksum. Let's just ignore that for now and put it aside for later.

The next three bytes were all zeros, followed by a byte representing the serial number. Or so i thought at first, until I passed a serial number of -1 to bmp2logo.exe, and got ff ff ff ff as these four bytes, so this pretty clearly is a 32-bit representation of this number. The next two bytes were pretty obvious as well, they are the height and width of the image, 27 and 88 respectively.

The last byte of the header seems to represent the number of bytes that comprise a single row. I'm not entirely sure about this, but it made the most sense, especially if we assume that there are other phones out there which may support grayscale images, and need a greater color depth and consequently more bits per row.

Now, on to the last part: The checksum. I was a bit lost here, so I just tried various ways of compressing data into 16 bits: Adding them, xor'ing them, swapping bytes around before adding them, always wrapping the results to 16 bits. In the end it was a typo that brought me to the solution: Instead of sum = (sum + swapped) & 0xffff, I wrote sum = (sum + swapped) % 0xffff. So instead of performing addition mod 65536, i was performing addition mod 65535 by accident. The result of this turned out to be just the binary inverse of the intended checksum, so let's add a final sum ^= 0xffff before returning the result and call it a day.

File Format

With all that information, and verifying that the payload did indeed match the uncompressed monochrome image, we finally got our file format:

| 10h 60h |  CHKSUM |       SERIAL      |
| H  | W  | RW |                        |
+----+----+----+                        |
|                                       |
:               PAYLOAD                 :
:                                       :
|                                       |
|                                       |

To summarize, the file consists of an 11 byte header, followed by the payload. The header consists of:

  1. The magic number 0x10 0x60
  2. A 16 bit checksum
  3. The 32 bit serial number in big endian
  4. H, the height of the image in pixels, 1 byte. Since the image is 27px high, this is 27, or 0x1b
  5. W, the width of the image in pixels, 1 byte. Since the image is 88px wide, this is 88, or 0x58
  6. RW, the number of bytes in a single row, 1 byte. Since the image is 88px wide and each byte holds 8 pixels at once, this is 11, or 0x0b

Following the header is the image payload, with each pixel expressed as a single bit, in row-major order starting in the top left corner.

A Python implementation of the checksum algorithm could like this:

sum = 0
# Iterate the file starting at byte 4 (directly after the checksum)
# Iterate two bytes at a time (a and b), stick them together in
# reverse order, and add to the total sum mod 0xffff.
for a, b in zip(data[4::2], data[5:2]):
    sum = (sum + ((b << 8) | a)) % 0xffff
# Flip all bits in the result
sum ^= 0xffff


Now that we know how such an image file is composed, we can write a tool to generate these files. And here's the result:

Cisco 7900 phone screen with a cat instead of the Cisco

You can find the converter script on Gitlab. It can be used in exactly the same way as the original bmp2logo.exe converter:

  1. ./ <serial> <infile.something> <outfile.dat>
  2. Copy <outfile.dat> to your TFTP server
  3. Update the phone configuration to point to the new file
  4. Recompile the configuration file and copy it to your TFTP server
  5. Reboot your phone

Remember to increment the serial number every time you change the image, or the phone won't pick up the new file.

Scaling and Slicing PDF Documents with pdfjam and mutool


I recently encountered the following challenge: I had a PDF document consisting of multiple A4 pages, each of which I needed to print scaled up to A2. However, I only had an A4 printer available, and the printer driver was not able to perform the scaling and/or slicing on its own.

After a long search, and lots of disappointments (since neither of Libre Office, Inkscape and Evince could do what I wanted, or I simply couldn't find the feature), I came up with the following solution:

$ pdfjam -o scaled.pdf --a2paper input.pdf
$ mutool poster -x 2 -y 2 scaled.pdf sliced.pdf

The first command takes the input file, scales it up to A2 size, and writes it to an intermediate file. The second command slices each page from the intermediate file into 2 slices both vertically and horizontally, totaling in 4 slices per page, which again results in A4-sized slices the printer can print.

Automated Debian package building with Gitlab CI and Reprepro


I previously deployed some ad-hoc services using ugly and difficult to maintain solutions, such as binaries manually extracted from container images and copied into /usr/local/bin. Of course, this is a lot of manual work for each upgrade of the service, and if you upgrade often, a lot of repetitive manual work. Or, in other words, the perfect kind of work to be automated.

So I decided to package the software in question for Debian, which is the Linux distribution I run these services on, and to automate the process of building the packages and publishing them to a repository. There's three different "categories" of software I wanted to have in my repository:

  • Binaries published as build artifacts in releases of their upstream repository. An example for this is Gitea. The binaries can be automatically downloaded and put into a package.

  • Software released as container images only. An example for this is the Drone CI Server. This is a bit more tricky, as the binaries must be extracted from the image before it can be put into a package.

  • Software already released as Debian packages, but not available through a repository. An example for this is a project of my own, the iCalendar Timeseries Server, where the CI pipeline automatically builds Debian packages and adds them as build artifacts to releases. Here, the already-built package just needs to be fetched and added to the repository.

For creating and maintaining the repository, I'm using Reprepro, as it is extremely simple to use, and generally is rather uncomplex and lightweight. It does however come with some limitations; I especially encountered the problem that Reprepro will only keep the latest version of a package in the repo index, so older versions won't be available to clients.

For automating the packaging and repository build process, I'm using Gitlab CI. The pipeline is running each night, building the latest stable version for each package, adding the packages to the in-container Reprepro repository, and then synchronize the repository to a web server of mine.

The source repository with the package build scripts and pipeline configuration is available on Gitlab. Though, before using this as a template, be advised that the packages built by this pipeline are not exceptionally high-quality. They are also usually built to fit my personal needs, so don't expect ready-to-use packages.