Tag: Boot

Layer 2 TFTP Relay for TFTP Recovery with IP Conflicts

en

Today I had to deal with an embedded device which refused to boot up due to a misconfiguration. Normally, this device, a Mitel DECT RFP 32 IP, obtains a DHCP lease on boot and fetches the firmware to run via TFTP. The firmware is downloaded at every reboot, since the device can not store it locally.

It can however store some boot options in flash, such as static IP and TFTP configuration. And this is where the problem started: Someone had accidentally configured 192.168.42.2 for the device's own IP AND the TFTP server IP. Meaning the device would boot up, assign itself a statically configured IPv4 address and then attempt to fetch the boot image via TFTP from exactly the same IP. Said TFTP traffic of course never left the device.

Flash Chip Short-Circuit

As a first measure, together with a friend, we took the device apart and checked whether any of the chips on there was a flash storage chip, and indeed we found one: An 8K I²C flash chip. Hoping that this is where this configuration was stored, we shorted the I²C bus' SDA to GND in order to disable communication with said chip.

Unfortunately this didn't do anything - apparently the boot configuration is not stored on this chip. And unfortunately we didn't find another storage chip on the PCB. So I decided to try another approach.

At least probing around the board we found a serial console, on which the device wrote log output during the boot process, which did help a lot.

ARP Spoofing

I first attempted ARP spoofing to convince the device to send its TFTP requests to the outside world:

$ sudo pacman -S dsniff
$ sudo arpspoof -i enp2s0 -c both 192.168.42.2

And indeed, amidst all the unsolicited ARP replies TFTP requests started popping up! So, at least one direction was working now. I did have to assign the same 192.168.42.2 to my notebook computer running the TFTP server. After doing so, the TFTP server started responding.

However, the TFTP response did not make it out onto the network, as the destination IPv4 was assigned to the notebook itself. I tried to mess around with ebtables/arptables a bit, but didn't find anything that worked as inteded.

Layer 2 Relay

So I decided to attempt to get it to work by having the TFTP server listen on localhost only, and write a small piece of software which would relay the network traffic between localhost and the LAN.

I originally planned to simply open two raw sockets (i.e. sending and receiving full Ethernet frames), one bound to the loopback interface, and one bound to the LAN interface. This piece software would then simply relay the Ethernet frames between the two interfaces, but replace MAC and IP addresses as needed, and recalculate the IPv4 header checksum along the way:

Visualization of the first layer 2 relay approach. The frames are passed almost as-is, only the MAC addresses and IP addresses are replaced, and the IPv4 header checksum is recomputed.
Figure 1: Visualization of the first layer 2 relay approach. The frames are passed almost as-is, only the MAC addresses and IP addresses are replaced, and the IPv4 header checksum is recomputed.

However, for some reason this did not work out quite as intended. It turned out that the frames sent out to the loopback interface also ended up being caught by the relay software and be sent out to the LAN, somehow disrupting TFTP traffic. The workaround I ended up with was to use a regular UDP datagram socket on the loopback interface instead. With this approach, frames coming in on the LAN interface would be stripped of their Ethernet, IP and UDP headers and the payload forwarded to the loopback UDP socket. For responses from the TFTP server, on the other hand, the relay had to construct these headers and send a full Ethernet frame out to the LAN:

Visualization of the layer 2 / UDP relay approach. This time, there is a UDP datagram socket on the loopback side, and protocol overhead is removed and added in the relay.
Figure 2: Visualization of the layer 2 / UDP relay approach. This time, there is a UDP datagram socket on the loopback side, and protocol overhead is removed and added in the relay.

And it worked! At least at first: The first client - server - client roundtrip went through. However, when the client attempted to acknowledge this transaction, things started to fall apart. Closer inspection of the network traffic revealed that the UDP payload sizes didn't match. The TFTP ACK was much longer than it should have been, and the excess bytes looked oddly familiar.

As it turns out, the DECT RFP is reusing the same buffer for outgoing TFTP messages over and over again, but when sending those messages out to the network, they are not limited to the actual message length; instead the full buffer is sent. The length fields in the UDP and IP headers were correctly set to only include the actual message though. The remainder of the buffer was just noise at the end of the Ethernet frame.

Visualization of the TFTP buffer reuse issue. Note how the IPv4 and UDP headers terminate at the end of the ACK message, but the Ethernet frame still contains the entire buffer.
Figure 3: Visualization of the TFTP buffer reuse issue. Note how the IPv4 and UDP headers terminate at the end of the ACK message, but the Ethernet frame still contains the entire buffer.

This caused problems in the relay software, since I was deducing the IP and UDP length fields from the total length of the received Ethernet frame. Reading the actual payload size from the UDP header instead of just assuming its size resolved the problem.

And with that, it was finally working: The DECT RFP successfully pulled its firmware image and booted. From this point, we were able to perform a factory reset, which switched it back to DHCP.

Conclusion

It turns out you can make two hosts on the same network talk to each other even when they have the same IP address. You just need to take a dive down to the Ethernet layer and take routing decisions away from your Operating System. The relay software I wrote for this purpose can be found on Gitlab. It still needs to be used in conjunction with arpspoof.

Though, in hindsight, it would probably have been possible to get the TFTP responses out onto the network simply by messing with my notebook computer's ARP table. However, I didn't think of this at the time and haven't had an opportunity to try since. (Not gonna intentionally brick that device again just for testing purposes.)

Chainload GRUB 2 from GRUB 2

en

I recently needed to set up a PC to dual boot Windows and Linux. Legacy BIOS, no UEFI. Unfortunately Windows likes to do Windows things and override the first-stage GRUB bootloader in the MBR with its own, and thereafter booting Windows only.

As this PC needs to be used by different persons with different technological skillsets, and in order to make repairing the boot process easier even if e.g. no Linux live boot medium were available, I came up with the following solution which loads GRUB via PXE to repair the "real" bootloader.

This solution is quite easy for end users. In the case of this specific PC, PXE boot can be triggered by keeping F8 pressed during boot. The instructions to do so fit on a small post-it note which I attached to the PC's screen.

Set Up DHCP and TFTP

You'll need:

  • A configurable DHCP server
  • A TFTP server
  • A device with GRUB 2 already installed

All of those can of course be on the same device. Sometimes the first two even have to be, since a lot of consumer-grade network cards can only load data via TFTP from the same device that's running the TFTP server. Basically they ignore and don't request DHCP option 66 (server-name), and only handle DHCP option 67 (bootfile-name) and attempt to load it via TFTP from the address that offered the DHCP lease.

First we need to set up the TFTP server. I'll use dnsmasq for both TFTP and DHCP. TFTP configuration in dnsmasq is as easy as the following two lines in /etc/dnsmasq.conf:

enable-tftp
tftp-root=/srv/tftp

We then need to configure a static DHCP lease with additional DHCP options. Again in dnsmasq.conf:

# "pool" is the pool name
dhcp-range=pool,192.168.0.10,192.168.0.254,24h
# "target" is the target's host name. "tftp-grub" is a "tag" used in the next two lines.
dhcp-host=01:23:45:67:89:0a,set:tftp-grub,192.168.0.10,target
# Set TFTP server and bootfile path
dhcp-option=tag:tftp-grub,option:tftp-server,192.168.0.1
dhcp-option=tag:tftp-grub,option:bootfile-name,/grub/i386-pc/core.0"

Create a GRUB 2 Netboot Environment

On the DHCP server (can also be done on a different host and then copied over), run the following command:

grub-mknetdir --net-directory /srv/tftp --subdir grub -d /usr/lib/grub/i386-pc/

The main difference between a system's /boot/grub and this netboot environment is the grub/i386-pc/core.0 file. This is GRUB's first-stage loader, the same thing that normally gets written to the MBR, but here it's in a file instead, which can be loaded via TFTP.

If you now tell your target machine to boot from PXE, you should get a GRUB console.

Chainload GRUB 2 from GRUB 2

I searched around a long time to figure out how this is done, and most of the results i got said to "just load the second GRUB's config file into the first GRUB instance". This may work in most cases, but will eventually break in this case, since the second GRUB's config file may attempt to load additional modules from it's own prefix (/boot/grub/i386-pc), which will be incompatible with the PXE-loaded GRUB if they're not the same version.

Long story short, unlike GRUB 1, GRUB 2's second-stage loader (the core.img file) cannot be loaded on its own. Only the first-stage loader (from MBR or core.0) can be chainloaded. And since we're trying to cover the case where the firs-stage loader has been overwritten, we need to get it from somewhere else.

In this case I decided to set up a small loopback image in the target system's root partition which contained a copy of the system's GRUB installation.

To set up this loopback image, run the following commands on the target system:

dd if=/dev/zero of=/grubloop bs=100M count=1
parted /grubloop mklabel msdos
# Offset from the start is needed so that there's enough space for
# the first-stage loader.
parted /grubloop mkpart primary ext4 1048576B 100%
parted /grubloop set 1 boot on
losetup -fo 1048576 /grubloop
# Loop device number may be different
mkfs.ext4 /dev/loop0
losetup -d /dev/loop0
mkdir /altboot
mount -o offset=1048576 /grubloop /altboot

Also enter this into your /etc/fstab:

/grubloop   /altboot    ext4    loop,offset=1048576 0   0

Finally install GRUB to /grubloop resp. /altboot:

grub-install --boot-directory=/altboot --target=i386-pc --force /grubloop
grub-mkconfig -o /altboot/grub/grub.cfg

This image, which now contains the same boot config as your /boot/grub, can now be booted like you'd boot any hard disk. To chainload this GRUB image from the PXE grub, create the file /srv/tftp/grub/grub.cfg on your TFTP server and add the following:

insmod biosdisk
insmod part_msdos
insmod ext2
insmod chain

# Adapt to your real Linux root partition
set root=(h0,msdos4)
chainloader /grubloop +1
boot

I'm intentionally hardcoding the root partition here, instead of using GRUB's search module, since I want the PXE boot to be fast and simple. The search module, on the other hand, loads a LOT of other modules via TFTP, which may become a performance issue.

When you now boot your target device via PXE, it will load the first GRUB instance from TFTP, which will then chainload the second GRUB instance from the loopback image, which will then boot into Linux.

Automatically Re-Install GRUB

I'm doing all of the above not only to boot via PXE, but also to automatically re-install GRUB to the disk's MBR. I achieved this with the following systemd unit, which is executed on every boot of the Linux system:

[Unit]
Description=Reinstall GRUB 2 to MBR and loopback altboot

[Service]
Type=oneshot
ExecStart=/usr/sbin/grub-install --boot-directory=/altboot --target=i386-pc --force /grubloop
ExecStart=/usr/sbin/grub-install --boot-directory=/boot --target=i386-pc /dev/disk/by-id/<DISK_ID>
ExecStart=/usr/sbin/grub-mkconfig -o /altboot/grub/grub.cfg
ExecStart=/usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg

[Install]
WantedBy=multi-user.target

Maybe a bit overkill, but it gets the job done and is easy to use for end users.