Layer 2 TFTP Relay for TFTP Recovery with IP Conflicts

en

Today I had to deal with an embedded device which refused to boot up due to a misconfiguration. Normally, this device, a Mitel DECT RFP 32 IP, obtains a DHCP lease on boot and fetches the firmware to run via TFTP. The firmware is downloaded at every reboot, since the device can not store it locally.

It can however store some boot options in flash, such as static IP and TFTP configuration. And this is where the problem started: Someone had accidentally configured 192.168.42.2 for the device's own IP AND the TFTP server IP. Meaning the device would boot up, assign itself a statically configured IPv4 address and then attempt to fetch the boot image via TFTP from exactly the same IP. Said TFTP traffic of course never left the device.

Flash Chip Short-Circuit

As a first measure, together with a friend, we took the device apart and checked whether any of the chips on there was a flash storage chip, and indeed we found one: An 8K I²C flash chip. Hoping that this is where this configuration was stored, we shorted the I²C bus' SDA to GND in order to disable communication with said chip.

Unfortunately this didn't do anything - apparently the boot configuration is not stored on this chip. And unfortunately we didn't find another storage chip on the PCB. So I decided to try another approach.

At least probing around the board we found a serial console, on which the device wrote log output during the boot process, which did help a lot.

ARP Spoofing

I first attempted ARP spoofing to convince the device to send its TFTP requests to the outside world:

$ sudo pacman -S dsniff
$ sudo arpspoof -i enp2s0 -c both 192.168.42.2

And indeed, amidst all the unsolicited ARP replies TFTP requests started popping up! So, at least one direction was working now. I did have to assign the same 192.168.42.2 to my notebook computer running the TFTP server. After doing so, the TFTP server started responding.

However, the TFTP response did not make it out onto the network, as the destination IPv4 was assigned to the notebook itself. I tried to mess around with ebtables/arptables a bit, but didn't find anything that worked as inteded.

Layer 2 Relay

So I decided to attempt to get it to work by having the TFTP server listen on localhost only, and write a small piece of software which would relay the network traffic between localhost and the LAN.

I originally planned to simply open two raw sockets (i.e. sending and receiving full Ethernet frames), one bound to the loopback interface, and one bound to the LAN interface. This piece software would then simply relay the Ethernet frames between the two interfaces, but replace MAC and IP addresses as needed, and recalculate the IPv4 header checksum along the way:

Visualization of the first layer 2 relay approach. The frames are passed almost as-is, only the MAC addresses and IP addresses are replaced, and the IPv4 header checksum is recomputed.
Figure 1: Visualization of the first layer 2 relay approach. The frames are passed almost as-is, only the MAC addresses and IP addresses are replaced, and the IPv4 header checksum is recomputed.

However, for some reason this did not work out quite as intended. It turned out that the frames sent out to the loopback interface also ended up being caught by the relay software and be sent out to the LAN, somehow disrupting TFTP traffic. The workaround I ended up with was to use a regular UDP datagram socket on the loopback interface instead. With this approach, frames coming in on the LAN interface would be stripped of their Ethernet, IP and UDP headers and the payload forwarded to the loopback UDP socket. For responses from the TFTP server, on the other hand, the relay had to construct these headers and send a full Ethernet frame out to the LAN:

Visualization of the layer 2 / UDP relay approach. This time, there is a UDP datagram socket on the loopback side, and protocol overhead is removed and added in the relay.
Figure 2: Visualization of the layer 2 / UDP relay approach. This time, there is a UDP datagram socket on the loopback side, and protocol overhead is removed and added in the relay.

And it worked! At least at first: The first client - server - client roundtrip went through. However, when the client attempted to acknowledge this transaction, things started to fall apart. Closer inspection of the network traffic revealed that the UDP payload sizes didn't match. The TFTP ACK was much longer than it should have been, and the excess bytes looked oddly familiar.

As it turns out, the DECT RFP is reusing the same buffer for outgoing TFTP messages over and over again, but when sending those messages out to the network, they are not limited to the actual message length; instead the full buffer is sent. The length fields in the UDP and IP headers were correctly set to only include the actual message though. The remainder of the buffer was just noise at the end of the Ethernet frame.

Visualization of the TFTP buffer reuse issue. Note how the IPv4 and UDP headers terminate at the end of the ACK message, but the Ethernet frame still contains the entire buffer.
Figure 3: Visualization of the TFTP buffer reuse issue. Note how the IPv4 and UDP headers terminate at the end of the ACK message, but the Ethernet frame still contains the entire buffer.

This caused problems in the relay software, since I was deducing the IP and UDP length fields from the total length of the received Ethernet frame. Reading the actual payload size from the UDP header instead of just assuming its size resolved the problem.

And with that, it was finally working: The DECT RFP successfully pulled its firmware image and booted. From this point, we were able to perform a factory reset, which switched it back to DHCP.

Conclusion

It turns out you can make two hosts on the same network talk to each other even when they have the same IP address. You just need to take a dive down to the Ethernet layer and take routing decisions away from your Operating System. The relay software I wrote for this purpose can be found on Gitlab. It still needs to be used in conjunction with arpspoof.

Though, in hindsight, it would probably have been possible to get the TFTP responses out onto the network simply by messing with my notebook computer's ARP table. However, I didn't think of this at the time and haven't had an opportunity to try since. (Not gonna intentionally brick that device again just for testing purposes.)