Remotely Access Live Supermicro Firmware+

TLDR; a vendor supported/supplied utility allows download of live BMC¹ firmware/configuration on (at least) some SuperMicro BMCs. It’s hard to tell who might be affected, but the utility used was written by ATEN Technology, ASRock ASPEED, and others all seem to be connected in various ways in not only SuperMicro firmware but the rest of the BMC world, and it’s not clear to me what other vendors might have this feature (sic) as well (QEMU lists many vendors using Aspeed BMCs that potentially could have something similar.)

I’ve only tested this on a couple of systems – a Supermicro X11SSZ-QF and a Supermicro X10-based virtual BMC (this firmware was run under QEMU (release notes claim to support their X10DRU-I+, X10DRT-P, X10DRG-H, X10DRW-I, X10DRI-T4+, X10DDW-I, X10DRFR, X10DRI-T, X10DRFF, X10DRFR-T, X10DRD, X10DRD-L motherboards.) That said, I’d say evidence strongly suggests this extends to other versions, as nearly identical strings² are hardcoded in /lib/libipmi.so that show the dd command dumping firmware to the same location. This is at least true for Supermicro servers in their X10, X11, X12, and X13 families; it’s been around for some time and presumably still is. Feel free to send me intel on test results of other systems, and update as appropriate.

The downloaded firmware includes the base OS, web pages/programs, BMC configuration, most files, and the kernel. Yes, you can change the firmware remotely by any number of methods, but here an attacker can see exactly what is going on on the BMC. Compromising the BMC becomes much easier if you can observe what the exact system is doing. Consider: would you want a potential attacker to be able to see your server/app configurations, password files, SSH keys, etcetera? It’s a dangerous thing, even if it’s not a bug but a feature. But hey, they didn’t ask me.

I think it’s especially interesting because it’s not simply being able to get the firmware of a BMC server, but it does so via valid IPMI by leveraging the OEM commands (the specification details how vendors can extend the protocol to allow vendor-specific commands/functionality³.) In the past I’ve heard rumors of various features from BMC vendors, such as capturing and freezing RAM, generating SMIs, etc., but these are all hidden from public view… how much do you trust your server vendor?) Ironically I’d actually seen this utility many years ago, but could never get it to work, apparently because I didn’t have the right hardware.

Details

ATEN Technology has a freely downloadable tool – AlUpdate⁴ that can do a variety of operations to a local or remote BMCs; here’s what I used in tests –

Under the hood (each item representing packet(s) sent/received) it has 4 stages –

1. authenticate to BMC and set communication parameters
2. get ready to send
3. send a I’m-ready-when-you-are-command
4. let me have it!

The first step is the setup request via a RMCP+ Open Session request via UDP on port 623, the standard IPMI port. I only mentioned it because while it could request any of a number encryption and data integrity algorithms, they instead shut off data integrity and confidentiality by using the 0x00 algorithms for both, meaning that all of this all sent in the clear with no protection.

After the connection is established, the AlUpdate tool sends three OEM IPMI commands in sequence –

• 0x1d
• 0x1e
• 0x1f

The first command packet is sent and receives a command was successful response.

The basic structure of each command is the same – here’s the full UDP payload of the first packet –

    06001200000000000000070020f8e881001d62

Or in tshark format –

    
        Version: 0x06
        Reserved: 0x00
        Sequence: 0xff
        Type: Normal RMCP, Class: IPMI
            ...0 0111 = Class: IPMI (0x07)
            0... .... = Message Type: Normal RMCP (0x0)
    IPMI v2.0+ Session Wrapper, session ID 0x360
        Authentication Type: RMCP+ (0x06)
        Payload type: IPMI Message (0x00), not encrypted, not authenticated
            0... .... = Encryption: Payload is unencrypted
            .0.. .... = Authenticated: Payload is unauthenticated
            ..00 0000 = Payload Type: IPMI Message (0x00)
        Session ID: 0x00000360
        Session Sequence Number: 0x00000000
        Message Length: 7
    Intelligent Platform Management Bus
        Target Address: 0x20
        Target LUN: 0x00, NetFN: OEM Request (0x3e)
            NetFn: OEM Request (0x3e)
            .... ..00 = Target LUN: 0x0
        Header Checksum: 0xe8 (correct)
        Source Address: 0x81
        Source LUN: 0x00, SeqNo: 0x00
            .... ..00 = Source LUN: 0x0
            0000 00.. = Sequence Number: 0x00
        Command: Unknown command (0x1d)
        Data checksum: 0x62 (correct)

Behind the scenes the BMC starts packing up the firmware on the BMC – if you’re on the BMC (perhaps more on that on a follow-up), running the ps command shows –

What is it doing? MTD is the Memory Technology Device subsystem and is used for storage backed by flash memory of various sorts (like NAND, NOR, etc); this shows the BMC extracting the firmware from its flash storage via dd to get ready to send.

The “/proc/mtd” psuedo file shows some details on the various MTD devices, including mtd5:

That “all_part” means basically the entire system – from it you can extract (via binwalk or whatever methods you’d like) the base OS, plus the web components, kernel, configuration details, etc. When the BMC boots up it mounts the root file system and then /nv and /web… here’s the kernel command line and the mount commands –

    console=ttyS1,115200 root=/dev/mtdblock2 rootfstype=cramfs noinitrd rw mem=79M

    mount -t jffs2 /dev/mtdblock1 /nv
    mount -t cramfs /dev/mtdblock4 /web

Every time a “0x1d” IPMI command is sent the “dd” is triggered again… (not in parallel, it kills off any previous)… and the lil’ BMC really struggles during that time. (Makes me wonder if repeated hammering of the disk might cause problems….)

The BMC itself runs Linux version 2.6.28.9 on a CPU: ARM926EJ-S –

The “dd if=/dev/mtdblock5 of=/tmp/dump_flash” bit is hardcoded in /lib/libipmi.so, and is executed via the /bin/ipmi_lan persistent daemon, which listens to UDP port 623 (IPMI’s default listening port.)

Next, AlUpdate waits about 50 seconds to send the second packet (command 0x1e.) Until the firmware is completely saved to disk the BMC will respond with a successful command completion code plus a 0x00 data field. The tool continues to query every 4-5 seconds until the BMC sends a 0x01 in the data field followed by 0x200000 (0x0200000) – the latter corresponding exactly to the size of the BMC firmware (33,554,432 in base 10.)

The request packet –

    0600760a000001000000070020f8e881001e61

And the final response packet –

    06002923be84070000000f0081fc8320001e0001020000000000bf

After seeing that response 0x1f IPMI commands are sent and repeated until the entire flash is sent over. This takes about XXX minutes in total. The request packets are the same old thing –

    0600760a000007000000070020f8e881001f60

While the response packets cram in 57 bytes per UDP packet (55 data bytes after the command and the checksum is taken into account.)

    06002923be840d000000410081fc8320001f0000ae17ee57504ba6a3a879a8dbb1b1d9c68a6252d1cb26de9049037a501fe781f78073c010e0c6fc32208e7abb00e8c03cb0e975f4bb0061dc

The whole exchange takes about 800,000 packets back and forth. After a successful transfer or if nothing happens for about 10m after the initial mtd dump, the image file will be removed from /tmp on the BMC.

Download BMC conf separately?

Oddly, using the -c flag to get just the configuration (e.g. ./AlUpdate -c -d backup.bin -i kcs -r y as per the usage output didn’t work. Checking the packet flow and what’s going on the BMC it seems to start fine, but then something goes awry… perhaps someone else can let me know how it should work.

Quick Test

You don’t strictly need the AlUpdate tool to test this… if you send a single authenticated packet via your IPMI tool o’ choice, such as –

    ipmi-raw --driver-type=LAN_2_0 -h 10.0.0.1 -u ADMIN -p ADMIN 0 3e 1d

or

    ipmitool -U ADMIN -P ADMIN -H 10.0.0.1 raw 0x3e 0x1d

On susceptible systems it’ll start the dump to /tmp/ on the BMC, and presumably throw an error (but who knows on various systems; other OEMs might use it for different purposes, since it’s up to them to define) if it doesn’t recognize the command.

Denouement

In the grand IPMI scheme of things, just another issue, I suppose; it’s already dangerous enough with all the other problems! Still, that someone could vacuum out such information from you BMCs seems at least casually alarming. Certainly if an adversary was able to do this it’s probably game over.

Others might claim that if they had your credentials or admin access to your BMC it’s pretty much over anyway, since you can upload new firmware or change the configuration anyway.

There is a substantial difference, however, between blindly overwriting existing data and being able to view and change existing data. And since the basic configuration and password of your BMCs is shared through multiple vendors, it means that even if your other BMCs aren’t Supermicro, such disclosure puts them at additional risk as well.

And if an adversary was able to install a backdoor into existing firmware (as I did to gain shell access to my own BMC to determine what was going on behind the scenes) by examining the configuration details of your BMC, that’s substantially more difficult to detect than someone actually changing the BMC carte blanche. Even if you don’t detect backdoors or changes to BMC configuration (which can be quite tricky), one might well notice a change in firmware version.

YMMV.

Be well and be safe.