Cisco Cloud Services Router 1000V and the Virtual Matryoshka

Recently we started playing around with Cisco’s virtual router, the CSR 1000V, while doing some protocol analysis. We found Cisco offering an BIN file for download (alternatively there is an ISO file which contains a GRUB boot loader and the BIN file, or an OVA file which contains a virtual machine description and the ISO file) and file(1) identifies it as DOS executable:

$ file csr1000v-universalk9.03.12.00.S.154-2.S-std.SPA.bin 
csr1000v-universalk9.03.12.00.S.154-2.S-std.SPA.bin: DOS executable (COM)

We didn’t manage to get the file running, neither in a (Free-)DOS environment, nor in a wine virtual DOS environment, except using the boot loader from the ISO file. So we became curious as for the structure and ingredients of the file.

Cisco is using a highly modified GRUB (more on this later) to launch an Linux kernel hidden inside the BIN file. After the kernel is loaded, an automated installation process starts preparing the virtual router environment (partitioning and formating the disk, installing a boot loader and extracting the virtual router software, this time a .PKG file). As soon as the installation is complete, the system restarts and boots into the virtual router.

So there are our first layers of the Matryoshka, an ISO file, holding the BIN file, containing a Linux kernel, a ram disk and the installed .PKG file.

In our next step we took a look at the virtual hard disk and the partitions created by the install script. And we found this:

Holy WOW, a lot of partitions!

So we tried mounting them by hand, but we only had success with two of them, the first partition, sdb1, which is seen by the virtual router as bootflash: and contains the router image (the .PKG file), another virtual appliance (the .OVA file – OHHH my Matryoshka) and by the time some trace files and reload logs:

The other partition we had success mounting was sdb5, also ext2 formated, but with a whole lot more interesting content. We found this to be the partition containing the GRUB stage2 image as well as the GRUB configuration file as well as the original .BIN file and some LUKS backup files:

This pointed in another direction, as the system seems using LUKS to encrypt some of the data and actually when using the luksDump functionality from cryptsetup(8) five of the partitions identified themselves as being LUKS encrypted (sdb2,6,7,8,9):

Ok, so this actually looks good, no proprietary encryption, no second key slot in use for ‘maintenance’ or ‘support’ and for the moment there is no way past this without the keys. Also, when reading the documentation of the CSR1000V there is this passage:

We can assume that at least one of the encrypted partitions contains the emulated NVRAM, which itself contains the running config with potentially some cisco7 ‘encrypted’ passwords.

As we did not have the keys yet (and yes, those obfl_temp_kX files did look a lot like keys, but they weren’t the LUKS keys) we continued tracing the boot process and found one weird thing in the GRUB config file:

They are booting a configuration file. WTF ?!?! Ok, as I said before, Cisco uses a modified version of GRUB and they sure patched it a lot, as it now does:

• Checksum verification on the configuration file
• Parsing of the configuration file to find the actual file containing the kernel
• Checksum verification on the kernel
• NVRAM access to find the value of the CONFREG (ROMMON emulation)
• Booting those magic .PKG files

But in the same way they stripped almost every feature GRUB is loved for. The interactive shell knows only two commands, confreg and help. also interactive config file editing is disabled and they messed up some of the sleep/input behavior.

Well, as GRUB is licensed under the GPL, I would love to see the source code of this ‘kind of GRUB’, but as usual with Cisco, the source code is nowhere to be found.

So, back to our Matryoshka. It took some time to figure out, but within the PKG file there are at least 3 more files: The Linux kernel getting booted by GRUB, an initrd file and last but not least another ISO file containing the root file system of the virtual router (and this finally is where the IOSd lives).

From this point on it wasn’t hard to read some of the init scripts and figure out were the LUKS keys live and as you may have already guessed, they are hidden inside of the obfl_temp_kX files. Cisco also ships a binary called khutil (short for Key Hiding UTILity, I guess), which is located on the initrd and is able to extract the LUKS keys from those files. I put together a small script, automating the process of unpacking the Matryoshka to a point where khutil is usable and then mounting the partitions and ISO files embedded:

That’s pretty much it for today, but one last thing: The Matryoshka doesn’t end here. Remember the OVA file from bootflash: ? It also contains an ISO file, containing another virtual machine, powered up inside of the(already) virtual router…

So what’s the moral of the story? First remember everything is a file, is a file, is a file, is a file (until it gets virtualized 😀 )
Second: sometimes it might make sense to verify vendor statements before blindly trusting them, especially when it comes to crypto. And last but not least: playing with virtual appliances can be as much fun as playing with their ‘physical’ counterparts…

cheers

/daniel