TAG | cloud
During a recent research project we performed an in-depth security assessment of Microsoft’s virtualization technologies, including Hyper-V and Azure. While we already had experience in discovering security vulnerabilities in other virtual environments (e.g. here and here), this was our first research project on the Microsoft virtualization stack and we took care to use a structured evaluation strategy to cover all potential attack vectors.
Part of our research concentrated on the Hyper-V hypervisor itself and we discovered a critical vulnerability which can be exploited by an unprivileged virtual machine to crash the hypervisor and potentially compromise other virtual machines on the same physical host. This bug was recently patched, see MS13-092 and our corresponding post.
First of all, I hope you all had a good start to 2014. Having some time off “between the years” (which is a German saying for the time between Christmas and NYE), I caught up on several virtualization security topics.
While virtualization is widely accepted as a sufficiently secure technology in many areas of IT operations (also for sensitive applications or exposed systems, like DMZs) by 2014, there are several recent vulnerabilities and incidents that are worth mentioning.
First of all, a rather old vulnerability (codename “VMDK Has Left the Building“) was eventually patched by VMware, the day before Christmas’ eve (honi soit qui mal y pense… ). While the initially described file inclusion vulnerability cannot be exploited anymore, first tests in our lab show that attempts to exploit the vulnerability lead to a complete freeze of the shared ESXi host. We still need to dig deeper into the patch and will keep you posted.
On November’s patch Tuesday, an important vulnerability in Hyper-V was patched by Microsoft. The bulletin does not provide a lot of details as for the vulnerability, but the relevant sentence is this one: “An attacker who successfully exploited this vulnerability could execute arbitrary code as System in another virtual machine (VM) on the shared Hyper-V host.”. This does not allow code execution in the hypervisor. However, Hyper-V’s architecture comprises the so-called root partition, which is a privileged virtual machine used for all kinds of management functionality. This means that code execution in this particular virtual machine most probably will still give an attacker complete control over the hypervisor. Even without this root partition, the vulnerability would be one of the worst-case vulnerabilities in the age of Cloud computing, provided that MS Azure employs Hyper-V (which can be considered a fair assumption. Still we have no distinct knowledge here). Again, we’ll have a closer look at this one in the near future.
At the end of December, OpenSSL suffered from a virtualization-related incident. The shared hypervisor was compromised using a weak password of the hosting provider. While password-related attacks are not specific to virtualized environments, it emphasizes the need for secure management practices for virtualization components. This sounds like a very basic recommendation, but many security assessments we conducted in this space resulted in the need to include “attacks against management interfaces” in the top ERNW virtualization risks, which we cover in our virtualization and cloud security workshops. Also we mentioned this in some presentations and research results.
As the described events show, virtualization security will remain an important topic in 2014 (even though marketing material suggest to simply adopt virtualization – I won’t give any links here, you’ve probably already seen plenty ). We will cover several aspects during this year’s Troopers edition. While our workshop on “Exploiting Hypervisors” is already online (for the detailed description, see here), one talk is missing: Due to some rather strict NDAs, we can’t provide any details so far (but if you’ve read the MS13-092 credits carefully, it shouldn’t be too hard to guess ).
I hope you’re looking forward to 2014 as much as I do, stay tuned,
with the rise of low-cost 3D-printers in the homes of thousands  of enthusiastic tinkerers the word spreads about these magical machines which can produce any mechanical, artsy, useful or useless parts you might come up with. Standing in living rooms worldwide, they don’t seem like a big threat  to anybody. But what happens if you connect them to the Internet?
What’s it about? 3D-Printing in a Nutshell.
0 Comments | Posted by Sergej Schmidt
The gritsforbreakfast blog post making the rounds on the Liberation Tech mailing list about security of Apple’s iMessaging service is gaining quite some attention. The post refers to a CNET article on how the iMessage service “stymied attempts by federal drug enforcement agents to eavesdrop” conversations due its end-to-end encryption and commends Apple for protecting the user’s privacy while pointing out that Gmail and Facebook Messaging don’t. However, I disagree on some points of the blog post and therefore want to discuss them here.
Last week Rapid7 posted an interesting analysis of the Amazon S3 storage system: Apparently roughly one out of six S3 buckets (a bucket is, simply said, a kind of folder) is accessible without any authentication mechanism. Accessing those files, the Rapid7 guys were able to download a wide range of data, also comprising confidential information such as source code or employee information, comparable to past research for other platforms (see also this presentation I gave on some of the biggest Cloud #Fails)
As you may already be familiar with some of our previous work which was mainly focused on isolation issues of hypervisors, we also want to present you an issue concerning availability in Cloud environments. This issue was already covered in some of our presentations, but will be explained in greater detail in this blog post.
at first a happy new year to all our readers!
And, of course, to everybody else, too ;-). May 2013 bring good things for you all, in particular (but not only) in the infosec space.
At the recent ATSAC 2012 conference a guy from the CERT Insider Threat Center gave a talk on the exact topic. Given that the ENISA Cloud Computing Risk Assessment lists “Cloud Provider Malicious Insider” as one of the top eight risks (out of overall 35 risks evaluated) and we just had some discussion about this in a customer environment, this might be of interest for some readers.
The slides of the talk can be found here.
As we are receiving a lot of questions about our VMDK has left the building post, we’re compiling this FAQ post — which will be updated as our research goes on.
How does the attack essentially work?
By bringing a specially crafted VMDK file into a VMware ESXi based virtualization environment. The specific attack path is described here.
What is a VMDK file?
A combination of two different types of VMDK files, the plain-text descriptor file containing meta data and the actual binary disk file, describes a VMware virtual hard disk. A detailed description can be found here.
Are the other similar file formats used in virtualization environments?
Yes, for example the following ones:
- VDI (used by e.g. Xen, VirtualBox)
- VHD (used by e.g. HyperV, VirtualBox)
- QCOW (used by e.g. KVM)
Are those vulnerable too?
We don’t know yet and are working on it.
Which part of VMDKs files is responsible for the attack/exposure?
The so-called descriptor file, describing attributes and structure of the virtual disk (See here for a detailed description).
How is this to be modified for a successful attack?
The descriptor file contains paths to filenames which, combined, resemble the actual disk. This path must be modified so that a file on the hypervisor is included (See here for a detailed description).
How would you call this type of attack?
In reference to web hacking vulnerabilities, we would call it a local file inclusion attack.
What is, in your opinion, the root cause for this vulnerability?
Insufficient input validation at both cloud providers and the ESXi hypervisor, and a, from our point of view, misunderstanding of trust boundaries, such as that one should “not import virtual machines from untrusted sources”.
Does this type of attack work in all VMware ESX/vSphere environments?
Basically, the ESXi5 and ESXi4 hypervisor are vulnerable to the described attack as of June 2012. Still, the actual exploitability depends on several additional factors described here.
Can this type of attack be performed if there’s no VMDK upload capability?
Which are typical methods of uploading VMDK files in (public) cloud environments?
E.g. Web-Interface, FTP, API, …
Which are typical methods of uploading VMDK files in corporate environments?
In addition to the mentioned ones, direct deployment to storage, vCloudDirector, …
Will sanitizing the VMDK (descriptor file) mitigate the vulnerability?
From our perspective this should not be too difficult to implement. There are basically two steps:
- Striping leading directory paths/relative paths from the path to be included
- Restricting included files to customer-owned directories
However a certain knowledge about the specific storage/deployment architecture is necessary in order to sanitize the VMDK descriptor file and not break functionality.
Will VMware patch this vulnerability?
Probably yes. They might do so “silently” though (that is without explicitly mentioning it in an associated VMSA) as they have done in the past for other severe vulnerabilities (e.g. for this one).
Could you please describe the full attack path?
All steps are described here.
More details can be found in a whitepaper to be published soon. Furthermore we will provide a demo with a simplified cloud provider like lab (including, amongst others, an FTP interface to upload files and a web interface to start machines) at upcoming conferences.
Do you need system/root access to the hypervisor in order to successfully carry out the attack?
No. All necessary information can be gathered during the attack.
What is the potential impact of a successful attack?
Read access to the physical hard drives of the hypervisor and thus access to all data/virtual machines on the hypervisor. We’re still researching on the write access.
Which platforms are vulnerable?
As of our current state of research, we can perform the complete attack path exclusively against the ESXi5 and ESXi4 hypervisors.
In case vCloud Director is used for customer access, are these platforms still vulnerable?
To our current knowledge, no. But our research on that is still in progress.
Are OVF uploads/other virtual disk formats vulnerable?
Our research on OVF is still in progress. At the moment, we cannot make a substantiated statement about that.
Is AWS/$MAJOR_CLOUD_PROVIDER vulnerable?
Since we did not perform any in the wild testing, we don’t know this yet. However, we have been contacted by cloud providers in order to discuss the described attack.
Given AWS does not run VMware anyways they will most probably not be vulnerable.
Is it necessary to start the virtual machine in a special way/using a special/uncommon API?
Which VMware products are affected?
At the moment, we can only confirm the vulnerability for the ESXi5 and ESXi4 hypervisors. Still, our research is going on ;-)
Matthias and Pascal
VMDK Has Left the Building — Some Nasty Attacks Against VMware vSphere 5 Based Cloud Infrastructures
20 Comments | Posted by Matthias Luft
Update #1: Slides are available for download here.
In the course of our ongoing cloud security research, we’re continuously thinking about potential attack vectors against public cloud infrastructures. Approaching this enumeration from an external customer’s (speak: attacker’s ) perspective, there are the following possibilities to communicate with and thus send malicious input to typical cloud infrastructures:
- Management interfaces
- Guest/hypervisor interaction
- Network communication
- File uploads
As there are already several successful exploits against management interfaces (e.g. here and here) and guest/hypervisor interaction (see for example this one; yes, this is the funny one with that ridiculous recommendation “Do not allow untrusted users access to your virtual machines.” ;-)), we’re focusing on the upload of files to cloud infrastructures in this post. According to our experience with major Infrastructure-as-a-Service (IaaS) cloud providers, the most relevant file upload possibility is the deployment of already existing virtual machines to the provided cloud infrastructure. However, since a quick additional research shows that most of those allow the upload of VMware-based virtual machines and, to the best of our knowledge, the VMware virtualization file format was not analyzed as for potential vulnerabilities yet, we want to provide an analysis of the relevant file types and present resulting attack vectors.
As there are a lot of VMware related file types, a typical virtual machine upload functionality comprises at least two file types:
The VMX file is the configuration file for the characteristics of the virtual machine, such as included devices, names, or network interfaces. VMDK files specify the hard disk of a virtual machine and mainly contain two types of files: The descriptor file, which describes the specific setup of the actual disk file, and several disk files containing the actual file system for the virtual machine. The following listing shows a sample VMDK descriptor file:
# Disk DescriptorFile
# Extent description
RW 33554432 VMFS "machine-flat01.vmdk"
RW 33554432 VMFS "machine-flat02.vmdk"
For this post, it is of particular importance that the inclusion of the actual disk file containing the raw device data allows the inclusion of multiple files or devices (in the listing, the so-called “Extent description”). The deployment of these files into a (public) cloud/virtualized environment can be broken down into several steps:
- Upload to the cloud environment: e.g. by using FTP, web interfaces, $WEB_SERVICE_API (such as the Amazon SOAP API, which admittedly does not allow the upload of virtual machines at the moment).
- Move to the data store: The uploaded virtual machine must be moved to the data store, which is typically some kind of back end storage system/SAN where shares can be attached to hypervisors and guests.
- Deployment on the hypervisor (“starting the virtual machine”): This can include an additional step of “cloning” the virtual machine from the back end storage system to local hypervisor hard drives.
To analyze this process more thoroughly, we built a small lab based on VMware vSphere 5 including
- an ESXi5 hypervisor,
- NFS-based storage, and
everything fully patched as of 2012/05/24. The deployment process we used was based on common practices we know from different customer projects: The virtual machine was copied to the storage, which is accessible from the hypervisor, and was deployed on the ESXi5 using the vmware-cmd utility utilizing the VMware API. Thinking about actual attacks in this environment, two main approaches come to mind:
- Fuzzing attacks: Given ERNW’s long tradition in the area of fuzzing, this seems to be a viable option. Still this is not in scope of this post, but we’ll lay out some things tomorrow in our workshop at #HITB2012AMS.
- File Inclusion Attacks.
Focusing on the latter, the descriptor file (see above) contains several fields which are worth a closer look. Even though the specification of the VMDK descriptor file will not be discussed here in detail, the most important field for this post is the so-called Extent Description. The extent descriptions basically contain paths to the actual raw disk files containing the file system of the virtual machine and were included in the listing above.
The most obvious idea is to change the path to the actual disk file to another path, somewhere in the ESX file system, like the good ol’ /etc/passwd:
# Disk DescriptorFile
# Extent description
RW 33554432 VMFS "machine-flat01.vmdk"
RW 0 VMFSRAW "/etc/passwd"
Unfortunately, this does not seem to work and results in an error message as the next screenshot shows:
As we are highly convinced that a healthy dose of perseverance (not to say stubbornness ) is part of any hackers/pentesters attributes, we gave it several other tries. As the file to be included was a raw disk file, we focused on files in binary formats. After some enumeration, we were actually able to include gzip-compressed log files. Since we are now able to access files included in the VMDK files inside the guest virtual machine, this must be clearly stated: We have/can get access to the log files of the ESX hypervisor by deploying a guest virtual machine – a very nice first step! Including further compressed log files, we also included the /bootbank/state.tgz file. This file contains a complete backup of the /etc directory of the hypervisor, including e.g. /etc/shadow – once again, this inclusion was possible from a GUEST machine! As the following screenshot visualizes, the necessary steps to include files from the ESXi5 host include the creation of a loopback device which points to the actual file location (since it is part of the overall VMDK file) and extracting the contents of this loopback device:
The screenshot also shows how it is possible to access information which is clearly belonging to the ESXi5 host from within the guest system. Even though this allows a whole bunch of possible attacks, coming back to the original inclusion of raw disk files, the physical hard drives of the hypervisor qualify as a very interesting target. A look at the device files of the hypervisor (see next screenshot) reveals that the device names are generated in a not-easily-guessable-way:
Using this knowledge we gathered from the hypervisor (this is heavily noted at this point, we’re relying on knowledge that we gathered from our administrative hypervisor access), it was also possible to include the physical hard drives of the hypervisor. Even though we needed additional knowledge for this inclusion, the sheer fact that it is possible for a GUEST virtual machine to access the physical hard drives of the hypervisor is a pretty big deal! As you still might have our stubbornness in mind, it is obvious that we needed to make this inclusion work without knowledge about the hypervisor. Thus let’s provide you with a way to access to any data in a vSphere based cloud environment without further knowledge:
- Ensure that the following requirements are met:
- ESXi5 hypervisor in use (we’re still researching how to port these vulnerabilities to ESX4)
- Deployment of externally provided (in our case, speak: malicious ) VMDK files is possible
- The cloud provider performs the deployment using the VMware API (e.g. in combination with external storage, which is, as laid out above, a common practice) without further sanitization/input validation/VMDK rewriting.
It must be noted that the hypervisor hard drives contain the so-called VMFS, which cannot be easily mounted within e.g. a Linux guest machines, but it can be parsed for data, accessed using VMware specific tools, or exported to be mounted on another hypervisor under our own administrative control.
Summarizing the most relevant and devastating message in short:
VMware vSphere 5 based IaaS cloud environments potentially contain possibilities to access other customers’ data…
We’ll conduct some “testing in the field” in the upcoming weeks and get back to you with the results in a whitepaper to be found on this blog. In any case this type of attacks might provide yet-another path for accessing other tenants’ data in multi-tenant environments, even though more research work is needed here. If you have the opportunity you might join our workshop at #HITB2012AMS.
Have a great day,
Pascal, Enno, Matthias, Daniel
During our ongoing research on the security of cloud service providers and cloud based applications, we performed a regular audit of our AWS account password. Thinking of popular incidents and evergreens in attack vectors, we were wondering which consequences an online bruteforce attack on our AWS password would have. So we decided to perform a bruteforce attack against our own account. Analyzing the login process of AWS, the following requirements for the bruteforce tool to be used could be derived:
- Cookie Handling
- HTTPS support
- HTTP 3xx support.
It turned out that it was pretty hard to find a password testing tool which fulfilled these requirements and would be able to actually handle the complex AWS login process — eventually there was none. Since we use and like Burp Suite pretty much, the Intruder suggested itself as an alternative which is straight forward to configure even though it might lack the speed and efficiency of special bruteforcing tools. Using burp’s history, we were able to identify the request which triggered the login process:
After the request is sent to the Intruder, the password field is marked
and the payloads to be used are configured.
Using exemplary payloads, it is possible to identify a successful login attempt, since it results in a redirect to the authenticated area/SSO server/whatever whereas a wrong passwords results in HTTP 200 presenting the AWS login page again:
Having this basic bruteforcing process established, the wordlist to be used must be generated. To decide which complexity should be covered, the Amazon password policy must be analyzed — if the restrictions in place deserve to be called a policy. The only restriction is that the password is between 6 and 20 characters (though the upper limit was determined regarding the maxlength field parameter when changing the password using the webfrontend, since there is no documentation about this available. Thinking of business needs, this behavior might be understandable since Amazon loses “endusers” and therefore money if their password policy is too strict). So we decided to use a wordlist which contains all passwords of 6 characters consisting of numbers (which can be generated pretty easy reactivating some old perl scripting skills: perl -le ‘printf “%06d”, $_ foreach(1..999999)’ ). Such passwords even might be pretty common when thinking of “birthday passwords”.
After performing about 400k requests, we paused the attack and searched for requests which resulted in a HTTP 302 response, just as the baseline request did.
And indeed, it was possible to bruteforce the password — which is not such a big surprise though. The bigger — and worse — surprise is, that it was still possible to login to our amazon account after performing about 2 million requests (including some dry runs) within two days originating from one single IP adress without having the account locked, being throttled down or notified in any way. And we were performing about 80k requests per hour.
Coming back to the title of the blogpost: At the moment of our investigation, there were no protection mechanisms against bruteforce attacks for the key to your datacenter — which your AWS credentials actually can be, if you are hosting a large amount of your services in EC2. Following a repsonsible disclosure policy, we contacted the AWS Security Team and got a very comprehensive answer. As we supposed, they pointed us to their MFA solution, which is basically, even though there was a major incident recently, a viable security control when authenticating users for data center access. But in addition, we had a long and beneficial dialog about potential mechanisms such as connection throttling and account locking. The outcome of our discussion is a CAPTCHA mechanism which kicks in after a brute force attempt is detected — and was also re-tested several times by our bruteforcing attempt. It was quite impressive to see that it was possible for Amazon to implement additional security measures in such a short time frame, regarding the huge size and complexity of the AWS environment. So we were really glad to get in touch with the committed AWS Security Team and were really happy to see that those guys are really into security and trying to communicate with their customers.