Cyber security vendor Eclypsium, who specialises in protecting firmware, has disclosed a severe vulnerability in how bare metal servers are managed. Put simply, it has discovered that the processes used by various cloud providers allow an attacker to alter the firmware code in servers and create a backdoor. The attacker can then release that server back to the cloud provider leaving the next customer with a compromised server.
This is not a theoretical attack. Eclypsium tested the attack against IBM SoftLayer, detailing what they did. They also make it clear that this is attack is not exclusive to IBM SoftLayer. Under the Common Vulnerability Scoring System v3, Eclypsium has rated this as 9.3 (critical) attack. Its rating data is CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H.
This attack is named Cloudborne.
What did Eclypsium do?
As with all claims like this, the question is how realistic is it? The details provided by Eclypsium show that it was a simple attack to craft and was persistent. The latter is important. When bare metal servers are released by customers, cloud providers routinely reset the server and put it back into the available pool.
At the heart of this attack are servers provided to IBM by Supermicro. Eclypsium has previously exposed problems with Supermicro servers. IBM SoftLayer has a long standing relationship with Supermicro, buying its x86 servers for a number of years.
The research team:
- Provisioned a bare metal server from IBM SoftLayer
- Verified the version of the baseboard management controller (BMC) and checked it was running the latest software release from Supermicro
- Recorded both the chassis and the product serial numbers
- Made a simple, benign change to the firmware by flipping a single bit
- Updated the BMC firmware using the AIUpdate tool
- Created an additional IPMI user with administrative access to the BMC channels
At the end of this, the server was released back to IBM to be reclaimed. Once IBM had run its reclaiming routine it made the server available for customers to use.
Eclypsium provisioned a number of servers after the reclamation process. Eventually it got back the server that they had altered. Although the additional user had been removed, the change to the BMC firmware were still in place. It also noted that the BMC root password had not changed and the BMC logs were still available.
The Eclypsium blog states: “By not deleting the logs, a new customer could gain insight into the actions and behaviors of the previous owner of the device. Meanwhile, knowledge of the BMC root password would enable an attacker to more easily gain control over the machine in the future.”
In its blog, Eclypsium has also published a timeline of events. This includes sending an advisory to IBM as email@example.com on September 6, 2018. The advisory was acknowledge. Apart from asking Eclypsium to fill out another form, IBM provided no further response on the incident.
IBM responds to press request
Enterprise Times contacted IBM on Feb 25 to ask for a comment on this story. IBM sent us a statement that said: “We are not aware of any client or IBM data being put at risk because of this reported potential vulnerability and we have taken actions to eliminate the vulnerability..
“Given the remediation steps we have taken and the level of difficulty required to exploit this vulnerability, we believe the potential impact to clients is low. While the report focuses on IBM, this was actually a potential industry-wide vulnerability for all cloud service providers and we thank Eclypsium for bringing it to the attention of the industry.”
A short while later, IBM sent Enterprise Times a link to a blog from its PSIRT team. In that blog, IBM rates the risk as Low Severity and claims it had remediated the situation. The publication of the blog triggered another email from Eclypsium.
It stated: “IBM has responded to this vulnerability by forcing all BMCs, including those that are already reporting up-to-date firmware, to be reflashed with factory firmware before they are re-provisioned to other customers. All logs in the BMC firmware are erased and all passwords to the BMC firmware are regenerated.”
It went on to say: “However, after this posting, an Eclypsium researcher was able to quickly confirm that he received the same system back that he worked on before (at 16th of Feb) and there was no indication that password or firmware had been changed from the last time he used it. The researcher is conducting more testing.”
Enterprise Times: What does this mean
This appears to be a two-fold issue. The first is that Supermicro servers are still exploitable even using the latest version of the BMC software. The second is that despite IBM claiming to reset the BMC firmware on all servers, something is not working well.
When it comes to customers looking to take on bare metal servers, irrespective of if they are from IBM or another cloud provider, this will raise concerns. IBM believes that this is a very hard vulnerability for hackers to exploit. Eclypsium, understandably, disagrees. Customers won’t care who is right, they want a definitive fix to the problem.
The common denominator here is Supermicro who has not responded to our request for comment. Have spent a significant period of time telling customers it doesn’t matter whose hardware is in the cloud, this incident risks undoing that. It will be interesting to see if customers now ask about the hardware and go back to a few years ago where they were trying to specify whose servers they ran on in the cloud.
There is another issue that is important. The start of this research was the discovery of a vulnerability in Supermicro servers. Eclypsium then followed that trail to the cloud. What it hasn’t told Enterprise Times is whether it has tried the same attack on other servers. If so, were they also vulnerable? This is a critical piece of information. This could be a very wide problem or one confined to a particular vendor. At the moment we just don’t know.
If we receive a response from Supermicro, we will publish an update. We will also update if IBM comes back with an updated blog on the issue.