Ax Sharma, Senior Security Researcher at Sonatype, has provided more details about the two critical vulnerabilities that SaltStack patched last Thursday. Despite issuing the patches and a warning from F-Secure to apply them, several organisations, including the blogging platform Ghost, found themselves losing control of their Salt implementations over the weekend.
Sonatype has been through
all the part of the code in Salt and discovered issues with how code was checked before it went into prodiction. that basic housekeeping was the core issue. In a short press statement Sharma said: “Having traced back through a series of commits, it is clear the vulnerabilities arose from either incomplete security enhancements made to Salt, or skeleton code with comments lurking in plain sight (CVE-2020-11651). The comments in the source code explained the selective “method whitelisting” functionality which was to be implemented as a security measure, but had no concrete code doing such a task.
“These findings underscore the importance of doing due diligence as a developer and tester when making security enhancements to open-source projects, and when leaving comments in source code which can be seen by the public. Care must be taken to ensure comments don’t reveal too much about what’s about to come in terms of security, otherwise adversaries are given a head start. And this wasn’t the only misstep Salt made – despite the developers adding checks for path traversal in earlier versions (CVE-2020-11652), the enhancements fell short of being actually effective, fuelling the chaos.”
Sonatype warning that manual patching is no longer good enough
When SaltStack pushed out its patches to the two CVEs last Thursday, users had two choices. If they had the auto-update feature enabled, Salt updated itself. If not, they had to download the patches and then apply them manually. It is not yet clear how many were set to manual update.
The explosion of APIs, micro-services and third-party code used by organisations is creating a security nightmare. Many organisations have little to no process for testing Open Source software (OSS) before they implement it. They also fail to accurately monitor how much OSS they are using across their environment. This makes it incredibly hard to patch if you are using a manual process.
According to Sharma: “As the series of Salt exploits show, when one vulnerable OSS package is used in thousands of places, thousands of companies are vulnerable to the same attack. Operations teams are now in a race against time between automated adversarial attacks, and must urgently update their applications. If your automation is faster than evil, you’re safe. If you continue to rely on slower, manual update and deployment methods, you are at risk in this period of active exploits.”
Enterprise Times: What does this mean?
A public announcement of critical vulnerabilities, patches, customers exploited and root cause in less than a week. For SaltStack, this has been six days to remember, and the fallout is far from over.
Sharma said: “SaltStack may have unintentionally tipped their hat to adversarial attacks for the two critical vulnerabilities that it patched last Thursday.” He believes that details around the method whitelist should have been in release notes, not comments in the code. He also pointed out on a telephone call that: “All software has flaws, even applications built by the best devs”
Sharma is being pragmatic. Yes, vulnerabilities will be discovered over time. However, proper release processes and code reviews should have identified both the CVEs that SaltStack has now remediated.
There are also several lessons here for other OSS projects. Better housekeeping, more testing and keep your comments out of the code are just three of them. Users of OSS software also need to review their patching and update processes.
This won’t be the last time software gets exploited, or customers get hit because they didn’t patch quickly enough.