Ax Sharma, Senior Security Researcher at Sonatype, has provided more details about the two critical vulnerabilities that SaltStack patched last Thursday. Despite issuing the patches and a warning from F-Secure to apply them, several organisations, including the blogging platform Ghost, found themselves losing control of their Salt implementations over the weekend.
Sonatype has been through all the part of the code in Salt and discovered issues with how code was checked before it went into prodiction. that basic housekeeping was the core issue. In a short press statement Sharma said: “Having traced back through a series of commits, it is clear the vulnerabilities arose from either incomplete security enhancements made to Salt, or skeleton code with comments lurking in plain sight (CVE-2020-11651). The comments in the source code explained the selective “method whitelisting” functionality which was to be implemented as a security measure, but had no concrete code doing such a task.
“These findings underscore the importance of doing due diligence as a developer and tester when making security enhancements to open-source projects, and when leaving comments in source code which can be seen by the public. Care must be taken to ensure comments don’t reveal too much about what’s about to come in terms of security, otherwise adversaries are given a head start. And this wasn’t the only misstep Salt made – despite the developers adding checks for path traversal in earlier versions (CVE-2020-11652), the enhancements fell short of being actually effective, fuelling the chaos.”
Sonatype warning that manual patching is no longer good enough
When SaltStack pushed out its patches to the two CVEs last Thursday, users had two choices. If they had the auto-update feature enabled, Salt updated itself. If not, they had to download the patches and then apply them manually. It is not yet clear how many were set to manual update.
The explosion of APIs, micro-services and third-party code used by organisations is creating a security nightmare. Many organisations have little to no process for testing Open Source software (OSS) before they implement it. They also fail to accurately monitor how much OSS they are using across their environment. This makes it incredibly hard to patch if you are using a manual process.
According to Sharma: “As the series of Salt exploits show, when one vulnerable OSS package is used in thousands of places, thousands of companies are vulnerable to the same attack. Operations teams are now in a race against time between automated adversarial attacks, and must urgently update their applications. If your automation is faster than evil, you’re safe. If you continue to rely on slower, manual update and deployment methods, you are at risk in this period of active exploits.”
Enterprise Times: What does this mean?
A public announcement of critical vulnerabilities, patches, customers exploited and root cause in less than a week. For SaltStack, this has been six days to remember, and the fallout is far from over.
Sharma said: “SaltStack may have unintentionally tipped their hat to adversarial attacks for the two critical vulnerabilities that it patched last Thursday.” He believes that details around the method whitelist should have been in release notes, not comments in the code. He also pointed out on a telephone call that: “All software has flaws, even applications built by the best devs”
Sharma is being pragmatic. Yes, vulnerabilities will be discovered over time. However, proper release processes and code reviews should have identified both the CVEs that SaltStack has now remediated.
There are also several lessons here for other OSS projects. Better housekeeping, more testing and keep your comments out of the code are just three of them. Users of OSS software also need to review their patching and update processes.
This won’t be the last time software gets exploited, or customers get hit because they didn’t patch quickly enough.
This is a pathetic attempt by @sonatype to monetize. The whole piece is full of inaccuracies and errors one would not even know where to begin with the corrections. The problem has nothing to do with the code or the lovely scans @sonatype wants to sell us. If you are stupid enough to have exposed your ops tools on the open internet, then you will suffer. This can be any ops tool. The lesson here is secure your ops tools, use firewalls and all. We have contributed to Salt project for years and used it for over 7, never had a CVE like that. Code scans would not find this and is proof that this whole piece is a shambles. Shame on all involved here, just shambles
Ax Sharma’s assessment of what lead to the vulnerabilities is completely in-accurate. He linked to examples that make no sense to any competent software engineer. The patches released when the vulnerabilities were disclosed did fix the issues. This information is just totally wrong and I don’t think it’s responsible to try and sensationalize things with bad information.
Daniel
We had one link from Ax which is the one that goes back to github. As he mentioned the two CVE’s I added those links so that anyone could see what the CVEs were about. The links outside of the quotes from Ax we added to link to the original warning from F-Secure last Thursday, our brief coverage of the Ghost outage and to Sonatype’s site. The comment from Brian Fox which we have published in full, contains more links that should be of more use.
Ian
We at Sonatype find the title of the article unfortunate as it colors the commentary by our security researcher in an inappropriate and unintended light. We are part of the OSS community, where everyone lives in glass houses. We go out of our way to not cast stones.
However, without examining the past, you are doomed to repeat it. This is the fundamental principle that underlies the entire responsible disclosure ecosystem. As such, Sonatype regularly provides analysis and commentary on vulnerable code and known exploits in the hope that it can reduce recurrences and expand knowledge of secure coding practices.
The best course of action for all readers here is to update any vulnerable Salt code in their environments immediately. The patches for Salt versions 3000 to 2015 have all been posted. Information can be found at https://help.saltstack.com/hc/en-us/articles/360043056331-New-SaltStack-Release-Critical-Vulnerability.
@Retro Ralph: There’s nothing in here attempting to monetize this conversation. Our intent is to highlight some areas for other oss developers of where things can be better. No one claimed a scan was going to detect this problem (static analysis of code is not even our business).
Stepping over your attempt to shoot the messenger, I acknowledge that you are correct that users should take more steps to harden their installations. That’s true in almost all vulnerability exploits. It does not change that the underlying facts in the vulnerable versions of code are correctly highlighted by Ax.
@Daniel: Ax was commenting on the state of the code resulting from pre-existing comments and previous path traversal fixes. The commit links in the original article seem to have been lost in translation. I’ll fix them here so readers can draw their own conclusions as to accuracy.
On line 1204 and again at line 1942 of master.py prior to the fix (https://github.com/saltstack/salt/commit/ffea7ffa215313f68b42f82984b0441e1017330c#diff-81ea017b5c8f831cea8ab4e035970b92L1204 and https://github.com/saltstack/salt/commit/ffea7ffa215313f68b42f82984b0441e1017330c#diff-81ea017b5c8f831cea8ab4e035970b92L1942) there are comments that seem to indicate a “TODO like” statement that methods allowed should be restricted, yet the code itself to whitelist doesn’t appear anywhere that we can find, nor is there the list of whitelisted methods. The net result seems to indicate an effective “ALLOW ALL” by default when a “DENY ALL” was probably intended as cited in the Packet Storm Security POC (https://packetstormsecurity.com/files/157560/Saltstack-3000.1-Remote-Code-Execution.html).
Takeaway #1: Even when developers know proper practices like whitelisting (as the original comments seem to imply), mistakes can happen and bugs and vulnerabilities result. Given that the comments are public, it could have drawn attention to the missing filtering, but one can’t be certain if or how the vulnerability was discovered by bad actors prior to disclosure.
There was a similar interpretation of the path traversal vulnerability – in that the code already had path filtering but it wasn’t complete. (from 2012: https://github.com/saltstack/salt/commit/854ae40097b07fdbc4ad4fa53f7efa1dea7ef9fe#diff-25a0730fb51c7750e5f1bd8a7b32bf51R269 and 2017 in the same file as the resent vuln: https://github.com/saltstack/salt/commit/dc649ded51#diff-25a0730fb51c7750e5f1bd8a7b32bf51R502)
Takeaway #2: It’s possible that developers came in after who weren’t aware of the path filtering code and inadvertently reopened the previously fixed vulnerabilities in new ways. This certainly wouldn’t be the first time such a thing happened in an oss project. Developers should also consider how future commits might unintentionally introduce changes that undo filters and look for ways to architecturally enforce sandboxing. This isn’t always possible of course, but it’s a worthy goal.
Brian
As you are aware, I spoke to Ax after the article was first published. We made two changes to address concerns he raised. It doesn’t happen often that we are asked to address an issue in a piece but we do respond when those requests are made to see what they are about.
The issue over the headline is that it is not misleading. As I pointed out to Ax and in a message to you on LinkedIn, when you use statements such as “Having traced back through a series of commits, it is clear the vulnerabilities arose from…” you are saying that the issue is obvious. Whether you call that pointing fingers or throwing stones in a glasshouse is up to you. The point is that we published Ax’ statements in full.
The email we received was also not an excited utterance. As I understand, it was an agreed press release that had gone through the approval process at Sonatype and its agency Babel PR, a company I know well and who are good at what they do. That is why we published the comments in full. What we didn’t have were all the links. They were not in the release so we could not have added them.
In an email exchange with Brian Fox, he pointed out the statement “Sonatype has been through all the code in Salt”. that was a misread by me of what Ax Sharma said as can be seen from the quote that follows the statement. That has now been changed.
SaltStack makes very powerful infrastructure automation for IT and security operations teams. With this power comes a great responsibility landing squarely on the shoulders of SaltStack, our customers, and users of Salt open source software to ensure a secure SaltStack implementation.
Patches for these SaltStack vulnerabilities are available for current, supported versions of Salt, as well as older, unsupported versions of Salt. We implore all SaltStack customers and Salt open source software users to update their SaltStack environments immediately. Apply the patches now, and harden your Salt infrastructure following SaltStack best practices as appropriate. For the latest, authoritative information about these CVEs please visit https://community.saltstack.com.
The Salt community has always been open and accommodating. We welcome open dialog and helpful contributions to the project. This is the open source way, and it is the Salty way. We welcome further engagement on this topic, or any other topic relevant to infrastructure automation and configuration management, in the SaltStack GitHub repo or in the SaltStack community workspace on Slack.
–Alex Peay, SaltStack