Virtual appliances and images are supposed to be an easy way to buy up-to-date software from a company. Instead, as shown by a new report from Orca Security, in many cases, they are a complete disaster. Orca looked closely at the security and patching levels of over 2,200 virtual appliances from 450 vendors. What it discovered was more than 400,000 vulnerabilities ranging from the mundane to those with CVSS scores of 9 or higher.
To be clear, this isn’t about virtual appliances that customers purchased ages ago and haven’t patched. These are the up-to-date, current, shipping virtual appliances from vendors as large as IBM, Oracle, Cognosys, A10 Networks, Qualys, Symantec, McAfee, Intel, Micro Focus, Tufin, SAP and many others. Some of these virtual appliances hadn’t been patched in over 36 months. Some even contained vulnerabilities such as EternalBlue, BlueKeep, DejaBlue and DirtyCow.
It’s also important to note that Orca gave vendors the normal 90-day notice of publication. That was extended to 120, yet a number chose to do nothing.
The details are all contained in the Orca Security 2020 State of Virtual Appliance Security report. It includes a link to all the virtual appliances that were tested, how they were scored and the type of risks they pose.
What does Orca say about the report findings?
Enterprise Times talked to Avi Shua, CEO and Co-Founder, Orca Security about the report and its findings. Shua began by saying: “We found within many of our customers, virtual appliances that they bought from other vendors with security deficiencies within them. Many of them haven’t been patched for years. Few of them are secure.
“We decided to do something about it. Our research team took almost every available, virtual appliance in any marketplaces. We literally downloaded more than 2200 virtual appliances and ran our scanner against those solutions. We detected security misconfigurations and vulnerabilities in the applications and the operating systems.”
How out of date were the virtual appliances?
Shua continued: “47% of the solutions haven’t been updated even once in the last 12 months. Everybody tells you to patch your machine, but if you haven’t updated in a year, then it naturally won’t be updated. The Qualys Vulnerability Management Platform, was vulnerable to a vulnerability that Qualys researchers found more than two years ago.
“We learned that transparency is important. When we allowed the vendors time, many of them took action. We’re not talking only about small vendors. I’m talking about Dell issuing critical security vulnerabilities. IBM decided to fix some and discontinue some of the virtual appliances that were simply not maintained.”
“It’s very important to emphasise we only checked the latest revisions. It’s not like we took the old version. We took the latest available revision of any solutions within the marketplaces.”
A lack of process or a lack of oversight?
“There are many vendors that simply don’t update. Only 3% updated their solution last month, 14% if you’re talking about the last three months. The fact that almost half don’t update than the yearly basis at least to me shows that they don’t have the necessary processes and maybe this virtual appliance is not perceived as something you need to maintain. At the end of the day, the customers are the ones that suffer.
“There are quite a few players who said, “Yes, it’s not bad, but we put in this release notes that the virtual appliance doesn’t come patched and you need to go to this and install this now.”
A deeper dive into the report
Enterprise Times had advanced access to the report, and there are some worrying statistics. Here are some of them:
- 401,571 vulnerabilities in 2218 virtual appliances
- < 8% of virtual appliances had ZERO known vulnerabilities
- < 5% ran on a maintained operating system
- 15% of virtual appliances were graded F (the bottom score). It included virtual appliances from large vendors such as Intel, CA Technologies, A10 Networks, Cloudflare, Micro Focus and Symantec.
- 268 images are still rated F
- Product management from some vendors was so variable that they had products that rated A+ and F.
- Orca scanned 293 images from a single vendor – Cognosys
- 183 images were removed by vendors
- 623 images were updated
- 228 images have not been updated for more than 36 months
How have vendors responded?
When informed about the problems, some vendors engaged with Orca, some patched, some removed products and some threw temper tantrums. The latter group includes vendors accepting the vulnerabilities were in the product but denied they could be exploited, threats of legal action and putting the responsibility for patching on the customer.
Enterprise Times contacted the PR teams for several of the vendors mentioned. Some took our calls, and others failed to respond to emails. Only one bothered to respond to our questions as to why this had happened.
ManageEngine is part of ZOHO. Three of its images were tested and, as a result, the worst-performing one was withdrawn.
Rajesh Ganesan, VP, ManageEngine commented: “We acknowledge that there has been a process gap at our end to promptly patch the virtual images of some of our older ManageEngine on-prem products available in cloud marketplaces like AWS and Azure. Ensuring the security of our products is of paramount importance to us. We are taking immediate steps to resolve the issue and keep all our solutions up-to-date by patching with the latest versions.”
It will be interesting to see how well ZOHO does in future tests by Orca.
Not just a problem for small vendors
It is easy to dismiss this type of report as being just a small vendor issue. Companies that have limited staff and who don’t have the time or resource to maintain virtual images properly. Among the large vendors and some of those with a large customer base who had virtual images tested :
| Vendor | No of products | Actions | Final rating |
| Actian | 4 | All removed | |
| Alteryx | 1 | A | |
| Barracuda Networks | 1 | Updated | A+ |
| BeyondTrust | 1 | Removed | |
| Bitnami | 115 | All updated | A to A+ |
| Canonical | 2 | Removed | |
| Firemon | 2 | D and F | |
| IBM | 3 | 1 removed
2 updated |
A and D |
| Intel | 2 | A and F | |
| Kaspersky Lab | 1 | Removed | |
| McAfee | 1 | Updated | C |
| Micro Focus | 2 | 1 removed | F |
| NVIDIA | 3 | 1 removed
1 updated |
A+ |
| Oracle | 1 | Removed | |
| Percona | 1 | Updated | A |
| Pivotal Software | 1 | Updated | D |
| Qualys | 1 | Updated | C |
| Redis Labs | 1 | Updated | F |
| SAP America | 2 | B and D | |
| Softaculous | 7 | C | |
| Splunk | 2 | 1 removed
1 updated |
A |
| Symantec | 4 | 2 removed
1 updated |
A and D |
| Xilinx | 3 | 1 removed | F |
| Zend | 8 | All updated | A and B |
| ZOHO | 18 | 6 removed
1 updated |
A, C, D, F |
| Zscaler | 1 | C |
A lack of communication
One of the most concerning aspects of this research is the lack of communication channels from vendors. Shua said: “When we tried to report issues it wasn’t easy. We had to hire an analyst just to find the email addresses and right contacts for more than 500 vendors. We managed to do that and told them we are going to publish in 90 days under responsible disclosure. Some asked for more time so we waited 120 days.”
This is 2020, not 1980. Reporting mechanisms should be the norm, not the exception. At RSA in February, there were several talks around the failure of vendors to take reports seriously. It seems that attitude persists.
Interestingly, when we asked ZOHO about this Ganesan replied: “We are aware Orca raised this issue previously and are reviewing our communications processes to ensure important information is received quickly by the right people in our team moving forwards so immediate action can be taken.”
Another nail in the coffin of trust with software vendors
On the whole, companies trust their software vendors to produce working products. There is a general acceptance that nothing is perfect, and in software, it never will be. However, when customers buy a product, they shouldn’t expect to find a vulnerability the vendor has known about for years.
A case in point is Qualys. The report states: “Qualys, itself a provider of a vulnerability scanning service, was distributing a 26-month-old appliance, with a critical vulnerability (CVE-2018-15473) Qualys itself had discovered and reported in 2018.” Discovering and knowing does not translate into fixing. [Note: Qualys were asked for a comment and didn’t respond]
Virtual appliances are widely used. Unlike fixed media such as CD, DVD and even diskettes (if you are old enough), they are not fixed in stone because of manufacturing. In the early 2000’s they were seen as the ideal way for vendors to distribute fully patched, up to date software. All that is needed is a process to manage the golden master and patch it when a vulnerability is fixed.
What happens after that, for example, once the customer downloads and starts using the product, is outside of the control of the vendor. The emergence of PaaS has provided a solution to this latter case as cloud vendors can now patch the platform directly.
Enterprise Times: What does this mean?
It is, quite frankly, a poor state of affairs. Vendors are quick to blame breaches on customers who don’t patch but then proceed to sell them software with known vulnerabilities. There is nothing here that a proper patching and product lifecycle process could not solve. That vendors have lost sight of their products, and the state of them is concerning.
Equally concerning is the number of products that are repackaged and sold on. It is common across the Linux distributions and, in the case of Cognosys, Windows 2008. There is software here with unsupported operating systems that can no longer be said to be secure. Vendors might assume that they are protected through other mechanisms, but that is unproven. There is an increasing number of attacks that traverse corporate networks looking for vulnerabilities. These virtual images are all open to those attacks.
What this does raise is the question of recourse. Under existing software agreements, customers have no recourse for being sold software the vendor knows is broken. Is it time for regulators to start tightening the screws and applying a fit-for-purpose test? The software lobby says that would stifle innovation. However, innovation cannot continue to be a rock to crawl under. At some point, vendors must take some responsibility for their product.
It is a good piece of work by Orca and is one that should be repeated regularly and perhaps on a wider scale.
