SoftLayer beats AWS, Google and Azure

VoltDB is an in-memory database vendor who claim to be: “..the only in-memory, NewSQL solution that delivers Smart Data Fast. VoltDB offers the performance of in-memory, the scalability of NoSQL, full streaming capability, and the transactional consistency of traditional relational databases.”

They have just released a blog authored by John Hugg and the results of their latest YCSB benchmark comparing IBM SoftLayer with Google Cloud, AWS and Microsoft Azure. The testing was carried out by a third party ACME Benchmarks who also published their story around the tests here.

The results will raise a lot of eyebrows. According to the results, anyone wanting to run in-memory databases at scale and at speed should be moving away from traditional cloud models and buying into bare-metal servers on IBM SoftLayer.

According to Hugg: “It can’t be overstated that this is one benchmark, and your mileage may vary. As stated above, this YCSB workload stresses some parts of the cloud platforms more than others. This benchmark also doesn’t consider the ecosystems these platforms come with; it focuses only on the compute components within the platforms.”

How did the various options perform?

What is especially interesting about this test is that neither VoltDB nor ACME Benchmarking appear to have any axe to grind with any of the vendors. The focus was on how to run an in-memory database the fastest and at the lowest cost.

In the first set of results looking at the number of Ops/Second the results were:

• ~1.55 million ops/sec – IBM SoftLayer bare metal servers
• ~1.03 million ops/sec – Google Cloud
• ~902,000 ops/sec – Microsoft Azure
• ~585,000 ops/sec – Amazon Web Services
• ~432,000 ops/sec – IBM SoftLayer virtual machine

The second set of tests looking at the number of billion ops/$ were:

• ~4.6 billion ops/$ – IBM SoftLayer bare metal servers
• ~3.4 billion ops/$ – Google Cloud
• ~1.6 billion ops/$ – IBM SoftLayer virtual machines
• ~1.6 billion ops/$ – Amazon Web Services
• ~0.42 billion ops/$ – Microsoft Azure

In-memory databases are seen as the best way to deal with very large volumes of data that need to be analysed quickly. All of the vendors above have their own in-memory database solutions and all support key industry in-memory solutions from the Open Source community such as Hadoop. As such it is not unreasonable to assume that they have an underlying infrastructure that is capable of delivering in-memory solutions quickly.

VoltDB is designed to run in-memory and is agnostic to the underlying architecture. That means that it does not have specific drivers for Google, AWS, Azure or SoftLayer. As part of the test process it is deployed on a set of resources, takes a collection of data and carries out a range of different tests.

According to Bruce Reading, chief executive officer and president of, VoltDB: “These results showcase the tremendous performance, price and scalability benefits of today’s cloud offerings for developing and deploying fast data applications.

“As organizations look to meet their fast data application demands in the cloud, VoltDB is the only modern in-memory OLTP platform that offers the ability to use streaming analytics in user and operational interactions, enabling new levels of application functionality and performance.”

What can we learn from these tests?

The most obvious lesson from these tests is that a bare metal server is far quicker for this type of testing that using a cloud based environment. Is that really as surprising as it seems. Probably not. After all, when running on bare metal servers there are no hypervisors or other cloud components to get in the way.

What is surprising is that having achieved that result there was no consideration to then running the YCSB benchmark across a range of bare metal cloud options. This would, at least, have shown if IBM SoftLayer are doing something special or if bare metal is indeed the solution to in-memory computing.

What is notable is that all these vendors have their own cloud infrastructures. IBM is OpenStack, Google, Amazon and Microsoft have their own cloud environments and it appears that the tests were run fairly on the primary cloud option providing by the vendors. As such it should be the most well tuned for their cloud and underlying hardware.

Perhaps a future test should include a number of OpenStack providers to ensure that the underlying cloud architecture is eliminated as far as is possible. This would then come down to a pure implementation variance between the cloud providers.

YCSB is not just about memory chips and processors

One of the interesting parts of this test is that YCSB is not just compute intensive it is also network intensive. This makes the test very interesting to customers who are looking to important data from a range of sources. The primary cloud providers are all beginning to push Data as a Service where they will make it possible for customer to acquire and important multiple data sets.

In order to do this they will need to bring together data across the cloud providers network. By using a benchmark that includes a network intensive element, VoltDB has managed to provide a test that can be seen as more representative of real-world database usage than many other tests out there.

YCSB is also focused more on read than write (95% v 5%). This eliminates issues with disks and secondary storage.

The nitty gritty

The requirements of the test in terms of resources were:

• 60 cores
• 60 GB RAM (minimum)
• 10 Gbps network support.

According to the testing document from ACME Benchmarking it ended up with much smaller numbers of cores ranging from AWS with 36 cores down to SoftLayer bare metal at 24 cores and SoftLayer virtual at just 16 cores.

RAM also differed widely with Microsoft Azure using 448GB while the two SoftLayer instances used just 64GB.

Differences were also evident in the network support. Azure had 40Gbps, Google Cloud was unknown, AWS and SoftLayer bare metal were 10Gbps while SoftLayer virtual was a miserly 1Gbps.

The reason for such a wide range of values according to Tim Callaghan from ACME Benchmarking was:

“I quickly learned that comparing cloud providers is no easy task when it comes to provisioning the same “size” of server. Each provider has its  own parameters for server resources. Some, like Google, keep it extremely simple. Others, like Amazon and IBM, have so many options that you’ll suffer from analysis paralysis when provisioning your first servers.”

Conclusion

These results will cause a lot of discussion, not least because they mix bare metal and cloud platforms. There will be cries of Apple’s vs Oranges and claims that the results are not fair. However, this was a test created by a company to test its software rather than test cloud platforms. They also chose to use a third-party run the tests to remove any accusation of bias on their behalf. For the absolute cynic, the tests are available online.

Putting all of that aside, the fact that bare metal has performed so well not just in terms of Billion Ops / $ as well as Ops / Second is a significant shock. The claims over the last few years from cloud providers is that they can outperform on-premises equipment at a far lower cost. In this instance it can be shown that neither claim is substantiated as bare metal is equivalent to a server being run by an organisations own IT team.

It will be interesting to see if we will see Google, AWS and Microsoft come back with their own tests to disprove these results. If they do then it will take some time to compare the different tests to ensure that all the results are comparable.

Of even more interest will be the results of these tests when they are rerun in a years time. At present, SoftLayer is not offering IBM POWER8-based servers inside its data centres. Sometime in the next month we will see those being deployed although they will not be supplied by IBM but by members of the OpenPOWER Foundation. At that point we will get our first widespread look at x86 vs POWER8 across bare metal and cloud infrastructures.