Code hosting platform GitHub has shaken off a record 1.35 terabit per second (Tbps) DDoS attack. The attack caused some disruption to its service but was quickly dealt with. GitHub called on DDoS mitigation service, Akamai Prolexic.
Akamai took over all inbound traffic to GitHub and routed it across its own systems. Those systems cleaned up the traffic by removing all the malicious packets. Once the attackers realised that they had lost the attack was dropped.
The scale of the attack and the successful defence will cheer IT security teams. This is the largest DDoS attack in two years and many feel it is long overdue. Last year IBM Security said it expected to see multi terabit attacks. While it didn’t happen then, this could be the start of a new wave of attacks.
The ability of GitHub to survive this attack shows that vendors such as Akamai have built solutions that can cope with scaled up attacks. The question now is how will the attackers respond? Will they look to create an even larger attack to test the limits of Akamai?
What caused the attack on GitHub?
GitHub’s engineering team has already published a blog from Sam Kottler. He has provided a first look at how the attack unfolded, what it consisted of and what GitHub did. There are several things we now know from Kottler:
- At 17:21 UTC the network team noticed an anomaly in the amount of inbound traffic
- At 17:26 UTC with the traffic at one location hitting 100Gbps, Akamai was called in
- At 17:30 UTC bandwidth monitoring and load balancers indicated a full recovery
- The first spike of 1.35Tbps also saw 126.9 million packets per second hit the GitHub network
- There was a second spike of 400Gbps at just after 18:00 UTC
- The attack was an amplification attack using memcached over UDP. This means that for every byte sent by the attacker, 51kb hit the GitHub network
- The attack originated from over 1,000 autonomous systems and involved tens of thousands of unique endpoints
- This was not an attack via a botnet
What risks do memcached attacks pose?
In his blog, Kottler points to the description by Cloudflare of an amplification attack using memcached.
ET also received comment from Tod Beardsley Research Director at Rapid7 about this attack on GitHub and the dangers memcached attacks pose:
“Today’s attack on GitHub using a memcached amplification attack is a harbinger of the new world of DDoS using this technique. Unless, and until these vulnerable memcached servers are themselves booted off the Internet, they will remain as an irresistibly attractive means for firing packet cannons at any target one might choose, all with no botnet infrastructure required.
Memcached is perfect for this sort of attack. By default, it produces many thousands of bytes of UDP response to a very short UDP request. This request can be easily spoofed and leveraged by attackers with low skill and few resources, and does not require any authentication. After all, the design purpose of memcached is to deliver popular content quickly and without much warning — but the design of memcached over UDP is patently inappropriate for today’s Internet.
For people worried about memcached-based amplification attacks — which should be everyone — firewall rules to block unsolicited traffic from UDP source port 11211 should be de rigueur for Internet networking today.
For people who rely on memcached for normal caching functions, and must expose it to their internet, using only the TCP interface will alleviate the spoofing problem, though at some performance cost. Many people were surprised that memcached ran over UDP in the first place, so most normal users simply need to disable the UDP functionality they weren’t using anyway.
For people who are absolutely insistent that their application must use memcached over UDP, I suspect they’re going to be out of luck very soon. There is every indication that reputable internet hosting companies the world over are going to be disabling this protocol at the ISP layer, since memcached over UDP is just too dangerous to run on the open Internet.”
What does this mean?
This is a significant boost for both Akamai and GitHub. It is unlikely that the attackers expected the attack to be mitigated so fast. Large-scale DDoS attacks are often used as a smokescreen. They rely on all resources being dedicated to restoring traffic and therefore missing the real attack.
In this case the DDoS attack was amplified using the memcached technique. As Kottler points out the amplification factor was 51,000:1. This is a significant jump. Given that GitHub and Akamai managed to defeat the attack, the question is how high can attackers go? Will we see a response from attackers with a 100,000:1 or greater amplification factor? Is there an upper limit to this type of attack?
One of the interesting things here is that it was not a botnet attack. Instead it came from tens of thousands of endpoints on over 1,000 systems. This is important. Many of those systems will have been known to GitHub with developers from them already connecting to store code. As such, there would have been a slight delay in detection and response. Had it come from botnets using unknown systems, it is likely that other alarms would have been sounded earlier.
There is a serious message here for enterprise customers and ISPs. Massively scaled DDoS attacks are back on the table. While this one failed, had it been launched at many businesses, NGOs or even government departments, the results could have been much more serious.