Distil Networks has released its 2016 Bad Bot Landscape Report and it makes for somewhat depressing reading. Subtitled The Rise of Advanced Persistent Bots the report makes the point that bots are cheap to deploy, are leveraging cloud providers and are becoming increasingly sophisticated. All of this increases the pressure on IT infrastructure teams as they struggle to keep the bad guys out.

The report runs to 16 pages and can be downloaded here (registration required) from the Distil Networks website. The biggest concern here is the rise of what Distil Networks call the Advanced Persistent Bot. These are not just bots capable of carrying out more advanced attacks but as with the latest generation of malware they are capable of evading detection and re-emerging even when they appear to have been taken down.

According to a statement in the press release from Rami Essaid, co-founder and CEO of Distil Networks: “When we dug into the bot activity in 2015, we identified an influx of Advanced Persistent Bots (APBs). ABPs can mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents.

“The persistency aspect is that they evade detection with tactics like dynamic IP rotation from huge pools of IP addresses, use Tor networks and peer to peer proxies to obfuscate their origins, and distribute attacks over hundreds of thousands of IP addresses. A whopping 88 percent of 2015 bad bot traffic were APBs. This shows that bot architects have already taken note of traditional bot detection techniques and are finding new sophisticated ways to invade websites and APIs, in an effort to take advantage of critical assets and impact a business’s bottom line.”

Distil Networks key findings are bleak

For many people, bots are a nuisance rather than a serious security risk. The reality is that the are not only used to probe networks but also to carry out brute force attacks and act as a distraction for other types of malicious infiltration. There are five key areas into which Distil Networks has grouped its main findings. We’ve pulled out some of the detail here:

Bot traffic

• “46 percent of all web traffic originates from bots, with over 18 percent from bad bots”: Given the congestion on many networks which leads to lower speeds and throughput reducing the traffic from bots would have an immediate impact on Internet performance.
• “Chrome edged out Firefox as the browser of choice for bad bot creators with over 26 percent of all user agents utilizing the Google browser”: This is partly due to the rise of Chrome as the browser of choice and partly due to the ease of coding in it. It is also worth noting that Safari has now appeared at No 4 in the list and this coincides with other security risk increases around the Apple platforms.

The rise of Advanced Persistent Bots (APBs)

• “88 percent of all bad bot traffic has one or more characteristics of an Advanced Persistent Bot”: These are features identified in the statement from Essaid above and is a significant increase from 2014 where 77 percent had these characteristics. Such a high number suggests that older bots are being replaced by new APBs and it will be interesting to see at what point Distil Networks or other security companies redefine what an APB is.
• “53 percent of bad bots are now able to load external resources like JavaScript meaning these bots will end up falsely attributed as humans in Google analytics and other tools“: This is a more complicated issue than it looks. The use of ad blocking and other traffic blockers inside browsers already means that many companies are no longer relying on Google analytics for accuracy. However the need for some degree of accurate analytics in order to present to advertisers so that websites can continue to afford content means that webmasters need better ways of eliminating bad bots from their traffic numbers.
• “39 percent of bad bots are able to mimic human behavior so tools such as WAFs, web log analysis, or Firewalls, which perform less detailed analysis of clients and their behavior, will likely result in huge amounts of false negatives”: This is beginning to be addressed by security analytics programs but big data alone is unable to remove the false positives. The problem here is that a continual stream of false alarms weakens security and trust making it easier for cybercriminals to attack systems.
• “The worst APBs change their identities over 100 times”: This makes it very hard for individual companies to track bots. What is needed is for better threat intelligence sharing that includes information on bots.
• “73 percent of bad bots rotate or distribute their attacks over multiple IP addresses and of those, a whopping 20 percent surpassed 100 IP addresses“: The danger of trying to block large numbers of addresses is that many of these bots utilise cloud and hosting services. This enables them to hide behind shared IP addresses which, if blocked, tend to impact a lot of other companies sharing the same IP address.

Amazon earns a hat-trick, China has the worst offending ISPs, while residential ISPs Comcast and Time Warner clean up their act

• “Amazon has appeared in the Top 5 Bad Bot Originators three years in a row”: This should not come as a surprise due to AWS being the largest public cloud provider in the world. It has long been reported that cybercriminals take advantage of stolen credit cards to buy access on Amazon and other cloud services. While they are taken down as soon as the card is discovered to be stolen, this still gives them access to resources for a period of time. Amazon is not the only cloud provider to have problems with SoftLayer and OVH SAS also making their way into the top 20.
• “Despite their repeated appearance in the top Bad Bot Originators list in 2013 and 2014, residential ISPs Comcast and Time Warner fell off the Top 20 bad bot originators for 2015”: This is good news for their customers as it means less congestion on the network. What is not clear here is whether this is due to a move away from infected end-user computers in favour of cloud services.
• “Six out of the top 20 ISPs with the highest percentage of bad bot traffic originated from China“: There is little surprise in this as China has been climbing the ranks of security risky locations for the past few years. It is not just about bad bot traffic but malicious code and other cyber attacks.

Digital publishing and real estate industry websites are bot targets

• “As an industry, digital publishers were hit hardest by bad bots, which make up over 31 percent of all their traffic”: Few of these companies have the resources or the knowledge to differentiate between bad bots, good bots and humans. As many of them rely on advertising for their revenue it leaves them open to charges of click fraud and will only make it harder for them to survive.
• “For small digital publishers (Alexa 50,001 – 150,000) 56 percent of traffic originates from bad bots while for medium-sized websites (10,001 to 50,000 Alexa ranking) are at a greater risk, as bad bot traffic made up 26 percent of all web traffic for this group”

Huge increase in bad bot traffic from China, but the United States still has biggest bot problem

• “Maldives, Israel and Kyrgyzstan had the highest bad bot GDP (number of bad bots per online user) at 526, 168, and 94 respectively”: As a major centre for cybersecurity research and development this is a major problem for Israel.
• “China, Norway, Germany, and the Netherlands are the most blocked countries for web traffic”
• “The United States boasts the largest originator of bots again, with over 39 percent of bot traffic, while India and Israel moved up to number two and three, respectively”

Conclusion

There are some significantly worrying numbers in this report. The fact that bad bots are transitioning to become APBs is no huge surprise. Bot herders who create and manage botnets have suffered some significant losses to law enforcement over the last few years. By updating their product they are simply doing what we see constantly in the cybercriminal community and delivering a better and more efficient product.

What this report doesn’t address is whether there is a premium for APBs at the moment and whether they are seeing price drops as they continue to grow. There is also no mention of APB-as-a-Service (APBaaS). It is inevitable that this will happen, especially as this is becoming the norm across other forms of cybercrime.

Companies need to take bad bots and particularly APBs into account when planning their security posture. If they fail to do so then they will find themselves unable to deal with the impact of an attack. Bots are no longer primarily about DDoS attacks as this report shows.