Gather ‘round, friends — I have an interesting story to tell you about how vulnerabilities persist in the wild and the risks posed by unpatched IoT devices!

On 28 June, we released a CrowdSec scenario to detect attempts to exploit a remote code execution bug in Netgear routers. Within the 24 hours that followed, CrowdSec Security Engines worldwide that are subscribed to our http-cve collection began detecting attacks. The signals from these attacks started pouring in on our side. Over the next two months, we’ve been averaging approximately 1000 attacks detected per day from around 9000 unique IPs. This quickly propelled the scenario to become the top scenario detected from the collection over this period, surpassing even famous exploits like log4shell. This would hardly be interesting on its own if it wasn’t for this very strange fact — the exploit the scenario detects is over 10 years old!

If you’re interested in understanding how exploitation attempts are rampant even ten years after being discovered, in this article, we report on this interesting anomaly, provide data on the exploit, show how the exploit works in practice, and speculate on why it is so persistent.

The Stone Age tool

Like many modern routers, the affected Netgear routers expose a web interface to see and change the various settings that users might want to change in their router. To change the settings, the router has to expose functionality that goes beyond serving static HTML. In the case of Netgear, they opted to use scripts based on the Common Gateway Interface (CGI) for this task.  

The CGI is a very primitive tool that traces back to the early days of the web. It works as follows. The web admin writes and compiles the script in whatever language they wish (C, Perl, Python, etc.) and puts it somewhere the webserver can access it, say /foo/bar/script.cgi. Now, if a visitor opens example.com/foo/bar/script.cgi, the server does not display the binary but, instead, runs the script and then displays everything the script pushes to the standard output as a webpage. A simple hello world example in Python would look like this:

print("Content-Type: text/html")
print()          
print("")
print("
This is my first CGI script
")
print("Hello, world!")

To allow for dynamic inputs, everything passed in the query parameters of the URL is forwarded to the CGI script as input. In principle, the CGI is a safe method of serving dynamic websites — it has been replaced mainly because it spawns a separate process for each request, causing the webserver to quickly struggle at high loads. 

As a router is rarely accessed by many people in tandem this is not an issue for the Netgear use case. However, the Netgear setup.cgi script has two fundamental flaws which, when combined, quickly turn innocent routers into dangerous botnets for criminals to exploit.

The loaded gun and its lazy bodyguard

The setup.cgi script provided by Netgear comes with the ability to execute arbitrary commands using the syscmd function. So, an authenticated user can access a shell inside their router and pass commands. For a simple hello world page, we can request the following page: 

/setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=echo+hello+world 

While this might have diagnostic use cases, it should be very clear that allowing any user to execute arbitrary scripts using the privileges of the router itself constitutes a loaded gun for exploitation. 

The ability to execute arbitrary commands now combines with a second problem in the setup.cgi script. The authentication protection can be easily bypassed by adding currentsetting.htm=1 as a query parameter at the end of the URL. This causes the setup script to skip the authentication check allowing anyone from the outside and the inside to execute code remotely on the router. 

As the router sits at the entrypoint of a network, it makes the exploitation of the vulnerability very hard to discover locally. It also opens up the other devices in the network for potential exploitation, as local connections are usually subject to less scrutiny by security systems. 

This exploit was discovered by Roberto Paleari in 2013. The authentication vulnerability was patched for some devices using a firmware update, though Netgear DGN2200 v1 remains vulnerable to this day. The flaws in setup.cgi persist in various different ways, a recent example of a new exploit can be found here. 

Detective on the scene

The Crowdsec Security Engine detects exploitation attempts using a trigger bucket. The scenario checks whether a given IP requests something starting with /setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=.

This scenario design allows the Security Engine to stay agnostic to changes in the authentication bypass strategy while only looking for the call to syscmd, which would execute code on the router. 

As the Security Engine does not know what kind of devices it is protecting, it will pick up exploitation attempts aimed at devices that are not vulnerable to the exploit. It is a common strategy for malware to reproduce itself by having exploited machines scan the internet to try and exploit devices vulnerable to the same exploit, so this case appears very often. Such requests produce what we call the background noise of the internet, which CrowdSec Enterprise users can filter from their alerts in the CrowdSec Console. 

Tracking the footprints

While we have already mentioned that the netgear_rce scenario handily surpassed other, more famous, scenarios in the number of threat reports we get from the CrowdSec community, the data also shows other curious patterns. 

A common pattern we see in our data is that certain attacks often trigger similar IPs. For some scenarios, this is by design, such as ssh-bf and ssh-slow-bf ( ssh-slow-bf is basically ssh-bf with a slower leak speed). For other scenarios, this can happen when an attacker tries multiple attacks at the same time or when there is a small exploratory phase that triggers more generic scenarios. As an example, the chart below shows the scenarios that are associated with IPs that trigger the log4j scenario.

The chart indicates that around 60% of attackers who are reported for log4j attacks also get reported for http-probing. Note that this is not necessarily from the same Security Engine. Log4j shows a very common pattern here. The most common co-scenario hovers around 60% and the tail is long.

Now compare this to the netgear_rce scenario below.

Here we see an entirely different picture. Attackers who are trying to exploit vulnerable Netgear routers almost exclusively do just that. It is a very unusual pattern. The most common co-scenario is an attack targeting xml-rpc, a common attack vector for WordPress sites, captured by our partner, Shield. 

A similarly unusual picture can be seen in the distribution of origin countries for the attackers. 

‍

This is arguably less surprising and gives hints as to why the Netgear attacks are so isolated. To understand the pattern for log4j, it helps to look at the Autonomous Systems (AS) instead of the origin countries.

As you can see, most of the attacks are instances bought at common web hosters. Chances are that most of these attackers are themselves captured web servers that are now trying to spread their infections far and wide. For the Netgear attacks, the targeted devices are routers and while you can spin up a DigitalOcean droplet for cheap, you rarely ask your local hosting company to set up an empty router in their datacenter. This means that the devices that are getting infected are not data center servers but home and business routers collecting dust on someone’s desk. 

What we are seeing is not the origin distribution of the operating attackers, but the distribution of vulnerable devices. 

This almost completes all the pieces of our puzzle. What remains to be clarified is how come there are so few vulnerable devices on the western side of the globe. 

The answer to this is rather simple. The two vulnerable routers connect to the internet using ADSL2+. This is a fairly old technology used to connect to the internet using classic telephone lines. The ADSL2+ standard, in particular, supports speeds up to a crawling 3MB/s. So, the reason why routers with a country of origin in the West are not flooding the web with exploitation attempts is simply that the ISPs of the Western world have accidentally closed the vulnerability by upgrading the devices used by their customers. 

This also prompts us to theorize why the attackers don’t trigger any other scenarios. Not only are the devices severely constrained in bandwidth, but it is also highly likely that these attacks are still self-replicating long after the people behind the original code have left for greener pastures. What we might be seeing is swathes of lonely, compromised routers, trying to phone home to places that no longer listen to their cries.

And that is all, folks!

I hope you enjoyed this small investigation into our data. If you want to learn more about the power of our crowdsourced data, get your hands on the latest Majority Report containing informational tidbits we have aggregated from our crowdsourced CTI.