Introducing the IP Range Reputation System: Identify Organized Cyber Crime vs. Petty Criminals

One of the big lessons we learned from collecting and analyzing huge amounts of data at CrowdSec is that cyberattacks are not distributed equally over the internet. 

There are Autonomous Systems (AS) — think internet service providers or hosters — and IP ranges that harbor significantly more criminal activity than others. This insight prompted us to aggregate the signals we receive not only by IP address but also by IP range and Autonomous System. 

In this article we provide an example of what this looks like in practice and how these new aggregations are used by us internally to improve our blocklist offerings. However, we first need to explore a crime scene to understand these insights.

Capturing petty criminals

The following table presents a sample of IPs that recently participated in http-path-traversal-probing attacks. These attacks are quite simplistic. They aim to get a web server to reveal files outside of the sitemap by trying to escape paths and walking backward along the file tree using ../. 

While most modern web servers are not vulnerable to such attacks, the attackers show up directly in the log even on failed attempts which makes this a good Indicator of Compromise (IoC) for the Crowdsec Security Engine to investigate.

Now, good detectives will have already noticed something afoot here. But in case you didn’t notice, the table below draws your attention to certain IPs.

While some IPs in this table seem to come from all over the internet, the IPs highlighted all share the first 3 sections of their IP address. What this means is that these IP addresses originate from the same neighborhood or IP range of the internet. To further explore this strange coincidence, we need to learn a little bit about how the internet works.

IP ranges — The neighborhoods of the internet

At the core, an IP address is a representation of a 32-digit binary number. Wikipedia has a nice graphic to show how each of the 4 numbers of the dot-decimal notation corresponds to an 8-digit binary part of the IP address. 

Usually, these IP addresses are not owned by individuals but by internet service providers and big technology companies, which then assign them to their customers and servers. To simplify the who-owns-what question, the service providers are not given lists of single IPs but so-called IP ranges. 

IP ranges are blocks of consecutive IP addresses. They are specified by the number of digits of the binary address that they have in common. So, a /31 range for a given address includes all IPs that have the same 31 first bits as this address. As an IP address is only 32 bits long that means that a `/31` range contains exactly two IPs. 

Generally, the lower the number on the range, the higher the number of IPs in it. At CrowdSec, we mainly consider range /24 neighborhoods, which are all the IPs that share the same 24 first bits. This is a convenient range size as it is still fairly small at 256 IPs per range, but also because the boundary falls right on one of the decimals in dot-decimal notation. So, the range 172.16.254.0/24 contains all the IPs from 172.16.254.0 to 172.16.254.256. This makes /24 an extremely useful range to look at.

Fighting organized crime

With the crash course in internet protocol and IP ranges finished, let’s turn our attention back to the matter at hand and recall the table of petty criminals seen earlier.

With your newly gained knowledge of the internet protocol, you can now identify that some of the criminals seen in the table operate from the same neighborhood of the internet. 

So, it’s not difficult to realize that we might not be dealing with petty criminals who do some path traversal on the side here, but we are facing an organized crime network operating from the XXX.234.140.0/24 neighborhood of the internet. 

Properly detecting such bad neighborhoods and taking the appropriate measures against these perpetrators is vital to our goal of always providing our users with blocklists that contain verified malicious IPs to help them proactively protect their systems from cyberattacks.

To generate the CrowdSec Intelligence Blocklist, we have built a fairly sophisticated system of rules that evaluate an IP on a diverse set of metrics to ultimately decide whether it gets added to the blocklist. This system is called the Expert System. The best way to understand the functionality of the Expert System is to imagine a judge ruling in a court. 

Every IP is put to a trial with evidence supporting the case. The judge (Expert System) has a code of law (CrowdSec ruleset) within which it has to decide whether to convict the suspect. For validation, we have a lawmakers’ convention (internal meeting) every week where we inspect the rulings and change the laws if needed. 

Staying within this law analogy, we face similar issues as real courts when fighting organized crime. Each individual IP in this range might not be criminal enough on its own to get convicted for a big sentence, but as an organization, it is clear that there is significant abuse coming from this range. Similar to real courts, we can keep playing whack-a-mole and jail abusers who cross the line for petty crimes, but to put a real dent into the criminal organization, we need a bigger mallet. In the real world, this big mallet is called the RICO Act — in the CrowdSec cybersecurity court, it is the Range Reputation System.

Introducing the IP Range Reputation System

The IP Range Reputation System is an aggregation of stats on individual IPs at the level of their /24 range. That means that for every /24 range, we track the following:

• The number of IPs from this range we have seen in reports from Security Engines
• The number of IPs from this range that landed in the CrowdSec Intelligence Blocklist
• The average duration of any given IP remaining in our data lake
• The trust score of the Security Engines reporting this IP 
• The behaviors identified for IPs in this range

We then use this information to build an organized crime case. The case is built on two components. 

• Neighborhood reputation: This is based on how many other IPs have already been reported and how many of them have landed in the CrowdSec Intelligence Blocklist. 
• Attacker behavior: How the behavior of this specific attacker compares to the other IPs in its range. The analogy for this is that while a pickpocket and a bike thief might operate in the same neighborhood, as long as they do their separate things, there is no reason to suspect that they are part of the same organized crime unit. 

These two components are then weighted, scored, and fed to the Expert System as additional evidence. If an IP is convicted of operating within a criminal network, it gets into the CrowdSec Intelligence Blocklist significantly faster, turning this simple system into an effective large mallet to bleed cybercriminals of their most valuable resource — fresh IP addresses. 

The Range Reputation System we built alongside this feature powers our insights in the Majority Report and allows us to act on emerging threats with confidence even when there is little primary information to go on.

Get your hands on the IP Range Reputation feature

The IP Range Reputation feature is now available in our CTI API to help incident response teams and threat researchers gain improved visibility in low-information environments and act fast and decisively against threats, even in situations where the defenders are outnumbered.

To use the IP Range Reputation feature today, simply sign in to your CrowdSec Console account or sign up for free if you don’t have an account yet. Then navigate to the Settings page to generate a CTI API key.

Equipped with your API key, you can get started by querying the API using curl or any other tool. If you don’t have any interesting IP addresses at hand, you can always find an interesting selection under https://app.crowdsec.net/cti.

curl -H "x-api-key: YOUR_API_KEY" https://cti.api.crowdsec.net/v2/smoke/YOUR_INTERESTING_IP | jq .

The response you receive from the API will contain, among other things, our new range reputation information.

{
  "ip": "YOUR_INTERESTING_IP",
 ...
  "ip_range_24": "YOUR_INTERESTING_IP.0/24",      
  "ip_range_24_reputation": "malicious",
  "ip_range_24_score": 5,               
  ...
}

Similar to our ratings of individual IPs, the range reputation comes as both a score ranging from 0 to 5 and as a single label of either malicious, suspicious, or known. For further information on our CTI API, including exhaustive documentation and a swagger portal, check out our CTI API documentation.

The CrowdSec team is working on expanding and improving the capabilities of the IP Range Reputation feature, so stay tuned as we’ll be sharing more updates on this soon!