Feb 27, 2023

Why develop new scenarios?

What is a CrowdSec Scenario?

Scenarios are core elements of the CrowdSec detection engine. These models enable the CrowdSec Security Engine to recognize suspicious behavior and ultimately make a decision whether to block an IP or not.

Our unique blocklist contains thousands of malicious IPs that have matched one of the many scenarios designed by our team or community.

The extreme variety of these scenarios allows us to classify offensive IPs according to their activity and share this information with all CrowdSec users to better protect their systems. The full collection of available scenarios can be found on the CrowdSec Hub.

While the CrowdSec scenarios cover attack patterns for typical types of attack (CVE, Brute Force, DoS etc), there is a lot more that can be done with them. Read on to see how you can modify existing scenarios, and help make the community safer by creating new ones.

How does it work?

A scenario is a configuration file in readable YAML format, which allows modeling behavior from elements captured in log files. These elements are measured, compared, and correlated to determine a framework outside of which any activity will be considered suspicious.

For example, too many access attempts to an authentication page from a single IP over a short period is likely a brute-force login attempt. The source IP can be slowed down, redirected to a honeypot, or banned. That is a robust protection scenario. You would find many of them in the CrowdSec Hub for all types of web servers.

But what makes CrowdSec’s scenarios extremely rich are the infinite possibilities of event correlations that describe specific behaviors, far from the detection standards. Because each application context is different, our power users have developed their own scenarios. They have identified new offensive behaviors and thus share unique malicious IPs with the entire CrowdSec community with a highly qualified context.

Why would you create your own scenarios?

There are many reasons why our users create new scenarios.

Once the CrowdSec Engine’s protection is in place, our users gain protection against any attempt from known malicious IPs shared by the community. They have deployed the usual scenarios corresponding to the types of servers they have and have covered the typical attack scenarios.

The next step in the security process will be to push the detection further to identify more sophisticated, internally and externally-driven behaviors. To do this, they will modify scenarios or create their own to model behaviors that correspond precisely to their applications or infrastructures and close the doors to the attackers for good.

There is no limit to the creation of scenarios. All available data can be processed to describe behavior that targets infrastructure or applications at the technical or business level. Scalping, the practice of using bots for bulk buying tickets or goods and reselling them for profit, is a perfect example of what a CrowdSec scenario can model.

Those who wish to can share these new scenarios with the CrowdSec community to better protect other users.

We are continuously developing new scenarios for all servers and applications. But above all, we enable our users to detect malicious behavior even more efficiently.

If we join our forces, we won’t give attackers a chance.

How to create a scenario?

Start by identifying the internal or external scenario that you would like to model.

Here is how you can create a scenario:

We’re going to create a scenario for an imaginary service “myservice” from the following logs of failed authentication:

Dec  8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '1.2.3.4'Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'

Pre-requisites

Create our test

From the root of the hub repository:

▶ cscli hubtest create myservice-bf --type syslog --ignore-parsers

  Test name                   :  myservice-bf
  Test path                   :  /home/dev/github/hub/.tests/myservice-bf
  Log file                    :  /home/dev/github/hub/.tests/myservice-bf/myservice-bf.log (please fill it with logs)
  Parser assertion file       :  /home/dev/github/hub/.tests/myservice-bf/parser.assert (please fill it with assertion)
  Scenario assertion file     :  /home/dev/github/hub/.tests/myservice-bf/scenario.assert (please fill it with assertion)
  Configuration File          :  /home/dev/github/hub/.tests/myservice-bf/config.yaml (please fill it with parsers, scenarios...)

Configure our test

Let’s add our parser and scenario to the test configuration (.tests/myservice-bf/config.yaml) file.

parsers:
  - crowdsecurity/syslog-logs
  - crowdsecurity/dateparse-enrich
  - ./parsers/s01-parse/crowdsecurity/myservice-logs.yaml
scenarios:
  - ./scenarios/crowdsecurity/myservice-bf.yaml
log_file: myservice-bf.log
log_type: syslog
ignore_parsers: true

Note: as our custom parser and scenario are not yet part of the hub, we specify their path relative to the root of the hub directory.

Scenario creation

Let’s create a simple scenario to detect bruteforce attempt on myservice:

# myservice bruteforce
type: leaky
name: crowdsecurity/myservice-bf
description: "Detect myservice bruteforce"
filter: "evt.Meta.log_type == 'myservice_failed_auth'"
leakspeed: "10s"
capacity: 5
groupby: evt.Meta.source_ip
blackhole: 1m
labels:
  service: myservice
  type: bruteforce
  remediation: true

Note: We filter on evt.Meta.log_type == ‘myservice_failed_auth’ because in the parser myservice-logs (created in the Creating parsers part) we set the log_type to myservice_failed_auth for bad password or bad user attempt.

We have the following fields:

a type: the type of bucket to use (trigger or leaky).
a name optional name
a description optional description
a filter: the filter to apply on events to be filled in this bucket.
a leakspeed how fast an event would leak from a bucket
a capacity: the number of events in the bucket before it overflows.
a groupby: a field from the event to partition the bucket. It is often the source_ip of the event.
a blackhole: the amount of time to not retrigger this scenario for the same groupby field.
some labels: some labels to apply on the trigger event. Don’t forget to set remediation: true if you want the IP to be blocked by remediation components.

We can then “test” our scenario like this:

▶ cscli hubtest run myservice-bf
INFO[01-10-2021 12:41:21 PM] Running test 'myservice-bf'                
WARN[01-10-2021 12:41:24 PM] Assert file '/home/dev/github/hub/.tests/myservice-bf/scenario.assert' is empty, generating assertion: 

len(results) == 1
"1.2.3.4" in results[0].Overflow.GetSources()
results[0].Overflow.Sources["1.2.3.4"].IP == "1.2.3.4"
results[0].Overflow.Sources["1.2.3.4"].Range == ""
results[0].Overflow.Sources["1.2.3.4"].GetScope() == "Ip"
results[0].Overflow.Sources["1.2.3.4"].GetValue() == "1.2.3.4"
results[0].Overflow.Alert.Events[0].GetMeta("datasource_path") == "myservice-bf.log"
results[0].Overflow.Alert.Events[0].GetMeta("datasource_type") == "file"
results[0].Overflow.Alert.Events[0].GetMeta("log_subtype") == "myservice_bad_user"
results[0].Overflow.Alert.Events[0].GetMeta("log_type") == "myservice_failed_auth"
results[0].Overflow.Alert.Events[0].GetMeta("service") == "myservice"
results[0].Overflow.Alert.Events[0].GetMeta("source_ip") == "1.2.3.4"
results[0].Overflow.Alert.Events[0].GetMeta("username") == "toto"
....
results[0].Overflow.Alert.GetScenario() == "crowdsecurity/myservice-bf"
results[0].Overflow.Alert.Remediation == true
results[0].Overflow.Alert.GetEventsCount() == 6

...


Please fill your assert file(s) for test 'myservice-bf', exiting

What happened here?

The scenario has been triggered and is generating some assertion (for functional test)
In production environment, an alert would have been send to the CrowdSec Local API.

We can again understand more of what is going on thanks to cscli hubtest explain:

▶ cscli hubtest explain myservice-bf
line: Dec  8 06:28:43 mymachine myservice[2806]: unknown user 'toto' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

line: Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

line: Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

line: Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

line: Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

line: Dec  8 06:28:43 mymachine myservice[2806]: bad password for user 'admin' from '1.2.3.4'
    ├ s00-raw
    |   └ 🟢 crowdsecurity/syslog-logs
    ├ s01-parse
    |   └ 🟢 crowdsecurity/myservice-logs
    ├ s02-enrich
    |   └ 🟢 crowdsecurity/dateparse-enrich
    ├-------- parser success 🟢
    ├ Scenarios
        └ 🟢 crowdsecurity/myservice-bf

We have now a fully functional scenario for myservice to detect brute forces! We can either deploy it to our production systems to do stuff.

Once your scenario is ready to rumble, test it with your logs. We even provide a simulation mode for that (“cscli simulation”) to ensure it is working properly. And then, share it with the community by posting it on the Hub!

Your benefits from doing that:

You can identify internal or external illicit behaviors you were unaware of.

You will be free to adapt your response to these behaviors.

Your infrastructure and applications will be even better protected.

You will get recognition as one of our amazing Power Users!