Exploring the CrowdSec Security Engine & open-appsec Open Source WAF Integration

In case you missed the news, CrowdSec and open-appsec have collaborated to strengthen cybersecurity defenses through an integration allowing our two systems to communicate.

‍open-appsec is a free and open-source WAF project, which is fully based on Machine Learning (instead of static signatures like most traditional WAFs). open-appsec provides automatic and preemptive protection against new zero-day attacks like Log4shell, and Spring4Shell, as well as OWASP-Top-10 attacks and more. It supports Kubernetes, Docker, and Linux and integrates with NGINX, Kong, and Ingress NGINX (more to come soon).

By making a bridge between the CrowdSec Security Engine and open-appsec's agent, it is now possible to share custom threat indicators and integrate the CrowdSec community blocklist into open-appsec. This partnership promises enhanced protection against potential threats, including zero-day attacks.

The release announcement presented this integration in detail, along with the documentation on how to deploy it on Kubernetes. You can find the open-appsec article here and the open-appsec collection for CrowdSec Security Engine here.

In this article, I will analyze attack behaviors on open-appsec-protected nodes during the past few months and present a Docker setup combining open-appsec and CrowdSec containers. This combination of strong community threat intelligence and an innovative machine learning threat prevention engine further enhances the effectiveness of both solutions.

The CrowdSec & open-appsec integration

Custom signals from open-appsec

open-appsec has its own detection capabilities, and with the CrowdSec integration, users can share signals with the CrowdSec network about malicious behavior they observe in exchange for the CrowdSec real-time community blocklist.

Those custom scenarios are passed on to CrowdSec via the open-appsec collection.

Since the release, we observed signals for those scenarios within the CrowdSec network and started integrating them into our consensus engine.

In the representation below, you can see the top scenarios pushed by the current pool of users. The top detected scenarios are:

• openappsec/openappsec-rce
• openappsec/openappsec-probing
• openappsec/openappsec-sql-injection

As the number of users of this integration increases, we might see some changes in those percentages depending on the nature of the user infrastructure where the installation is set up.

CrowdSec real-time blocklist events

Periodically, open-appsec retrieves the blocklist from the CrowdSec Security Engine Local API. The strength of the CrowdSec blocklist resides in the effort CrowdSec puts into avoiding false positives or stale information.

Blocking decisions found in the blocklist are translated into Security Events in open-appsec, under the name Access Control External Vendor Reputation (CrowdSec), as shown below.

Quick Docker setup guide

In the integration announcement, you can find a quick guide on how to set it up on Kubernetes. Further deployment options can be found in the open-appsec docs.

This quick guide will showcase a dummy example for setting it up in Docker containers.

Asset setup on my.openappsec.io

Let’s start by setting up our Profile (representing our agent deployment) as well as an Asset for specifying the website/URL we want to protect (containing also its desired protection settings) on the open-appsec Management Portal WebUI at https://my.openappsec.io/ in order to have everything ready for deployment. 

Note: The profile token needs to be provided to the agent at deployment time to assign it to the desired Profile in the Management Portal WebUI.

Profile creation

Let's create a profile. For our example, we’ll create a new Docker profile:

General settings

• For further reference, we’ll call it Docker Agents Test
• Subtype: NGINX
• Tags: we won’t need tags for this example
• Note: The Authentication token is available on this page (see screenshot below), please copy it to the clipboard, we’ll need it for later.

Advanced Settings: No need to make any changes here.

Log Trigger creation

We now need to set up a Log Trigger that will later allow us to have logs in the open-appsec console as well as locally parseable by CrowdSec.

• Create a new Log Trigger
• Name: For further reference, we’ll call it Cloud and Local Logs
• Threat Prevention / AppSec
  - Check Detect events and Prevent events 
  - To eventually cover further detection scenarios from CrowdSec, you can also check All web requests, as it can be useful for some of our advanced detection scenarios
  - Additional logging: From Severity: High, check Response Code
• The Log To section is the important one: - Check Cloud in order to see logs in the WebUI
  - Check Gateway/agent in order to provide CrowdSec the means to create signals based on parsing/evaluating open-appsec’s locally stored security logs

Blocking behavior (optional)

Let’s define the behavior that will be assigned to the asset blocking.

• Create a new Behavior of type Web User Response
• You can choose the type you want, but for testing purposes, we’ll set:
  - Mode Block Page
  - Choose what you want for the title, body, and response code

Asset creation

Now, we’ll create an asset and assign it to the profile we created before.

Create a new Web Application.

General Settings

• Basic/Name: For further reference, we’ll call it Test Serve
• Basic/Profiles: Let’s select our previously created Docker Agents Test profile‍
• Basic/Tags and Zone can stay blank
• Web Application: Setup your application’s URL. For the test we’ll use http://localhost
• Source Identity: 
  - Select Source IP in the select box (select X-Forwarded-For if you have another proxy in front of the open-appsec Agent)
  - You can leave trusted sources blank (machine learning will consider traffic from any configured trusted sources as benign/normal usage of the protected asset)

Threat Prevention

• Make sure to switch the mode from Learn/Detect to Prevent: Web Application Best Practice: Mode - Prevent ‍
• The following subsections define what you’re protecting against, as open-appsec provides various security features like BotDetection, API Schema enforcement. However, in this case:  
  - You can leave the default configuration- Note: Set the trigger and behavior previously created

Note: Save settings by clicking the Enforce button at the top right in the open-appsec WebUI.

Pop containers with Docker Compose

The Docker environment will be comprised of 3 containers:

• An NGINX container with a default webpage, which contains an open-appsec attachment module connecting it to the open-appsec agent for the security enforcement
• An open-appsec agent container
• A CrowdSec container

In order to avoid having to compose up and down too many times, let’s first setup some config files.

Setting up the configuration

In the directory of your choice, create the following.

NGINX config file

Create a ./nginxconf/default.conf file where we’ll set up a proxy_pass to allow open-appsec to have an eye on the web traffic. 

While in production deployments, you would typically have the NGINX proxy configured to pass traffic to protected backend web servers; however, here, we stick to using just one NGINX container. 

You can, of course, have any proper configuration if you’re using it on less of a dummy setup.

#./nginxconf/default.conf
---
server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    location / {
        proxy_pass http://127.0.0.1:8085;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

}

server {
    listen       8085;
    listen  [::]:8085;
    server_name  localhost;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

CrowdSec acquisition file for open-appsec logs

Create a ./crowdsec/acquis.d/openappsec.yaml file we’ll use to describe where to look for the open-appsec logs.

#./crowdsec/acquis.d/openappsec.yaml
---
source: file
filenames:
  - /var/log/cp-nano-http-transaction-handler.log*
labels:
  type: openappsec
  

We could, of course, look into logs in many different ways. CrowdSec supports many streams, among which Docker logs, but here, we’ll just target the mounted shared folders. This method is not the most efficient, but good for testing purposes.

Docker Compose

Create a docker-compose.yml file. Make sure to replace "YOUR AUTHENTICATION TOKEN HERE" with the token you have copied from the my.openappsec.io WebUI profile associated with the protected asset.

Note: We added the environment variable DISABLE_PARSERS=crowdsecurity/whitelists for test purposes only. The reason is that we will do malicious behaviors from private IPs, and we don’t want them to be added to the allowlist.

#docker-compose.yml
---
version: '3.8'

services:
  open-appsec-agent:
    image: ghcr.io/openappsec/agent:latest
    container_name: open-appsec-agent
    ipc: host
    volumes:
      - ./conf:/etc/cp/conf
      - ./data:/etc/cp/data
      - ./logs:/var/log/nano_agent
    environment:
      - CROWDSEC_ENABLED=true
      - CROWDSEC_MODE=prevent
      - CROWDSEC_LOGGING=enabled
      - CROWDSEC_API_URL=http://crowdsec:8080/v1/decisions/stream
      - CROWDSEC_AUTH_METHOD=apiKey
      - CROWDSEC_AUTH_DATA=ec612ba31c47a6307df2a5aacc3d11e6
    command: /cp-nano-agent --token [YOUR AUTHENTICATION TOKEN HERE] 
    restart: unless-stopped
    depends_on:
      - crowdsec

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    #ipc: host
    volumes:
      - ./crowdsec:/etc/crowdsec
      - ./crowdsec/data:/var/lib/crowdsec/data
      - ./crowdsec/logs:/var/log/crowdsec/
      - ./logs/:/var/log/
    environment:
      - COLLECTIONS=openappsec/openappsec
      - BOUNCER_KEY_openappsec=ec612ba31c47a6307df2a5aacc3d11e6
      - DISABLE_PARSERS=crowdsecurity/whitelists

  open-appsec-nginx:
    image: ghcr.io/openappsec/nginx-attachment:latest
    container_name: open-appsec-nginx
    volumes:
      - ./nginxconf:/etc/nginx/conf.d
    ipc: host
    ports:
      - 8081:80
    restart: unless-stopped
    depends_on:
      - open-appsec-agent
 

Pop containers and test

Let’s now run docker-compose up in the folder where docker-compose.yml and the config folders are.

After a few seconds, you should see in the open-appsec console that the open-appsec agent is connected:

The profile should be downloaded shortly in order to protect your endpoint :

• http://localhost:8081/ should display the default NGINX page
• http://localhost:8081/?shell_cmd=cat/etc/passwd should (after a few tries maybe) display the blocking page

You can also run pentest tools like nikto to check what attacks are blocked.

If you’re having issues, try troubleshooting with the following steps:

• Verify in the open-appsec console that everything is Saved/Enforced
• Try to run the /cp-nano-agent --token <you token> again or down and up the docker-compose again and wait a few minutes

What CrowdSec should see

On the CrowdSec container run cscli alerts list.

You should see a list of alerts like the one below:

Visualize remediation from CrowdSec’s Blocklist within open-appsec

From the open-appsec console, you’ll be able to see the traffic prevented based on CrowdSec Blocklist.

In the Monitoring section, under the Important Events tab, you’ll be able to see Prevent actions for the Event Name Access Control External Vendor Reputation. In addition, you can add the filter (eventname:"Access Control External Vendor Reputation") to see only those events.

Summing up

In this article, I wanted to showcase how the CrowdSec Security Engine and open-appsec Open Source WAF integration combines the strength of our community-driven threat intelligence with open-appsec’s innovative machine learning threat prevention engine to analyze attack behaviors. 

I hope you enjoyed this walkthrough, and if you haven’t tried out open-appsec yet, now’s your chance!

Website: https://www.openappsec.io   

Github: https://www.github.com/openappsec

Docs: https://www.openappsec.io/docs

Playgrounds: https://www.openappsec.io/playground 

Contact: info@openappsec.io 

‍