Secure Docker Compose stacks with CrowdSec - The open-source & collaborative IPS
September 1st, 2021
10 mn read

Secure Docker Compose stacks with CrowdSec

This article explains how to make CrowdSec and Docker Compose work together to protect applications exposed in containers. It should allow us to:

  • Automatically ban malevolent IPs from accessing our container services
  • Manually add/remove and inspect ban decisions
  • Monitor CrowdSec’s behavior (via cli and dashboards)

This article has been written by our tech team and is already published on the Alibaba Cloud blog here.

Target Architecture

The chart below shows a glimpse of how our target architecture will look:

Let’s create a Docker Compose file that will setup the following:

  • A reverse-proxy that uses Nginx
  • A sample application that exposes an Apache2 “hello world”
  • A CrowdSec container that reads the reverse-proxy’s logs to detect attacks on the HTTP service
  • A Metabase container that will generate fancy dashboards showing what has been happening

We have chosen the simplest way to collect logs: by sharing volumes between containers. If you are in production, you are probably using a logging-driver to centralize logs with rsyslog or another driver, so don’t forget to adapt the CrowdSec Docker Compose configuration to read your logs properly.
The docker-compose.yml file is as follow:

version: '3'
 
services:
 #the application itself : static html served by apache2.
 #the html can be found in ./app/
 app:
   image: httpd:alpine
   restart: always
   volumes:
     - ./app/:/usr/local/apache2/htdocs/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.2
 
 #the reverse proxy that will serve the application
 #you can see nginx's config in ./reverse-proxy/nginx.conf
 reverse-proxy:
   image: nginx:alpine
   restart: always
   ports:
     - 8000:80
   depends_on:
     - 'app'
   volumes:
     - ./reverse-proxy/nginx.conf:/etc/nginx/nginx.conf
     - logs:/var/log/nginx
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.3
  #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.0.8
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"
   depends_on:
     - 'reverse-proxy'
   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5
 
volumes:
 logs:
 crowdsec-db:
 crowdsec-config:
 
networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24

The reverse-proxy (nginx) container writes its logs to a logs volume mounted by the crowdsec container.

CrowdSec’s SQLite database is in a crowdsec-db volume mounted by the dashboard (metabase) container

Initial deployment

Prerequisites: Docker / Docker Compose

We have put the configuration files altogether on this repository, so that you can simply clone it to deploy.

From the Docker Compose directory, you can deploy with docker-compose up -d and then check that everything is running with docker-compose ps.

▶ git clone https://github.com/crowdsecurity/example-docker-compose
▶ cd example-docker-compose
▶ sudo docker-compose up
▶ sudo docker-compose ps

# cd examples/docker-compose
# docker-compose up -d
...
# docker-compose ps     
             Name                           Command               State           Ports         
------------------------------------------------------------------------------------------------
docker-compose_app_1             httpd-foreground                 Up      80/tcp                
docker-compose_crowdsec_1        /bin/sh -c /bin/sh docker_ ...   Up                            
docker-compose_dashboard_1       /app/run_metabase.sh             Up      0.0.0.0:3000->3000/tcp
docker-compose_reverse-proxy_1   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:8000->80/tcp 

Let’s check and make sure everything is working!

Demo app check

With the following command, we can check whether access to our demo app is working properly.

▶ curl http://localhost:8000/
Hello world !%

Demo app check

We need to validate whether our CrowdSec setup reads logs as expected.

docker-compose exec crowdsec cscli metrics

▶ sudo docker-compose exec crowdsec cscli metrics
INFO[25-02-2021 03:38:50 PM] Buckets Metrics:                             
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-crawl-non_statics | -             | -         |            2 |      2 |       2 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[25-02-2021 03:38:50 PM] Acquisition Metrics:                         
+-----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE               | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/nginx/example.access.log |          2 |            2 | -              |                      2 |
+-----------------------------------+------------+--------------+----------------+------------------------+
INFO[25-02-2021 03:38:50 PM] Parser Metrics:                              
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  |    6 |      2 |        4 |
| child-crowdsecurity/nginx-logs |    2 |      2 | -        |
| crowdsecurity/dateparse-enrich |    2 |      2 | -        |
| crowdsecurity/geoip-enrich     |    2 |      2 | -        |
| crowdsecurity/http-logs        |    2 | -      |        2 |
| crowdsecurity/nginx-logs       |    2 |      2 | -        |
| crowdsecurity/non-syslog       |    2 |      2 | -        |
+--------------------------------+------+--------+----------+
INFO[25-02-2021 03:38:50 PM] Local Api Metrics:                           
+--------------------+--------+------+
|       ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/watchers/login | POST   |    2 |
+--------------------+--------+------+

What happened and what is relevant here? 

The command cscli metrics queries the Prometheus metrics exposed locally by CrowdSec and presents them in a fancy terminal output:

  • The “acquisition metrics” show us that our requests are indeed generating logs that are being read (“LINES READ”), parsed (“LINES PARSED”) and even matched with installed scenarios (“LINES POURED TO BUCKET”)
  • The “buckets metrics” and “parser metrics” allow us to see which parsers and scenarios are being triggered

CrowdSec configuration check

The command cscli hub list allows us to see which parsers and scenarios are deployed.

▶ sudo docker-compose exec crowdsec cscli hub list
INFO[25-02-2021 03:46:41 PM] Loaded 14 collecs, 19 parsers, 23 scenarios, 3 post-overflow parsers 
INFO[25-02-2021 03:46:41 PM] unmanaged items : 20 local, 0 tainted        
INFO[25-02-2021 03:46:41 PM] PARSERS:                                     
-------------------------------------------------------------------------------------------------------------
 NAME                            📦 STATUS   VERSION  LOCAL PATH                                             
-------------------------------------------------------------------------------------------------------------
 crowdsecurity/sshd-logs         ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml         
 crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml         
 crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml 
 crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml     
 crowdsecurity/nginx-logs        ✔️  enabled  0.2      /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml        
 crowdsecurity/http-logs         ✔️  enabled  0.4      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml        
-------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] SCENARIOS:                                   
--------------------------------------------------------------------------------------------------------------------------
 NAME                                       📦 STATUS   VERSION  LOCAL PATH                                               
--------------------------------------------------------------------------------------------------------------------------
 ltsich/http-w00tw00t                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml               
 crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml      
 crowdsecurity/http-probing                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml                
 crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml 
 crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml            
 crowdsecurity/http-bad-user-agent          ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-bad-user-agent.yaml         
 crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml                      
 crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml     
 crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml        
 crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml           
--------------------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] COLLECTIONS:                                 
------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH                                         
------------------------------------------------------------------------------------------------------------
 crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml                
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml 
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml               
 crowdsecurity/nginx                ✔️  enabled  0.1      /etc/crowdsec/collections/nginx.yaml               
------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] POSTOVERFLOWS:                               
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------

Metabase check

Metabase is one of the components that has been deployed, which helps us generate dashboards for better observability. You can hop onto http://127.0.0.1:3000/ and log in with crowdsec@crowdsec.net and password !!Cr0wdS3c_M3t4b4s3??

Metabase comes with a default password because of how it is deployed. Remember to change the default password and restrict the metabase’s access to relevant IP addresses or network ranges.

At first, dashboards will be empty since no attacks were detected yet. The main one should look like this:

If any of those checks have failed for you, take a look at container logs with docker-compose logs crowdsec (for example).

Detection features

Note: In real-world setups, whitelists are deployed to prevent banning private IPs.

After checking to make sure everything is ready to go, let’s try some detection features. As we work with an exposed HTTP service, let’s fire a Nikto from another machine in the LAN nikto -host http://192.168.2.227:8000

Note: The IP is dependent on your LAN setup and addressing plan.

On the other hand, we kept an eye on CrowdSec’s logs through the following command.

docker-compose logs -f crowdsec

Here we can spot our client’s IP (192.168.2.211) being flagged for triggering various scenarios:

We can see my IP is banned with docker-compose exec crowdsec cscli decisions list.

We can review and inspect alerts with docker-compose exec crowdsec cscli alerts list and docker-compose exec crowdsec cscli alerts inspect -d XX:

Note: cscli alerts list returns the list of all alerts triggered.

▶ sudo docker-compose exec crowdsec cscli alerts inspect -d 43

################################################################################################

 - ID         : 43
 - Date       : 2021-02-26T08:26:07Z
 - Machine    : ee0ebe5b529c4995964ff0b3e01b1801sxSpiCdYj9lpSD9W
 - Simulation : false
 - Reason     : crowdsecurity/http-sensitive-files
 - Events Count : 5
 - Scope:Value: Ip:192.168.2.211
 - Country    : 
 - AS         : 

 - Active Decisions  :
+-----+------------------+--------+--------------------+----------------------+
| ID  |   SCOPE:VALUE    | ACTION |     EXPIRATION     |      CREATED AT      |
+-----+------------------+--------+--------------------+----------------------+
| 802 | Ip:192.168.2.211 | ban    | 3h53m15.124782708s | 2021-02-26T08:26:07Z |
+-----+------------------+--------+--------------------+----------------------+

 - Events  :

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| ASNNumber     |               0 |
| http_args_len |               0 |
| log_type      | http_access-log |
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.bak   |
| IsInEU        | false           |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| log_type      | http_access-log |
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.sql   |
| IsInEU        | false           |
| ASNNumber     |               0 |
| http_args_len |               0 |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
|      KEY      |      VALUE      |
+---------------+-----------------+
| service       | http            |
| source_ip     | 192.168.2.211   |
| http_status   |             404 |
| http_path     | /CyNqbugR.exe   |
| IsInEU        | false           |
| ASNNumber     |               0 |
| http_args_len |               0 |
| log_type      | http_access-log |
+---------------+-----------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-------------------+
|      KEY      |       VALUE       |
+---------------+-------------------+
| IsInEU        | false             |
| ASNNumber     |                 0 |
| http_args_len |                 0 |
| log_type      | http_access-log   |
| service       | http              |
| source_ip     | 192.168.2.211     |
| http_status   |               404 |
| http_path     | /CyNqbugR.printer |
+---------------+-------------------+

- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+--------------------+
|      KEY      |       VALUE        |
+---------------+--------------------+
| ASNNumber     |                  0 |
| http_args_len |                  0 |
| log_type      | http_access-log    |
| service       | http               |
| source_ip     | 192.168.2.211      |
| http_status   |                404 |
| http_path     | /CyNqbugR.htaccess |
| IsInEU        | false              |
+---------------+--------------------+

Note: cscli alerts inspect -d <ID> allows you to generate more details about a given alert.

Monitoring activities with dashboards

Now that we have triggered several scenarios, we can go back to our Metabase dashboards (http://127.0.0.1:3000 with the default setup) and check the activity.

If the traffic came from a public IP (rather than a private one, as in this example), crowdsecurity/geoip-enrich would have enriched events with geo-localization data and AS/range information.

Block attacks with bouncers

Now that we have a fully functional CrowdSec service, we can detect incoming attacks on our service based on installed collection scenarios.

After detecting these attacks, our objective is to block them. You will use the cs-firewall-bouncer to achieve this. Start by installing it on the host, and block malevolent traffic directly in the DOCKER-USER chain, the default chain created by Docker to filter traffic targeting the containers.

You can find the firewall bouncer on the CrowdSec Hub. As of today, the most up-to-date version is v0.10.

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz
tar xvzf cs-firewall-bouncer.tgz
cd cs-firewall-bouncer-v0.0.10
sudo ./install.sh

Since July, custom and firewall bouncers are packaged with Debian, Ubuntu, CentOS, RHEL and Amazon Linux. Please check our GitHub repo for more information. The install is straightforward. It will deploy a systemd unit for the service and make sure you meet the requirements. Here, I didn’t have ipset running, and it installed it for me.

Here, we installed the bouncer on a host where CrowdSec isn’t running. As a result, the service isn’t happy.

Now, let’s configure the bouncer to speak with the Local API running on our CrowdSec container. We start by creating an API token for our bouncer with cscli.

docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer

▶ sudo docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer
Api key for 'HostFirewallBouncer':

   aaebb3708fe67eeeccbb52a21e5e7862

Please keep this key since you will not be able to retrive it!

Then, you need to configure the bouncer to use this token to authenticate with CrowdSec’s Local API. In the /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml, edit api_urlapi_key, and iptables_chains. In this case, IPv6 was also disabled with disable_ipv6:

mode: iptables
piddir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
api_url: http://172.20.0.4:8080/
api_key: aaebb3708fe67eeeccbb52a21e5e7862
disable_ipv6: true
#if present, insert rule in those chains
iptables_chains:
 - DOCKER-USER

Note: We edited the chains to only have DOCKER-USER, and we set api_url accordingly to our docker-compose.yml file, along with the newly generated api token.

You shall not pass: CrowdSec in action

Now, we can get our freshly configured bouncer started with sudo systemctl start cs-firewall-bouncer.service and take a look at our new firewall configuration:

▶ sudo iptables -L -n                                                                  
...
Chain DOCKER-USER (1 references)
target     prot opt source               destination         
DROP       all  --  0.0.0.0/0            0.0.0.0/0            match-set crowdsec-blacklists src
...

We can see that our DOCKER-USER chain has been populated with a rule to match incoming traffic against our ipset, and our ipset is filled with relevant information.

▶ sudo ipset -L crowdsec-blacklists            
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 300
Size in memory: 6016
References: 1
Number of entries: 61
Members:
178.20.157.98 timeout 80103
52.184.35.59 timeout 80103
...

As you might have noticed by now, our ipset isn’t only filled with our local decisions but also with ones from the community. However, we can see that our “local” decisions made their way to the ipset list.

▶ sudo ipset -L crowdsec-blacklists  | grep 192
192.168.2.211 timeout 12859

Now, we can check the attacker machine and see we are blocked and cannot access the application.

$ curl -vv 192.168.2.227:8000
* Rebuilt URL to: 192.168.2.227:8000/
*   Trying 192.168.2.227...
* TCP_NODELAY set
^C

The attacker is prevented from accessing all Docker Compose applications. We limited the decision to the DOCKER-USER chain on purpose, which means local applications exposed by the host would still be accessible. If we wish to extend the ban to all incoming traffic, we could have added the INPUT chain to the list, as is the case with the default setup.

Conclusion

Throughout this tutorial, we deployed a minimal yet complete applicative stack using Docker Compose. Then, we covered how to secure it with CrowdSec. While most people will use CrowdSec as a host-based defense mechanism, we can see that it’s also suitable for Docker environments. If you want to meet with the team to share your feedback, you can find us on our Gitter channel.

About the author

Thibault Koechlin graduated from EPITECH, specializing in the security of IT systems & networks. He started his career at NBS in 2004, as an expert in penetration testing before being appointed Head of the offensive security team. He then became CISO by expanding his skills around defensive security before initiating the development of several open-source products and building teams with rare skills. He completed his ascent within the company through an operational partner role, leading the creation of the company’s flagship product: Cerberhost. In December 2019, he co-founded CrowdSec with Philippe Humeau and Laurent Soubrevilla. He is the CTO of the company.

You also like

Let's make the internet safer together

AFEB9F8A-65D8-49C4-BF47-4958C484D8C8
Download v1.2.1