This article explains how to make CrowdSec and Docker Compose work together to protect applications exposed in containers. It should allow us to:
- Automatically ban malevolent IPs from accessing our container services
- Manually add/remove and inspect ban decisions
- Monitor CrowdSec’s behavior (via cli and dashboards)
Target Architecture
The chart below shows a glimpse of how our target architecture will look:
Let’s create a Docker Compose file that will setup the following:
- A reverse-proxy that uses Nginx
- A sample application that exposes an Apache2 “hello world”
- A CrowdSec container that reads the reverse-proxy’s logs to detect attacks on the HTTP service
- A Metabase container that will generate fancy dashboards showing what has been happening
We have chosen the simplest way to collect logs: by sharing volumes between containers. If you are in production, you are probably using a logging-driver to centralize logs with rsyslog or another driver, so don’t forget to adapt the CrowdSec Docker Compose configuration to read your logs properly.
The docker-compose.yml file is as follow:
version: '3'
services:
#the application itself : static html served by apache2.
#the html can be found in ./app/
app:
image: httpd:alpine
restart: always
volumes:
- ./app/:/usr/local/apache2/htdocs/
networks:
crowdsec_test:
ipv4_address: 172.20.0.2
#the reverse proxy that will serve the application
#you can see nginx's config in ./reverse-proxy/nginx.conf
reverse-proxy:
image: nginx:alpine
restart: always
ports:
- 8000:80
depends_on:
- 'app'
volumes:
- ./reverse-proxy/nginx.conf:/etc/nginx/nginx.conf
- logs:/var/log/nginx
networks:
crowdsec_test:
ipv4_address: 172.20.0.3
#crowdsec : it will be fed nginx's logs
#and later we're going to plug a firewall bouncer to it
crowdsec:
image: crowdsecurity/crowdsec:v1.0.8
restart: always
environment:
#this is the list of collections we want to install
#https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
COLLECTIONS: "crowdsecurity/nginx"
GID: "${GID-1000}"
depends_on:
- 'reverse-proxy'
volumes:
- ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
- logs:/var/log/nginx
- crowdsec-db:/var/lib/crowdsec/data/
- crowdsec-config:/etc/crowdsec/
networks:
crowdsec_test:
ipv4_address: 172.20.0.4
#metabase, because security is cool, but dashboards are cooler
dashboard:
#we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
build: ./crowdsec/dashboard
restart: always
ports:
- 3000:3000
environment:
MB_DB_FILE: /data/metabase.db
MGID: "${GID-1000}"
depends_on:
- 'crowdsec'
volumes:
- crowdsec-db:/metabase-data/
networks:
crowdsec_test:
ipv4_address: 172.20.0.5
volumes:
logs:
crowdsec-db:
crowdsec-config:
networks:
crowdsec_test:
ipam:
driver: default
config:
- subnet: 172.20.0.0/24The reverse-proxy (nginx) container writes its logs to a logs volume mounted by the crowdsec container.
CrowdSec’s SQLite database is in a crowdsec-db volume mounted by the dashboard (metabase) container
Initial deployment
Prerequisites: Docker / Docker Compose
We have put the configuration files altogether on this repository, so that you can simply clone it to deploy.
From the Docker Compose directory, you can deploy with docker-compose up -d and then check that everything is running with docker-compose ps.
▶ git clone https://github.com/crowdsecurity/example-docker-compose
▶ cd example-docker-compose
▶ sudo docker-compose up
▶ sudo docker-compose pssudo dnf install crowdsec# cd examples/docker-compose
# docker-compose up -d
...
# docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------docker-compose_app_1 httpd-foreground Up 80/tcp
docker-compose_crowdsec_1 /bin/sh -c /bin/sh docker_ ... Up
docker-compose_dashboard_1 /app/run_metabase.sh Up 0.0.0.0:3000->3000/tcp
docker-compose_reverse-proxy_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:8000->80/tcpLet’s check and make sure everything is working!
Demo app check
With the following command, we can check whether access to our demo app is working properly.
▶ curl http://localhost:8000/
Hello world !%Demo app check
We need to validate whether our CrowdSec setup reads logs as expected.
docker-compose exec crowdsec cscli metrics▶ sudo docker-compose exec crowdsec cscli metrics
INFO[25-02-2021 03:38:50 PM] Buckets Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-crawl-non_statics | - | - | 2 | 2 | 2 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[25-02-2021 03:38:50 PM] Acquisition Metrics:
+-----------------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/nginx/example.access.log | 2 | 2 | - | 2 |
+-----------------------------------+------------+--------------+----------------+------------------------+
INFO[25-02-2021 03:38:50 PM] Parser Metrics:
+--------------------------------+------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs | 6 | 2 | 4 |
| child-crowdsecurity/nginx-logs | 2 | 2 | - |
| crowdsecurity/dateparse-enrich | 2 | 2 | - |
| crowdsecurity/geoip-enrich | 2 | 2 | - |
| crowdsecurity/http-logs | 2 | - | 2 |
| crowdsecurity/nginx-logs | 2 | 2 | - |
| crowdsecurity/non-syslog | 2 | 2 | - |
+--------------------------------+------+--------+----------+
INFO[25-02-2021 03:38:50 PM] Local Api Metrics:
+--------------------+--------+------+
| ROUTE | METHOD | HITS |
+--------------------+--------+------+
| /v1/watchers/login | POST | 2 |
+--------------------+--------+------+What happened and what is relevant here?
The command cscli metrics queries the Prometheus metrics exposed locally by CrowdSec and presents them in a fancy terminal output:
- The “acquisition metrics” show us that our requests are indeed generating logs that are being read (“LINES READ”), parsed (“LINES PARSED”) and even matched with installed scenarios (“LINES POURED TO BUCKET”)
- The “buckets metrics” and “parser metrics” allow us to see which parsers and scenarios are being triggered
CrowdSec configuration check
The command cscli hub list allows us to see which parsers and scenarios are deployed.
▶ sudo docker-compose exec crowdsec cscli hub list
INFO[25-02-2021 03:46:41 PM] Loaded 14 collecs, 19 parsers, 23 scenarios, 3 post-overflow parsers
INFO[25-02-2021 03:46:41 PM] unmanaged items : 20 local, 0 tainted
INFO[25-02-2021 03:46:41 PM] PARSERS:
-------------------------------------------------------------------------------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
-------------------------------------------------------------------------------------------------------------
crowdsecurity/sshd-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/dateparse-enrich ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
crowdsecurity/geoip-enrich ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/nginx-logs ✔️ enabled 0.2 /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
crowdsecurity/http-logs ✔️ enabled 0.4 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
-------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] SCENARIOS:
--------------------------------------------------------------------------------------------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
--------------------------------------------------------------------------------------------------------------------------
ltsich/http-w00tw00t ✔️ enabled 0.1 /etc/crowdsec/scenarios/http-w00tw00t.yaml
crowdsecurity/http-crawl-non_statics ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-crawl-non_statics.yaml
crowdsecurity/http-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-probing.yaml
crowdsecurity/http-path-traversal-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-path-traversal-probing.yaml
crowdsecurity/http-xss-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-xss-probing.yaml
crowdsecurity/http-bad-user-agent ✔️ enabled 0.3 /etc/crowdsec/scenarios/http-bad-user-agent.yaml
crowdsecurity/ssh-bf ✔️ enabled 0.1 /etc/crowdsec/scenarios/ssh-bf.yaml
crowdsecurity/http-backdoors-attempts ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-backdoors-attempts.yaml
crowdsecurity/http-sensitive-files ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sensitive-files.yaml
crowdsecurity/http-sqli-probing ✔️ enabled 0.2 /etc/crowdsec/scenarios/http-sqli-probing.yaml
--------------------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] COLLECTIONS:
------------------------------------------------------------------------------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
------------------------------------------------------------------------------------------------------------
crowdsecurity/sshd ✔️ enabled 0.1 /etc/crowdsec/collections/sshd.yaml
crowdsecurity/base-http-scenarios ✔️ enabled 0.3 /etc/crowdsec/collections/base-http-scenarios.yaml
crowdsecurity/linux ✔️ enabled 0.2 /etc/crowdsec/collections/linux.yaml
crowdsecurity/nginx ✔️ enabled 0.1 /etc/crowdsec/collections/nginx.yaml
------------------------------------------------------------------------------------------------------------
INFO[25-02-2021 03:46:41 PM] POSTOVERFLOWS:
--------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
--------------------------------------
--------------------------------------Metabase check
Metabase is one of the components that has been deployed, which helps us generate dashboards for better observability. You can hop onto http://127.0.0.1:3000/ and log in with crowdsec@crowdsec.net and password !!Cr0wdS3c_M3t4b4s3??
Metabase comes with a default password because of how it is deployed. Remember to change the default password and restrict the metabase’s access to relevant IP addresses or network ranges.
At first, dashboards will be empty since no attacks were detected yet. The main one should look like this:
If any of those checks have failed for you, take a look at container logs with docker-compose logs crowdsec (for example).
Detection features
Note: In real-world setups, whitelists are deployed to prevent banning private IPs.
After checking to make sure everything is ready to go, let’s try some detection features. As we work with an exposed HTTP service, let’s fire a Nikto from another machine in the LAN nikto -host http://192.168.2.227:8000
Note: The IP is dependent on your LAN setup and addressing plan.
On the other hand, we kept an eye on CrowdSec’s logs through the following command.
docker-compose logs -f crowdsec
Here we can spot our client’s IP (192.168.2.211) being flagged for triggering various scenarios:
- crowdsecurity/http-bad-user-agent: the IP comes from a known bad user agent
- crowdsecurity/http-probing: tried to access a lot of distinct non-existent files
- crowdsecurity/http-crawl-non_statics: attempted to access a lot of distinct non-static ressources
- crowdsecurity/http-sensitive-files: IP tried to access a lot of sensitive files
- crowdsecurity/http-path-traversal-probing: IP tried to perform a path traversal attack
We can see my IP is banned with docker-compose exec crowdsec cscli decisions list.
We can review and inspect alerts with docker-compose exec crowdsec cscli alerts list and docker-compose exec crowdsec cscli alerts inspect -d XX:
Note: cscli alerts list returns the list of all alerts triggered.
▶ sudo docker-compose exec crowdsec cscli alerts inspect -d 43
################################################################################################
- ID : 43
- Date : 2021-02-26T08:26:07Z
- Machine : ee0ebe5b529c4995964ff0b3e01b1801sxSpiCdYj9lpSD9W
- Simulation : false
- Reason : crowdsecurity/http-sensitive-files
- Events Count : 5
- Scope:Value: Ip:192.168.2.211
- Country :
- AS :
- Active Decisions :
+-----+------------------+--------+--------------------+----------------------+
| ID | SCOPE:VALUE | ACTION | EXPIRATION | CREATED AT |
+-----+------------------+--------+--------------------+----------------------+
| 802 | Ip:192.168.2.211 | ban | 3h53m15.124782708s | 2021-02-26T08:26:07Z |
+-----+------------------+--------+--------------------+----------------------+
- Events :
- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
| KEY | VALUE |
+---------------+-----------------+
| ASNNumber | 0 |
| http_args_len | 0 |
| log_type | http_access-log |
| service | http |
| source_ip | 192.168.2.211 |
| http_status | 404 |
| http_path | /CyNqbugR.bak |
| IsInEU | false |
+---------------+-----------------+
- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
| KEY | VALUE |
+---------------+-----------------+
| log_type | http_access-log |
| service | http |
| source_ip | 192.168.2.211 |
| http_status | 404 |
| http_path | /CyNqbugR.sql |
| IsInEU | false |
| ASNNumber | 0 |
| http_args_len | 0 |
+---------------+-----------------+
- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-----------------+
| KEY | VALUE |
+---------------+-----------------+
| service | http |
| source_ip | 192.168.2.211 |
| http_status | 404 |
| http_path | /CyNqbugR.exe |
| IsInEU | false |
| ASNNumber | 0 |
| http_args_len | 0 |
| log_type | http_access-log |
+---------------+-----------------+
- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+-------------------+
| KEY | VALUE |
+---------------+-------------------+
| IsInEU | false |
| ASNNumber | 0 |
| http_args_len | 0 |
| log_type | http_access-log |
| service | http |
| source_ip | 192.168.2.211 |
| http_status | 404 |
| http_path | /CyNqbugR.printer |
+---------------+-------------------+
- Date: 2021-02-26 08:26:07 +0000 UTC
+---------------+--------------------+
| KEY | VALUE |
+---------------+--------------------+
| ASNNumber | 0 |
| http_args_len | 0 |
| log_type | http_access-log |
| service | http |
| source_ip | 192.168.2.211 |
| http_status | 404 |
| http_path | /CyNqbugR.htaccess |
| IsInEU | false |
+---------------+--------------------+Note: cscli alerts inspect -d
Monitoring activities with dashboards
Now that we have triggered several scenarios, we can go back to our Metabase dashboards (http://127.0.0.1:3000 with the default setup) and check the activity.
If the traffic came from a public IP (rather than a private one, as in this example), crowdsecurity/geoip-enrich would have enriched events with geo-localization data and AS/range information.
Block attacks with bouncers
Now that we have a fully functional CrowdSec service, we can detect incoming attacks on our service based on installed collection scenarios.
After detecting these attacks, our objective is to block them. You will use the cs-firewall-bouncer to achieve this. Start by installing it on the host, and block malevolent traffic directly in the DOCKER-USER chain, the default chain created by Docker to filter traffic targeting the containers.
You can find the firewall bouncer on the CrowdSec Hub. As of today, the most up-to-date version is v0.10.
wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz
tar xvzf cs-firewall-bouncer.tgz
cd cs-firewall-bouncer-v0.0.10
sudo ./install.shSince July, custom and firewall bouncers are packaged with Debian, Ubuntu, CentOS, RHEL and Amazon Linux. Please check our GitHub repo for more information. The install is straightforward. It will deploy a systemd unit for the service and make sure you meet the requirements. Here, I didn’t have ipset running, and it installed it for me.
Here, we installed the bouncer on a host where CrowdSec isn’t running. As a result, the service isn’t happy.
Now, let’s configure the bouncer to speak with the Local API running on our CrowdSec container. We start by creating an API token for our bouncer with cscli.
docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer
▶ sudo docker-compose exec crowdsec cscli bouncers add HostFirewallBouncerApi key for 'HostFirewallBouncer':
aaebb3708fe67eeeccbb52a21e5e7862
Please keep this key since you will not be able to retrive it!Then, you need to configure the bouncer to use this token to authenticate with CrowdSec’s Local API. In the /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml, edit api_url, api_key, and iptables_chains. In this case, IPv6 was also disabled with disable_ipv6:
mode: iptables
piddir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
api_url: http://172.20.0.4:8080/
api_key: aaebb3708fe67eeeccbb52a21e5e7862
disable_ipv6: true
#if present, insert rule in those chains
iptables_chains:
- DOCKER-USERNote: We edited the chains to only have DOCKER-USER, and we set api_url accordingly to our docker-compose.yml file, along with the newly generated api token.
You shall not pass: CrowdSec in action
Now, we can get our freshly configured bouncer started with sudo systemctl start cs-firewall-bouncer.service and take a look at our new firewall configuration:
▶ sudo iptables -L -n
...
Chain DOCKER-USER (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 match-set crowdsec-blacklists
src
...We can see that our DOCKER-USER chain has been populated with a rule to match incoming traffic against our ipset, and our ipset is filled with relevant information.
▶ sudo ipset -L crowdsec-blacklists
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 300
Size in memory: 6016
References: 1
Number of entries: 61
Members:
178.20.157.98 timeout 80103
52.184.35.59 timeout 80103
...As you might have noticed by now, our ipset isn’t only filled with our local decisions but also with ones from the community. However, we can see that our “local” decisions made their way to the ipset list.
▶ sudo ipset -L crowdsec-blacklists | grep 192
192.168.2.211 timeout 12859Now, we can check the attacker machine and see we are blocked and cannot access the application.
$ curl -vv 192.168.2.227:8000
* Rebuilt URL to: 192.168.2.227:8000/
* Trying 192.168.2.227...
* TCP_NODELAY set
^CThe attacker is prevented from accessing all Docker Compose applications. We limited the decision to the DOCKER-USER chain on purpose, which means local applications exposed by the host would still be accessible. If we wish to extend the ban to all incoming traffic, we could have added the INPUT chain to the list, as is the case with the default setup.
Conclusion
Throughout this tutorial, we deployed a minimal yet complete applicative stack using Docker Compose. Then, we covered how to secure it with CrowdSec. While most people will use CrowdSec as a host-based defense mechanism, we can see that it’s also suitable for Docker environments. If you want to meet with the team to share your feedback, you can find us on our Gitter channel.
About the author
Thibault Koechlin graduated from EPITECH, specializing in the security of IT systems & networks. He started his career at NBS in 2004, as an expert in penetration testing before being appointed Head of the offensive security team. He then became CISO by expanding his skills around defensive security before initiating the development of several open-source products and building teams with rare skills. He completed his ascent within the company through an operational partner role, leading the creation of the company’s flagship product: Cerberhost. In December 2019, he co-founded CrowdSec with Philippe Humeau and Laurent Soubrevilla. He is the CTO of the company.
