May 10, 2022

How to mitigate security threats with CrowdSec in Kubernetes using Traefik

Introduction

Previously we published 2 articles part1 & part2 covering Kubernetes Crowdsec integration with Nginx as an ingress controller. Now we will explain how to integrate Crowdsec in a k8s cluster with Traefik as an ingress controller to increase the level of security.

In this article, we’ll set up a k8s cluster locally using Kind and set up ingress using the Traefik ingress controller. The latter acts as a modern HTTP reverse proxy and a load balancer that simplifies the deployment of microservices. Then we will install CrowdSec to parse Traefik ingress logs and install the Crowdsec Traefik bouncer to remediate the attacks on the ingress controller.

Special thanks to goes to Fabien, the Crowdsec Traefik bouncer developer, and ambassador who made this integration a reality. Read Fabien’s interview for CrowdSec here.

Before we deep dive into the tutorial. We invite you to join our webinar with Traefik Labs where we will show you how to integrate CrowdSec into your Kubernetes cluster with Traefik as an ingress controller to detect and remediate security threats. You will get a chance to ask the CrowdSec team questions you may have. Register here.

Prerequisites

Before you start this step-by-step guide, make sure you have:

If you are ready, let’s get started!

Setup environment

Deploying K8s cluster

Kind cluster configuration kind.yaml

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 30000
    hostPort: 80
    protocol: TCP
  - containerPort: 30001
    hostPort: 443
    protocol: TCP

Create kind cluster:

kind create cluster --config kind.yaml

Deploying Traefik Ingress Controller

Traefik Ingress helm values ingress-traefik-kind-values.yaml

image:
  name: traefik
  pullPolicy: IfNotPresent
logs:
  access:
    enabled: true
service:
  type: NodePort
ports:
  traefik:
    expose: true
  web:
    nodePort: 30000
  websecure:
    nodePort: 30001
nodeSelector:
  ingress-ready: 'true'
providers:
  kubernetesCRD:
    allowCrossNamespace: true
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Equal
    effect: NoSchedule

Install Traefik Ingress:

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install -n ingress-traefik ingress-traefik traefik/traefik -f ./ingress-traefik-kind-values.yaml --create-namespace

Microservice example app (HelloWorld)

Helloworld app helm values configuration helloworld-values.yaml:

ingress:
  enabled: true
  # If you don't enable the bouncer as global middleware,
  # you can add it per ingress config like this:
  # annotations:
  #   traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-traefik-bouncer@kubernetescrd

Install the helloworld app (included in the Crowdsec helm charts repo):

helm repo add crowdsec https://crowdsecurity.github.io/helm-charts
helm repo update
helm install helloworld crowdsec/helloworld -f helloworld-values.yaml

Don’t forget to edit /etc/hosts to be able to access the helloworld app:

172.17.0.1      helloworld.local

Then try to access it

$ curl -i http://helloworld.local
HTTP/1.1 200 OKContent-Length: 13
Content-Type: text/plain; charset=utf-8
Date: Fri, 08 Apr 2022 13:46:03 GMT
X-App-Name: http-echo
X-App-Version: 0.2.3

helloworld !

Installing CrowdSec

Before installing Crowdsec, let’s create an account and connect to the Console, an easy-to-use web interface to inspect multiple CrowdSec agent signals spread across different networks, to have better visualization of our alerts.

To link our Crowdsec instance to the console, we need to enroll it. So, retrieve the enrollment key from the console by clicking on add instance once connected, then copy the enrollment key provided (see the screenshot below).

In our example, we’ll add an instance_name as k8s_cluster and some tags (linux, k8s, test) to easily find our instance in the Console.

CrowdSec helm values configuration crowdsec-values.yaml

container_runtime: containerd
container_runtime: containerd
agent:
  # To specify each pod you want to process it logs (pods present in the node)
  acquisition:
    # The namespace where the pod is located
    - namespace: ingress-traefik
      # The pod name
      podName: ingress-traefik-*
      # as in crowdsec configuration, we need to specify the program name so the parser will match and parse logs
      program: traefik
  # Those are ENV variables
  env:
    # As we are running Nginx, we want to install the Nginx collection
    - name: PARSERS
      value: "crowdsecurity/cri-logs"
    - name: COLLECTIONS
      value: "crowdsecurity/traefik"
    - name: DISABLE_PARSERS
      value: "crowdsecurity/whitelists"
  persistentVolume:
    config:
      enabled: false
lapi:
  dashboard:
    enabled: false
    ingress:
      host: dashboard.local
      enabled: true
  env:
    - name: ENROLL_KEY
      value: ""
    - name: ENROLL_INSTANCE_NAME
      value: "k8s_cluster"
    - name: ENROLL_TAGS
      value: "k8s linux test"
    # If it's a test, we don't want to share signals with CrowdSec so disable the Online API.
    #- name: DISABLE_ONLINE_API
    #  value: "true"

Install CrowdSec helm:

helm install crowdsec crowdsec/crowdsec -f crowdsec-values.yaml -n crowdsec --create-namespace

After installing, CrowdSec starts running and if we go back to our console, we can see there is a new instance to accept (see screenshot below).

Accept the instance and restart the crowdsec-lapi-*, so the signals will begin to be available in the console.

kubectl -n crowdsec delete pod crowdsec-lapi-6ccc959cdb-x8h76

We will now try to attack the helloworld app and see if CrowdSec detects and raises a ban against the attack.

Let’s attempt the attack using nikto:

$ ./nikto.pl -host http://helloworld.local/
...

Then get shell on the crowdsec-lapi pod and see if there is a decision

$ kubectl -n crowdsec exec -it crowdsec-lapi-fc44857bf-gf8jv -- sh
/ #
/ # cscli decisions list
+----+----------+---------------+---------------------------------------+--------+---------+----+--------+--------------------+----------+
| ID |  SOURCE  |  SCOPE:VALUE  |                REASON                 | ACTION | COUNTRY | AS | EVENTS |     EXPIRATION     | ALERT ID |
+----+----------+---------------+---------------------------------------+--------+---------+----+--------+--------------------+----------+
|  9 | crowdsec | Ip:10.244.0.1 | crowdsecurity/f5-big-ip-cve-2020-5902 | ban    |         | 0  |      1 | 3h58m57.535543005s |        9 |
+----+----------+---------------+---------------------------------------+--------+---------+----+--------+--------------------+----------+
8 duplicated entries skipped

We can see that CrowdSec detected multiple attacks (it only shows the last attack type). Using the Console, we can already see the alerts (see screenshot below).

Since the cluster is installed locally, we have private IP in the alerts.

Now we need to block this IP on the Traefik ingress controller. Still in the crowdsec-lapi shell, generate a bouncer API key.

/ # cscli bouncers add traefik-ingress
INFO[08-04-2022 02:32:47 PM] push and pull to Central API disabled
Api key for 'traefik-ingress':

  882882ac8acdf60dacc008dd3de68cf0

Please keep this key since you will not be able to retrieve it!

Install CrowdSec Traefik bouncer

Traefik CrowdSec middleware bouncer k8s configuration crowdsec-traefik-bouncer-values.yaml

The bouncer needs the API key generated previously and the CrowdSec local API endpoint service.

bouncer:
  crowdsec_bouncer_api_key: 882882ac8acdf60dacc008dd3de68cf0
  crowdsec_agent_host: "crowdsec-service.crowdsec.svc.cluster.local:8080"

Install bouncer helm (in the same namespace as the Traefik ingress controller)

helm install -n ingress-traefik traefik-bouncer crowdsec/crowdsec-traefik-bouncer -f crowdsec-traefik-bouncer-values.yaml

Now the bouncer is installed, it will show on helm notes how to integrate it as middleware in Traefik.

# Global configuration (middleware for all Ingress and IngressRoutes)

To add traefik bouncer as middleware globally, you need to add this configuration below to your traefik helm values and upgrade.

---
additionalArguments:  
  - "--entrypoints.web.http.middlewares=ingress-traefik-traefik-bouncer@kubernetescrd"
  - "--entrypoints.websecure.http.middlewares=ingress-traefik-traefik-bouncer@kubernetescrd"
---

# Configuration by Ingress or IngressRoute
To add traefik crowdsec bouncer as a middleware. You need to add following your ingress type:

**Kubernetes Ingress** :
---
ingress:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-traefik-bouncer@kubernetescrd
---

**Kubernetes IngressRoute** :
---
spec:
  routes:
    middlewares:
      - name: ingress-traefik-traefik-bouncer@kubernetescrd
---

We choose to install it as a global middleware (for all my applications). So we need to upgrade Traefik helm values (ingress-traefik-kind-values.yaml) adding:

additionalArguments:
  - "--entrypoints.web.http.middlewares=ingress-traefik-traefik-bouncer@kubernetescrd"
  - "--entrypoints.websecure.http.middlewares=ingress-traefik-traefik-bouncer@kubernetescrd"

Now upgrade command:

helm upgrade -n ingress-traefik ingress-traefik traefik/traefik -f ./ingress-traefik-kind-values.yaml

Now we can check in the Traefik interface that there is a new middleware. For that we need to set a port-forward so we can access to the Traefik dashboard.

kubectl -n ingress-traefik port-forward ingress-traefik-f44768cd4-gz9kr 9000:9000 &

And go to: http://localhost:9000/dashboard/#/http/middlewares. You will see the Crowdsec Traefik bouncer as a new middleware available.

Also all the routers on entrypoints web and websecure will have the new middleware enabled by default.

Remediation

As we already used nikto and attacked our helloworld app, CrowdSec already raised a decision against our IP. We are now blocked by the middleware.

$ curl -i http://helloworld.local
HTTP/1.1 403 Forbidden
Content-Length: 9
Content-Type: text/plain; charset=utf-8
Date: Fri, 08 Apr 2022 14:45:59 GMT

We can unban our IP and retry to access it again:

$ k -n crowdsec exec -it crowdsec-lapi-fc44857bf-8ln6j -- sh
/ # cscli decisions delete --ip 10.244.0.1
INFO[08-04-2022 02:54:56 PM] 9 decision(s) deleted

$ curl -i http://helloworld.local
HTTP/1.1 200 OK
Content-Length: 13
Content-Type: text/plain; charset=utf-8
Date: Fri, 08 Apr 2022 14:55:09 GMT
X-App-Name: http-echo
X-App-Version: 0.2.3

helloworld !

Conclusion

As you’ve seen, the integration is pretty easy to implement Crowdsec, Traefik and the bouncer which allow you to have a powerful and secure ingress controller. Join our joint webinar with Traefik Labs to see how to mitigate security threats with CrowdSec and Traefik.

Sign up here. This will learn how to integrate CrowdSec into your Kubernetes cluster with Traefik as an ingress controller to detect and remediate security threats and get to ask questions you may have! Join us on May 18.