Scalable and Low-Friction Authentication for MSSPs with the CrowdSec IDPS

Expanding operations for Managed Security Service Providers (MSSPs) involves meticulous planning that can span weeks or even months, yet unforeseen challenges may still emerge during the deployment phase. 

This article aims to illuminate strategies for overcoming obstacles that prevented a small MSSP from efficiently implementing CrowdSec’s behavior-based IDPS and the CrowdSec Remediation Components on exposed workloads. By the end of this process, our MSSP user gained the ability to implement a scalable, low-friction authentication mechanism that proved invaluable.

Note: To protect the anonymity of our user, we will not disclose the name of the MSSP involved in this use case. 

The challenge

To give you insight into our journey toward the ideal solution, let’s start with how this MSSP initially got in touch with us. They reached out through our Discord channel, initially inquiring about a way to bypass authentication for Remediation Components. Their goal was to ensure that any request sent to the Local API (LAPI) from a Remediation Component would receive a response, regardless of whether a valid key was provided or not.

This initiated a significant debate within CrowdSec regarding whether it would be advisable to offer support for this feature. Concerns were raised about the potential risks to end users, particularly if they enable this option but neglect to secure the endpoint with proper firewall settings. Ultimately, it was concluded that we would not support this option. This decision was underpinned by the fact that we already facilitate TLS authentication, which enables Remediation Components to authenticate securely without necessitating a key, aligning better with the security needs and operational efficiencies sought by MSSPs.

Upon presenting our solution to this particular MSSP, their feedback was clear: they were searching for an approach that minimized both configuration and maintenance effort, which is why they had initially inquired about the possibility of eliminating authentication altogether. Given that the endpoints in question were read-only, they perceived the security risk as minimal. 

This perspective underscores the unique challenges MSSPs face in balancing security needs with the desire for streamlined, low-overhead solutions that can be deployed efficiently and maintained with minimal resource allocation.

The solution

In response to the MSSP’s preferences for simplicity and minimal maintenance, we explored multiple strategies, including the potential modification of CrowdSec’s code. However, our efforts led to a topic we have never explored that not only addressed the present challenge but also opened up new avenues for innovation — a topic we plan to delve even more into in a forthcoming article. 

This solution required no alterations to CrowdSec itself. The topic was particularly relevant given the MSSP’s infrastructure setup: they operated with a single Security Engine positioned alongside their log repository, which was essentially a straightforward WireGuard server. This server facilitated connections from multiple on-premise deployments, transmitting logs through rsyslog.

The solution we devised involved implementing a reverse proxy in front of the LAPI to handle authenticated TLS requests on behalf of the Remediation Components. In practice, the MSSP would configure their Remediation Components to direct requests to http(s)?:///. Upon receiving a request, the reverse proxy would then proxy it to the LAPI, enhancing the request with a valid certificate. This approach allowed the MSSP to benefit from the dual advantages of encrypted communication through the WireGuard tunnel and the ease of managing just the reverse proxy situated before the LAPI.

Upon discussing this internally, we also came to the conclusion that this solution can also be used in conjunction with Remediation Components that do not support TLS authentication — the most known is our Nginx Lua integration, but as I eluded to earlier, we will be covering this in a forthcoming article on how you can achieve it.

The implementation

In this part, I will demonstrate the solution we implemented to overcome the challenges faced by our MSSP user. You can replicate this solution for your systems, making the necessary adjustments.

Note: The user had already created their own Certificate Authority (CA) and the necessary certificates for TLS authentication, so I will skip this part of the process. If you wish to replicate this tutorial for your systems and you haven’t set those up, we provide comprehensive guidance in our dedicated documentation on utilizing CFSSL. Alternatively, if you prefer to use OpenSSL, there is extensive documentation available.

To get things started, it was essential to transition the CrowdSec Security Engine to utilize HTTPS; this involved configuring the engine to incorporate the CA and delineate the Organizational Units (OUs) permitted. This step ensures the engine can authenticate incoming certificates for access to read-only endpoints. Here is the snippet of the configuration values provided to the Security Engine:

api:
  server:
	trusted_proxies: # List of trusted IP addresses
  	  - “127.0.0.1” # Reverse Proxy exists on same machine
	tls:
  	  cert_file: /root/ca/intermediate/certs/server.cert.pem
  	  key_file: /root/ca/intermediate/private/server.key.pem
  	  ca_cert_path: /root/ca/intermediate/certs/ca-chain.cert.pem
  	  #client_verification: RequireAndVerifyClientCert
  	  bouncers_allowed_ou: #OU allowed for bouncers
    	    - bouncer-ou
          

Note: Bouncers are legacy terminology for Remediation Components that was updated within the Taxonomy Article.

So let’s break down the above configuration:

• trusted proxies: A pivotal step for informing the CrowdSec LAPI about the presence of an upstream proxy. When properly set up, the LAPI can retrieve the actual IP address of incoming requests from the X-Forwarded-For header provided by the proxy. This configuration is vital for the LAPI endpoints to function correctly, as it prevents the system from mistakenly identifying all Remediation Components as originating from a single entity.
• cert_file parameter: Refers to the file path of the server certificate, which I presumed you generated following the instructions in our CFSSL guide using server.json, or by employing the server_cert extension as outlined in the OpenSSL guide.
• key_file parameter: Specifies the location of the private key used during the generation of the cert_file.
• ca_cert_path parameter: Indicates the location of the CA certificate within your system. It’s essential to highlight that this setting typically refers to a bundled certificate file, which combines the intermediate and the root certificate assembled in that specific sequence. This configuration ensures that the chain of trust is established correctly, allowing the system to verify the authenticity of certificates presented by clients or servers during the SSL/TLS handshake process.
• bouncers_allowed_ou parameter: Specifies a list of OUs that are permitted within the certificates presented by clients. This setting plays a crucial role in the authentication process, as it ensures that only clients with certificates containing the specified OUs can interact with the system. Importantly, the bouncers_allowed_ou list must not be left empty; if a certificate does not include an OU or has an incorrect OU, it will necessitate the regeneration of the certificate

Before restarting the Security Engine to implement the new configuration, it was critical to ensure that there is no degradation in service during the time between then and the deployment and configuration of the reverse proxy. To maintain seamless operations, it was necessary to update the local_api_credentials.yaml file to include the bundled certificate. This update allowed the system to recognize and trust the new certificate presented by the LAPI.

url: “https://127.0.0.1:8080/” # Note this line has changed to https
username: xxxxx
password: xxxxx
ca_cert_path: “/root/ca/intermediate/certs/ca-chain.cert.pem” # Same bundle specified within configuration

Once the necessary preparations, including the update of the local_api_credentials.yaml to incorporate the bundled certificates, are complete, we restarted the CrowdSec Security Engine. 

Note: This restart should be executed using the method specific to your platform or environment.

To verify that the new configuration has been successfully implemented, it was important to check that the service status of the CrowdSec Security Engine is marked as running and to confirm that the log files have been updated with entries around the current timestamp.

Setting up a reverse proxy

After confirming the successful implementation and stability of the CrowdSec Security Engine configuration, the next step was setting up the reverse proxy. In this instance, Nginx was used as the web server due to the MSSP’s familiarity with its configuration and features. 

Note: The principles and configurations discussed here are likely applicable, with some adjustments, to other web servers as well.

To streamline the setup process, the Nginx reverse proxy was installed on the same machine as the Security Engine. The installation process varies depending on the operating system in use. In this use case, the platform of choice was Debian 12, so installing Nginx was straightforward and was done by executing a simple command: apt install nginx. This command fetches and installs the latest stable version of Nginx from the Debian repositories.

After successfully installing Nginx, the next step was configuring it to function effectively as a reverse proxy. The first configuration change we needed to make was modifying the root Nginx configuration file — typically located at /etc/nginx/nginx.conf — though the exact path can vary based on the operating system and Nginx setup. It’s important to consult the official documentation for your specific environment to locate this file accurately.

Nginx outlines the directives needed to add here, they are designed to identify the type of request incoming from the client and to assign a corresponding variable for use in the X-Forwarded-For header. This setup is particularly important for maintaining RFC compliance, especially in scenarios where the architecture might involve multiple upstream proxies. The goal here was to ensure that the reverse proxy correctly handles and forwards the client’s original IP address to the backend services, like the CrowdSec Security Engine, through the X-Forwarded-For header.

Next, we created a “snippet” file; it is an efficient approach to managing reusable configurations within Nginx. As the name implies, these snippets contain configuration directives that can be included in multiple server blocks or locations, facilitating a more streamlined and maintainable Nginx configuration.

For this specific setup, we stored the snippet in /etc/nginx/snippets/proxy.conf. The contents of this snippet file included directives that are common to the functioning of a reverse proxy, such as defining header fields and configuring the default proxy pass behaviors — for example:

proxy_http_version             	1.1;
proxy_cache_bypass             	$http_upgrade;
# Proxy SSL
proxy_ssl_server_name          	on;
# Proxy headers
proxy_set_header Upgrade       	$http_upgrade;
proxy_set_header Connection    	$connection_upgrade;
proxy_set_header X-Real-IP     	$remote_addr;
proxy_set_header Forwarded     	$proxy_add_forwarded;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host  $host;
proxy_set_header X-Forwarded-Port  $server_port;
# Proxy timeouts
proxy_connect_timeout          	60s;
proxy_send_timeout             	60s;
proxy_read_timeout             	60s;

I am not going to delve into each configuration option as this is specific to Nginx. Here is a link to the proxy module if you would like to know what each option does.

Finally, we configured the default server block since, for the MSSP, this reverse proxy is dedicated to the LAPI, so there are no subdomains to handle. After a few trials and errors, here is the final configuration file we came to:

upstream lapi {
    server 127.0.0.1:8080;
}    
server {
    listen 80;
    server_name _;
    include /etc/nginx/snippets/proxy.conf;
    location / {    
   	 proxy_pass https://lapi;
   	 proxy_ssl_session_reuse off;    
    }
    location ^~ /v1/decisions {
   	 proxy_pass https://lapi;
   	 proxy_ssl_certificate /root/ca/intermediate/certs/bouncer.cert.pem;
   	 proxy_ssl_certificate_key /root/ca/intermediate/private/bouncer.key.pem;   	 
    }

    proxy_ssl_name Crowdsec Server;
    proxy_ssl_protocols       	TLSv1.3;
    proxy_ssl_ciphers         	HIGH:!aNULL:!MD5;
    proxy_ssl_trusted_certificate /root/ca/intermediate/certs/ca-chain.cert.pem;
    proxy_ssl_verify    	on;
    proxy_ssl_verify_depth  2;
}

Again, I won’t delve into every detail of the directives but will highlight the critical elements from top to bottom. 

For the location /, it was crucial to set proxy_ssl_session_reuse to off. This adjustment was necessary because Nginx attempts to reuse the same SSL session for multiple requests to enhance performance. However, when implementing TLS authentication, any request directed toward a resource that the Remediation Components lack access to will be automatically rejected by the LAPI. 

In the subsequent location directive, a priority modifier, indicated by ^~, plays a crucial role. This notation signals to Nginx that any locations commencing with the specified string should take precedence in the configuration hierarchy, surpassing the default / route, which typically acts as a catch-all pathway. The reason this was needed is the LAPI has two decision endpoints: one for stream and one for “get all” — this is outlined in our LAPI Swagger.

Within this particular location directive, it was imperative to designate the appropriate certificates that would accompany the request, thereby authenticating the Remediation Components. Should you follow the guides previously outlined, you will find that for CFSSL, the relevant certificate is produced using bouncer.json, whereas, for OpenSSL, it’s generated through the usr_cert extension.

Next, proxy_ssl_name should match the OU defined on the server certificate. Within our CFSSL documentation, we use Crowdsec Server, so this is why we have that as the value. Following this, we defined proxy_ssl_trusted_certificate — this is the same bundled certificate we used within the Security Engine configuration, so we defined the same path as the reverse proxy exists on the same machine.

We could now finally test and reload the configuration file using the following command:

nginx -t && nginx -s reload

Note: If you are replicating this use case for your systems and you run into any issues with the test command, please take a moment to troubleshoot. Should you be unable to resolve the problem, we invite you to join our Discord server and create a ticket for assistance in one of the support channels.

Time to test

After completing all the configuration steps, we sent a request to the decision endpoint via the reverse proxy and received a response. In our test environment, our VM (Virtual Machine) was running on 192.168.121.71, and a simple curl request:

$ curl http://192.168.121.71/v1/decisions
null⏎

At that point, the LAPI had no decisions, hence the null response since it’s running inside a test environment. To rectify the situation, we added some test data via cscli and sent another curl request.

$ cscli decisions add –ip 10.10.10.10

$ curl http://192.168.121.71/v1/decisions
[{"duration":"3h59m32.607779967s","id":75044,"origin":"cscli","scenario":"manual 'ban' from '31d6f1cfc3674d439b23cd5446889c60UbiH5msYf5bzLbLp'","scope":"Ip","type":"ban","value":"10.10.10.10"}]

But we were not quite done yet. We needed to ensure that any requests to modify the data were rejected because Remediation Components did not have write permissions.

$ curl -X DELETE “http://192.168.121.71/v1/decisions?ip=10.10.10.10”
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.18.0
< Date: Tue, 12 Mar 2024 16:47:26 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 46
< Connection: keep-alive
< Www-Authenticate: JWT realm=Crowdsec API local
<
* Connection #0 to host 192.168.121.71 left intact
{"code":401,"message":"cookie token is empty"}⏎

As you can see, the LAPI responded correctly with a 401 response code since Remediation Components cannot send delete requests. So, since we have validated that the configuration was working as intended, we provided these to the MSSP, which then used Ansible to mass deploy the firewall Remediation Component and update the URL within the configuration to be the reverse proxy. 

Here is an image they sent us after rolling out the endpoints:

Wrap up

Throughout this article, we've explored the process of establishing a reverse proxy in front of the Security Engine to facilitate authentication across a scalable array of Remediation Components. 

The beauty of this setup lies in its minimalistic configuration requirements, making it an accessible and efficient solution for teams of any size. For MSSPs, the ability to implement such a scalable, low-friction authentication mechanism is invaluable.

If you follow this tutorial and implement this solution in your system, join our Discord server and share your results!