Welcome to the second part of our tutorial on how to set up and secure a multi-server CrowdSec Security Engine installation. In the first part, I walked you through the setup of CrowdSec Security Engines across multiple servers, with one server serving as the parent and two additional machines forwarding alerts to it.
In this part, I will address security issues posed by clear HTTP communication in the previous multi-server Security Engine installation. To solve this, I propose establishing the communication between Security Engines over encrypted channels. This solution allows server-2 or server-3 to trust the server-1 identity and avoid man-in-the-middle attacks.
Using self-signed certificates
Create the certificate
First, you need to create a certificate. This can be achieved with the following one-liner.
openssl req -x509 -newkey rsa:4096 -keyout encrypted-key.pem -out cert.pem -days 365 -addext "subjectAltName = IP:172.31.100.242
For now, the Security Engine is not able to ask for the passphrase of the private key when starting. So, you have the choice to decipher the private key by hand each time you start or reload the Security Engine or store the key unencrypted. In any way, to strip the passphrase, you can use the following:
openssl rsa -in encrypted-key.pem -out key.pem
Then, the unencrypted key file can be safely deleted after the Security Engine is started.
Configure the Security Engine to use a self-signed certificate
On server-1 you need to configure the Security Engine to use the generated certificate. As seen below, the tls.cert_file and tls.key_file option in the api.server section of the following /etc/crowdec/config.yaml excerpt is set to the generated certificate file.
api:
server:
log_level: info
listen_uri: 10.0.0.1:8080
profiles_path: /etc/crowdsec/profiles.yaml
online_client: # Crowdsec API credentials (to push signals and receive bad
tls:
cert_file: /etc/crowdsec/ssl/cert.pem
key_file: /etc/crowdsec/ssl/key.pem
On the client side, configuration changes happen in two files. First, modify /etc/crowdec/config.yaml to accept self-signed certificates by setting the insecure_skip_verify to true.
You also need to change HTTP for HTTPS in the /etc/crowdsec/local_api_credentials.yaml file in order to reflect the changes. This small change has to be done on all three servers (server-1, server-2 and server-3).
Note: Please keep in mind that this LAPI configuration has to be done on server-1 as well if it’s used as a log processor too.
url: https://10.0.0.1:8080/
login: [login]
password: [password]
Side note: Obviously using self-signed certificates doesn’t provide any confidence over ownership on the LAPI server. Servers using the service (server-2 or server-3 in this setup) are still vulnerable to man-in-the-middle attacks, but at least this setup provides encrypted communications. That’s the reason why the InsecureSkipVerify option is needed.
Using a Certificate Authority-issued certificate
Let’s Encrypt, or services like Amazon ACM, can be leveraged to workaround the InsecureSkipVerify, by issuing a certificate for a fully qualified domain name that can be added to /etc/hosts or to a local DNS server. /etc/crowdsec/local_api_credentials.yaml can then be filled with this specified fully qualified domain name.
This indeed works and prevents the InsecureSkipVerify option from being set. This ensures that communication between client and server can’t be tampered with as long as the DNS configuration can be trusted, but should still be considered as a workaround.
Using a PKI
The process of configuring and managing an SSL Public Key Infrastructure (PKI) falls outside the scope of this tutorial, but I highly recommend you take a look at the official OpenSSL documentation. The simple PKI scenario is enough for this Security Engine setup.
Following the OpenSSL documentation, there are a few things worth mentioning.
To be usable in our CrowdSec TLS scenario, the certificate requests have to be issued with a subject alternative name corresponding to the IP of the Crowdsec LAPI server. This can be done by positioning the SAN environment variable when invoking OpenSSL for the certificate request (see step 3.3 in the OpenSSL simple PKI scenario).
SAN=IP:10.0.0.1 openssl req -new -config etc/server.conf -out certs/crowdsec.csr -keyout certs/crowdsec.key
The public part of the root and the signing certificates (bundle file created at step 4.5 in the OpenSSL simple PKI scenario) have to be added to the local certificate store before starting the CrowdSec Security Engine. In this setup, this is required to connect to the LAPI server. There’re many ways to do so, golang sources specify where certificates are expected, or you can use the SSL_CERT_FILE environment variable in the systemd service file to specify where to find the certificate when launching the Security Engine.
Updated note on CrowdSec and TLS authentication
After the first publication of this article, we added a new feature to the Security Engine you are now able not only to secure communication over TLS but also ensure authentication with certificate. In the official documentation you can find a great example that shows how TLS authentication can be done using certificates between Security Engines or between Security Engine and Remediation Component.
Conclusion
This article gives some highlights on how to secure communications between different CrowdSec Security Engine installations. The considered use case is Security Engine installations in a private network, but this can also be deployed on a public network with communication over the internet. In such a case, a third-party certificate would easily do the trick.
Depending on the needs, I proposed three different ways to achieve secure TLS communications between your Security Engines — using self-signed certificates, using certificates issued by a Certificate Authority, and using an SSL Public Key Infrastructure.
The first scenario, with self-signed certificates, only applies if you want to ensure encrypted communication with no need for authentication. The second scenario proposed may only be considered as a workaround when you have the possibility to modify local DNS resolutions. The third proposed scenario is the most complicated but would fit in most use cases and may be the way to go when security concerns are high.
I hope this tutorial comes in handy. Thanks for reading and stay tuned!
If you have any questions or feedback, don’t hesitate to reach out to us on our community platforms on Discord and Discourse.
