CrowdSec v.1.0 is out!
The team is happy to announce the official release of CrowdSec v.1.0.x which introduces several improvements to the previous version, including a major architectural change: the introduction of a local REST API.
This local API allows all components to communicate together in a more efficient way, supporting more complex architectures, while keeping it simple for mono-machines users. It also makes the creation of bouncers (the remediation component) much simpler and renders them more resilient to upcoming changes, limiting the necessary maintenance time.
> A quick look in the mirror
Prior to v1.0, the various components of CrowdSec were communicating directly with the database as shown below:
(note: red circles show where the database was communicating directly with others elements)
While it was a good way for us to quickly create a prototype, it brought several limitations that we wanted to remove as soon as possible.
> The star of the show: the local API
In the new 1.0 release, the CrowdSec architecture has been deeply remodeled:
All CrowdSec components (the agent reading logs, cscli for humans, and bouncers to deter the bad guys) can now communicate together via a REST API, instead of reading or writing directly in the database. With this new version, only the local API service will interact with the database (supports SQLite, PostgreSQL and MySQL).
> What it brings to the table
The benefits of having a REST API for the bouncers are major:
- Bouncers become truly stateless, making it possible to build new ones for serverless/lambda environments
- Future changes in DB schemas won’t impact bouncers, and API versioning ensures smooth control over this
- Bouncer implementation becomes really quick and easy: just one HTTP GET request with an API token is needed
- Regarding CrowdSec and cscli, having a REST API to write and read decisions & alerts brings the following benefits:
- It allows distributed setups: more advanced users can run the local API on one machine, while CrowdSec agent(s) are crunching logs on several others, using bouncers here & there
- It is now possible to build more scalable setups by splitting responsibilities across several components
Overall, these changes make CrowdSec much more maintainable in the long run as it allows the abstraction of data models and database schemas. It also introduces API versioning to ease the detection of breaking changes and enables distributed setups.
Last but not least, having the bouncers query the decision via a REST API allows us to expose a similar endpoint on “our” central API (the non open-source one). The goal is to authorize users that can’t or don’t want to run the CrowdSec agent itself to only use bouncers and still protect their assets by leveraging the CrowdSec global IP reputation database.
> But there is more!
Better auditability of what is sent & received
One of the legitimate fears of early adopters is data privacy. While we’re not sharing more than the strict necessary to curate the blocklist distributed to users, it is always important to be able to walk the talk. The newly released local API is the only component communicating back with our central API. This gave us a good reason to document all of this properly. As a result, the documentation can be found here, but since trust does not exclude control, you can now easily check all of this yourself. After a detection, we only send 3 variables back to the API: the timestamp of an attack, the scenario and the offending IP.
Introduction of a multi-tenant architecture
With the local API introduction and the reflected changes in the dashboards, it is now possible to create proper multi-tenant architectures, as each machine is pushing alerts with its own identifier, allowing for proper reporting and such.
Some users reported slow cscli commands when the amount of active decisions/alerts was too high. With this version, the database backend has been completely revamped:
- Exit GoRM, welcome entgo: we switched from the gorm ORM to entgo, which lead, amongst other things, to a substantial performance improvement
- We’re now using gin as our API service, and while we haven’t yet performed a proper benchmark that we can publish, we’re very happy with the first results
- Separating the log processing (CrowdSec agent) from the decision read/write (local API) allows for a better separation of tasks and roles
Various changes were made to the documentation, aiming to make it more comprehensive and accessible. We made sure to save former versions’ documentation in case of interest.
Cscli commands reorganization
Some of the commands within cscli were moved to create a more coherent ensemble.