bruneaug/dshield-siemDShield Sensor Log Collection with ELK
This is fork and a significant update from the initial publication on the ISC Storm Center website by Scott Jensen as a BACS paper and the scripts published in Github.
https://github.com/fkadriver/Dshield-ELK
https://isc.sans.edu/diary/DShield+Sensor+Monitoring+with+a+Docker+ELK+Stack+Guest+Diary/30118
This is a good reference on howto use DShield-SIEM for analysis: https://isc.sans.edu/diary/30962/
This provides an overview how the log collection with the DShield sensor is done.
https://github.com/bruneaug/DShield-SIEM/blob/main/Troubleshooting/DShield-SIEM-Flow.png
This docker is custom built to be used with the DShield Honeypot to collect and parse the logs and collect the data to make it easy to search for research purposes. The suggested installation is to install the DShield sensor in a Rasperry using PI Raspbian OS or a system running Ubuntu 24.04 LTS either in your network or in the cloud of your choice.
Note: This must be installed on a separate server as per the instruction below.
Note: To update ELK server components, follow these steps:
https://github.com/bruneaug/DShield-SIEM/blob/main/README.md#download-github-update
Install docker as user ($) account:
sudo apt-get install ca-certificates curl gnupg network-manager txt2html
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
Note: You may need to take the next line, remove the backlash () and put everythin in a single line in Notepad to run this echo.
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt upgrade
sudo reboot (if update were applied)
sudo apt-get install -y jq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin pip
sudo systemctl enable docker
sudo systemctl start docker.service
Confirm the docker service is running
sudo systemctl status docker.service
Add the following ELK servername es01 kibana fleet-server heartbeat logstash filebeat filebeat01
to /etc/hosts after the IP address and the server name:
sudo vi /etc/hosts
Add this to /etc/hosts after the ELK server IP addresses
127.0.0.1 sensor es01 ...
192.168.25.80 sensor es01 ...
es01 kibana fleet-server heartbeat logstash filebeat filebeat01
Edit the sysctl.conf file:
sudo vi /etc/sysctl.conf
Add the following at the bottom of the file:
vm.max_map_count=262144
Add save it. This is for Elasticsearch when using Docker.
sudo iptables -nL
Using user ($) account, run the following commands:
git clone https://github.com/bruneaug/DShield-SIEM.git
mkdir scripts
mv DShield-SIEM/AddOnScripts/*.sh scripts
chmod 754 scripts/*.sh
cd ~/DShield-SIEM
Note: Before installation, you can edit the .env (ls -la to see it) file to make any derided changes.
- Current default password for elastic is student
Memory Limits in .env are the most memory that docker will allocate for each of the ELK containers.
Default to 3GB for Elasticsearch but can be expanded if you have the resources
Edit .env file and configure for your own ELK Stack
vi .env
cd logstash/pipeline
Edit heartbeat.yml to change the sensor IP (named PI) with your own sensor IP. Kibana heartbeat check the status of your services and determine whether they are available.
Update this IP and hostname to reflect your sensor.
Note: You can copy and add same 5 lines if you have more than one sensor you would like to monitor
This can be viewed Observability -> Uptime Monitors
This line is the sensor IP address. If it doesn't received a respose while connecting to TCP 12222, it will mark it as down in the interface.
This is initiated by the ELK server to check the response from the sensor.
Update this line: hosts: ["192.168.25.165"]
vi ~/DShield-SIEM/heartbeat/heartbeat.yml
The script change_perms.sh will configure some of the files before starting the installation of the docker components.
The script will also update the IP address used in the dashboard query to the IP of your ELK stack.
~/scripts/change_perms.sh
~/scripts/get_researchers.sh
Now execute docker compose to build the ELK server applications.
This will build: Kibana, Elasticsearch, elastic-agent, Logstash and load the Cowrie parsers, configuration files and dashboard.
sudo docker compose up -d
Installation Completed
sudo cp /var/lib/docker/volumes/dshield-elk_certs/_data/ca/ca.crt /usr/local/share/ca-certificates/dshield.crt
sudo update-ca-certificates
This loads the DShield SIEM dashboards and the SIEM signatures into Kibana.
~/scripts/dashboard-setup.sh
This command shows the list of docker directories in used after all of Elasticsearch components have been installed.
As data gets added to Elasticsearch, you can also monitor either with the command below or within ELK the amount of disk is available for storage.
sudo du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker
Using netstat, these 4 services should now be listening.
netstat -an | grep '80\|9200\|9300\|8220\|5601\|5044\|5066\|5067\|5068\|9600'
This should be the output you should see:
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN ---> Local CyberChef tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN ---> Kibana Web Access tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN ---> Local Attack Navigator tcp 0 0 0.0.0.0:8220 0.0.0.0:* LISTEN ---> elastic-agent Service tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN ---> Elasticsearch Service tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN ---> Elasticsearch Cluster Service tcp 0 0 0.0.0.0:5044 0.0.0.0:* LISTEN ---> Logstash Data Collection tcp 0 0 0.0.0.0:5066 0.0.0.0:* LISTEN ---> Metricbeat Stack Node Information tcp 0 0 0.0.0.0:5067 0.0.0.0:* LISTEN ---> Filebeat Stack Node Information tcp 0 0 0.0.0.0:5068 0.0.0.0:* LISTEN ---> Filebeat Stack Node Information (Process ISC Threat Intel to Logstash) tcp 0 0 0.0.0.0:9600 0.0.0.0:* LISTEN ---> Logstash Stack Node and Pipeline Information tcp6 0 0 :::80 :::* LISTEN tcp6 0 0 :::5601 :::* LISTEN tcp6 0 0 :::8080 :::* LISTEN tcp6 0 0 :::8220 :::* LISTEN tcp6 0 0 :::9200 :::* LISTEN tcp6 0 0 :::9300 :::* LISTEN tcp6 0 0 :::5044 :::* LISTEN tcp6 0 0 :::5066 :::* LISTEN tcp6 0 0 :::5067 :::* LISTEN tcp6 0 0 :::5068 :::* LISTEN tcp6 0 0 :::9600 :::* LISTEN
This will install the default templates and dashboards can be used to store netflow data from remote DShield sensors.
In order for Logstash to start storing data, the following commands must be executed. First step is to login filebeat:
sudo docker exec -ti filebeat bash
Next, run this Filebeat command to load the templates into Kibana:
./filebeat setup -e
After docker finish installing all the ELK docker components, now it is time to login the ELK stack using your Ubuntu server IP.
In order to see the ELK stack health, it is necessary to configure the stack monitoring by navigating to: Management -> Stack Monitoring
The fleet-agent (this is an elastic-agent running on the ELK stack) will be used to ingest threat intelligence.
sudo cp /var/lib/docker/volumes/dshield-elk_certs/_data/ca/ca.crt /tmp
sudo openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g
The output will look like this:
673FB617E15CCCE73F9B647EF99449642A19CFC1D75BF5772047DA99DB950844
Get Content of Elasticsearch CA Certificate to Apply to Advanced YAML configuration. Type the command because it doesn't copy well
sudo cat /tmp/ca.crt | sed -r 's/(.*)/ \1/g'
Follow the example from the Troubleshooting fleet-server-examples guide URL above for the correct format.
sed will add the 4 spaces with the previous command against the CA certificate
After adding the certificate information, save and apply these settings.
Followed by Save and deploy
Under Settings, configure the Fleet server hosts by either edit or Add Fleet Server configured as follows:
Next phase is to Select Agent Policy → Add Agent → Enroll in Fleet → Add Fleet Server
We are going to need this information to setup our fleet server.
Login via SSH to the fleet-server and make sure the fleet-server is running before setting up our agent:
sudo docker exec -ti fleet-server bash
./elastic-agent status (check it is running)
./elastic-agent restart (if it doesn't appear to be running, force a restart, and recheck the status)
This is an example of what need to be copied to the fleet server. Ensure the fleet server es is: https://es01:9200
Add the bold section after port=8220 because are certificates are self-generated. This will ensure the agent takes the update.
The token and fingerprint will be different than this example
Copy the elastic-enrol agent below and eplace these 2 lines taken from your own serve and everything else remain the same.
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MzEwOTcwODU3MzI6VjMyLU13cmFUOUM1eUFvMUhVUFl5QQ \ --fleet-server-es-ca-trusted-fingerprint=0D9A25F4C147EB3A496253525DF6F039CF3C19776E64A1F77CEFCCD08B76BC61 \
While logged in the fleet-agent, enroll the elastic-agent in your fleet-agent:
Use this template to install your certificate and make sure you replace with your own
fleet-server-service-token & fleet-server-es-ca-trusted-fingerprint
elastic-agent enroll \
--url=https://fleet-server:8220 \
--fleet-server-es=https://es01:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MzEwOTcwODU3MzI6VjMyLU13cmFUOUM1eUFvMUhVUFl5QQ \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca=/certs/es01/es01.crt \
--fleet-server-es-ca-trusted-fingerprint=0D9A25F4C147EB3A496253525DF6F039CF3C19776E64A1F77CEFCCD08B76BC61 \
--fleet-server-port=8220 \
--certificate-authorities=/certs/ca/ca.crt \
--fleet-server-cert=/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/certs/fleet-server/fleet-server.key \
--elastic-agent-cert=/certs/fleet-server/fleet-server.crt \
--elastic-agent-cert-key=/certs/fleet-server/fleet-server.key \
--fleet-server-es-cert=/certs/fleet-server/fleet-server.crt \
--fleet-server-es-cert-key=/certs/fleet-server/fleet-server.key
This will replace your current settings. Do you want to continue? [Y/n]: Y
Successfully enrolled the Elastic Agent.
From your current location, verify it installed correctly
./elastic-agent status
./elastic-agent restart (if you got what appears to be errors, force a restart and check the status)
Now that the Fleet Server is connected, close this Windows and lets proceed to the next step.
In Elastic Management → Fleet, refresh Agents and this is what shows up:
The server is now ready to install Threat Intel Agents to be used in Security (SIEM portion) against the honeypot logs.
The next step is to select Agent policies → Fleet Server Policy → Add integration:
Use this Dashboard to view the metrics collected by elastic-agent:
If some of the threat intel above were added to the elastic-agent, and cowrie* was added to the Management -> Advanced Settings during the initial installation, it can now track rules that match in the alert trends of the SIEM part of the ELK stack.
This is the output from Management → Stack Management → Advanced Settings
If any threat intel match, the SIEM will display the following activity that can now be investigated:
Next step is to add the Filebeat package to the DShield Sensor to send the logs the Elasticsearch.
Use the following steps to install Filebeat using the following commands taken from this reference [3]:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
echo "deb https://artifacts.elastic.co/packages/oss-8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update
After the installation, we place a hold on filebeat so you can update the sensor without changing the filebeat version. The filebeat
version should always match the Elasticsearc/Kibana docker version.
Linux - X86_64
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.19.15-amd64.deb
sudo apt install ./filebeat-8.19.15-amd64.deb
sudo apt-mark hold filebeat
PI - aarch64
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.19.15-arm64.deb
sudo apt install ./filebeat-8.19.15-arm64.deb
sudo apt-mark hold filebeat
In order to update filebeat, when the time comes, you will need to remove the hold. Check if there is a hold on filebeat first before removing it
sudo apt-mark showhold
sudo apt-mark unhold filebeat
If you want to use Zeek on your DShield sensor, review these installation steps
To connect from a cloud sensor to your local ELK Stack, you need to do the following:
Follow this URL to download all the files related to the DShield Sensor
https://github.com/bruneaug/DShield-Sensor
git clone https://github.com/bruneaug/DShield-Sensor.git
sudo cp ~/DShield-Sensor/filebeat/filebeat.yml /etc/filebeat/filebeat.yml
sudo vi /etc/filebeat/filebeat.yml
output.logstash:
hosts: ["192.168.25.23:5044"]
#hosts: ["logstash.com:5044"]
sudo su -
filebeat test config
Expected output: Config OK
filebeat test output
Expected output:
logstash: 192.168.25.231:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 192.168.25.231
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
sudo systemctl enable filebeat
sudo systemctl start filebeat
sudo systemctl status filebeat
Filebeat tracks the events it has processed with a file located called log.json, if deleted, all the events that were previous sent to Elasticsearch will be reprocessed when filebeat is restarted.
The location of this file:
sudo su -
cd /var/lib/filebeat/registry/filebeat
If you are planning to resend all the logs because your ELK server got rebuild, stop filebeat, delete log.json and restart filebeat.
This will usually take a while to catch up in the ELK Stack dashboard. If you are looking to see if it is sending the logs to the ELK Stack, you could
To access the Dashboard select Analytics -> Dashboard -> [Logs DShield Sensor] Overview
Follow the steps in this Page to add to Kibana the ca.crt used by the ELK server.
Refer to this page for a list of useful docker commands.
https://github.com/bruneaug/DShield-SIEM/edit/main/Troubleshooting/docker_useful_commands..md
These command are run from the Dev Tool -> Console
GET _nodes/http?pretty (Get a list and information of all the nodes)
GET _security/_authenticate
GET /_cat/nodes?v=true
This is a list of script and code that students from the BACS program have written to assist with log analysis.
Jesse La Grew cowrieprocessor
jrjobe DShield-Cowrie-json-Parser
ham-sauce vt_cowrie
NickTheSecurityDude dshield-honeypot-scripts and LinkedIn Post
Josh Gilman DShieldKickStarter
kcustom11 dshield-azure Azure Terraform
Boggby ec2-honeypot-template
amelete11235
homelab DShiel SIEM Raspberry 5
William Constantino Cowrie logfile_investigation with Python
William Constantino Cowrie single_multi_IP with Python
Steven C. Peterson DShield Model Context Protocol
Nathan Smisson dshield-tooling
[1] https://isc.sans.edu/tools/honeypot/
[2] https://www.elastic.co/downloads/beats/filebeat
[3] https://www.elastic.co/guide/en/beats/filebeat/8.8/setup-repositories.html#_apt
[4] https://isc.sans.edu/diary/DShield+Honeypot+Activity+for+May+2023/29932
[5] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+to+Elasticsearch/29458
[6] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+Analysis/29412
[7] https://github.com/jslagrew/cowrieprocessor/blob/main/submit_vtfiles.py
[8] https://handlers.sans.edu/gbruneau/elastic.htm
[9] https://www.elastic.co/guide/en/fleet/current/secure-connections.html
[10] https://www.docker.elastic.co/
[11] https://github.com/gchq/CyberChef
[12] https://github.com/mitre-attack/attack-navigator/pkgs/container/attack-navigator