Merge branch 'master' into hpfeeds-cmd

CHANGELOG.md

@@ -1,11 +1,38 @@
 # Changelog
 
-## 20190516
+## 20190626
 - **HPFEEDS Opt-In commandline option**
   - Pass a hpfeeds config file as a commandline argument
   - hpfeeds config is saved in `/data/ews/conf/hpfeeds.cfg`
   - Update script restores hpfeeds config
 
+## 20190604
+- **Finalize Fatt support**
+  - Build visualizations, searches, dashboards
+  - Rebuild index patterns
+  - Some finishing touches
+
+## 20190601
+- **Start supporting Fatt, remove Glastopf**
+  - Build Dockerfile, Adjust logstash, installer, update and such.
+  - Glastopf is no longer supported within T-Pot
+
+## 20190528+20190531
+- **Increase total number of fields**
+  - Adjust total number of fileds for logstash templae from 1000 to 2000.
+
+## 20190526
+- **Fix build for Cowrie**
+  - Upstream changes required a new package `py-bcrypt`.
+
+## 20190525
+- **Fix build for RDPY**
+  - Building was prevented due to cache error which occurs lately on Alpine if `apk` is using `--no-ache' as options.
+
+## 20190520
+- **Adjust permissions for /data folder**
+  - Now it is possible to download files from `/data` using SCP, WINSCP or CyberDuck.
+
 ## 20190513
 - **Added Ansible T-Pot Deployment on Open Telekom Cloud**
   - Reusable Ansible Playbooks for all cloud providers

README.md

@@ -12,7 +12,6 @@ and includes dockerized versions of the following honeypots
 * [cowrie](https://github.com/cowrie/cowrie),
 * [dionaea](https://github.com/DinoTools/dionaea),
 * [elasticpot](https://github.com/schmalle/ElasticpotPY),
-* [glastopf](http://mushmush.org/),
 * [glutton](https://github.com/mushorg/glutton),
 * [heralding](https://github.com/johnnykv/heralding),
 * [honeypy](https://github.com/foospidy/HoneyPy),
@@ -30,6 +29,7 @@ Furthermore we use the following tools
 * [Cyberchef](https://gchq.github.io/CyberChef/) a web app for encryption, encoding, compression and data analysis.
 * [ELK stack](https://www.elastic.co/videos) to beautifully visualize all the events captured by T-Pot.
 * [Elasticsearch Head](https://mobz.github.io/elasticsearch-head/) a web front end for browsing and interacting with an Elastic Search cluster.
+* [Fatt](https://github.com/0x4D31/fatt) a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
 * [Spiderfoot](https://github.com/smicallef/spiderfoot) a open source intelligence automation tool.
 * [Suricata](http://suricata-ids.org/) a Network Security Monitoring engine.
 
@@ -141,7 +141,6 @@ In T-Pot we combine the dockerized honeypots ...
 * [cowrie](http://www.micheloosterhof.com/cowrie/),
 * [dionaea](https://github.com/DinoTools/dionaea),
 * [elasticpot](https://github.com/schmalle/ElasticPot),
-* [glastopf](http://mushmush.org/),
 * [glutton](https://github.com/mushorg/glutton),
 * [heralding](https://github.com/johnnykv/heralding),
 * [honeypy](https://github.com/foospidy/HoneyPy),
@@ -157,6 +156,7 @@ In T-Pot we combine the dockerized honeypots ...
 * [Cyberchef](https://gchq.github.io/CyberChef/) a web app for encryption, encoding, compression and data analysis.
 * [ELK stack](https://www.elastic.co/videos) to beautifully visualize all the events captured by T-Pot.
 * [Elasticsearch Head](https://mobz.github.io/elasticsearch-head/) a web front end for browsing and interacting with an Elastic Search cluster.
+* [Fatt](https://github.com/0x4D31/fatt) a pyshark based script for extracting network metadata and fingerprints from pcap files and live network traffic.
 * [Spiderfoot](https://github.com/smicallef/spiderfoot) a open source intelligence automation tool.
 * [Suricata](http://suricata-ids.org/) a Network Security Monitoring engine.
 
@@ -220,7 +220,7 @@ Depending on your installation type, whether you install on [real hardware](#har
 
 ##### NextGen Installation (Glutton replacing Honeytrap, HoneyPy replacing Elasticpot)
 - Honeypots: adbhoney, ciscoasa, conpot, cowrie, dionaea, glutton, heralding, honeypy, mailoney, rdpy, snare & tanner
-- Tools: cockpit, cyberchef, ELK, elasticsearch head, ewsposter, NGINX, spiderfoot, p0f and suricata
+- Tools: cockpit, cyberchef, ELK, elasticsearch head, ewsposter, fatt, NGINX, spiderfoot, p0f and suricata
 
 - 6-8 GB RAM (less RAM is possible but might introduce swapping)
 - 128 GB SSD (smaller is possible but limits the capacity of storing events)
@@ -389,6 +389,7 @@ The Update script will:
 
 You simply run the update script:
 ```
+sudo su -
 cd /opt/tpot/
 ./update.sh -y
 ```
@@ -513,7 +514,7 @@ We hope you understand that we cannot provide support on an individual basis. We
 # Licenses
 The software that T-Pot is built on uses the following licenses.
 <br>GPLv2: [conpot](https://github.com/mushorg/conpot/blob/master/LICENSE.txt), [dionaea](https://github.com/DinoTools/dionaea/blob/master/LICENSE), [honeypy](https://github.com/foospidy/HoneyPy/blob/master/LICENSE), [honeytrap](https://github.com/armedpot/honeytrap/blob/master/LICENSE), [suricata](http://suricata-ids.org/about/open-source/)
-<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://github.com/schmalle/ElasticPot), [ewsposter](https://github.com/dtag-dev-sec/ews/), [glastopf](https://github.com/glastopf/glastopf/blob/master/GPL), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
+<br>GPLv3: [adbhoney](https://github.com/huuck/ADBHoney), [elasticpot](https://github.com/schmalle/ElasticPot), [ewsposter](https://github.com/dtag-dev-sec/ews/), [fatt](https://github.com/0x4D31/fatt/blob/master/LICENSE), [rdpy](https://github.com/citronneur/rdpy/blob/master/LICENSE), [heralding](https://github.com/johnnykv/heralding/blob/master/LICENSE.txt), [snare](https://github.com/mushorg/snare/blob/master/LICENSE), [tanner](https://github.com/mushorg/snare/blob/master/LICENSE)
 <br>Apache 2 License: [cyberchef](https://github.com/gchq/CyberChef/blob/master/LICENSE), [elasticsearch](https://github.com/elasticsearch/elasticsearch/blob/master/LICENSE.txt), [logstash](https://github.com/elasticsearch/logstash/blob/master/LICENSE), [kibana](https://github.com/elasticsearch/kibana/blob/master/LICENSE.md), [docker](https://github.com/docker/docker/blob/master/LICENSE), [elasticsearch-head](https://github.com/mobz/elasticsearch-head/blob/master/LICENCE)
 <br>MIT license: [ciscoasa](https://github.com/Cymmetria/ciscoasa_honeypot/blob/master/LICENSE), [glutton](https://github.com/mushorg/glutton/blob/master/LICENSE)
 <br> Other: [cowrie](https://github.com/micheloosterhof/cowrie/blob/master/LICENSE.md), [mailoney](https://github.com/awhitehatter/mailoney), [Debian licensing](https://www.debian.org/legal/licenses/)
@@ -537,7 +538,7 @@ Without open source and the fruitful development community (we are proud to be a
 * [elasticsearch](https://github.com/elastic/elasticsearch/graphs/contributors)
 * [elasticsearch-head](https://github.com/mobz/elasticsearch-head/graphs/contributors)
 * [ewsposter](https://github.com/armedpot/ewsposter/graphs/contributors)
-* [glastopf](https://github.com/mushorg/glastopf/graphs/contributors)
+* [fatt](https://github.com/0x4D31/fatt/graphs/contributors)
 * [glutton](https://github.com/mushorg/glutton/graphs/contributors)
 * [heralding](https://github.com/johnnykv/heralding/graphs/contributors)
 * [honeypy](https://github.com/foospidy/HoneyPy/graphs/contributors)

bin/clean.sh

@@ -129,12 +129,12 @@ fuELK () {
   chown tpot:tpot /data/elk -R
 }
 
-# Let's create a function to clean up and prepare glastopf data
-fuGLASTOPF () {
-  if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/glastopf/*; fi
-  mkdir -p /data/glastopf/db /data/glastopf/log
-  chmod 770 /data/glastopf -R
-  chown tpot:tpot /data/glastopf -R
+# Let's create a function to clean up and prepare fatt data
+fuFATT () {
+  if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/fatt/*; fi
+  mkdir -p /data/fatt/log
+  chmod 770 -R /data/fatt
+  chown tpot:tpot -R /data/fatt
 }
 
 # Let's create a function to clean up and prepare glastopf data
@@ -262,7 +262,7 @@ if [ "$myPERSISTENCE" = "on" ];
     fuDIONAEA
     fuELASTICPOT
     fuELK
-    fuGLASTOPF
+    fuFATT
     fuGLUTTON
     fuHERALDING
     fuHONEYPY

bin/hptest.sh

@@ -82,6 +82,8 @@ fuCHECKFORARGS
 
 echo "Starting scans ..."
 echo "$myMEDPOTPACKET" | nc "$myHOST" 2575 &
+curl -XGET "http://$myHOST:9200/logstash-*/_search" &
+echo "
I20100" | timeout --foreground 3 nc "$myHOST" 10001 &
 fuSCAN "180" "7,8,102,135,161,1025,1080,5000,9200" "$myHOST" "-sC -sS -sU -sV"
 fuSCAN "180" "2048,4096,5432" "$myHOST" "-sC -sS -sU -sV --version-light"
 fuSCAN "120" "20,21" "$myHOST" "--script=ftp* -sC -sS -sV"

bin/restore_es.sh

@@ -60,6 +60,25 @@ curl -s XPUT ''$myES'_template/.*' -H 'Content-Type: application/json' -d'
 }'
 echo
 
+# Set logstash template
+echo -n $myCOL1"### Setting up logstash template: "$myCOL0
+curl -s XPUT ''$myES'_template/logstash' -H 'Content-Type: application/json' -d'
+{
+  "index_patterns": "logstash-*",
+    "settings" : {
+      "index" : { 
+        "number_of_shards": 1,
+        "number_of_replicas": 0,
+          "mapping" : { 
+            "total_fields" : {
+              "limit" : "2000"
+            } 
+          }  
+      }
+    }
+}'
+echo
+
 # Restore indices
 curl -s -X DELETE ''$myES'.kibana*' > /dev/null
 for i in $myINDICES;

docker/cowrie/Dockerfile

@@ -17,6 +17,7 @@ RUN apk -U --no-cache add \
                        openssl-dev \
                        python \
                        python-dev \
+                       py-bcrypt \
                        py-mysqldb \
                        py-pip \
                        py-requests \

docker/elk/logstash/dist/elasticsearch-template-es5x.json

@@ -4,7 +4,12 @@
   "settings" : {
     "index.refresh_interval" : "5s",
     "index.number_of_shards" : "1",
-    "index.number_of_replicas" : "0"
+    "index.number_of_replicas" : "0",
+        "mapping" : {
+          "total_fields" : {
+            "limit" : "2000"
+          }
+        }
   },
   "mappings" : {
     "_default_" : {

docker/elk/logstash/dist/elasticsearch-template-es6x.json

@@ -4,7 +4,8 @@
   "settings" : {
     "index.refresh_interval" : "5s",
     "index.number_of_shards" : "1",
-    "index.number_of_replicas" : "0"
+    "index.number_of_replicas" : "0",
+    "index.mapping.total_fields.limit": "2000"
   },
   "mappings" : {
     "_default_" : {

docker/elk/logstash/dist/logstash.conf

@@ -1,6 +1,13 @@
 # Input section
 input {
 
+# Fatt                                 
+  file {                                   
+    path => ["/data/fatt/log/fatt.log"]
+    codec => json     
+    type => "Fatt"
+  }  
+
 # Suricata
   file {
     path => ["/data/suricata/log/eve.json"]
@@ -57,12 +64,6 @@ input {
     type => "ElasticPot"
   }
 
-# Glastopf
-  file {
-    path => ["/data/glastopf/log/glastopf.log"]
-    type => "Glastopf"
-  }
-
 # Glutton
   file {
     path => ["/data/glutton/log/glutton.log"]
@@ -128,6 +129,27 @@ input {
 # Filter Section
 filter {
 
+
+# Fatt
+  if [type] == "Fatt" {
+    date {
+      match => [ "timestamp", "ISO8601" ]
+    }
+    mutate {
+      rename => {
+        "sourceIp" => "src_ip"
+	"destinationIp" => "dest_ip"
+	"sourcePort" => "src_port"
+	"destinationPort" => "dest_port"
+        "gquic" => "fatt_gquic"
+        "http" => "fatt_http"
+        "rdp" => "fatt_rdp"
+        "ssh" => "fatt_ssh"
+        "tls" => "fatt_tls"
+      }
+    }
+  }
+
 # Suricata
   if [type] == "Suricata" {
     date {
@@ -243,17 +265,6 @@ filter {
     }
   }
 
-# Glastopf
-  if [type] == "Glastopf" {
-    grok {
-      match => [ "message", "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{IP:src_ip}%{SPACE}%{WORD}%{SPACE}%{URIPROTO:http_method}%{SPACE}%{NOTSPACE:http_uri}%{SPACE}%{NOTSPACE}%{SPACE}%{HOSTNAME}:%{NUMBER:dest_port:integer}" ]
-    }
-    date {
-      match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
-      remove_field => ["timestamp"]
-    }
-  }
-
 # Glutton
   if [type] == "Glutton" {
     date {
@@ -406,7 +417,7 @@ if "_grokparsefailure" in [tags] { drop {} }
   }
 
 # Add T-Pot hostname and external IP
-  if [type] == "Adbhoney" or [type] == "Ciscoasa" or [type] == "ConPot" or [type] == "Cowrie" or [type] == "Dionaea" or [type] == "ElasticPot" or [type] == "Glastopf" or [type] == "Glutton" or [type] == "Honeytrap" or [type] == "Heralding" or [type] == "Honeypy" or [type] == "Mailoney" or [type] == "Medpot" or [type] == "P0f" or [type] == "Rdpy" or [type] == "Suricata" or [type] == "Tanner" {
+  if [type] == "Adbhoney" or [type] == "Ciscoasa" or [type] == "ConPot" or [type] == "Cowrie" or [type] == "Dionaea" or [type] == "ElasticPot" or [type] == "Fatt" or [type] == "Glutton" or [type] == "Honeytrap" or [type] == "Heralding" or [type] == "Honeypy" or [type] == "Mailoney" or [type] == "Medpot" or [type] == "P0f" or [type] == "Rdpy" or [type] == "Suricata" or [type] == "Tanner" {
     mutate {
       add_field => {
         "t-pot_ip_ext" => "${MY_EXTIP}"

docker/fatt/Dockerfile

@@ -0,0 +1,41 @@
+FROM alpine
+#
+# Include dist
+#ADD dist/ /root/dist/
+#
+# Get and install dependencies & packages
+RUN apk -U add \
+              git \
+              py3-libxml2 \
+              py3-lxml \
+              python3 \
+              python3-dev && \
+    apk -U add tshark --repository http://dl-3.alpinelinux.org/alpine/edge/community/ && \
+#
+# Setup user
+    addgroup -g 2000 fatt && \
+    adduser -S -s /bin/ash -u 2000 -D -g 2000 fatt && \
+#
+# Install fatt
+    mkdir -p /opt && \
+    cd /opt && \
+    git clone --depth=1 https://github.com/0x4D31/fatt && \
+    cd fatt && \
+    mkdir -p log && \
+    pip3 install --upgrade pip && \
+    pip3 install pyshark==0.4.2.2 && \
+#
+# Setup configs
+    chown fatt:fatt -R /opt/fatt/* && \
+#
+# Clean up
+    apk del --purge git \
+                    python3-dev && \
+    rm -rf /root/* && \
+    rm -rf /var/cache/apk/* 
+#
+# Start fatt
+STOPSIGNAL SIGINT
+ENV PYTHONPATH /opt/fatt
+WORKDIR /opt/fatt
+CMD python3 fatt.py -i $(/sbin/ip address | grep '^2: ' | awk '{ print $2 }' | tr -d [:punct:]) --print_output --json_logging -o log/fatt.log

docker/fatt/docker-compose.yml

@@ -0,0 +1,17 @@
+version: '2.3'
+
+services:
+
+# Fatt service
+  fatt:
+    build: .
+    container_name: fatt
+    restart: always
+    network_mode: "host"
+    cap_add:
+     - NET_ADMIN
+     - SYS_NICE
+     - NET_RAW
+    image: "dtagdevsec/fatt:1903"
+    volumes:
+     - /data/fatt/log:/opt/fatt/log

docker/glastopf/README.md

@@ -1,6 +1,6 @@
 [![](https://images.microbadger.com/badges/version/dtagdevsec/glastopf:1903.svg)](https://microbadger.com/images/dtagdevsec/glastopf:1903 "Get your own version badge on microbadger.com") [![](https://images.microbadger.com/badges/image/dtagdevsec/glastopf:1903.svg)](https://microbadger.com/images/dtagdevsec/glastopf:1903 "Get your own image badge on microbadger.com")
 
-# glastopf
+# glastopf (deprecated)
 
 [glastopf](https://github.com/mushorg/glastopf) is a python web application honeypot.
 

docker/rdpy/Dockerfile

@@ -4,7 +4,7 @@ FROM alpine
 ADD dist/ /root/dist/
 
 # Get and install dependencies & packages
-RUN apk -U --no-cache add \
+RUN apk -U add \
             build-base \
             git \
             libffi-dev \
@@ -14,8 +14,6 @@ RUN apk -U --no-cache add \
             python-dev \
             py-pip \
             py-setuptools && \
-    apk -U --no-cache add --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ \
-            py-qt && \
 
 # Setup user
     addgroup -g 2000 rdpy && \
@@ -52,8 +50,7 @@ RUN apk -U --no-cache add \
                     libffi-dev \
                     openssl-dev \
                     python-dev \
-                    py-pip \
-                    py-qt && \
+                    py-pip && \ 
     rm -rf /var/cache/apk/*
 
 # Start rdpy