Release T-Pot 20.06.0

# Release Notes

## Upgrade from 19.03.x
- If you are running T-Pot 19.x you can upgrade to T-Pot 20.06.0 by running `/opt/tpot/update.sh`. Please be aware upgrades can break things, so please backup all of your data or take snapshot of your machine **before** you run the update procedure.
- To protect possible changes of your Kibana objects you need to manually [export](https://github.com/dtag-dev-sec/tpotce/wiki/Import---Export-Kibana-Objects) (backup) your objects and manually [import](https://github.com/dtag-dev-sec/tpotce/wiki/Import---Export-Kibana-Objects) (overwrite) the provided T-Pot Kibana Objects after upgrading.

## Changelog
- **Release T-Pot 20.06.0**
  - After 4 months of public testing with the NextGen edition T-Pot 20.06 can finally be released.
- **Debian Buster**
  - With the release of Debian Buster T-Pot now has access to all packages required right out of the box.
- **Add new honeypots**
  - [Dicompot](https://github.com/nsmfoo/dicompot) by @nsmfoo is a low interaction honeypot for the Dicom protocol which is the international standard to process medical imaging information. Together with Medpot which supports the HL7 protocol T-Pot is now offering a Medical Installation type.
  - [Honeysap](https://github.com/SecureAuthCorp/HoneySAP) by SecureAuthCorp is a low interaction honeypot for the SAP services, in case of T-Pot configured for the SAP router.
  - [Elasticpot](https://gitlab.com/bontchev/elasticpot) by Vesselin Bontchev replaces ElasticpotPY as a low interaction honeypot for Elasticsearch with more features, plugins and scripted responses.
- **Rebuild Images**
  - All docker images were rebuilt based on the latest (and stable running) versions of the tools and honeypots. Mostly the images now run on Alpine 3.12 / Debian Buster. However some honeypots / tools still reuire Alpine 3.11 / 3.10 to run properly.
- **Install Types**
  - All docker-compose files (`/opt/tpot/etc/compose`) were remixed and most of the NextGen honeypots are now available in Standard.
  - There is now a **Medical** Installation Type with Dicompot and Medpot which will be of most interest for medical institutions to get started with T-Pot.
- **Update Tools**
  - Connecting to T-Pot via `https://<ip>:64297` brings you to the T-Pot Landing Page now which is based on Heimdall and the latest NGINX enforcing TLS 1.3.
  - The ELK stack was updated to 7.8.0 and stripped down to the necessary core functions (where possible) for T-Pot while keeping ELK RAM requirements to a minimum (8GB of RAM is recommended now). The number of index pattern fields was reduced to **697** which increases performance significantly. There are **22** Kibana Dashboards, **397** Kibana Visualizations and **24** Kibana Searches readily available to cover all your needs to get started and familiar with T-Pot.
  - Cyberchef was updated to 9.21.0.
  - Elasticsearch Head was updated to the latest version available on GitHub.
  - Spiderfoot was updated to latest 3.1 dev.
- **Landing Page**
  - After logging into T-Pot via web you are now greeted with a beautifully designed landing page.
- **Countless Tweaks and improvements**
  - Under the hood lots of tiny tweaks, improvements and a few bugfixes will increase your overall experience with T-Pot.

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## 20200630
+- **Release T-Pot 20.06**
+  - After 4 months of public testing with the NextGen edition T-Pot 20.06 can finally be released.
+- **Debian Buster**
+  - With the release of Debian Buster T-Pot now has access to all packages required right out of the box.
+- **Add new honeypots**
+  - [Dicompot](https://github.com/nsmfoo/dicompot) by @nsmfoo is a low interaction honeypot for the Dicom protocol which is the international standard to process medical imaging information. Together with Medpot which supports the HL7 protocol T-Pot is now offering a Medical Installation type.
+  - [Honeysap](https://github.com/SecureAuthCorp/HoneySAP) by SecureAuthCorp is a low interaction honeypot for the SAP services, in case of T-Pot configured for the SAP router.
+  - [Elasticpot](https://gitlab.com/bontchev/elasticpot) by Vesselin Bontchev replaces ElasticpotPY as a low interaction honeypot for Elasticsearch with more features, plugins and scripted responses.
+- **Rebuild Images**
+  - All docker images were rebuilt based on the latest (and stable running) versions of the tools and honeypots. Mostly the images now run on Alpine 3.12 / Debian Buster. However some honeypots / tools still reuire Alpine 3.11 / 3.10 to run properly.
+- **Install Types**
+  - All docker-compose files (`/opt/tpot/etc/compose`) were remixed and most of the NextGen honeypots are now available in Standard.
+  - There is now a **Medical** Installation Type with Dicompot and Medpot which will be of most interest for medical institutions to get started with T-Pot.
+- **Update Tools**
+  - Connecting to T-Pot via `https://<ip>:64297` brings you to the T-Pot Landing Page now which is based on Heimdall and the latest NGINX enforcing TLS 1.3.
+  - The ELK stack was updated to 7.8.0 and stripped down to the necessary core functions (where possible) for T-Pot while keeping ELK RAM requirements to a minimum (8GB of RAM is recommended now). The number of index pattern fields was reduced to **697** which increases performance significantly. There are **22** Kibana Dashboards, **397** Kibana Visualizations and **24** Kibana Searches readily available to cover all your needs to get started and familiar with T-Pot.
+  - Cyberchef was updated to 9.21.0.
+  - Elasticsearch Head was updated to the latest version available on GitHub.
+  - Spiderfoot was updated to latest 3.1 dev.
+- **Landing Page**
+  - After logging into T-Pot via web you are now greeted with a beautifully designed landing page.
+- **Countless Tweaks and improvements**
+  - Under the hood lots of tiny tweaks, improvements and a few bugfixes will increase your overall experience with T-Pot.
+
 ## 20200316
 - **Move from Sid to Stable**
   - Debian Stable has now all the packages and versions we need for T-Pot. As a consequence we can now move to the `stable` branch.
@@ -207,3 +232,5 @@
   - If T-Pot, opposed to the requirements, does not have full internet access netselect-apt fails to determine the fastest mirror as it needs ICMP and UDP outgoing. Should netselect-apt fail the default mirrors will be used.
 - **Improve install speed with apt-fast**
   - Migrating from a stable base install to Debian (Sid) requires downloading lots of packages. Depending on your geo location the download speed was already improved by introducing netselect-apt to determine the fastest mirror. With apt-fast the downloads will be even faster by downloading packages not only in parallel but also with multiple connections per package.
+
+`git log --date=format:"## %Y%m%d" --pretty=format:"%ad %n- **%s**%n  - %b"`

bin/clean.sh

@@ -114,6 +114,15 @@ fuCOWRIE () {
   chown tpot:tpot /data/cowrie -R
 }
 
+# Let's create a function to clean up and prepare dicompot data
+fuDICOMPOT () {
+  if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/dicompot/log; fi
+  mkdir -p /data/dicompot/log
+  mkdir -p /data/dicompot/images
+  chmod 770 /data/dicompot -R
+  chown tpot:tpot /data/dicompot -R
+}
+
 # Let's create a function to clean up and prepare dionaea data
 fuDIONAEA () {
   if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/dionaea/*; fi
@@ -172,6 +181,14 @@ fuHONEYPY () {
   chown tpot:tpot /data/honeypy -R
 }
 
+# Let's create a function to clean up and prepare honeysap data
+fuHONEYSAP () {
+  if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/honeysap/*; fi
+  mkdir -p /data/honeysap/log
+  chmod 770 /data/honeysap -R
+  chown tpot:tpot /data/honeysap -R
+}
+
 # Let's create a function to clean up and prepare honeytrap data
 fuHONEYTRAP () {
   if [ "$myPERSISTENCE" != "on" ]; then rm -rf /data/honeytrap/*; fi
@@ -271,12 +288,14 @@ if [ "$myPERSISTENCE" = "on" ];
     fuCITRIXHONEYPOT
     fuCONPOT
     fuCOWRIE
+    fuDICOMPOT
     fuDIONAEA
     fuELASTICPOT
     fuELK
     fuFATT
     fuGLUTTON
     fuHERALDING
+    fuHONEYSAP
     fuHONEYPY
     fuHONEYTRAP
     fuMAILONEY

bin/export_kibana-objects.sh

@@ -17,15 +17,16 @@ fi
 myDATE=$(date +%Y%m%d%H%M)
 myINDEXCOUNT=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=index-pattern' | jq '.saved_objects[].attributes' | tr '\\' '\n' | grep "scripted" | wc -w)
 myINDEXID=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=index-pattern' | jq '.saved_objects[].id' | tr -d '"')
-myDASHBOARDS=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=dashboard&per_page=300' | jq '.saved_objects[].id' | tr -d '"')
-myVISUALIZATIONS=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=visualization&per_page=300' | jq '.saved_objects[].id' | tr -d '"')
-mySEARCHES=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=search&per_page=300' | jq '.saved_objects[].id' | tr -d '"')
+myDASHBOARDS=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=dashboard&per_page=500' | jq '.saved_objects[].id' | tr -d '"')
+myVISUALIZATIONS=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=visualization&per_page=500' | jq '.saved_objects[].id' | tr -d '"')
+mySEARCHES=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=search&per_page=500' | jq '.saved_objects[].id' | tr -d '"')
+myCONFIGS=$(curl -s -XGET ''$myKIBANA'api/saved_objects/_find?type=config&per_page=500' | jq '.saved_objects[].id' | tr -d '"')
 myCOL1=""
 myCOL0=""
 
 # Let's ensure normal operation on exit or if interrupted ...
 function fuCLEANUP {
-  rm -rf patterns/ dashboards/ visualizations/ searches/
+  rm -rf patterns/ dashboards/ visualizations/ searches/ configs/
 }
 trap fuCLEANUP EXIT
 
@@ -65,12 +66,22 @@ for i in $mySEARCHES;
   done;
 echo
 
+# Export configs
+mkdir -p configs
+echo $myCOL1"### Now exporting"$myCOL0 $(echo $myCONFIGS | wc -w) $myCOL1"configs." $myCOL0
+for i in $myCONFIGS;
+  do
+    echo $myCOL1"###### "$i $myCOL0
+    curl -s -XGET ''$myKIBANA'api/saved_objects/config/'$i'' | jq '. | {attributes, references}' > configs/$i.json &
+  done;
+echo
+
 # Wait for background exports to finish
 wait
 
 # Building tar archive
 echo $myCOL1"### Now building archive"$myCOL0 "kibana-objects_"$myDATE".tgz"
-tar cvfz kibana-objects_$myDATE.tgz patterns dashboards visualizations searches > /dev/null
+tar cvfz kibana-objects_$myDATE.tgz patterns dashboards visualizations searches configs > /dev/null
 
 # Stats
 echo
@@ -79,4 +90,5 @@ echo $myCOL1"###### Exported"$myCOL0 $myINDEXCOUNT $myCOL1"index patterns." $myC
 echo $myCOL1"###### Exported"$myCOL0 $(echo $myDASHBOARDS | wc -w) $myCOL1"dashboards." $myCOL0
 echo $myCOL1"###### Exported"$myCOL0 $(echo $myVISUALIZATIONS | wc -w) $myCOL1"visualizations." $myCOL0
 echo $myCOL1"###### Exported"$myCOL0 $(echo $mySEARCHES | wc -w) $myCOL1"searches." $myCOL0
+echo $myCOL1"###### Exported"$myCOL0 $(echo $myCONFIGS | wc -w) $myCOL1"configs." $myCOL0
 echo

bin/hptest.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 myHOST="$1"
-myPACKAGES="netcat nmap"
+myPACKAGES="dcmtk netcat nmap"
 myMEDPOTPACKET="
 MSH|^~\&|ADT1|MCM|LABADT|MCM|198808181126|SECURITY|ADT^A01|MSG00001-|P|2.6
 EVN|A01|198808181123
@@ -83,7 +83,11 @@ fuCHECKFORARGS
 echo "Starting scans ..."
 echo "$myMEDPOTPACKET" | nc "$myHOST" 2575 &
 curl -XGET "http://$myHOST:9200/logstash-*/_search" &
+curl -XPOST -H "Content-Type: application/json" -d '{"name":"test","email":"test@test.com"}' "http://$myHOST:9200/test" &
 echo "
I20100" | timeout --foreground 3 nc "$myHOST" 10001 &
+findscu -P -k PatientName="*" $myHOST 11112 &
+getscu -P -k PatientName="*" $myHOST 11112 &
+telnet $myHOST 3299 &
 fuSCAN "180" "7,8,102,135,161,1025,1080,5000,9200" "$myHOST" "-sC -sS -sU -sV"
 fuSCAN "180" "2048,4096,5432" "$myHOST" "-sC -sS -sU -sV --version-light"
 fuSCAN "120" "20,21" "$myHOST" "--script=ftp* -sC -sS -sV"

bin/import_kibana-objects.sh

@@ -20,7 +20,7 @@ myCOL0=""
 
 # Let's ensure normal operation on exit or if interrupted ...
 function fuCLEANUP {
-  rm -rf patterns/ dashboards/ visualizations/ searches/
+  rm -rf patterns/ dashboards/ visualizations/ searches/ configs/
 }
 trap fuCLEANUP EXIT
 
@@ -98,12 +98,29 @@ for i in $mySEARCHES;
 echo
 wait
 
+# Restore configs
+myCONFIGS=$(ls configs/*.json | cut -c 9- | rev | cut -c 6- | rev)
+echo $myCOL1"### Now importing "$myCOL0$(echo $myCONFIGS | wc -w)$myCOL1 "configs." $myCOL0
+for i in $myCONFIGS;
+  do
+    curl -s -XDELETE ''$myKIBANA'api/saved_objects/configs/'$i'' -H "Content-Type: application/json" -H "kbn-xsrf: true" > /dev/null &
+  done;
+wait
+for i in $myCONFIGS;
+  do
+    echo $myCOL1"###### "$i $myCOL0
+    curl -s -XPOST ''$myKIBANA'api/saved_objects/configs/'$i'' -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @configs/$i.json > /dev/null &
+  done;
+echo
+wait
+
 # Stats
 echo
 echo $myCOL1"### Statistics"
 echo $myCOL1"###### Imported"$myCOL0 $myINDEXCOUNT $myCOL1"index patterns." $myCOL0
 echo $myCOL1"###### Imported"$myCOL0 $(echo $myDASHBOARDS | wc -w) $myCOL1"dashboards." $myCOL0
 echo $myCOL1"###### Imported"$myCOL0 $(echo $myVISUALIZATIONS | wc -w) $myCOL1"visualizations." $myCOL0
 echo $myCOL1"###### Imported"$myCOL0 $(echo $mySEARCHES | wc -w) $myCOL1"searches." $myCOL0
+echo $myCOL1"###### Imported"$myCOL0 $(echo $myCONFIGS | wc -w) $myCOL1"configs." $myCOL0
 echo
 

bin/updateip.sh

@@ -10,7 +10,7 @@ if [ "$myEXTIP" = "" ];
 fi
 mySSHUSER=$(cat /etc/passwd | grep 1000 | cut -d ':' -f1)
 echo "" > /etc/issue
-toilet -f ivrit -F metal --filter border:metal "T-Pot   19.03" | sed 's/\\/\\\\/g' >> /etc/issue
+toilet -f ivrit -F metal --filter border:metal "T-Pot   20.06" | sed 's/\\/\\\\/g' >> /etc/issue
 echo >> /etc/issue
 echo ",---- [ \n ] [ \d ] [ \t ]" >> /etc/issue
 echo "|" >> /etc/issue

docker/adbhoney/Dockerfile

@@ -8,6 +8,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
     apk -U add \
             git \
             libcap \
+	    py3-pip \
             python3 \
             python3-dev && \
 #

docker/ciscoasa/Dockerfile

@@ -12,6 +12,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
             libffi-dev \
             openssl \
             openssl-dev \
+	    py3-pip \
             python3 \
             python3-dev && \
 #

docker/citrixhoneypot/Dockerfile

@@ -6,6 +6,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
             git \
             libcap \
 	    openssl \
+	    py3-pip \
             python3 \
             python3-dev && \
 #

docker/conpot/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.10
+FROM alpine:latest
 #
 # Include dist
 ADD dist/ /root/dist/
@@ -12,10 +12,12 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
              libev \
              libtool \
              libcap \
+	     libffi-dev \
              libxslt \
              libxslt-dev \
              mariadb-dev \
              pkgconfig \
+	     py3-pip \
              python3 \
              python3-dev \
              py-cffi \
@@ -42,13 +44,13 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
     pip3 install --no-cache-dir . && \
     cd / && \
     rm -rf /opt/conpot /tmp/* /var/tmp/* && \
-    setcap cap_net_bind_service=+ep /usr/bin/python3.7 && \
+    setcap cap_net_bind_service=+ep /usr/bin/python3.8 && \
 #    
 # Get wireshark manuf db for scapy, setup configs, user, groups
     mkdir -p /etc/conpot /var/log/conpot /usr/share/wireshark && \
     wget https://github.com/wireshark/wireshark/raw/master/manuf -o /usr/share/wireshark/manuf && \
     cp /root/dist/conpot.cfg /etc/conpot/conpot.cfg && \
-    cp -R /root/dist/templates /usr/lib/python3.7/site-packages/conpot/ && \
+    cp -R /root/dist/templates /usr/lib/python3.8/site-packages/conpot/ && \
     addgroup -g 2000 conpot && \
     adduser -S -s /bin/ash -u 2000 -D -g 2000 conpot && \
 #

docker/conpot/dist/conpot.cfg

@@ -3,7 +3,7 @@ sensorid = conpot
 
 [virtual_file_system]
 data_fs_url = %(CONPOT_TMP)s
-fs_url = tar:///usr/lib/python3.7/site-packages/conpot/data.tar
+fs_url = tar:///usr/lib/python3.8/site-packages/conpot/data.tar
 
 [session]
 timeout = 30

docker/cowrie/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine
+FROM alpine:latest
 #
 # Include dist
 ADD dist/ /root/dist/
@@ -16,6 +16,7 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
                        mpfr-dev \
                        openssl \
                        openssl-dev \
+		       py3-pip \
                        python3 \
                        python3-dev \
                        py3-bcrypt \
@@ -30,11 +31,11 @@ RUN sed -i 's/dl-cdn/dl-2/g' /etc/apk/repositories && \
 # Install cowrie
     mkdir -p /home/cowrie && \
     cd /home/cowrie && \
-    git clone --depth=1 https://github.com/micheloosterhof/cowrie -b v2.0.2 && \
+    git clone --depth=1 https://github.com/micheloosterhof/cowrie -b v2.1.0 && \
     cd cowrie && \
     mkdir -p log && \
-    pip3 install --upgrade pip && \
-    pip3 install --upgrade -r requirements.txt && \
+    cp /root/dist/requirements.txt . && \
+    pip3 install -r requirements.txt && \
 #
 # Setup configs
     export PYTHON_DIR=$(python3 --version | tr '[A-Z]' '[a-z]' | tr -d ' ' | cut -d '.' -f 1,2 ) && \