GSoC 2019 Work Product Submission

Hello everyone! I have been working on the project this summer and the major work was to add more emulators to extend the support for new vulnerabilities such as Template Injection, PHP Object Injection, and XXE Injection.

TANNER

TANNER is a remote data analysis and classification service used to compose the response served by SNARE events. Basically, it is the brain of the honeypot.

Tanner Coverage

Initially, the overall coverage was at 57%, now it has been increased to 78%.

Code:

1. Base Emulator tests: https://github.com/mushorg/tanner/commit/d7b6c4b3bb0497941d12b4747fae196f5accd604.
2. MySQL DB Helper tests: https://github.com/mushorg/tanner/commit/813647d11065f5200e37564beedb730524e8b8a7.
3. Tanner API tests: https://github.com/mushorg/tanner/commit/017b5a1e41b384c90fd7ad921b295f55eb6a5cfd.
4. MySQLI Emulator tests: https://github.com/mushorg/tanner/commit/029de8c95f7254b5f27825ab50b21f87e10bf239.
5. SQLite Emulator & SQLite DB Helper tests: https://github.com/mushorg/tanner/commit/8ce13d1f7d4423ddaf0e7910781199be9b90ce40.
6. Dorks Manager utility tests: https://github.com/mushorg/tanner/commit/d0cca3a8e46b072cb031ca0b780979397a789997.
7. Tanner Web Server tests: https://github.com/mushorg/tanner/commit/18324abb38343c2db441614e1b30f87bf0aa30c8.

PHP Object Injection Emulator

A new emulator to emulate PHP Object Injection vulnerability. To mimic the functionality a vulnerable PHP code is built (custom class can also be added) in which attacker's serialized objects will be injected directly. The execution results are obtained using PHP sandbox. Since now we are using phpox in multiple places a sandbox helper (php_sandbox_helper.py) is also implemented. The input serialized object is detected with a regex pattern. The only issue we faced with this task was to find a suitable method to expose the vulnerable code to the attacker but then we decided to leave it for the user.

(^|;|{|})O:[0-9]+:

Code:

1. Emulator: https://github.com/mushorg/tanner/commit/4f39e4ef01360244b98e4f1c24a7f47d1aac756c.
2. Tests for the emulator: https://github.com/mushorg/tanner/commit/2584da9d12b861d3c7604fce2eb27e299fa785ec.
3. PHP objection Injection Docs: https://github.com/mushorg/tanner/commit/e68e79cc3c479ea6d3b9d6825e5beba712ca6fd9

XML External Entity (XXE) Injection Emulator

It emulates XXE Injection vulnerability. The idea behind this emulator is similar, a vulnerable PHP code is created which parses the XML data and then it gets the injection results from PHP sandbox. The XML input is detected by a regex pattern by checking the presence of xml and doctype tags. Out of Band XXE injection is also supported by this emulator and can be setup using config.

.*<(\?xml|(!DOCTYPE.*)).*>

Code:

1. XXE Emulator & Tests: https://github.com/mushorg/tanner/commit/ecc8f9190193f170166f272ad0aece3241bca87c.
2. Docs: https://github.com/mushorg/tanner/commit/48bf597a287b54f3621d17547697b81062963672.

Template Injection Emulator

Emulator for Template Injection vulnerability. Currently, it supports tornado and mako python templating engines and can be extended for other engines as well. We also tried other engines like twig templating engine used in PHP which will get its injection results from php sandbox but the complex payloads were not working properly. Different engines have different injection formats hence different regex patterns are used for detection.

Basic Injections -

Tornado: {{7*7}} -> 49 Mako: <% x=7*7 %>${x} -> 49

All the templates are stored in files/engines directory and the execution of these vulnerable templates is done using custom docker image (base image: alpine) for ensuring the safety of the honeypot. One issue that we faced was when XSS payloads got matched with the MAKO regex but the reverse way was not possible so we reordered the XSS regex position after MAKO.

Note: We are using github remote path to build the custom image for its simplicity and the other option was to use a tar file to build the image but then we will have to build an updated tar file every time tanner starts and it is not something that needs to be updated too often so it was not suitable.

.*({{.*}}).* - Tornado
.*(<%.*|\s%>).* - Mako

Code:

1. Emulator & tests: https://github.com/mushorg/tanner/commit/1908773e90bd5e0222e4e9ff623dad1b90092cd4
2. Docs: https://github.com/mushorg/tanner/commit/75b738683ecc2170c8c901ae43e8522a6d2dd253
3. Custom Dockerfile: https://github.com/mushorg/tanner/commit/c708d10a00fe19cb90ca217ba7a75179bc7e2610.

AIODocker Helper

Earlier we were using docker-py for using docker API, now it has been replaced by aiodocker to use docker asynchronously. A helper with all the required functions (build image, execute cmd, create container, get container, delete container) is implemented.

Updated the Command execution emulator and LFI emulator as they were using older implementation. To get the execution results in the emulators we are using run + delete method which takes 1.5324 sec than using create_or_replace method which takes 1.4206 sec plus time to delete the container and checking the logs for the output.

Code:

1. Aiodocker Helper & Tests: https://github.com/mushorg/tanner/commit/d272dcdb324fb72e5f019d8e0195eba524434af4
2. Helper Comments: https://github.com/mushorg/tanner/commit/b9977ce0ee197278a36b98049bf2e640cf5d5c2e

Logging

Improved logging in emulators, added more debug and informational logs. Added the logs of filename along with hashname when downloading files as in case of RFI emulator. Refs - #216

Code:

1. https://github.com/mushorg/tanner/commit/2d4c601a3cc9a12e0cd0900588aa4e28f4508fae

Other Commits

1. Changed Implementation in PHP Code Injection Emulator and RFI Emulator using PHPsandbox: https://github.com/mushorg/tanner/commit/573cdaa17b22fedb6ee772698508d1648d83828a.

2. Asyncmock Utility: https://github.com/mushorg/tanner/commit/b6ee8948ccdc5796fcb34e7865eaac3488a9d6a0.

3. Fix for test_config.py: https://github.com/mushorg/tanner/commit/0e29aac9b16a3bb2b1f6901cd1f148fb5be5a354.

4. Added Comments: https://github.com/mushorg/tanner/commit/b5fcde7b7504ecb8b1ddfaa19e149b278dc62112, https://github.com/mushorg/tanner/commit/333d2741500f539338f031b701894b3b1696eed3.

5. Fix for #312: https://github.com/mushorg/tanner/commit/9808f38e3f839e7817d4b54bdc281ef4b702fb5b.

6. Minor fixes: https://github.com/mushorg/tanner/commit/096526b047445bb9c8b1694e207aab397c3f73b1, https://github.com/mushorg/tanner/commit/bcb4e630aa6d0c8b5b3baa8eee8a6c68bdd573c5, https://github.com/mushorg/tanner/commit/9fa7289896d194e0ca7b03faec3dfd72553271a8, https://github.com/mushorg/tanner/commit/daa26cd582394fa6adde99272d797970c861fdcb.

SNARE

Coverage Improvement

Added tests for Cloner, also fixed test file naming structure. Previously the overall coverage was 78%, now it has been increased to 92%.

Code:

1. Cloner Tests: https://github.com/mushorg/snare/commit/0a77101f70d7e8123262eeb97068dcc92594642e.
2. Improved Naming: https://github.com/mushorg/snare/commit/ecc726489bb017077b6d6121a3d73080fd647377.

What’s left to do?

So, all of the initially proposed work has been merged successfully. And I would love to keep contributing to this amazing project after GSoC. There are a few tasks that I would like to work on: adding more and more emulators like padding oracle and replacing the database in tanner from Redis to Postgres :)

Cheers, Rajat