ScoutSuite integration

[ShayNehmad]

We'd like to use ScoutSuite to collect and present Cloud findings in our Zero Trust report.

PDR

Problem statement

The problem currently is that we don't cover the Workloads and Orchestration pillars of Zero Trust and our report can generate more value to the users, especially enterprise ones.

The business value of this will be for the Enterprise cloud users and will be measured via download #s of this new version.

Scope

MH

• ScoutSuite findings show up in ZT report
• Minimal requirements added

NTH

• Extra documentation/enrichment on ScoutSuite findings

Out of scope

• Framework for integrating outside products (Infection Monkey as pen-testing framework)

Reference materials

https://github.com/nccgroup/ScoutSuite

Specific Components PDR

Information Gathering in the Monkeys

Functional requirements

Inputs: Collect Cloud info is configured. This will run ScoutSuite. The exact software boundaries between ScoutSuite and Infection Monkey will be discussed in the DDR including if and how to update our ScoutSuite fork.
Outputs: Telemetry, how many checks were performed, how many resources were found.
Logs + error logs

Quality Requirements

The setup MUST BE very clearly defined and documented so users can use this feature by adding the relevant AWS role to their EC2 machines. THIS MIGHT BE INSECURE.

Zero Trust Report

Functional requirements

Inputs: Cloud data from Monkeys via telemetries.
Outputs: Zero Trust Findings
Logs on telemetry parsing + errors

Quality Requirements

Performance must be unhindered with MANY findings. A 2000 findings report MUST be generated in under 7 seconds.

Checklist: Requirements

The requirements checklist contains a list of questions to ask yourself about your project’s requirements.

Use the list as a sanity check at construction time to determine how solid the ground that you’re standing on is—where you are on the requirements Richter scale. Not all of the checklist questions will apply to your project. If you’re working on an informal project, you’ll find some that you don’t even need to think about. You’ll find others that you need to think about but don’t need to answer formally. If you’re working on a large, formal project, however, you may need to consider every one.

Specific Functional Requirements

• Are all the inputs to the system specified, including their source, accuracy, range of values, and frequency?
• Are all the outputs from the system specified, including their destination, accuracy, range of values, frequency, and format?
• Are all output formats specified for Web pages, reports, and so on?
• Are all the external hardware and software interfaces specified?
• Are all the external communication interfaces specified, including handshaking, error-checking, and communication protocols?
• Are all the tasks the user wants to perform specified?
• Is the data used in each task and the data resulting from each task specified?

Specific Nonfunctional (Quality) Requirements

• Is the expected response time, from the user’s point of view, specified for all necessary operations?
• Are other timing considerations specified, such as processing time, datatransfer rate, and system throughput?
• Is the level of security specified?
• Is the reliability specified, including the consequences of software failure, the vital information that needs to be protected from failure, and the strategy for error detection and recovery?
• Are minimum machine memory and free disk space specified?
• Is the maintainability of the system specified, including its ability to adapt to changes in specific functionality, changes in the operating environment, and changes in its interfaces with other software?
• Is the definition of success included? Of failure?

Requirements Quality

• Are the requirements written in the user’s language? Do the users think so?
• Does each requirement avoid conflicts with other requirements?
• Are acceptable tradeoffs between competing attributes specified—for example, between robustness and correctness?
• Do the requirements avoid specifying the design?
• Are the requirements at a fairly consistent level of detail? Should any requirement be specified in more detail? Should any requirement be specified in less detail?
• Are the requirements clear enough to be turned over to an independent group for construction and still be understood? Do the developers think so?
• Is each item relevant to the problem and its solution? Can each item be traced to its origin in the problem environment?
• Is each requirement testable? Will it be possible for independent testing to determine whether each requirement has been satisfied?
• Are all possible changes to the requirements specified, including the likelihood of each change?

Requirements Completeness

• Where information isn’t available before development begins, are the areas of incompleteness specified?
• Are the requirements complete in the sense that if the product satisfies every requirement, it will be acceptable?
• Are you comfortable with all the requirements? Have you eliminated requirements that are impossible to implement and included just to appease your customer or your boss?

[ShayNehmad]

DDR

We are mostly going to talk about boundaries.

High-level design

We will program some utility functions into ScoutSuite to make it usable as a library and not only as a standalone program with HTML output. The main goal is to create a single endpoint called run_as_library which doesn't use a file output but rather returns all of the findings in a dictionary. These changes ought to be pretty generic as we'll try to PR them back to NCC.

The Monkey agent will run this function and send the results to the Island as telemetry. Since ScoutSuite's results are already separated into RedYellowGreen, we will not need to parse and choose the severity ourselves.

We need to decide what is a test, a finding and an event in the context of ScoutSuite.

Options which have been considered but rejected

• Package ScoutSuite as a binary and download it from the Island: Run the entire tool, zip and return the results, and display them in an iframe. This won't work nicely for multiple Monkeys as the results won't combine and will look bad

DD

Affected components

ScoutSuite

TODO: Detail changes in our fork. I'm working with NCC to understand what's the best approach there.

Seems like almost no changes will be required - maybe logger config or output file path. Nothing major for sure.

We'll need to create our own subset of their test - this is a param to the main "run" function, but might get rather complex since we're choosing a subset of a lot of tests.

Monkey agent

The decision right now is to call ScoutSuite async here: https://github.com/ShayNehmad/ScoutSuite/blob/43a931fe24ae332133a098f337584baf5a4bf2ff/ScoutSuite/__main__.py#L76

If we're on AWS, almost no params.

If we're on Azure, we'll do msi=True (Managed Service Identity)

Wait for the results, read the results file (JSON) and send telemetry. If ScoutSuite failed to execute, capture the error and send that in telemetry as well.

• Decision - where do we want to "shove" this in the workflow? It is systeminfo collection? If so, how will we do it async?

Island

The island should get the results from telemetry, but the processing should be rather "generic" since the results are coming already with status etc. - this is UNLIKE the previous ZT tests which were collected on Monkey side and processed on Island side.

A ScoutSuite results parser + "enricher" (which ZT test each ScoutSuite test belongs to) should exist.

Report UI

No major changes will be done to the Zero Trust report or its UI.

We will add a "which environments have been recognized" panel and we need to add an infobox explaining tests are filtered based on this.

• We need to also think if we want to include the ScoutSuite results in the regular report (with remediations etc.).

Testing

🤕 Assumption - we'll only be as good as ScoutSuite. We can open PRs to ScoutSuite to fix issues we're unhappy with.

Sanity

Set up a situation where one check is red, one is yellow, one is green, one is grey.

• AWS
• Azure?
• GCP?

Edge cases

All parsing code should be edge-cased via Unit Tests and NOT manual cloud testing; this will be much more efficient.

Risks + opportunities

RISK: Cloud setups will take a lot of time to get up and running and to debug and test on.
RISK: Complex parsing/translation layer from SS to IM ZT
RISH: High maintenance cost of pulling back changes from ncc/scoutsuite into shay/scoutsuite
OPPO: Improve report performance
OPPO: Improve report loading screen with separate indicators for all the different endpoints
OPPO: Improve the system_info module with a plugin architecture

Tasks

• Finalize DDR
• Separate to tasks + evaluate each one

Dev

• Add an environment check to Monkey so Island knows where Monkeys are running #521 - In CR right now, took about 1.5 days so far.
• After Add an environment check to Monkey so Island knows where Monkeys are running #521, build a UI panel for the reports which should which environments we have found. Need to decide how it looks, which info it displays, and where to put it. 1 day
• Build the call to ScoutSuite: which file, which tests. About 2 days
• Call ScoutSuite programmatically from the Monkey, read the results file, send it "back home" as telemetry. Since we don't want to slow down, we'll have to introduce a new telem + config (on/off for starters). Error reporting is very important as well, we need to know how to parse and send back errors from ScoutSuite. About a day
• Build the parser. UNKNOWN, we need to categorize tests into ZT tests, make sure we deal with exceptions well... THIS PART NEEDS HEAVY UNIT TESTING. Not less than 3 days
• UI elements in ZT report which explain to the user which tests are filtered out of the results and why. About 1/2 day if we keep it simple
• Test on AWS and Azure machines with differing permissions 💀 A few days to test and tweak

[ShayNehmad]

DDR 1 with @danielguardicore :

• Don't run all the tests - filter out interesting
• Finding per Cloud/Cloud service
• Display if any of the Monkeys were cloud (and which) - a single panel for both reports.
• If there were no Monkeys on cloud X, don't display unexecuted tests from cloud X.
• We'll need to detect running in Azure.
• Check if scoutsuite works without super strong IAM role/if findings are different according to differing IAM permissions

[ShayNehmad]

WIP

ScoutSuite Exploratory Surgery for DDR

Setup

I set up the following machines, and I'm debugging Directly from PyCharm using remote SSH shell:

• An AWS machine with Full Admin IAM role. i-05c3679394ad7c6d5
• An Azure machine with the Reader role b95047a4-b17a-4a39-8b03-901dc475c046

Conclusions

Installations

There are quite a lot of extra dependencies this bestows on the Monkey (mostly cloud stuff)).

First execution - AWS

Went with AWS. Ran using Python 3.7 - python3.7 scout.py aws. This means we need to know ahead of time which provider we are planning to scan and pass the parameter accordingly.

Started debugging.

The main mechanism seems to be an asyncio event loop which uses a ThreadPool - they already thought about MultiThreading. Need to make sure I can run this async myself. I didn't debug the internals by accident, so let's start with analyzing the results (and the results FILE) for now and approach the internals + interesting parameters later.

Results analysis

Interesting categories

The "baseline" will be AWS (Azure equivalent will be provided later).

For sure

We'll have the following "categories", meaning tests, per cloud provider:

• Compute - EC2 | Virtual machines
• Identity - IAM | Azure Active Directory
• Storage - S3 | Azure Blob Storage
• Networking - VPC | Virtual Network
• Orchastration - Basically the "other"/"misc" test for general misconfigurations

Maybe list

Might fits in Storage category

• RDS - DB service
• EFS - Basically a file share
• ElastiCache - Redis
• EMR - Cloud native big data
• RedShift - Data warehouse, which means DBs for BI.

Might fit in general Orchastration category...

• CloudTrail - See actions. Good for complience. Seems like this + CloudWatch should be the same thing.
• CloudWatch - Orchastration and visibility, unified view all resources and apps.
• CloudFormation - Like TerraForm but worse since it's only for AWS?
• Config - Same as CloudTrail or CloudWatch but for configurations. What? Why isn't all of this in one service?

Nope

• Lambda - Serverless computing
• Directconnect - Connect to AWS through dedicated hardware + connection (not over Internet).
• ELB - Load balancer
• ELBV2 - Load balancer
• Route53 - Cloud DNS
• SES - Email
• SNS - PubSub
• SQS - MQ

Second execution - Azure

scout.py azure: error: one of the arguments -c/--cli -m/--msi -s/--service-principal --file-auth --user-account is required.

To fix:

scout.py azure --msi uses the Managed Service Account (IAM equivalent).

Results are MUCH less extensive, seems like ScoutSuite care much less about Azure.

[VakarisZ]

Implementation Plan and other details

SS findings -> ZT findings translation

1. Island receives SS findings
2. Island goes through each SS finding, that's relevant to ZT findings, and creates new ZT finding with SS finding in details or adds SS finding to the details of an already present ZT finding. For that we also need to define which SS findings map to which ZT findings (many to one). For example ZT finding "Scoutsuite found overly permissive firewall rules" would map to SS findings: "Security Group Opens All Ports to All", "Security Group Opens RDP Port to All", "Security Group Opens SSH Port to All" etc.
3. Island goes through each relevant service and saves it's resources in mongodb.

UI

1. UI receives ZT findings and cloud services info.
2. Based on ZT finding type it either renders events or resources button.
   
3. Resources button opens up a modal window with dropdowns of each SS finding. Dropdowns contain resources, relevant to SS finding. Below screenshot shows how resources would look(without dropdown)
   

Task list

Setup

• Setup AWS testing env e/0.5
• Setup GCP testing env e/0.5 ?
• Setup Azure testing env e/0.5 ?

Infrastructure

• Modify Scoutsuite into an API e/1
• Test API run and send telemetries from monkey e/0.5

SS finding to ZT finding translation

• Change ZT document structure and related code (ZT finding needs to support subdocuments in details prop) e/2
• Parse SS data into ZT document e/1
• Decide which SS findings should be aggregated into which ZT findings, which ZT findings should be created for CSPM and if we need to add more ZTX principles, somehow map SS findings and ZT findings together e/2
• Repeat above step for GCP and Azure ? e/3

SS finding display

• Adapt ZT front end to display the changed ZT document structure e/1
• Create modal window to display SS resources e/5:
  • Either modify SS front end code (which is ugly/jquery based) and re-use it in modal window + dropdowns OR
  • Write our own JS that would map SS finding to cloud resource
  • Improve back-end data parsing to further ease SS finding <-> Cloud resources mapping

Total estimate: 13 - 16 sp.

[VakarisZ]

Common terms:

Monkey finding

Monkey finding is zero trust finding created by monkey. It contains usual ZT finding structure, it's details contain events.

Monkey finding data structure corresponds to:

{
    "_id" : ObjectId("5f576dd1c62c74f8cde19244"),
    "_cls" : "Finding",
    "type": "monkey",
    "test" : "endpoint_security_exists",
    "status" : "Failed",
    "details" : {
        "_cls" : "MonkeyFindingDetails",
        "_ref" : {
            "$ref" : "monkey_finding_details",
            "$id" : ObjectId("5f576dd1c62c74f8cde19243")
        }
    }
}

Monkey finding details correspond to:

{
    "_id" : ObjectId("5f576dd1c62c74f8cde19243"),
    "events" : [ 
        {
            "timestamp" : ISODate("2020-09-08T14:41:05.791Z"),
            "title" : "Process list",
            "message" : "Monkey on ip-10-0-0-167 scanned the process list",
            "event_type" : "monkey_local"
        }, 
        {
            "timestamp" : ISODate("2020-09-08T15:05:06.774Z"),
            "title" : "Process list",
            "message" : "Monkey on ip-10-0-0-167 scanned the process list",
            "event_type" : "monkey_local"
        }
    ]
}

Scoutsuite finding

Scoutsuite finding is zero trust finding created using scoutsuite report data. It contains usual ZT finding structure, it's details contain scoutsuite rules.

Scoutsuite finding data structure corresponds to:

{
    "_id" : ObjectId("5f576dd1c62c74f8cde19244"),
    "_cls" : "Finding",
    "type": "scoutsuite",
    "test" : "scoutsuite_permissive_firewall_rules",
    "status" : "Failed",
    "details" : {
        "_cls" : "ScoutsuiteFindingDetails",
        "_ref" : {
            "$ref" : "scoutsuite_finding_details",
            "$id" : ObjectId("5f576dd1c62c74f8cde19243")
        }
    }
}

Scoutsuite finding details data structure corresponds to::

{
    "_id" : ObjectId("5f576dd1c62c74f8cde19244"),
    "_cls" : "ScoutsuiteFindingDetails",
    "scoutsuite_rules" : [
       {'description': 'Security Group Opens All Ports to All', 
        'path': 'ec2.regions.id.vpcs.id.security_groups.id.rules.id.protocols.id.ports.id.cidrs.id.CIDR', 
        'level': 'danger', 
        'display_path': 'ec2.regions.id.vpcs.id.security_groups.id', 
        'items': ['ec2.regions.eu-central-1.vpcs.vpc-00015526b6695f9aa.security_groups.sg-019eb67135ec81e65.rules.ingress.protocols.ALL.ports.1-65535.cidrs.0.CIDR'], 
        'dashboard_name': 'Rules', 
        'checked_items': 172, 
        'flagged_items': 1, 
        'service': 'EC2', 
        'rationale': 'It was detected that all ports in the security group are open, and any source IP address could send traffic to these ports, which creates a wider attack surface for resources assigned to it. Open ports should be reduced to the minimum needed to correctly operate and,  when possible, source address restrictions should be implemented.', 
        'remediation': None, 
        'compliance': None, 
        'references': None
     },
      {'description': 'Get Actions Authorized to All Principals', 
       'path': 's3.buckets.id.policy.Statement.id',
       'level': 'danger', 
       'display_path': 's3.buckets.id',
       'items': ['s3.buckets.b5db8203e27b115e4c6a99199da1e7081077ce92.policy.Statement.0'], 
       'dashboard_name': 'Buckets',
       'checked_items': 3, 
       'flagged_items': 1, 
       'service': 'S3',
       'rationale': "Allowing IAM actions to all principals is contrary to the principle of least privilege and presents and opportunity for abuse. This policy should be reviewed to ensure it is secure and in line with the resource's intended use.",
       'remediation': None, 
       'compliance': None, 
       'references': None
    }
    ]
}

Resources

Resources represent items on cloud. Resources can also contain other resources, for example regions have vpc's which have instances and so on.

Obviously, data between different resource types varies greatly.

Resources are part of data ScoutSuite generates, and will be stored on the database.

Example data of ec-2 instance:

Example data of s3 bucket:

ScoutSuite data -> Scoutsuite ZT finding.

Data analysis:

Scoutsuite findings are ordered by cloud service. Each service contains some form of resources (buckets, vpc's, instances, security groups etc.) and findings and/or filters (which we call scoutsuite rules because their contents follow rule data structure.

In the ec2 service, relevant resources are regions. We should make sure to only store relevant services, and only send resources to UI once, not with every finding.

findings/filters (same thing, we use the term rule for them) contain a variety of fields, but they are consistent throughout different findings. description, path, level, display_path, dashboard_name and rationale and some other fields seem to be present in all findings, whereas remediation, compliance and references might be 'None'.

Items contain a path to relevant resources in scoutsuite data. This is how we know which field/resource is vulnerable/misconfigured.

Process overview

High level overview of ZT workflow, involving ScoutSuite data parsing

1. Scoutsuite API: if monkey is ran on island, it calls scoutsuite API to gather scoutsuite data.
2. Scoutsuite data parsing on backend: monkey island receives scoutsuite data.

• Island creates Scoutsuite ZT findings: Island picks out scoutsuite rules relevant to the finding out of scoutsuite report data and based on them determines the status of scoutsuite ZT finding created. Then, scoutsuite rules are aggregated into scoutsuite finding details collection, which is linked to Scoutsuite ZT finding.
• The whole scoutsuite report data is saved in bulk on the database. We need it, because it contains resources we'll want to show in the UI.

3. UI requests ZT report: Island returns all ZT findings: monkey ZT findings and scoutsuite ZT findings. Additionally, scoutsuite report data is also sent to UI.
4. UI displays ZT report: Scoutsuite ZT findings are displayed as shown in the mockups above. In each scoutsuite rule only affected resources are displayed:

• According to path, display_paht and items values of a scoutsuite rule, corresponding resource data is pulled from scoutsuite report data. We display only basic resource fields.
• (Optional) resources are displayed similarly to ScoutSuite report, by using their handlebars templates (same as in UI mockup)