Skip to content

Commit

Permalink
feat!: Encapsulation (#15)
Browse files Browse the repository at this point in the history
* feat: add capsule support

* feat: add capsules, firehose and sqs

* feat!: add capsule support to condition

* feat: add Firehose & SQS support

* chore: add SQS to examples/aws

* docs: update examples/aws README

* feat: add capsule support to sink, firehose & sqs

* chore: update examples/condition

* feat!: add capsule support to transform

* feat: add capsule support to process

* chore: update remaining examples

* chore: remove firehose ingest support

* refactor: processors

* refactor: transforms

* refactor: sink

* refactor: internal/aws

* refactor: condition

* refactor: libsonnet files

* refactor: internal/json

* docs: misc markdown formatting

* refactor: error wrapping

* refactor: error wrapping

* fix: http errors

* fix: foreach test

* refactor: update metadata pattern

* refactor: add ctx to conditions

* style: standardized variables

* style: update results

* docs: typo

* fix: staticcheck fixes
  • Loading branch information
jshlbrd authored Aug 22, 2022
1 parent 5cc9648 commit e46e780
Show file tree
Hide file tree
Showing 158 changed files with 5,111 additions and 3,000 deletions.
17 changes: 11 additions & 6 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,31 @@
<!--- Provide a general summary of your changes in the Title above -->

## Description

<!--- Describe your changes in detail -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->

## How Has This Been Tested?

<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
* [ ] Bug fix (non-breaking change which fixes an issue)
* [ ] New feature (non-breaking change which adds functionality)
* [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Checklist:

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
* [ ] My code follows the code style of this project.
* [ ] My change requires a change to the documentation.
* [ ] I have updated the documentation accordingly.
15 changes: 7 additions & 8 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,15 @@

# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards
Expand All @@ -21,7 +20,7 @@ community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
community
Expand All @@ -32,7 +31,7 @@ Examples of unacceptable behavior include:
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
Expand All @@ -41,7 +40,7 @@ Examples of unacceptable behavior include:

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
Expand All @@ -53,7 +52,7 @@ decisions when appropriate.

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

Expand Down Expand Up @@ -115,7 +114,7 @@ community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Expand Down
6 changes: 4 additions & 2 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Contributors

Thank you to [all of our contributors](https://github.com/brexhq/substation/graphs/contributors). For reviewing per-file contributions, run the following commands:

```sh
git blame <file>
git log -p <file>
```

# Brex Team
- [Josh Liburdi](https://twitter.com/jshlbrd)
- [Julie Agnes Sparks](https://twitter.com/JulieASparks)
* [Josh Liburdi](https://twitter.com/jshlbrd)
* [Julie Agnes Sparks](https://twitter.com/JulieASparks)
43 changes: 21 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,28 @@ Substation also provides Go packages for filtering and modifying JSON data.

## Features

As an event-driven ITL application, Substation has these features:
As an event-driven ingest, transform, and load application, Substation has these features:

- real-time event filtering and processing
- cross-dataset event correlation and enrichment
- concurrent event routing to downstream systems
- runs on containers, built for extensibility
- support for new event filters and processors
- support for new ingest sources and load destinations
- supports creation of custom applications (e.g., multi-cloud)
* real-time event filtering and processing
* cross-dataset event correlation and enrichment
* concurrent event routing to downstream systems
* runs on containers, built for extensibility
+ support for new event filters and processors
+ support for new ingest sources and load destinations
+ supports creation of custom applications (e.g., multi-cloud)

As a package, Substation has these features:

- [evaluate and filter JSON objects](condition/)
- [modify data from, to, and in-place as JSON objects](process/)
* [evaluate and filter JSON objects](condition/)
* [modify data from, to, and in-place as JSON objects](process/)

## Use Cases

Substation was originally designed to support the mission of achieving high quality data for threat hunting, threat detection, and incident response, but it can be used to move data between many distributed systems and services. Here are some example use cases:

- data availability: sink data to an intermediary streaming service such as AWS Kinesis, then concurrently sink it to a data lake, data warehouse, and SIEM
- data consistency: normalize data across every dataset using a permissive schema such as the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html)
- data completeness: enrich data by integrating AWS Lambda functions and building self-populating AWS DynamoDB tables for low latency, real-time event context
* data availability: sink data to an intermediary streaming service such as AWS Kinesis, then concurrently sink it to a data lake, data warehouse, and SIEM
* data consistency: normalize data across every dataset using a permissive schema such as the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html)
* data completeness: enrich data by integrating AWS Lambda functions and building self-populating AWS DynamoDB tables for low latency, real-time event context

## Example Data Pipelines

Expand Down Expand Up @@ -71,10 +71,9 @@ graph TD

The complexity of a data pipeline, including its features and how it connects with other pipelines, is up to the user. The diagram below shows two complex data pipelines that have these feature:

- both pipelines write unmodified data to intermediary streaming data storage (e.g., AWS Kinesis) to support concurrent consumers and downstream systems
- both pipelines transform data by enriching it from their own inter-pipeline metadata lookup (e.g., AWS DynamoDB)
- pipeline Y additionally transforms data by enriching it from pipeline X's metadata lookup

* both pipelines write unmodified data to intermediary streaming data storage (e.g., AWS Kinesis) to support concurrent consumers and downstream systems
* both pipelines transform data by enriching it from their own inter-pipeline metadata lookup (e.g., AWS DynamoDB)
* pipeline Y additionally transforms data by enriching it from pipeline X's metadata lookup

```mermaid
Expand Down Expand Up @@ -127,7 +126,7 @@ As a toolkit, Substation makes no assumptions about how data pipelines are confi

## Quickstart

Users can use the steps below to test Substation's functionality. We recommend doing the steps below in a Docker container (we've included [Visual Studio Code configurations](https://code.visualstudio.com/docs/remote/containers) for developing and testing Substation in `.devcontainer/` and `.vscode/`).
Users can use the steps below to test Substation's functionality. We recommend doing the steps below in a Docker container (we've included [Visual Studio Code configurations](https://code.visualstudio.com/docs/remote/containers) for developing and testing Substation in `.devcontainer/` and `.vscode/` ).

### Step 0: Set Environment Variable

Expand Down Expand Up @@ -173,10 +172,10 @@ The [examples/aws](/examples/aws/) directory contains a fully-featured data pipe

More documentation about Substation can be found across the project, including:

- [Configuration Syntax](/config/)
- [Deploying to AWS](/examples/aws/)
- [Using Conditions to Evaluate JSON Objects](/condition/)
- [Using Processors to Modify JSON Objects](/process/)
* [Configuration Syntax](/config/)
* [Deploying to AWS](/examples/aws/)
* [Using Conditions to Evaluate JSON Objects](/condition/)
* [Using Processors to Modify JSON Objects](/process/)

## Licensing

Expand Down
1 change: 1 addition & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
# Responsible Disclosure

To report security issues in Substation, please follow [Brex's Responsible Disclosure process](https://www.brex.com/security/responsible-disclosure/).
5 changes: 5 additions & 0 deletions build/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,19 @@
# build

This directory contains build configurations and scripts for managing Substation.

## config

Contains build scripts that support compiling Substation Jsonnet config files into JSON and deploying configurations to services (e.g., AWS AppConfig).

## container

Contains Docker build files for each application under [cmd/](/cmd/).

## terraform

Contains Terraform modules for deploying Substation pipelines.

## scripts

Contains miscellaneous scripts that assist with deploying and maintaining Substation pipelines.
Loading

0 comments on commit e46e780

Please sign in to comment.