feat!: Encapsulation (#15)

* feat: add capsule support

* feat: add capsules, firehose and sqs

* feat!: add capsule support to condition

* feat: add Firehose & SQS support

* chore: add SQS to examples/aws

* docs: update examples/aws README

* feat: add capsule support to sink, firehose & sqs

* chore: update examples/condition

* feat!: add capsule support to transform

* feat: add capsule support to process

* chore: update remaining examples

* chore: remove firehose ingest support

* refactor: processors

* refactor: transforms

* refactor: sink

* refactor: internal/aws

* refactor: condition

* refactor: libsonnet files

* refactor: internal/json

* docs: misc markdown formatting

* refactor: error wrapping

* refactor: error wrapping

* fix: http errors

* fix: foreach test

* refactor: update metadata pattern

* refactor: add ctx to conditions

* style: standardized variables

* style: update results

* docs: typo

* fix: staticcheck fixes

.github/pull_request_template.md

@@ -1,26 +1,31 @@
 <!--- Provide a general summary of your changes in the Title above -->
 
 ## Description
+
 <!--- Describe your changes in detail -->
 
 ## Motivation and Context
+
 <!--- Why is this change required? What problem does it solve? -->
 <!--- If it fixes an open issue, please link to the issue here. -->
 
 ## How Has This Been Tested?
+
 <!--- Please describe in detail how you tested your changes. -->
 <!--- Include details of your testing environment, and the tests you ran to -->
 <!--- see how your change affects other areas of the code, etc. -->
 
 ## Types of changes
+
 <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
-- [ ] Bug fix (non-breaking change which fixes an issue)
-- [ ] New feature (non-breaking change which adds functionality)
-- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+* [ ] Bug fix (non-breaking change which fixes an issue)
+* [ ] New feature (non-breaking change which adds functionality)
+* [ ] Breaking change (fix or feature that would cause existing functionality to change)
 
 ## Checklist:
+
 <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
-- [ ] My code follows the code style of this project.
-- [ ] My change requires a change to the documentation.
-- [ ] I have updated the documentation accordingly.
+* [ ] My code follows the code style of this project.
+* [ ] My change requires a change to the documentation.
+* [ ] I have updated the documentation accordingly.

CODE_OF_CONDUCT.md

@@ -1,16 +1,15 @@
-
 # Contributor Covenant Code of Conduct
 
 ## Our Pledge
 
 We as members, contributors, and leaders pledge to make participation in our
 community a harassment-free experience for everyone, regardless of age, body
 size, visible or invisible disability, ethnicity, sex characteristics, gender
-identity and expression, level of experience, education, socio-economic status,
+identity and expression, level of experience, education, socio-economic status, 
 nationality, personal appearance, race, caste, color, religion, or sexual
 identity and orientation.
 
-We pledge to act and interact in ways that contribute to an open, welcoming,
+We pledge to act and interact in ways that contribute to an open, welcoming, 
 diverse, inclusive, and healthy community.
 
 ## Our Standards
@@ -21,7 +20,7 @@ community include:
 * Demonstrating empathy and kindness toward other people
 * Being respectful of differing opinions, viewpoints, and experiences
 * Giving and gracefully accepting constructive feedback
-* Accepting responsibility and apologizing to those affected by our mistakes,
+* Accepting responsibility and apologizing to those affected by our mistakes, 
   and learning from the experience
 * Focusing on what is best not just for us as individuals, but for the overall
   community
@@ -32,7 +31,7 @@ Examples of unacceptable behavior include:
   any kind
 * Trolling, insulting or derogatory comments, and personal or political attacks
 * Public or private harassment
-* Publishing others' private information, such as a physical or email address,
+* Publishing others' private information, such as a physical or email address, 
   without their explicit permission
 * Other conduct which could reasonably be considered inappropriate in a
   professional setting
@@ -41,7 +40,7 @@ Examples of unacceptable behavior include:
 
 Community leaders are responsible for clarifying and enforcing our standards of
 acceptable behavior and will take appropriate and fair corrective action in
-response to any behavior that they deem inappropriate, threatening, offensive,
+response to any behavior that they deem inappropriate, threatening, offensive, 
 or harmful.
 
 Community leaders have the right and responsibility to remove, edit, or reject
@@ -53,7 +52,7 @@ decisions when appropriate.
 
 This Code of Conduct applies within all community spaces, and also applies when
 an individual is officially representing the community in public spaces.
-Examples of representing our community include using an official e-mail address,
+Examples of representing our community include using an official e-mail address, 
 posting via an official social media account, or acting as an appointed
 representative at an online or offline event.
 
@@ -115,7 +114,7 @@ community.
 
 ## Attribution
 
-This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], 
 version 2.1, available at
 [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
 

CONTRIBUTORS.md

@@ -1,10 +1,12 @@
 # Contributors
+
 Thank you to [all of our contributors](https://github.com/brexhq/substation/graphs/contributors). For reviewing per-file contributions, run the following commands:
+
 ```sh
 git blame <file>
 git log -p <file>
 ```
 
 # Brex Team
-- [Josh Liburdi](https://twitter.com/jshlbrd)
-- [Julie Agnes Sparks](https://twitter.com/JulieASparks)
+* [Josh Liburdi](https://twitter.com/jshlbrd)
+* [Julie Agnes Sparks](https://twitter.com/JulieASparks)

README.md

@@ -10,28 +10,28 @@ Substation also provides Go packages for filtering and modifying JSON data.
 
 ## Features
 
-As an event-driven ITL application, Substation has these features:
+As an event-driven ingest, transform, and load application, Substation has these features:
 
-- real-time event filtering and processing
-- cross-dataset event correlation and enrichment
-- concurrent event routing to downstream systems
-- runs on containers, built for extensibility
-  - support for new event filters and processors
-  - support for new ingest sources and load destinations
-  - supports creation of custom applications (e.g., multi-cloud)
+* real-time event filtering and processing
+* cross-dataset event correlation and enrichment
+* concurrent event routing to downstream systems
+* runs on containers, built for extensibility
+  + support for new event filters and processors
+  + support for new ingest sources and load destinations
+  + supports creation of custom applications (e.g., multi-cloud)
 
 As a package, Substation has these features:
 
-- [evaluate and filter JSON objects](condition/)
-- [modify data from, to, and in-place as JSON objects](process/)
+* [evaluate and filter JSON objects](condition/)
+* [modify data from, to, and in-place as JSON objects](process/)
 
 ## Use Cases
 
 Substation was originally designed to support the mission of achieving high quality data for threat hunting, threat detection, and incident response, but it can be used to move data between many distributed systems and services. Here are some example use cases:
 
-- data availability: sink data to an intermediary streaming service such as AWS Kinesis, then concurrently sink it to a data lake, data warehouse, and SIEM
-- data consistency: normalize data across every dataset using a permissive schema such as the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html)
-- data completeness: enrich data by integrating AWS Lambda functions and building self-populating AWS DynamoDB tables for low latency, real-time event context
+* data availability: sink data to an intermediary streaming service such as AWS Kinesis, then concurrently sink it to a data lake, data warehouse, and SIEM
+* data consistency: normalize data across every dataset using a permissive schema such as the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html)
+* data completeness: enrich data by integrating AWS Lambda functions and building self-populating AWS DynamoDB tables for low latency, real-time event context
 
 ## Example Data Pipelines
 
@@ -71,10 +71,9 @@ graph TD
 
 The complexity of a data pipeline, including its features and how it connects with other pipelines, is up to the user. The diagram below shows two complex data pipelines that have these feature:
 
-- both pipelines write unmodified data to intermediary streaming data storage (e.g., AWS Kinesis) to support concurrent consumers and downstream systems
-- both pipelines transform data by enriching it from their own inter-pipeline metadata lookup (e.g., AWS DynamoDB)
-- pipeline Y additionally transforms data by enriching it from pipeline X's metadata lookup
-
+* both pipelines write unmodified data to intermediary streaming data storage (e.g., AWS Kinesis) to support concurrent consumers and downstream systems
+* both pipelines transform data by enriching it from their own inter-pipeline metadata lookup (e.g., AWS DynamoDB)
+* pipeline Y additionally transforms data by enriching it from pipeline X's metadata lookup
 
 ```mermaid
 
@@ -127,7 +126,7 @@ As a toolkit, Substation makes no assumptions about how data pipelines are confi
 
 ## Quickstart
 
-Users can use the steps below to test Substation's functionality. We recommend doing the steps below in a Docker container (we've included [Visual Studio Code configurations](https://code.visualstudio.com/docs/remote/containers) for developing and testing Substation in `.devcontainer/` and `.vscode/`).
+Users can use the steps below to test Substation's functionality. We recommend doing the steps below in a Docker container (we've included [Visual Studio Code configurations](https://code.visualstudio.com/docs/remote/containers) for developing and testing Substation in `.devcontainer/` and `.vscode/` ).
 
 ### Step 0: Set Environment Variable
 
@@ -173,10 +172,10 @@ The [examples/aws](/examples/aws/) directory contains a fully-featured data pipe
 
 More documentation about Substation can be found across the project, including:
 
-- [Configuration Syntax](/config/)
-- [Deploying to AWS](/examples/aws/)
-- [Using Conditions to Evaluate JSON Objects](/condition/)
-- [Using Processors to Modify JSON Objects](/process/)
+* [Configuration Syntax](/config/)
+* [Deploying to AWS](/examples/aws/)
+* [Using Conditions to Evaluate JSON Objects](/condition/)
+* [Using Processors to Modify JSON Objects](/process/)
 
 ## Licensing
 

SECURITY.md

@@ -1,2 +1,3 @@
 # Responsible Disclosure
+
 To report security issues in Substation, please follow [Brex's Responsible Disclosure process](https://www.brex.com/security/responsible-disclosure/).

build/README.md

@@ -1,14 +1,19 @@
 # build
+
 This directory contains build configurations and scripts for managing Substation.
 
 ## config
+
 Contains build scripts that support compiling Substation Jsonnet config files into JSON and deploying configurations to services (e.g., AWS AppConfig).
 
 ## container
+
 Contains Docker build files for each application under [cmd/](/cmd/). 
 
 ## terraform
+
 Contains Terraform modules for deploying Substation pipelines. 
 
 ## scripts
+
 Contains miscellaneous scripts that assist with deploying and maintaining Substation pipelines.