feat!: Encapsulation

[jshlbrd]

Description

• Makes encapsulation the default pattern for handling data in all Substation applications and public APIs (see changes in /config/)
  • Adds backwards compatible functions for handling non-encapsulated data
• Adds AWS Firehose sink support
• Adds AWS SQS source and sink support
• Adds context support condition API
• Updates code style of public APIs

This is a non-breaking change for ITL applications and a breaking change for the public APIs.

Motivation and Context

The motivation for adding encapsulation was the need to let users simultaneously handle both data (structured or unstructured) and metadata. This has two advantages:

• sources can add metadata when data is ingested
• metadata can be tracked alongside unstructured (e.g., binary) data

This change means that users can now access and interpret information only the source application knows; for example:

• substation/file: the filename and file size of the source file that data is read from
• substation/aws/kinesis: the Kinesis stream data is delivered by and approximate arrival time of data
• substation/aws/s3: the S3 bucket and object that data is read from

It also provides total separation of metadata from data, which is most useful in sink applications; for example:

• http: storing HTTP headers as metadata instead of in the data object
• s3: storing the S3 prefix as metadata instead of in the data object
• sumologic: storing the category as metadata instead of in the data object

This gives us some flexibility for growth in the future. For example, instead of relying on configured settings for Kinesis streams and S3 buckets, we can add features to let users dynamically apply these from metadata.

How Has This Been Tested?

• All public APIs had their unit tests updated
• Firehose and SQS features were tested in a development AWS account

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[jshlbrd]

Go lint is failing due to an unavoidable issue mocking an SQS API call for a unit test. The fix for this is to stop using golint (it's deprecated) and switch to staticcheck (recommended replacement), we'll handle that in a different PR.