Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Adds Gzip Processor and Content Inspector #2

Merged
merged 6 commits into from
Apr 25, 2022
Merged

Conversation

jshlbrd
Copy link
Collaborator

@jshlbrd jshlbrd commented Apr 22, 2022

Description

This PR adds two new features:

  • Gzip processor, to be used for converting data to and from Gzip
    • This is the first "data conversion" processor we've built, more detail on why is mentioned below
  • Content inspector, to be used for checking the content (MIME) type of data
    • This accompanies the Gzip processor and should be used before applying data conversion processors

Motivation and Context

Substation assumes that all data coming into and going out of a pipeline is JSON, but there are use cases for converting bodies of data to and from different formats. This PR adds support for Gzip and inspection of any MIME type, but the design should be reused in future "data conversion" processors. These could include:

  • encoded data (base64, hex, etc.)
  • compressed data (zip, zlib, etc.)
  • formatted data (CSV, XML, etc.)

For processor options, the simplest way to describe conversion is with an option called direction that accepts one of two strings: "to" and "from". This should be applicable to any type of data conversion (e.g., "from base64", "to zlib", "from CSV", "to XML") and I think it's better than having bespoke options ("decode base64", "compress zlib", etc.), but I'm open to alternative suggestions.

How Has This Been Tested?

Added a bunch of unit tests:

  • Gzip processor has unit tests for every option
  • Byter and Channeler factories have a unit test for the Gzip processor options
  • Content inspector has unit tests for every option (success / fail of content type, with negation)
  • Inspector factory has a unit test for the Content inspector options

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd marked this pull request as ready for review April 22, 2022 00:58
@jshlbrd jshlbrd requested a review from a team as a code owner April 22, 2022 00:58
Copy link
Contributor

@julieagnessparks julieagnessparks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Great commenting added for the additional functionality :)

@jshlbrd jshlbrd merged commit cdd2999 into main Apr 25, 2022
@jshlbrd jshlbrd deleted the jshlbrd/feat-gzip branch April 25, 2022 18:40
@jshlbrd jshlbrd mentioned this pull request Apr 25, 2022
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants