🎉 Redshift Source: create secure-only version

[VitaliiMaltsev]

What

We want to create secure-only versions of connectors that can be used in the Airbyte cloud. The idea is that these connectors inherently prevent certain insecure connections such as connecting to a database over the public internet without encryption.

How

Modified the connector's spec to hide any options which allow the user to disable TLS. Changed the connector to enable TLS by default if the TLS option is not specified

Recommended reading order

1. x.java
2. y.python

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions
• Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions
• Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

vmaltsev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[VitaliiMaltsev]

/test connector=connectors/source-redshift-strict-encrypt

🕑 connectors/source-redshift-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1345035522
✅ connectors/source-redshift-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1345035522
No Python unittests run

[irynakruk]

LGTM

[tuliren]

@VitaliiMaltsev, please follow and check off the Pre-merge Checklist to properly register this new connector. Not everything is needed to add a strict encrypt connector, but currently some files are missing (e.g. registering the connector in the source_definitions.yaml file.

Also don't forget to register this new connector in NormalizationRunnerFactory.

[VitaliiMaltsev]

Also don't forget to register this new connector in NormalizationRunnerFactory.
@tuliren I don't think we need to register this connector in NormalizationRunnerFactory, cause it's source connector. I added Redshift Destination secure-only to NormalizationRunnerFactory in other PR https://github.com/airbytehq/airbyte/pull/7121/files

[alexandr-shegeda]

@VitaliiMaltsev, please follow and check off the Pre-merge Checklist to properly register this new connector. Not everything is needed to add a strict encrypt connector, but currently some files are missing (e.g. registering the connector in the source_definitions.yaml file.

hi @tuliren
thank you for your comment, just a quick note about adding new connectors to _definitions.yaml, as it was discussed previously with @sherifnada - we will add -strict-encrypt connectors later to the different config files because all of them should be available only for the cloud version

[sherifnada]

Do all redshift clusters support encrypted connections? (I don't remember off the top of my head)

In other words, can we safely assume that we can always use ssl=true in the JDBC URL?

If so, then why don't we always do SSL by default? It eliminates the need for a connector fork and even having this as an option in redshift's config

[VitaliiMaltsev]

Do all redshift clusters support encrypted connections? (I don't remember off the top of my head)

In other words, can we safely assume that we can always use ssl=true in the JDBC URL?

If so, then why don't we always do SSL by default? It eliminates the need for a connector fork and even having this as an option in redshift's config

@sherifnada Yes, the traffic will be encrypted for all redfshift clusters with the included ssl = true option in jdbc url. Do you mean not to create new strict-encrypt connectors, but to enable this option by default in the source-redshift and destination-redshift connectors?

[sherifnada]

@VitaliiMaltsev correct. And also removing the option from the redshift connector and always setting it to true

[VitaliiMaltsev]

@VitaliiMaltsev correct. And also removing the option from the redshift connector and always setting it to true

@sherifnada done in #7234. Please review

[sherifnada]

LGTM we can close this one