Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Mongodb secure only source #6859

Merged
merged 6 commits into from
Oct 11, 2021
Merged

Added Mongodb secure only source #6859

merged 6 commits into from
Oct 11, 2021

Conversation

irynakruk
Copy link
Contributor

@irynakruk irynakruk commented Oct 7, 2021 •
edited
Loading

What

We want to create secure-only versions of connectors that can be used in the Airbyte cloud. The idea is that these connectors inherently prevent certain insecure connections such as connecting to a database over the public internet without encryption.

How

Created a new connector source--secure based on the current connector, modified the connector's spec to hide any options which allow the user to disable TLS change the connector to enable TLS by default if the TLS option is not specified.

Recommended reading order

  1. MongodbSourceStrictEncrypt.java
  2. tests

Pre-merge Checklist

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

@github-actions github-actions bot added the area/connectors Connector related issues label Oct 7, 2021
@irynakruk irynakruk linked an issue Oct 7, 2021 that may be closed by this pull request
Create a secure-only MongoDb source #6715
Closed
6 tasks
@irynakruk irynakruk temporarily deployed to more-secrets October 7, 2021 14:48 Inactive
@irynakruk irynakruk temporarily deployed to more-secrets October 7, 2021 14:53 Inactive
@irynakruk irynakruk temporarily deployed to more-secrets October 7, 2021 16:49 Inactive
@irynakruk irynakruk marked this pull request as ready for review October 7, 2021 16:52
@irynakruk irynakruk requested a review from Phlair October 7, 2021 16:53
@irynakruk
Copy link
Contributor Author

irynakruk commented Oct 7, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1317159659
❌ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1317159659
🐛 https://gradle.com/s/4jigtvj25qeqa

@jrhizor jrhizor temporarily deployed to more-secrets October 7, 2021 17:03 Inactive
@irynakruk irynakruk temporarily deployed to more-secrets October 7, 2021 18:14 Inactive
@irynakruk
Copy link
Contributor Author

irynakruk commented Oct 7, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1317468551
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1317468551
No Python unittests run

@jrhizor jrhizor temporarily deployed to more-secrets October 7, 2021 18:43 Inactive
@irynakruk irynakruk requested review from tuliren and removed request for Phlair October 8, 2021 13:52
tuliren
tuliren approved these changes Oct 8, 2021
View reviewed changes
Copy link
Contributor

@tuliren tuliren left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nitpick. Can we simplify the name? Maybe something like source-mongodb-secure or source-mongodb-encrypt.

Never mind. I see that -strict-encrypt is the universal suffix we use for all these connectors. Let's keep it then.

...encrypt/src/main/java/io.airbyte.integrations.source.mongodb/MongodbSourceStrictEncrypt.java
((ObjectNode) spec.getConnectionSpecification().get("properties")).remove("tls");
((ObjectNode) spec.getConnectionSpecification().get("properties").get("instance_type").get("oneOf").get(0).get("properties")).remove("tls");
return spec;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a comment for this method about the rationale behind removing the tls field?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I've added, thanks

airbyte-integrations/connectors/source-mongodb-strict-encrypt/acceptance-test-config.yml Outdated
connector_image: airbyte/source-mongodb-strict-encrypt:dev
tests:
spec:
- spec_path: "src/test/resources/expected_spec.json"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this file is not needed for Java source connector.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, removed

airbyte-integrations/connectors/source-mongodb-v2/src/main/resources/spec.json
"type": "string",
"enum": ["standalone"],
"default": "standalone"
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch.

@irynakruk irynakruk temporarily deployed to more-secrets October 11, 2021 14:28 Inactive
@irynakruk
Copy link
Contributor Author

irynakruk commented Oct 11, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1329462755
✅ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1329462755

@jrhizor jrhizor temporarily deployed to more-secrets October 11, 2021 14:54 Inactive
@irynakruk irynakruk temporarily deployed to more-secrets October 11, 2021 15:00 Inactive
@irynakruk irynakruk temporarily deployed to more-secrets October 11, 2021 15:22 Inactive
@irynakruk irynakruk merged commit 77fd7d9 into master Oct 11, 2021
@irynakruk irynakruk deleted the irynakruk/6715-mongodb-secure-source branch October 11, 2021 16:04
@jrhizor jrhizor mentioned this pull request Oct 13, 2021
Merged
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
@irynakruk @schlattk
Added Mongodb secure only source (airbytehq#6859)
a3da08d 
* Added mongodb secure source
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tuliren tuliren tuliren approved these changes

@alexandr-shegeda alexandr-shegeda alexandr-shegeda approved these changes

@VitaliiMaltsev VitaliiMaltsev Awaiting requested review from VitaliiMaltsev

Assignees
No one assigned
Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create a secure-only MongoDb source
4 participants
@irynakruk @tuliren @alexandr-shegeda @jrhizor