Fix ssh tunneling for normalization #6396

Phlair · 2021-09-23T10:02:08Z

What

New lines as part of OAuth were added into /airbyte-integrations/bases/base-normalization/entrypoint.sh that remove the config file so we don't expose secrets to non-airbyte docker images during custom dbt transformations.

As a side-effect this broke ssh tunnelling in normalisation because it relied on using the config file to get relevant parameters to create the tunnel.

How

Creating a config ssh.json specific for ssh tunnelling in normalization's transform.py and using that instead to get parameters for tunnel. This only contains necessary params for tunnelling and no OAuth secrets.

Recommended reading order

transform.py
sshtunneling.sh
others

Phlair · 2021-09-23T10:08:32Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265369827
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265369827
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     40    71%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    395    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     40    71%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    395    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     117      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     46    67%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    181    83%

ChristopheDuong

It'd be nice to at least add it to normalization unit_test TestTransformConfig for the generated file for ssh too

Phlair · 2021-09-23T10:37:40Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265456207
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265456207
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     35    75%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    390    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     35    75%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    390    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     117      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         139     46    67%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1035    181    83%

Phlair · 2021-09-23T11:09:51Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265553758
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1265553758

Phlair · 2021-09-23T11:53:55Z

Looks like same problem as this on /publish command but the new version is up on docker hub. Confirmed the new image has these changes in so going to merge this.

Phlair added 4 commits September 22, 2021 21:21

switch to custom file for ssh config in normalization

283f45d

Merge branch 'master' into george/normalisation-ssh-fix

cf2adfa

bump version

82299ec

get local port properly

de3fc8e

github-actions bot added area/worker Related to worker normalization labels Sep 23, 2021

Phlair requested review from ChristopheDuong and cgardens September 23, 2021 10:02

ChristopheDuong approved these changes Sep 23, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets September 23, 2021 10:10 Inactive

added unit test for write_ssh_config

3a3af7c

Phlair temporarily deployed to more-secrets September 23, 2021 10:38 Inactive

jrhizor temporarily deployed to more-secrets September 23, 2021 10:39 Inactive

jrhizor temporarily deployed to more-secrets September 23, 2021 11:12 Inactive

Phlair temporarily deployed to more-secrets September 23, 2021 11:54 Inactive

format

8b758e1

Phlair temporarily deployed to more-secrets September 23, 2021 12:13 Inactive

ChristopheDuong mentioned this pull request Sep 23, 2021

Skip updating spec cache when publishing normalization docker image #6399

Merged

Phlair merged commit 3d8625e into master Sep 23, 2021

Phlair deleted the george/normalisation-ssh-fix branch September 23, 2021 13:08

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ssh tunneling for normalization #6396

Fix ssh tunneling for normalization #6396

Phlair commented Sep 23, 2021

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

ChristopheDuong left a comment

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

Phlair commented Sep 23, 2021 •

edited

Loading

Fix ssh tunneling for normalization #6396

Fix ssh tunneling for normalization #6396

Conversation

Phlair commented Sep 23, 2021

What

How

Recommended reading order

Phlair commented Sep 23, 2021 • edited by github-actions bot Loading

ChristopheDuong left a comment

Choose a reason for hiding this comment

Phlair commented Sep 23, 2021 • edited by github-actions bot Loading

Phlair commented Sep 23, 2021 • edited by github-actions bot Loading

Phlair commented Sep 23, 2021 • edited Loading

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

Phlair commented Sep 23, 2021 •

edited by github-actions bot

Loading

Phlair commented Sep 23, 2021 •

edited

Loading