Publish PR 6937: new source amazon sqs #7389

marcosmarxm · 2021-10-26T14:52:44Z

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…st-pr-6937

marcosmarxm · 2021-10-26T14:53:24Z

/test connector=connectors/source-amazon-sqs

🕑 connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386196143
❌ connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386196143
🐛 https://gradle.com/s/l6tphkbcucscs

marcosmarxm · 2021-10-26T15:05:36Z

/test connector=connectors/source-amazon-sqs

🕑 connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386246928
❌ connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386246928
🐛 https://gradle.com/s/5bhixrun5dbim

marcosmarxm · 2021-10-26T15:25:48Z

/test connector=connectors/source-amazon-sqs

🕑 connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386330294
❌ connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386330294
🐛 https://gradle.com/s/muxn3a4g3fznc

marcosmarxm · 2021-10-26T16:02:12Z

/test connector=connectors/source-amazon-sqs

🕑 connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386470201
✅ connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386470201
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                            Stmts   Miss  Cover
	 ---------------------------------------------------
	 source_amazon_sqs/__init__.py       2      0   100%
	 source_amazon_sqs/source.py        99     20    80%
	 ---------------------------------------------------
	 TOTAL                             101     20    80%

marcosmarxm · 2021-10-26T16:11:48Z

/publish connector=connectors/source-amazon-sqs

🕑 connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386507622
✅ connectors/source-amazon-sqs https://github.com/airbytehq/airbyte/actions/runs/1386507622

* Initial commit, working source with static Creds * Typo in example queue url * Adds auto delete of messages after read * Adds visibility timeout * remove insecure comments from AWS IAM Key spec * explicitly set supported sync modes * explicit sync mode should be lower case * Adds unit tests for check, discover, read * remove incremental acceptance test block * remove incremental from conf catalog sample * remove test requirement moto from main req * align int catalog sample with sample_files * fixing catalog configs * acceptance testing config * adds expected records txt * automated formatting changes * remove expected records block from acpt test * Adds Docs page * Ammends formatting on readme * Adds doc link to summary * Improve error handling & debug logging * Adds bootstrap.md * Add a todo suggestion for batch output * Adds SQS to integrations readme list * lower case properties * removed unused line * uses enum for aws region * updates sample configs to use lowercase * required props to lower case * add missed property to lowercase * gradle formatting * Fixing issues from acceptance tests * annotate secrets in spec.json with airbyte_secret * Adds explicit warnings about data less when using Delete Message option * create config files * remove eof line * correct catalog with airbyte creds * format Co-authored-by: Alasdair Brown <me@alasdairb.com>

sdairs added 30 commits October 9, 2021 17:52

Initial commit, working source with static Creds

9a7caf4

Typo in example queue url

4a79ce7

Adds auto delete of messages after read

137bf3c

Adds visibility timeout

92e1c93

remove insecure comments from AWS IAM Key spec

e11f936

explicitly set supported sync modes

2780160

explicit sync mode should be lower case

9eac21a

Adds unit tests for check, discover, read

932e376

remove incremental acceptance test block

1ebc016

remove incremental from conf catalog sample

d093456

remove test requirement moto from main req

1c0cab4

align int catalog sample with sample_files

6971564

fixing catalog configs

7cae0d9

acceptance testing config

a0b2c95

adds expected records txt

d0dec92

automated formatting changes

36f64ef

remove expected records block from acpt test

c3f7ed1

Adds Docs page

f1c8193

Ammends formatting on readme

3ae20c7

Adds doc link to summary

68d34ad

Improve error handling & debug logging

d637b51

Adds bootstrap.md

91e48c8

Add a todo suggestion for batch output

e99bb40

Adds SQS to integrations readme list

734bf52

lower case properties

8994d3f

removed unused line

06f15b4

uses enum for aws region

6eea384

updates sample configs to use lowercase

8686364

required props to lower case

e0a1b8f

add missed property to lowercase

2c9c4d1

marcosmarxm added 2 commits October 26, 2021 11:43

Merge branch 'sqs-source' of github.com:sdairs/airbyte into marcos/te…

1fa48ca

…st-pr-6937

create config files

68354f3

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 26, 2021

marcosmarxm temporarily deployed to more-secrets October 26, 2021 14:53 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 14:55 Inactive

remove eof line

82e02d9

marcosmarxm temporarily deployed to more-secrets October 26, 2021 15:00 Inactive

solve conflict

aeef0c8

marcosmarxm temporarily deployed to more-secrets October 26, 2021 15:06 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 15:08 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 15:28 Inactive

marcosmarxm added 2 commits October 26, 2021 12:57

correct catalog with airbyte creds

111dded

format

43af1b4

marcosmarxm temporarily deployed to more-secrets October 26, 2021 16:03 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 16:05 Inactive

jrhizor temporarily deployed to more-secrets October 26, 2021 16:13 Inactive

solve conflict integration test

102de3f

github-actions bot removed the area/documentation Improvements or additions to documentation label Oct 26, 2021

marcosmarxm temporarily deployed to more-secrets October 26, 2021 16:30 Inactive

marcosmarxm merged commit 9157b3d into master Oct 26, 2021

marcosmarxm deleted the marcos/test-pr-6937 branch October 26, 2021 16:39

jrhizor mentioned this pull request Nov 1, 2021

Bump Airbyte version from 0.30.23-alpha to 0.30.24-alpha #7532

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish PR 6937: new source amazon sqs #7389

Publish PR 6937: new source amazon sqs #7389

marcosmarxm commented Oct 26, 2021

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

Publish PR 6937: new source amazon sqs #7389

Publish PR 6937: new source amazon sqs #7389

Conversation

marcosmarxm commented Oct 26, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

marcosmarxm commented Oct 26, 2021 • edited by github-actions bot Loading

marcosmarxm commented Oct 26, 2021 • edited by github-actions bot Loading

marcosmarxm commented Oct 26, 2021 • edited by github-actions bot Loading

marcosmarxm commented Oct 26, 2021 • edited by github-actions bot Loading

marcosmarxm commented Oct 26, 2021 • edited by github-actions bot Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Oct 26, 2021 •

edited by github-actions bot

Loading