🎉 Source Square: migrate to Beta (YAML) #19369

artem1205 · 2022-11-13T22:59:25Z

What

migrate to YAML

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

artem1205 · 2022-11-14T21:55:56Z

/test connector=connectors/source-square

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3465491575
✅ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3465491575
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_square/source.py           4      0   100%
source_square/__init__.py         2      0   100%
source_square/components.py      84     36    57%
-------------------------------------------------
TOTAL                            90     36    60%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

artem1205 · 2022-11-15T14:15:34Z

/test connector=connectors/source-square

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471226367
❌ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471226367
🐛 https://gradle.com/s/js22p7452bw4i

Build Failed

Test summary info:

Could not find result summary

artem1205 · 2022-11-15T15:29:52Z

/test connector=connectors/source-square

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471826443
❌ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471826443
🐛 https://gradle.com/s/kdjryvmby2maq

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >300.0s
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
============ 2 failed, 41 passed, 31 warnings in 736.22s (0:12:16) =============

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471826443
✅ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3471826443
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_square/source.py           4      0   100%
source_square/__init__.py         2      0   100%
source_square/components.py      84      6    93%
-------------------------------------------------
TOTAL                            90      6    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

All Passed

airbyte-integrations/connectors/source-square/source_square/components.py

girarda · 2022-11-18T02:07:45Z

airbyte-integrations/connectors/source-square/source_square/components.py

+
+
+@dataclass
+class SquareSubstreamSlicer(SquareSlicer):


what missing features is this covering for? Would love to find a way to reuse existing components or make this generalizable enough that it can be reused

This class is similar to SubstreamSlicer, and even can be used here, but it utilised 1:1 relation (parent:child). In my case I need to get all parent records (locations stream), get id from them and iterate in batch (10 ids in request)

Why do we want to batch 10 the slices in batches of 10?

we want to batch 10 locations to optimise requests

Could you create an issue advocating for adding this functionality into the lowcode CDK as described here?

If I'm understanding correctly, the additional features needed from the substream slicer are:

parsing a subfield of the parent record/slice

batching multiple parent streams in the output

same thing as mentioned above

Done https://github.com/airbytehq/airbyte/issues/19760

girarda

this looks pretty good. can you document the custom components so we know the intent and can find ways to generalize them in the future?

I'll approve afterwards

brianjlai

Only light comments, but also echoing Alex's feedback to document what specific functionality these custom component solve for in the context of the integration.

airbyte-integrations/connectors/source-square/source_square/components.py

artem1205 · 2022-12-13T23:13:58Z

/test connector=connectors/source-square

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3690162060
❌ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3690162060
🐛 https://gradle.com/s/5hrxj5fwifmwk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
============ 1 failed, 42 passed, 31 warnings in 378.30s (0:06:18) =============

artem1205 · 2022-12-14T12:05:50Z

/test connector=connectors/source-square

🕑 connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3694620714
✅ connectors/source-square https://github.com/airbytehq/airbyte/actions/runs/3694620714
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_square/source.py           4      0   100%
source_square/__init__.py         2      0   100%
source_square/components.py      44      4    91%
-------------------------------------------------
TOTAL                            50      4    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

All Passed

artem1205 · 2022-12-14T12:46:55Z

/publish connector=connectors/source-square auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-square
https://github.com/airbytehq/airbyte/actions/runs/3694926560

Connector	Did it publish?	Were definitions generated?
connectors/source-square	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

artem1205 · 2022-12-16T18:05:05Z

/publish connector=connectors/source-square

🕑 Publishing the following connectors:
connectors/source-square
https://github.com/airbytehq/airbyte/actions/runs/3715444597

Connector	Did it publish?	Were definitions generated?
connectors/source-square	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

fixed

octavia-squidington-iv added area/connectors Connector related issues connectors/source/square area/documentation Improvements or additions to documentation labels Nov 13, 2022

artem1205 temporarily deployed to more-secrets November 15, 2022 11:18 Inactive

artem1205 temporarily deployed to more-secrets November 15, 2022 11:34 Inactive

artem1205 added 5 commits November 15, 2022 15:15

🐛 Source Square: migrate to Beta (YAML)

57f8a14

Source Square: update CDK, reorder properties

bc8a475

Source Square: update docs

89857c5

Source Square: add tests; change release stage to beta

b94f22a

Source Square: fix tests

280f228

artem1205 force-pushed the artem1205/source-square-lowcode-18705 branch from 6348742 to 280f228 Compare November 15, 2022 14:15

artem1205 temporarily deployed to more-secrets November 15, 2022 14:17 Inactive

artem1205 requested review from girarda and pedroslopez November 15, 2022 15:03

Source Square: fix tests formatting

08c24ec

artem1205 temporarily deployed to more-secrets November 15, 2022 15:31 Inactive

Source Square: fix tests formatting

9b9b485

girarda requested review from brianjlai and removed request for pedroslopez November 16, 2022 15:06

artem1205 temporarily deployed to more-secrets November 16, 2022 15:06 Inactive

artem1205 mentioned this pull request Nov 17, 2022

[EPIC] Source Square to Beta #14166

Closed

10 tasks

girarda reviewed Nov 18, 2022

View reviewed changes

Source Square: Add token_expiry_date_format to oauth authenticator

ba1c14c

artem1205 temporarily deployed to more-secrets November 22, 2022 16:42 Inactive

girarda previously requested changes Nov 22, 2022

View reviewed changes

brianjlai reviewed Nov 23, 2022

View reviewed changes

airbyte-integrations/connectors/source-square/source_square/components.py Outdated Show resolved Hide resolved

Source Square: ref component

12bc2f7

artem1205 temporarily deployed to more-secrets December 9, 2022 18:39 — with GitHub Actions Inactive

Source Square: Refactor Slicer; inherit from datetime slicer

6b48d6f

artem1205 temporarily deployed to more-secrets December 9, 2022 20:46 — with GitHub Actions Inactive

artem1205 requested a review from sherifnada December 9, 2022 20:50

sherifnada approved these changes Dec 10, 2022

View reviewed changes

Merge branch 'master' into artem1205/source-square-lowcode-18705

f8bbff4

artem1205 temporarily deployed to more-secrets December 13, 2022 23:15 — with GitHub Actions Inactive

Source Square: test fix

5c6cc1b

artem1205 temporarily deployed to more-secrets December 14, 2022 12:07 — with GitHub Actions Inactive

Source Square: publish low-code rc1

a47841d

artem1205 temporarily deployed to more-secrets December 14, 2022 12:46 — with GitHub Actions Inactive

artem1205 temporarily deployed to more-secrets December 14, 2022 12:47 — with GitHub Actions Inactive

Source Square: update version

36d1ba5

artem1205 temporarily deployed to more-secrets December 16, 2022 18:06 — with GitHub Actions Inactive

artem1205 temporarily deployed to more-secrets December 16, 2022 18:07 — with GitHub Actions Inactive

auto-bump connector version

1f3f37d

octavia-squidington-iii temporarily deployed to more-secrets December 16, 2022 18:33 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets December 16, 2022 18:34 — with GitHub Actions Inactive

artem1205 merged commit 697b91a into master Dec 16, 2022

artem1205 deleted the artem1205/source-square-lowcode-18705 branch December 16, 2022 18:58

octavia-squidington-iii mentioned this pull request Dec 18, 2022

Bump Airbyte version from 0.40.25 to 0.40.26 #20630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Square: migrate to Beta (YAML) #19369

🎉 Source Square: migrate to Beta (YAML) #19369

artem1205 commented Nov 13, 2022 •

edited

Loading

artem1205 commented Nov 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Nov 15, 2022 •

edited by github-actions bot

Loading

artem1205 commented Nov 15, 2022 •

edited by github-actions bot

Loading

girarda Nov 18, 2022

artem1205 Nov 18, 2022

girarda Nov 21, 2022

artem1205 Nov 22, 2022

sherifnada Dec 9, 2022

artem1205 Dec 9, 2022

girarda left a comment

brianjlai left a comment

artem1205 commented Dec 13, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 16, 2022 •

edited by github-actions bot

Loading

🎉 Source Square: migrate to Beta (YAML) #19369

🎉 Source Square: migrate to Beta (YAML) #19369

Conversation

artem1205 commented Nov 13, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

artem1205 commented Nov 14, 2022 • edited by github-actions bot Loading

Build Passed

artem1205 commented Nov 15, 2022 • edited by github-actions bot Loading

Build Failed

artem1205 commented Nov 15, 2022 • edited by github-actions bot Loading

Build Failed

Build Passed

girarda Nov 18, 2022

Choose a reason for hiding this comment

artem1205 Nov 18, 2022

Choose a reason for hiding this comment

girarda Nov 21, 2022

Choose a reason for hiding this comment

artem1205 Nov 22, 2022

Choose a reason for hiding this comment

sherifnada Dec 9, 2022

Choose a reason for hiding this comment

artem1205 Dec 9, 2022

Choose a reason for hiding this comment

girarda left a comment

Choose a reason for hiding this comment

brianjlai left a comment

Choose a reason for hiding this comment

artem1205 commented Dec 13, 2022 • edited by github-actions bot Loading

Build Failed

artem1205 commented Dec 14, 2022 • edited by github-actions bot Loading

Build Passed

artem1205 commented Dec 14, 2022 • edited by github-actions bot Loading

artem1205 commented Dec 16, 2022 • edited by github-actions bot Loading

artem1205 commented Nov 13, 2022 •

edited

Loading

artem1205 commented Nov 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Nov 15, 2022 •

edited by github-actions bot

Loading

artem1205 commented Nov 15, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 13, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 14, 2022 •

edited by github-actions bot

Loading

artem1205 commented Dec 16, 2022 •

edited by github-actions bot

Loading