Source Close.com: migration to low-code #18968

roman-yermilov-gl · 2022-11-04T14:46:42Z

What

This is a migration to Low-code

How

Old code is still present but replaced with new Source Declarative class in __init__.py. Made custom Slicer and Pagination to support custom needs of the connector

davydov-d · 2022-11-07T09:20:28Z

airbyte-integrations/connectors/source-close-com/source_close_com/source_lc.py

+# Declarative Source
+class SourceCloseCom(YamlDeclarativeSource):
+    def __init__(self):
+        super().__init__(**{"path_to_yaml": "close_com.yaml"})


Suggested change

super().__init__(**{"path_to_yaml": "close_com.yaml"})

super().__init__(path_to_yaml="close_com.yaml")

This is an auto-generated code that no need to touch. This is how all the Source classes should look like

davydov-d · 2022-11-07T09:21:10Z

airbyte-integrations/connectors/source-close-com/Dockerfile

@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

-LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.version=0.1.1


I think a minor version should be bumped, not patch

Nothing changed at all from functional perspective. Moreover we are using the same code as we used before, but just run it in another way. Should we really bump it?

but it does not look like a bugfix as well :)

davydov-d · 2022-11-07T09:21:56Z

airbyte-integrations/connectors/source-close-com/acceptance-test-config.yml

-                      "incoming_sms_tasks", "zoom_connected_accounts", "custom_email_connected_accounts",
-                      "google_connected_accounts", "send_as", "email_sequences", "dialer", "email_bulk_actions",
-                      "sequence_subscription_bulk_actions", "delete_bulk_actions"]
+      empty_streams:


there should be no empty streams if we're going to certify this connector to beta

I was trying to avoid empty streams but didn't succeed. We cannot to populate some tasks from our side and send_as requires payment. It also makes impossible for now to migrate SAT config to new version

davydov-d · 2022-11-07T09:22:18Z

airbyte-integrations/connectors/source-close-com/acceptance-test-config.yml

@@ -14,10 +14,24 @@ tests:
  basic_read:


could you please migrate SAT config to the new version?

New version does not support empty_streams

it does support empty streams but you need to define the bypass reason

airbyte-integrations/connectors/source-close-com/source_close_com/single_stream_slicer.py

davydov-d · 2022-11-07T09:26:20Z

...yte-integrations/connectors/source-close-com/source_close_com/offset_increment_workaround.py

+
+
+@dataclass
+class OffsetIncrementWorkaround(PaginationStrategy, JsonSchemaMixin):


could you please add a docstring explaining the difference between this class and other paginating strategies?

davydov-d · 2022-11-07T09:27:15Z

airbyte-integrations/connectors/source-close-com/source_close_com/close_com.yaml

+    $ref: "*ref(definitions.incremental_stream)"
+    stream_cursor_field: "date_created"
+    $options:
+      primary_key: "id"


the primary key should be defined outside the options parameter

I can put it here as well, if I need to specify it by primary_key per stream. But you are right, this key isn't used and was finally defined globally in base stream. It should be removed from here

airbyte-integrations/connectors/source-close-com/source_close_com/close_com.yaml

davydov-d · 2022-11-07T09:30:35Z

what's the purpose of not removing the python code?

roman-yermilov-gl · 2022-11-10T16:06:58Z

what's the purpose of not removing the python code?

It is just an agreement

roman-yermilov-gl · 2022-11-11T14:33:53Z

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3445784181
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3445784181
Python tests coverage:

Name                                              Stmts   Miss  Cover
---------------------------------------------------------------------
source_close_com/offset_increment_workaround.py      24      0   100%
source_close_com/__init__.py                          4      0   100%
source_close_com/single_stream_slicer.py             64     13    80%
source_close_com/source_lc.py                         4      1    75%
---------------------------------------------------------------------
TOTAL                                                96     14    85%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              394    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 293, 331-348, 357-365, 369-374, 380, 413-418, 456-463, 506-508, 511, 576-584, 596-599, 604, 660-661, 667, 670, 706-716, 729-754
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1557    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:364: Backward compatibility tests are disabled for version 0.1.0.
================== 30 passed, 1 skipped in 642.55s (0:10:42) ===================

girarda · 2022-11-17T05:38:59Z

...yte-integrations/connectors/source-close-com/source_close_com/offset_increment_workaround.py

+
+    def __post_init__(self, options: Mapping[str, Any]):
+        self._offset = 0
+        self.page_size = InterpolatedString.create(self.page_size, options=options)


@roman-yermilov-gl how about updating the paginator's field to be an interpolated string instead of duplicating the class for a one-line change?

It may affect existing connectors that using page_size as integer?

page_size should always return an int, but the underlying field could be Union[InterpolatedString, str, int]

Removed custom component, but need the following PR to be accepted:
#19646

girarda · 2022-11-17T18:47:13Z

airbyte-integrations/connectors/source-close-com/source_close_com/single_stream_slicer.py

+
+
+@dataclass
+class SingleStreamSlicer(SingleSlice):


is there a benefit to using a single slice instead of chunking the time range like DatetimeStreamSlicer does? Using multiple slices is nice because it gives us checkpointing for free

There are streams that require a very small step, a millisecond or less. It looks like I cannot avoid customization

Fine with me if it has to be custom, but I'd like to better understand the limitation of the implementation in the CDK.

The DatetimeStreamSlicer's update_cursor method also updates the cursor based on the last record read, so I would expect it to fetch the same records even if they are in different slices

Figured out how to avoid custom Slicer, so custom component is removed

girarda

love it! Approved pending #19646

girarda · 2022-11-22T22:21:40Z

airbyte-integrations/connectors/source-close-com/source_close_com/close_com.yaml

+      field_name: "_skip"
+    pagination_strategy:
+      type: OffsetIncrement
+      page_size: "{{ options['items_per_page'] }}"


roman-yermilov-gl · 2022-12-06T16:53:01Z

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3631620509
❌ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3631620509
🐛 https://gradle.com/s/fnpiy7gj2ndr4

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream googl...
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:370: Backward compatibility tests are disabled for version 0.1.0.
============= 4 failed, 26 passed, 1 skipped in 214.79s (0:03:34) ==============

roman-yermilov-gl · 2022-12-07T12:25:52Z

/test connector=connectors/source-close-com

🕑 connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3638859589
✅ connectors/source-close-com https://github.com/airbytehq/airbyte/actions/runs/3638859589
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:370: Backward compatibility tests are disabled for version 0.1.0.
================== 30 passed, 1 skipped in 356.87s (0:05:56) ===================

roman-yermilov-gl · 2022-12-07T12:57:14Z

/publish connector=connectors/source-close-com

🕑 Publishing the following connectors:
connectors/source-close-com
https://github.com/airbytehq/airbyte/actions/runs/3639089400

Connector	Did it publish?	Were definitions generated?
connectors/source-close-com	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

roman-yermilov-gl requested review from grubberr, midavadim, davydov-d, arsenlosenko, bazarnov, brianjlai, lazebnyi and darynaishchenko November 4, 2022 14:46

roman-yermilov-gl self-assigned this Nov 4, 2022

github-actions bot added the area/connectors Connector related issues label Nov 4, 2022

octavia-squidington-iv added the connectors/source/close-com label Nov 4, 2022

github-actions bot added the area/documentation Improvements or additions to documentation label Nov 4, 2022

davydov-d requested changes Nov 7, 2022

View reviewed changes

roman-yermilov-gl temporarily deployed to more-secrets November 10, 2022 13:12 Inactive

roman-yermilov-gl temporarily deployed to more-secrets November 10, 2022 14:26 Inactive

roman-yermilov-gl temporarily deployed to more-secrets November 10, 2022 16:04 Inactive

davydov-d approved these changes Nov 10, 2022

View reviewed changes

roman-yermilov-gl force-pushed the ryermilov/source-close-com-migration branch from d6957c9 to 07f3b3f Compare November 11, 2022 14:32

roman-yermilov-gl temporarily deployed to more-secrets November 11, 2022 14:35 Inactive

bazarnov approved these changes Nov 11, 2022

View reviewed changes

grubberr temporarily deployed to more-secrets November 13, 2022 17:02 Inactive

grubberr temporarily deployed to more-secrets November 14, 2022 08:43 Inactive

roman-yermilov-gl requested a review from girarda November 15, 2022 13:31

lazebnyi approved these changes Nov 16, 2022

View reviewed changes

girarda reviewed Nov 17, 2022

View reviewed changes

roman-yermilov-gl force-pushed the ryermilov/source-close-com-migration branch from 94c3ad6 to e9e37a7 Compare November 21, 2022 10:29

roman-yermilov-gl temporarily deployed to more-secrets November 21, 2022 10:31 Inactive

roman-yermilov-gl temporarily deployed to more-secrets November 21, 2022 15:39 Inactive

roman-yermilov-gl requested a review from girarda November 22, 2022 13:37

girarda approved these changes Nov 22, 2022

View reviewed changes

roman-yermilov-gl temporarily deployed to more-secrets December 6, 2022 19:22 Inactive

roman-yermilov-gl temporarily deployed to more-secrets December 6, 2022 19:37 Inactive

roman-yermilov-gl added 11 commits December 7, 2022 15:36

Source Close.com: migration to low-code

5887c1d

Source Close.com: bump version; update changelog

be53d92

Source Close.com: review fixes

0950a49

Source Close.com: review fixes

4d019c1

Source Close.com: documentation structure standartization

9053ffe

Source Close.com: making connector to be prepared to beta

3a41de6

Source Close.com: migrate SAT config to new version

5a201ee

Source Close.com: unit tests for custom components

ddccf0a

Source Close.com: avoid custom pagination component

91d64cb

Source Close.com: avoid custom slicer component

368ef9d

Source Close.com: customize datetime parser function for sliser

d373d86

roman-yermilov-gl force-pushed the ryermilov/source-close-com-migration branch from 9d79545 to d373d86 Compare December 7, 2022 12:20

roman-yermilov-gl temporarily deployed to more-secrets December 7, 2022 12:22 — with GitHub Actions Inactive

auto-bump connector version

5168a32

octavia-squidington-iii temporarily deployed to more-secrets December 7, 2022 13:22 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets December 7, 2022 13:23 — with GitHub Actions Inactive

lazebnyi mentioned this pull request Dec 7, 2022

Source Close.com: no export of custom fields values #10339

Closed

roman-yermilov-gl merged commit ab44197 into master Dec 7, 2022

roman-yermilov-gl deleted the ryermilov/source-close-com-migration branch December 7, 2022 18:00

octavia-squidington-iii mentioned this pull request Dec 7, 2022

Bump Airbyte version from 0.40.23 to 0.40.24 #20206

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Close.com: migration to low-code #18968

Source Close.com: migration to low-code #18968

roman-yermilov-gl commented Nov 4, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 7, 2022

davydov-d Nov 7, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 7, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 7, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 10, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 7, 2022

davydov-d Nov 10, 2022

roman-yermilov-gl Nov 10, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 10, 2022

davydov-d Nov 7, 2022

roman-yermilov-gl Nov 7, 2022 •

edited

Loading

davydov-d commented Nov 7, 2022

roman-yermilov-gl commented Nov 10, 2022

roman-yermilov-gl commented Nov 11, 2022 •

edited by github-actions bot

Loading

girarda Nov 17, 2022

roman-yermilov-gl Nov 18, 2022

girarda Nov 19, 2022

roman-yermilov-gl Nov 22, 2022 •

edited

Loading

girarda Nov 17, 2022

roman-yermilov-gl Nov 18, 2022

girarda Nov 19, 2022

roman-yermilov-gl Nov 22, 2022

girarda left a comment

girarda Nov 22, 2022

roman-yermilov-gl commented Dec 6, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Dec 7, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Dec 7, 2022 •

edited by github-actions bot

Loading

	super().__init__(**{"path_to_yaml": "close_com.yaml"})
	super().__init__(path_to_yaml="close_com.yaml")



		@dataclass
		class OffsetIncrementWorkaround(PaginationStrategy, JsonSchemaMixin):

Source Close.com: migration to low-code #18968

Source Close.com: migration to low-code #18968

Conversation

roman-yermilov-gl commented Nov 4, 2022

What

How

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roman-yermilov-gl Nov 7, 2022 • edited Loading

Choose a reason for hiding this comment

davydov-d commented Nov 7, 2022

roman-yermilov-gl commented Nov 10, 2022

roman-yermilov-gl commented Nov 11, 2022 • edited by github-actions bot Loading

Build Passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roman-yermilov-gl Nov 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

girarda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roman-yermilov-gl commented Dec 6, 2022 • edited by github-actions bot Loading

Build Failed

roman-yermilov-gl commented Dec 7, 2022 • edited by github-actions bot Loading

Build Passed

roman-yermilov-gl commented Dec 7, 2022 • edited by github-actions bot Loading

roman-yermilov-gl Nov 7, 2022 •

edited

Loading

roman-yermilov-gl commented Nov 11, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl Nov 22, 2022 •

edited

Loading

roman-yermilov-gl commented Dec 6, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Dec 7, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Dec 7, 2022 •

edited by github-actions bot

Loading