Standard Tests: Implement v2 test suites #2181 #2792

keu · 2021-04-07T22:40:48Z

closes #2181

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

test.java
component.ts
the rest

keu · 2021-04-08T18:14:57Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+        records_3 = filter_output(output, type_=Type.RECORD)
+        states_3 = filter_output(output, type_=Type.STATE)
+
+        assert not records_3, "There should be no records after second incremental read"


well, here should be a better check

Ideally this verifies the incremental sync invariant described here

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

sherifnada · 2021-04-13T17:30:09Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+        records_3 = filter_output(output, type_=Type.RECORD)
+        states_3 = filter_output(output, type_=Type.STATE)
+
+        assert not records_3, "There should be no records after second incremental read"


Ideally this verifies the incremental sync invariant described here

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

sherifnada · 2021-04-13T17:48:33Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_core.py

@@ -51,6 +50,13 @@ def test_check(self, connector_config, docker_runner: ConnectorRunner):
        assert len(con_messages) == 1, "Connection status message should be emitted exactly once"
        assert con_messages[0].connectionStatus.status == Status.SUCCEEDED

+    def test_check_without_network(self, connector_config, docker_runner: ConnectorRunner):


my guess is that this check will work well for most sources but sources which don't always require internet like source-file might not need it. I see a couple of ways around this:

add a flag to the config requires_internet that can be used to disable this check

don't use this check and rely on the other ones which malform the configs

I vote for №2

vitaliizazmic

LGTM

sherifnada

There's a few comments but feel free to merge once they are addressed

sherifnada · 2021-04-15T01:14:51Z

airbyte-integrations/bases/standard-test/README.md

@@ -1,6 +1,6 @@
 # Running Standard Source Tests
 ```bash
-PYTHONPATH=. pytest standard_test.py --standard_test_config=/connectors/source-hubspot/ -vvv
+python -m pytest standard_test/tests --standard_test_config=../../connectors/source-hubspot/ -vvv


should the hubspot bit be removed?

sherifnada · 2021-04-18T03:40:42Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_core.py

@@ -51,6 +52,13 @@ def test_check(self, connector_config, docker_runner: ConnectorRunner):
        assert len(con_messages) == 1, "Connection status message should be emitted exactly once"
        assert con_messages[0].connectionStatus.status == Status.SUCCEEDED

+    # def test_check_without_network(self, connector_config, docker_runner: ConnectorRunner):


should we delete instead of comment out?

sherifnada · 2021-04-18T03:41:31Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_core.py



+@pytest.mark.timeout(10)


the only problem with this is if the docker image takes a long time to download. Maybe we should have a fixture that downloads the docker image first so downstream tests can set reasonable timeouts like this.

fixed with pull_docker_image

airbyte-integrations/bases/standard-test/standard_test/tests/test_core.py

sherifnada · 2021-04-18T03:43:03Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_core.py

@@ -69,9 +78,9 @@ def test_discover(self, connector_config, catalog, docker_runner: ConnectorRunne
            assert catalog_messages[0].catalog == catalog, "Catalog should match the one that was provided"


+@pytest.mark.timeout(300)


this is a very short timeout given that some sources might have lots of streams or be reading lots of data. WDYT? Maybe this method shouldn't have a timeout? or maybe the timeout should be configurable?

I think not forcing timeout defeats the purpose of it. Another aspect is that we want to keep tests fast and scalable. They shouldn't take more than 20 minutes. If a connector can't guarantee this time either it is defective or it's test account configuration.

I agree in principle, but pragmatically speaking I worry this may be difficult/full of friction to do in practice, so my suggestion is borne out of practicality rather than philosophy

can we make it configurable? 300 second timeout sounds fine in this case

as agreed with @sherifnada I will create a ticket for this, for now we will move on

sherifnada · 2021-04-18T03:45:26Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+
+
+@pytest.fixture(name="cursor_paths")
+def cursor_paths_fixture(inputs, configured_catalog_for_incremental):


can we put return type hints for all these fixtures?

sherifnada · 2021-04-18T03:45:42Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+    result = {}
+
+    for stream in configured_catalog_for_incremental.streams:
+        path = cursor_paths.get(stream.stream.name, [stream.cursor_field[-1]])


the cursor field could be nested so we should return the full cursor field instead of just the last element

well, this is intentional, the default behavior in states don't use nested cursors, wherein records it is often the case.
I think the chance of events is the following:
(flat cursor in records) > (nested cursor in records) => (flat cursor in state) > (nested cursor in states)
so default behaviour here should appear more frequently

Ok I see 👍🏼

sherifnada · 2021-04-18T04:01:52Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+        records = filter_output(output, type_=Type.RECORD)
+        states = filter_output(output, type_=Type.STATE)
+
+        assert states, "Should produce at least one state"


should this test be merged with the one below? I understand that ideally it's not, but we have to be pragmatic because these tests can take a long time.

sherifnada · 2021-04-18T04:02:28Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+    return catalog
+
+
+@pytest.mark.timeout(300)


same comment as basic read about timeout -- this will be too short for some sources

sherifnada · 2021-04-18T04:02:55Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+            helper = JsonSchemaHelper(schema=stream.stream.json_schema)
+            record_value = helper.get_cursor_value(record=record.record.data, cursor_path=stream.cursor_field)
+            state_value = helper.get_state_value(state=latest_state[stream_name], cursor_path=cursor_paths[stream_name])
+            assert record_value <= state_value, "First incremental sync should produce records younger or equal to cursor value from the state"


good idea with this check!

sherifnada · 2021-04-18T18:52:39Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+    result = {}
+
+    for stream in configured_catalog_for_incremental.streams:
+        path = cursor_paths.get(stream.stream.name, [stream.cursor_field[-1]])


Ok I see 👍🏼

sherifnada · 2021-04-18T18:53:02Z

airbyte-integrations/bases/standard-test/standard_test/tests/test_incremental.py

+@pytest.fixture(name="configured_catalog_for_incremental")
+def configured_catalog_for_incremental_fixture(configured_catalog) -> ConfiguredAirbyteCatalog:
+    catalog = incremental_only_catalog(configured_catalog)
+    for stream in catalog.streams:


oh, we already do this filtering elsewhere? I must have missed it

) * Standard Tests: Implement v2 test suites #2181 (#2792) * rename standard tests (v2) to source_acceptance_test

keu self-assigned this Apr 7, 2021

auto-assign bot requested review from jrhizor and sherifnada April 7, 2021 22:40

keu linked an issue Apr 8, 2021 that may be closed by this pull request

Implement v2 test suites #2181

Closed

keu changed the title ~~WIP: Implement v2 test suites #2181~~ Standard Tests: Implement v2 test suites #2181 Apr 8, 2021

keu commented Apr 8, 2021

View reviewed changes

keu requested review from vitaliizazmic and yevhenii-ldv April 8, 2021 18:16

sherifnada suggested changes Apr 13, 2021

View reviewed changes

keu requested a review from sherifnada April 15, 2021 00:47

sherifnada removed the request for review from jrhizor April 15, 2021 14:44

keu mentioned this pull request Apr 15, 2021

Source Hubspot: rename state fields and change format to match format in records #2907

Closed

vitaliizazmic reviewed Apr 16, 2021

View reviewed changes

keu mentioned this pull request Apr 17, 2021

Set a default timeout in standard tests #2517

Closed

sherifnada approved these changes Apr 18, 2021

View reviewed changes

fixes and address comments from @sherifnada

a825b25

keu force-pushed the keu/standard_python-tests/test-suites branch from dc82433 to a825b25 Compare April 18, 2021 17:56

sherifnada approved these changes Apr 18, 2021

View reviewed changes

eugene-kulak added 9 commits April 18, 2021 22:10

fix tests

a0a2c67

fix leaking of secrets in the output

4f048c0

fix package path

a3799bc

fix imports

f62a666

hide secrets

f524ef2

fix logs

885fd49

fix sequential read test

4f2787c

fix build

43f8080

format

95257a2

keu merged commit 0247670 into keu/standard-python-tests Apr 19, 2021

keu deleted the keu/standard_python-tests/test-suites branch April 19, 2021 14:56

keu added a commit that referenced this pull request Apr 20, 2021

Standard Tests: create modular standard test suite v2 scaffolding (#2702

1755847

) * Standard Tests: Implement v2 test suites #2181 (#2792) * rename standard tests (v2) to source_acceptance_test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standard Tests: Implement v2 test suites #2181 #2792

Standard Tests: Implement v2 test suites #2181 #2792

keu commented Apr 7, 2021

keu Apr 8, 2021 •

edited

Loading

sherifnada Apr 13, 2021

sherifnada Apr 13, 2021

sherifnada Apr 13, 2021

keu Apr 15, 2021

vitaliizazmic left a comment

sherifnada left a comment

sherifnada Apr 15, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

keu Apr 18, 2021

sherifnada Apr 18, 2021

keu Apr 18, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

keu Apr 19, 2021

sherifnada Apr 18, 2021

keu Apr 18, 2021

sherifnada Apr 18, 2021

keu Apr 18, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

keu Apr 18, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

sherifnada Apr 18, 2021

		@@ -69,9 +78,9 @@ def test_discover(self, connector_config, catalog, docker_runner: ConnectorRunne
		assert catalog_messages[0].catalog == catalog, "Catalog should match the one that was provided"


		@pytest.mark.timeout(300)



		@pytest.fixture(name="cursor_paths")
		def cursor_paths_fixture(inputs, configured_catalog_for_incremental):



		@pytest.mark.timeout(10)

Standard Tests: Implement v2 test suites #2181 #2792

Standard Tests: Implement v2 test suites #2181 #2792

Conversation

keu commented Apr 7, 2021

What

How

Pre-merge Checklist

Recommended reading order

keu Apr 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliizazmic left a comment

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keu Apr 8, 2021 •

edited

Loading