Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Source Asana: migrate to new SAT, added base HTTP errors handling #19561

Merged
merged 18 commits into from
Nov 22, 2022

Conversation

bazarnov
Copy link
Collaborator

What

Resolving: #19413

How

  • migrated to new SAT with high level of strictness
  • added base HTTP errors handling for: 402,403,404,451
  • covered with unit test
  • little code refactoring

🚨 User Impact 🚨

No impact is expected

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

@octavia-squidington-iv octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Nov 17, 2022
@bazarnov bazarnov linked an issue Nov 17, 2022 that may be closed by this pull request
@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 18, 2022

/test connector=connectors/source-asana

🕑 connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3494891321
❌ connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3494891321
🐛 https://gradle.com/s/zihr3nnxsfak4

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream proje...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
============= 2 failed, 26 passed, 1 skipped in 1297.82s (0:21:37) =============

@bazarnov bazarnov requested a review from a team November 18, 2022 07:09
@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 20, 2022

/test connector=connectors/source-asana

🕑 connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3508669443
❌ connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3508669443
🐛 https://gradle.com/s/se5wo6gxqt4aa

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream proje...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
============= 2 failed, 26 passed, 1 skipped in 2540.45s (0:42:20) =============

@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 21, 2022

/test connector=connectors/source-asana

🕑 connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3516560919
❌ connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3516560919
🐛 https://gradle.com/s/5o6ddwt5cyv6u

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream proje...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
============= 2 failed, 26 passed, 1 skipped in 1297.44s (0:21:37) =============

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for migrating the config to high test strictness level.
I'm concerned about the use of an internal asana account for testing. We should create a new one for testing (ideally a paid one to benefit from all the exposed streams).
I created an issue here for this: https://github.com/airbytehq/airbyte/issues/19662

@@ -0,0 +1,84 @@
{"stream": "projects", "data": {"gid": "1201856001387054", "archived": false, "color": "dark-green", "created_at": "2022-02-22T23:20:52.238Z", "current_status": null, "custom_fields": [], "custom_field_settings": [{"gid": "1201856001387132"}, {"gid": "1201856001387112"}, {"gid": "1201856001387116"}, {"gid": "1201856001387139"}, {"gid": "1201856001387164"}], "default_view": "list", "due_on": null, "due_date": null, "followers": [{"gid": "1201617089911900"}, {"gid": "1200344886012993"}, {"gid": "1201607360212792"}], "html_notes": "<body>Standardize your RFP and evaluation process with our template to ensure you\u2019re fully prepared to receive and evaluate proposals\u2014all in the same place\u2014so you can pick the best organization for the job.</body>", "icon": "page_layout", "members": [{"gid": "1201617089911900"}, {"gid": "1200344886012993"}, {"gid": "1201607360212792"}], "modified_at": "2022-05-01T07:33:20.428Z", "name": "Speaking Engagements ", "notes": "Standardize your RFP and evaluation process with our template to ensure you\u2019re fully prepared to receive and evaluate proposals\u2014all in the same place\u2014so you can pick the best organization for the job.", "owner": {"gid": "1201617089911900"}, "permalink_url": "https://app.asana.com/0/1201856001387054/1201856001387054", "public": true, "resource_type": "project", "start_on": null, "team": {"gid": "1200344940838474"}, "workspace": {"gid": "1199152354181012"}}, "emitted_at": 1668517065006}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid this file potentially exposes sensitive internal data as the account used for testing is not a sandbox one. Please delete it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere Should we move this certification to on hold until the dedicated sandbox account is up and running?
or should we skip checking expected records with bypass reason? there is also a problem with two sequential reads since we're testing the active valid account and data is not frozen.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to keep this certification on hold, as a dedicated test account will provide better testing guarantees. But I will let @YowanR decide, I'm not sure of the efforts required for properly seeding a sandbox account.

Comment on lines 25 to 26
- name: custom_fields
bypass_reason: "This stream in not available for our Test Account. 402 - Custom Fields are not available for free users or guests."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create a dedicated paid sandbox account to cover this stream?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @YowanR to make decisions here)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the main issue that we are running into is that we are using an internal account instead of a sandbox account. For most connectors, we should have sandbox accounts instead of prod account. Now, a big issue here is that we actually took a very long time to get this account unblocked for OAuth. If we were to make another account, I would expect that we would block for a long time again which is detrimental to our customers. Instead, I suggest that we do two things:

  1. Bypass the tests that are failing due to having a prod account and not a sandbox acc (plus hide sensitive info!)
  2. Start setting up a sandbox account to revisit. Once we have the sandbox account, we can solve the issues that we've seen. Thoughts on this @airbytehq/connector-operations and @bazarnov ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YowanR it's unclear to me if you want this PR to certify Asana to GA.
If it's not, I'm fine with logging future work for creating a sandbox account and adding the bypass reasons. I don't want to block any feature improvement on the connector. But I'd love new GA connectors to have the testing guarantees we outlined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere I'd like us to use strict testing to get Asana to GA but, because of how hard it was to get OAuth for this account, we need to work with the restrictions of using a prod account instead of a sandbox account. Based on the comments you've left in this PR, it seems that we will need to bypass a few tests, but overall, we'll still covering most tests correct? I think that the main question I have is: do you think the strict SAT framework can work with a prod account or is that a no-no?

Copy link
Contributor

@evantahler evantahler Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding my 2 cents - I think that GA means that we can reliably test the connector and guarantee that everything works, and we regularly to test all the relevant parts of the source's API. To me, GA means that we can both test oAuth and have a sandbox/test account that provides data for all the streams we provide. So, I would suggest leaving the connector in beta until we get a test account with oAuth. If I had to choose, I think ensuring correct data (e.g. expected_records) is more important than ensuring oAuth works (we value correctness more than anything else).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can split data tests and Oauth tests and set them using:

  • OAuth connectivity checks and discover - prod account
  • other SAT-related tests - sandbox

WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea to split the difference @bazarnov !

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's fair! Let's do it

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YowanR
Sure, do we already have the sandbox so I could do it)? Or are we moving this to "on hold" until the sandbox exists?

def should_retry(self, response: requests.Response) -> bool:
if response.status_code in ASANA_ERRORS_MAPPING.keys():
self.logger.error(
f"Skipping stream {self.name}. {ASANA_ERRORS_MAPPING.get(response.status_code)}. Full error message: {response.text}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @erohmensing this connector is implementing a custom approach at skipping unavailable stream.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add it to the list 👀

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YowanR:

because of how hard it was to get OAuth for this account...

Just to clarify, is enabling oAuth per-account in Asana? Do you pay for it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is per account. I don't believe we had to pay extra for it, but the process itself was incredibly slow (This is main reason we had this connector move out from Q3). @igrankova and @ycherniaiev can give more context here.

@alafanechere
Copy link
Contributor

@bazarnov , in the future feel free to ask for a review from all the @airbytehq/connector-operations team for the PR related to high test strictness level.

@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 21, 2022

/test connector=connectors/source-asana

🕑 connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3516867823
❌ connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3516867823
🐛 https://gradle.com/s/2npllp6msxxsm

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
============= 1 failed, 27 passed, 1 skipped in 1220.24s (0:20:20) =============

@bazarnov
Copy link
Collaborator Author

@bazarnov , in the future feel free to ask for a review from all the @airbytehq/connector-operations team for the PR related to high test strictness level.

Can we just use the @airbytehq/connector-operations when asking for a review? Or should we mention each member independently?

@alafanechere
Copy link
Contributor

@bazarnov , in the future feel free to ask for a review from all the @airbytehq/connector-operations team for the PR related to high test strictness level.

Can we just use the @airbytehq/connector-operations when asking for a review? Or should we mention each member independently?

Just using @airbytehq/connector-operations is perfect!

@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 22, 2022

/test connector=connectors/source-asana

🕑 connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3523087982
✅ connectors/source-asana https://github.com/airbytehq/airbyte/actions/runs/3523087982
Python tests coverage:

Name                       Stmts   Miss  Cover
----------------------------------------------
source_asana/oauth.py         10      0   100%
source_asana/__init__.py       2      0   100%
source_asana/streams.py      130      7    95%
source_asana/source.py        27      2    93%
----------------------------------------------
TOTAL                        169      9    95%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: Bypassed until dedicated sandbox account is up and running, because `two-sequential-reads` is not happy.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
================== 27 passed, 2 skipped in 464.31s (0:07:44) ===================

@bazarnov
Copy link
Collaborator Author

@alafanechere Please review again, as you've requested changes.

@bazarnov
Copy link
Collaborator Author

@YowanR @alafanechere @evantahler
For the Beta release, we will skip testing full-refresh + expected records, but will keep strictness on a high level.
Once the dedicated sandbox is created and filled with data, we can modify SAT config to adapt it to use the sandbox for data tests while testing OAuth with prod.

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on the SAT side. I made some wording suggestion in the bypass reasons.

@bazarnov
Copy link
Collaborator Author

bazarnov commented Nov 22, 2022

/publish connector=connectors/source-asana

🕑 Publishing the following connectors:
connectors/source-asana
https://github.com/airbytehq/airbyte/actions/runs/3525253670


Connector Did it publish? Were definitions generated?
connectors/source-asana

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets November 22, 2022 17:02 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets November 22, 2022 17:02 Inactive
@bazarnov bazarnov temporarily deployed to more-secrets November 22, 2022 17:09 Inactive
@bazarnov bazarnov temporarily deployed to more-secrets November 22, 2022 17:09 Inactive
@bazarnov bazarnov merged commit 0dddc1e into master Nov 22, 2022
@bazarnov bazarnov deleted the baz/source-asana-new-sat-migration branch November 22, 2022 17:34
letiescanciano added a commit that referenced this pull request Nov 28, 2022
* master: (74 commits)
  Fix support icon in sidebar
  fix: add BuildPulse report for helm ac tests (#19785)
  fix: yaml syntax (#19775)
  🪟 🐛 Fix custom connection creation endpoint (#19702)
  Source facebook marketing: check "breakdowns" combinations (#19645)
  fix typo: notify instead of sync (#19737)
  find_non_rate_limited_PAT (#19736)
  Source Google Ads: fix schema for "campaigns" stream (#19700)
  🎉 Source Asana: migrate to new SAT, added base HTTP errors handling (#19561)
  fix order not to randomly fail backward compatibility check (#19377)
  Bump helm chart version reference to 0.42.0 (#19706)
  fix: add extraEnv block (#19703)
  Bump Airbyte version from 0.40.21 to 0.40.22 (#19687)
  Bump helm chart version reference to 0.41.3 (#19685)
  Add connector builder server to airbyte proxy, kube overlays, and helm charts (#19554)
  dbt Cloud integration doc (#19619)
  🪟 🎉 Display service token validation errors in the UI (#19578)
  17644 Update Destination data type test to use the new data types (#19281)
  Docs: fix broken connector builder UI docs links (#19631)
  Bump Airbyte version from 0.40.20 to 0.40.21 (#19634)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/asana
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🎉 Source Asana: migrate to high strictness SAT
9 participants