Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source freshdesk, hubspot, mixpanel, paypal-transaction, salesforce, zendesk-support: adding fixtures to mock time.sleep for connectors that explicitly sleep #12335

Merged
merged 12 commits into from
Apr 28, 2022

Conversation

brianjlai
Copy link
Contributor

@brianjlai brianjlai commented Apr 26, 2022

What

Addresses #11784

We have plenty of unit tests that spend a bulk of their time sleeping because of a backoff for retryable errors or because of custom connector logic. Since unit tests are run in isolation, we don't need to waste our time waiting.

Connectors changed

  • freshdesk
  • hubspot
  • mixpanel
  • paypal-transaction
  • salesforce
  • zendesk-support

How

As a first pass I specifically addressed connectors that had custom time.sleep() calls within their code. The concept is pretty straightforward that if pytest-mock isn't imported then I added it for a given connector. And then I introduced a new test fixture that mocks time.sleep() and that fixture is set to be auto used for each test in that file.

Note: time.sleep() is used in the default behavior for the user_defined_backoff_handler. I'm certain that other connectors are also unnecessarily sleeping in unit tests, but I haven't figured out a great way to fix them all at once. My proposition is in the immediate term as part of the maintenance process for connectors, if long running connector unit tests are identified, then we emulate the behavior here to mock time.sleep() on a case by case basis. But potentially we might want to explore the concept of test configs that will instrument this and other test primitives by default, but still offering a way to enable/disable.

Recommended reading order

This change affects a couple of connectors, order of reading doesn't matter really.

🚨 User Impact 🚨

Only changes to tests, no breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit
         ======================== 5 passed, 3 warnings in 4.00s =========================
> Task :airbyte-integrations:connectors:source-freshdesk:unitTest

         ======================= 91 passed, 208 warnings in 3.06s =======================
> Task :airbyte-integrations:connectors:source-hubspot:unitTest

         ======================= 17 passed, 102 warnings in 0.41s =======================
> Task :airbyte-integrations:connectors:source-mixpanel:unitTest

         ======================== 4 passed, 79 warnings in 0.36s ========================
> Task :airbyte-integrations:connectors:source-paypal-transaction:unitTest

         ================= 51 passed, 153 warnings in 70.34s (0:01:10) ==================
> Task :airbyte-integrations:connectors:source-salesforce:unitTest

         ======================= 22 passed, 33 warnings in 2.17s ========================
> Task :airbyte-integrations:connectors:source-zendesk-support:unitTest
Integration

Integration test run results can be seen in the comments. freshdesk has existing failures

Acceptance

Put your acceptance tests output here.

@CLAassistant
Copy link

CLAassistant commented Apr 26, 2022

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the area/connectors Connector related issues label Apr 26, 2022
@brianjlai brianjlai changed the title adding fixtures to mock time.sleep for connectors that explicitly sleep source freshdesk, hubspot, mixpanel, paypal-transaction, salesforce, zendesk-support: adding fixtures to mock time.sleep for connectors that explicitly sleep Apr 26, 2022
@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/2229593934
❌ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/2229593934
🐛 https://gradle.com/s/i6h2q3o6ymxw4
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
=================== 4 failed, 18 passed in 388.89s (0:06:28) ===================

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2229827485
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2229827485
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               150     19    87%
source_salesforce/streams.py           294     62    79%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             77     33    57%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  561    128    77%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             77      6    92%
source_salesforce/api.py               150     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           294     43    85%
--------------------------------------------------------
TOTAL                                  561     67    88%

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2229841187
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2229841187
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                         Stmts   Miss  Cover
------------------------------------------------
source_hubspot/errors.py         6      0   100%
source_hubspot/__init__.py       2      0   100%
source_hubspot/streams.py      707     56    92%
source_hubspot/source.py        69     12    83%
------------------------------------------------
TOTAL                          784     68    91%

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2229869022
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2229869022
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                          Stmts   Miss  Cover
-------------------------------------------------
source_mixpanel/__init__.py       2      0   100%
source_mixpanel/source.py       304     76    75%
-------------------------------------------------
TOTAL                           306     76    75%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:21: `future_state_path` not specified, skipping
================== 21 passed, 1 skipped in 2900.79s (0:48:20) ==================

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-paypal-transaction

🕑 connectors/source-paypal-transaction https://github.com/airbytehq/airbyte/actions/runs/2229931447
✅ connectors/source-paypal-transaction https://github.com/airbytehq/airbyte/actions/runs/2229931447
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                    Stmts   Miss  Cover
-----------------------------------------------------------
source_paypal_transaction/__init__.py       2      0   100%
source_paypal_transaction/source.py       193     94    51%
-----------------------------------------------------------
TOTAL                                     195     94    52%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 19 passed, 2 skipped in 58.40s ========================

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 26, 2022

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2229942385
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2229942385
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_zendesk_support/__init__.py       2      0   100%
source_zendesk_support/streams.py      293     56    81%
source_zendesk_support/source.py        45     25    44%
--------------------------------------------------------
TOTAL                                  340     81    76%

@brianjlai brianjlai marked this pull request as ready for review April 26, 2022 23:38
@brianjlai brianjlai requested a review from girarda April 26, 2022 23:38
Copy link
Contributor

@girarda girarda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codecov
Copy link

codecov bot commented Apr 27, 2022

Codecov Report

❗ No coverage uploaded for pull request base (master@e0902e6). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #12335   +/-   ##
=========================================
  Coverage          ?   80.52%           
=========================================
  Files             ?       24           
  Lines             ?     2470           
  Branches          ?        0           
=========================================
  Hits              ?     1989           
  Misses            ?      481           
  Partials          ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0902e6...4168e8f. Read the comment docs.

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Apr 27, 2022
@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/freshdesk

🕑 connectors/freshdesk https://github.com/airbytehq/airbyte/actions/runs/2234445411
❌ Failed to publish connectors/freshdesk
❌ Couldn't auto-bump version for connectors/freshdesk

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/2234482505
❌ Failed to publish connectors/source-freshdesk
❌ Couldn't auto-bump version for connectors/source-freshdesk

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2234651986
🚀 Successfully published connectors/source-hubspot
🚀 Auto-bumped version for connectors/source-hubspot
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2234651986

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2234652916
❌ Failed to publish connectors/source-mixpanel
❌ Couldn't auto-bump version for connectors/source-mixpanel

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2234653407
🚀 Successfully published connectors/source-salesforce
🚀 Auto-bumped version for connectors/source-salesforce
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2234653407

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2234718372
🚀 Successfully published connectors/source-zendesk-support
🚀 Auto-bumped version for connectors/source-zendesk-support
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/2234718372

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-paypal-transaction

🕑 connectors/source-paypal-transaction https://github.com/airbytehq/airbyte/actions/runs/2234719328
🚀 Successfully published connectors/source-paypal-transaction
🚀 Auto-bumped version for connectors/source-paypal-transaction
✅ connectors/source-paypal-transaction https://github.com/airbytehq/airbyte/actions/runs/2234719328

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:36 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:36 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:40 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:40 Inactive
@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2234783075
❌ Failed to publish connectors/source-mixpanel
❌ Couldn't auto-bump version for connectors/source-mixpanel

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:49 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:49 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:52 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 17:52 Inactive
@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2235024415
❌ Failed to publish connectors/source-mixpanel
❌ Couldn't auto-bump version for connectors/source-mixpanel

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2235392241
❌ Failed to publish connectors/source-mixpanel
❌ Couldn't auto-bump version for connectors/source-mixpanel

@brianjlai
Copy link
Contributor Author

brianjlai commented Apr 27, 2022

/publish connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2236141112
🚀 Successfully published connectors/source-mixpanel
🚀 Auto-bumped version for connectors/source-mixpanel
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2236141112

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 23:54 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets April 27, 2022 23:54 Inactive
@brianjlai brianjlai temporarily deployed to more-secrets April 28, 2022 16:55 Inactive
@brianjlai brianjlai temporarily deployed to more-secrets April 28, 2022 16:55 Inactive
@brianjlai brianjlai temporarily deployed to more-secrets April 28, 2022 18:21 Inactive
@brianjlai brianjlai temporarily deployed to more-secrets April 28, 2022 18:22 Inactive
@brianjlai brianjlai merged commit bb2da42 into master Apr 28, 2022
@brianjlai brianjlai deleted the brian/mock_sleep_to_speedup_tests branch April 28, 2022 19:21
suhomud pushed a commit that referenced this pull request May 23, 2022
…zendesk-support: adding fixtures to mock time.sleep for connectors that explicitly sleep (#12335)

* adding fixtures to mock time.sleep for connectors that explicitly sleep

* bump connector versions

* update changelog doc for each connector

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* auto-bump connector version

* remove version bump for freshdesk because connector tests are in a bad state

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants