Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SAT: backward compatiblity hypothesis testing: Disable filter too much health check #17871

Merged
merged 5 commits into from
Oct 12, 2022

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Oct 12, 2022

What

Connector builds transiently fails when running SAT unit tests while running check_fake_previous_config_against_actual_spec

> Task :airbyte-integrations:bases:source-acceptance-test:_unitTestCoverage
	     �[94mdef�[39;49;00m �[92mtest_validate_previous_configs�[39;49;00m(previous_connector_spec, actual_connector_spec, should_fail):
	         expectation = pytest.raises(NonBackwardCompatibleError) �[94mif�[39;49;00m should_fail �[94melse�[39;49;00m does_not_raise()
	         �[94mwith�[39;49;00m expectation:
	 >           validate_previous_configs(previous_connector_spec, actual_connector_spec, �[94m200�[39;49;00m)
	 
	 �[1m�[31munit_tests/test_backward_compatibility.py�[0m:990: 
	 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
	 �[1m�[31msource_acceptance_test/utils/backward_compatibility.py�[0m:201: in validate_previous_configs
	     check_fake_previous_config_against_actual_spec()
	 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
	 
	     �[37m@given�[39;49;00m(from_schema(previous_connector_spec.dict()[�[33m"�[39;49;00m�[33mconnectionSpecification�[39;49;00m�[33m"�[39;49;00m]))
	 >   �[37m@settings�[39;49;00m(max_examples=number_of_configs_to_generate, verbosity=Verbosity.quiet, suppress_health_check=(HealthCheck.too_slow,))
	 �[1m�[31mE   hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 7 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls�[0m
	 �[1m�[31mE   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.�[0m
	 
	 �[1m�[31msource_acceptance_test/utils/backward_compatibility.py�[0m:191: FailedHealthCheck

How

Disable health checks on filtering too much. We'll fine-tune this test later, but this should prevent transient connector build failure in the meantime.

To investigate this problem later, I opened a tech-debt issue.

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-zendesk-sunshine
  • source-chargebee
  • source-harvest
  • source-onesignal
  • source-delighted
  • source-commercetools
  • source-zenloop
  • source-posthog
  • source-lever-hiring
  • source-pardot
  • source-iterable
  • source-google-ads
  • source-salesloft
  • source-strava
  • source-mailchimp
  • source-surveymonkey
  • source-retently
  • source-braintree
  • source-notion
  • source-prestashop
  • source-facebook-pages
  • source-klaviyo
  • source-recharge
  • source-freshservice
  • source-okta
  • source-plaid
  • source-mailgun
  • source-confluence
  • source-google-search-console
  • source-github
  • source-amazon-ads
  • source-paystack
  • source-quickbooks-singer
  • source-tplcentral
  • source-monday
  • source-cart
  • source-greenhouse
  • source-lemlist
  • source-outreach
  • source-pipedrive
  • source-twilio
  • source-instagram
  • source-openweather
  • source-azure-table
  • source-asana
  • source-facebook-marketing
  • source-sentry
  • source-amazon-sqs
  • source-salesforce
  • source-zendesk-talk
  • source-drift
  • source-youtube-analytics
  • source-amazon-seller-partner
  • source-linnworks
  • source-sendgrid
  • source-appsflyer
  • source-amplitude
  • source-airtable
  • source-pinterest
  • source-freshcaller
  • source-freshsales
  • source-gitlab

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-azure-table
  • source-airtable
  • source-sentry
  • source-salesforce
  • source-zendesk-talk
  • source-tplcentral
  • source-sendgrid
  • source-gitlab
  • source-monday
  • source-mailchimp
  • source-instagram
  • source-lever-hiring
  • source-google-search-console
  • source-braintree
  • source-commercetools
  • source-openweather
  • source-pipedrive
  • source-pardot
  • source-freshsales
  • source-harvest
  • source-quickbooks-singer
  • source-asana
  • source-retently
  • source-onesignal
  • source-amplitude
  • source-appsflyer
  • source-freshservice
  • source-github
  • source-paystack
  • source-okta
  • source-twilio
  • source-lemlist
  • source-surveymonkey
  • source-linnworks
  • source-strava
  • source-outreach
  • source-klaviyo
  • source-facebook-pages
  • source-iterable
  • source-salesloft
  • source-confluence
  • source-freshcaller
  • source-delighted
  • source-pinterest
  • source-plaid
  • source-cart
  • source-chargebee
  • source-zenloop
  • source-prestashop
  • source-amazon-ads
  • source-youtube-analytics
  • source-recharge
  • source-google-ads
  • source-facebook-marketing
  • source-notion
  • source-greenhouse
  • source-posthog
  • source-zendesk-sunshine
  • source-amazon-sqs
  • source-drift
  • source-mailgun
  • source-amazon-seller-partner

Copy link
Contributor

@pedroslopez pedroslopez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linking the failed health check doc here for future reference: https://hypothesis.readthedocs.io/en/latest/healthchecks.html

@alafanechere
Copy link
Contributor Author

alafanechere commented Oct 12, 2022

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3236707928


Connector Did it publish? Were definitions generated?
bases/source-acceptance-test

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-greenhouse
  • source-amazon-ads
  • source-okta
  • source-freshsales
  • source-pardot
  • source-monday
  • source-outreach
  • source-youtube-analytics
  • source-facebook-pages
  • source-iterable
  • source-facebook-marketing
  • source-salesloft
  • source-google-ads
  • source-amazon-sqs
  • source-cart
  • source-amazon-seller-partner
  • source-tplcentral
  • source-twilio
  • source-linnworks
  • source-plaid
  • source-google-search-console
  • source-salesforce
  • source-lever-hiring
  • source-braintree
  • source-freshservice
  • source-appsflyer
  • source-prestashop
  • source-drift
  • source-chargebee
  • source-surveymonkey
  • source-github
  • source-delighted
  • source-airtable
  • source-confluence
  • source-instagram
  • source-azure-table
  • source-mailgun
  • source-klaviyo
  • source-openweather
  • source-pinterest
  • source-harvest
  • source-zendesk-talk
  • source-commercetools
  • source-zenloop
  • source-zendesk-sunshine
  • source-onesignal
  • source-paystack
  • source-gitlab
  • source-lemlist
  • source-mailchimp
  • source-amplitude
  • source-pipedrive
  • source-recharge
  • source-notion
  • source-sendgrid
  • source-strava
  • source-quickbooks-singer
  • source-asana
  • source-freshcaller
  • source-retently
  • source-sentry
  • source-posthog

@alafanechere alafanechere merged commit 0644a84 into master Oct 12, 2022
@alafanechere alafanechere deleted the augustin/sat/validate-previous-config branch October 12, 2022 18:21
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants