Parse list of dicts in json_schema_helper.find_nodes() #19386
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-airtable |
0.1.3 |
✅ | ✅ |
source-amazon-ads |
0.1.25 |
✅ | ✅ |
source-amazon-seller-partner |
0.2.28 |
✅ | ✅ |
source-amazon-sqs |
0.1.0 |
✅ | ✅ |
source-amplitude |
0.1.17 |
✅ | ✅ |
source-appsflyer |
0.1.0 |
⚠ (doc not found) |
⚠ (not in seed) |
source-asana |
0.1.4 |
✅ | ✅ |
source-azure-table |
0.1.3 |
✅ | ✅ |
source-braintree |
0.1.3 |
✅ | ✅ |
source-cart |
0.2.0 |
❌ (changelog missing) |
✅ |
source-chargebee |
0.1.16 |
✅ | ❌ (diff seed version) |
source-commercetools |
0.1.0 |
✅ | ✅ |
source-confluence |
0.1.1 |
✅ | ✅ |
source-datadog |
0.1.0 |
✅ | ✅ |
source-delighted |
0.1.4 |
✅ | ✅ |
source-drift |
0.2.5 |
✅ | ✅ |
source-facebook-marketing |
0.2.72 |
✅ | ✅ |
source-facebook-pages |
0.1.6 |
✅ | ✅ |
source-freshcaller |
0.1.0 |
✅ | ✅ |
source-freshsales |
0.1.2 |
✅ | ✅ |
source-freshservice |
0.1.1 |
✅ | ✅ |
source-github |
0.3.8 |
✅ | ✅ |
source-gitlab |
0.1.6 |
✅ | ✅ |
source-google-ads |
0.2.4 |
✅ | ✅ |
source-google-search-console |
0.1.18 |
✅ | ✅ |
source-greenhouse |
0.3.0 |
✅ | ✅ |
source-harvest |
0.1.11 |
✅ | ✅ |
source-instagram |
1.0.0 |
✅ | ✅ |
source-iterable |
0.1.21 |
✅ | ✅ |
source-klaviyo |
0.1.10 |
✅ | ✅ |
source-lemlist |
0.1.1 |
❌ (changelog missing) |
❌ (diff seed version) |
source-lever-hiring |
0.1.3 |
✅ | ✅ |
source-linnworks |
0.1.5 |
❌ (changelog missing) |
✅ |
source-mailchimp |
0.2.15 |
✅ | ✅ |
source-mailgun |
0.1.0 |
⚠ (doc not found) |
✅ |
source-monday |
0.1.4 |
✅ | ✅ |
source-notion |
0.1.10 |
✅ | ✅ |
source-okta |
0.1.13 |
✅ | ✅ |
source-onesignal |
0.1.2 |
✅ | ✅ |
source-openweather |
0.1.6 |
✅ | ✅ |
source-outreach |
0.1.2 |
✅ | ✅ |
source-pardot |
0.1.0 |
⚠ (doc not found) |
⚠ (not in seed) |
source-paystack |
0.1.1 |
✅ | ✅ |
source-pinterest |
0.1.9 |
✅ | ✅ |
source-pipedrive |
0.1.13 |
✅ | ✅ |
source-plaid |
0.3.2 |
❌ (changelog missing) |
✅ |
source-posthog |
0.1.7 |
✅ | ✅ |
source-prestashop |
0.3.0 |
⚠ (doc not found) |
✅ |
source-quickbooks-singer |
0.1.5 |
⚠ (doc not found) |
✅ |
source-recharge |
0.2.4 |
❌ (changelog missing) |
✅ |
source-retently |
0.1.2 |
✅ | ✅ |
source-salesforce |
1.0.26 |
✅ | ✅ |
source-salesloft |
0.1.3 |
✅ | ✅ |
source-sendgrid |
0.2.16 |
✅ | ✅ |
source-sentry |
0.1.7 |
✅ | ✅ |
source-strava |
0.1.2 |
✅ | ✅ |
source-surveymonkey |
0.1.11 |
✅ | ✅ |
source-tplcentral |
0.1.1 |
✅ | ⚠ (not in seed) |
source-twilio |
0.1.13 |
✅ | ✅ |
source-weatherstack |
0.1.0 |
✅ | ✅ |
source-youtube-analytics |
0.1.3 |
✅ | ✅ |
source-zendesk-sunshine |
0.1.1 |
✅ | ✅ |
source-zendesk-talk |
0.1.5 |
✅ | ✅ |
source-zenloop |
0.1.3 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
|
/test connector=connectors/source-bigquery
Build PassedTest summary info: |
|
/test connector=connectors/source-clickhouse
Build PassedTest summary info: |
|
/test connector=connectors/source-cockroachdb
Build PassedTest summary info: |
|
/test connector=connectors/source-db2
Build PassedTest summary info: |
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
|
/test connector=connectors/source-oracle
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
|
/test connector=connectors/source-redshift
Build PassedTest summary info: |
|
/test connector=connectors/source-snowflake
Build PassedTest summary info: |
|
/test connector=connectors/source-tidb
Build PassedTest summary info: |
|
/test connector=connectors/source-elasticsearch
Build PassedTest summary info: |
|
/test connector=connectors/source-kafka
Build PassedTest summary info: |
|
/test connector=connectors/source-mongodb-v2
Build PassedTest summary info: |
|
/test connector=connectors/source-sftp
Build PassedTest summary info: |
erohmensing
changed the title
erohmensing
commented
Nov 15, 2022
airbyte-integrations/bases/source-acceptance-test/unit_tests/test_json_schema_helper.py
Show resolved
Hide resolved
erohmensing
changed the title
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
|
Ran tests for all the connectors after implementing the fix that got the test that reproduced the error to pass. Changed some things afterward but the functionality should be the same (the test still passed and then some), but re-running postgres, where we definitely ran into the error before, as a sanity check. |
evantahler
approved these changes
Nov 15, 2022
airbyte-integrations/bases/source-acceptance-test/unit_tests/test_json_schema_helper.py
Show resolved
Hide resolved
airbyte-integrations/bases/source-acceptance-test/unit_tests/test_json_schema_helper.py
Show resolved
Hide resolved
erohmensing
commented
Nov 15, 2022
colesnodgrass
pushed a commit
that referenced
this pull request
Nov 16, 2022
* Get test on nested list/dict passing - use index to query next object for list * Fix flakecheck * Test that get_node provides correct value * Improve test and test cases * Rewrite method for better comprehension * Add test for base-level key. Rewrite method for comprehension and handling this case
colesnodgrass
added a commit
that referenced
this pull request
Nov 30, 2022
* wip; add micronaut * add additional json deserializer methods * wip; converting to micronaut * misc cleanup * wip; broken * wip; still broken * wip * formatting * minor code cleanup; no actual changes * wip; still broken * removed commented out code; no longer broken * wip; clean-up micronaut code * cleanup; format * fix pmd issues * remove unused file * init ApplicationTest * edited link (#19444) * move 'Example values' into intl (#19446) * Revert "Update action.yml (#19416)" (#19450) This reverts commit 78fb528. * Notifications Workflow (#18735) * notification workflow * Bmoric/remove unused code (#19188) * Tmp * Move when the deletion is performed * Re-enable disable test * PR comments * Use cancel * rename * Fix test and version check position * remove unused temporal deletion code * Remove false todo * Rm repeated test * Rm unused import * Make sure that long running activity are not retried (#19452) * Parse list of dicts in json_schema_helper.find_nodes() (#19386) * Get test on nested list/dict passing - use index to query next object for list * Fix flakecheck * Test that get_node provides correct value * Improve test and test cases * Rewrite method for better comprehension * Add test for base-level key. Rewrite method for comprehension and handling this case * adding tests * fix test * formatting * remove unused dependencies * add missing test resource * format * add missing test resource (real) * format * add back protocol-models dep * format * pr feedback; log stacktrace Co-authored-by: Sophia Wiley <106352739+sophia-wiley@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Topher Lubaway <asimplechris@gmail.com> Co-authored-by: Anne <102554163+alovew@users.noreply.github.com> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com>
akashkulk
pushed a commit
that referenced
this pull request
Dec 2, 2022
* Get test on nested list/dict passing - use index to query next object for list * Fix flakecheck * Test that get_node provides correct value * Improve test and test cases * Rewrite method for better comprehension * Add test for base-level key. Rewrite method for comprehension and handling this case
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
close 🐛 [SAT] json_schema_helper.find_nodes() doesn't correctly parse lists of dicts #19244
pass connectors failing SAT tests with the following error
find_nodesto find keys at base-level dictHow
When adding items to the path while traversing the schema, if the next object is a list, add the index instead of the item to the variant path. E.g. to parse the following dict:
Before:
After:
Previously, it was keeping track of which item in the
oneOflist it was on by putting in the entire element. However this wasn't indexible when trying to find what existed at that node via recursive indexing, hence the error inget_node().Recommended reading order
test_find_and_get_nodes()find_nodes()🚨 User Impact 🚨
None, hopefully! Tests which could have errored should now work, meaning they may now pass or fail based on the connector