Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Oracle & MongoDb destinations : Enable DAT tests #11965

Merged
merged 24 commits into from
Apr 19, 2022

Conversation

DoNotPanicUA
Copy link
Contributor

@DoNotPanicUA DoNotPanicUA commented Apr 13, 2022

What

Enable DAT tests #9443

  • Destination Oracle
  • Destination MongoDB

Recommended reading order

  1. MongodbDestinationAcceptanceTest.java
  2. OracleTestDataComparator.java
  3. SshOracleDestinationAcceptanceTest.java
  4. UnencryptedOracleDestinationAcceptanceTest.java

@github-actions github-actions bot added the area/connectors Connector related issues label Apr 13, 2022
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2161062710
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2161062710
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 13, 2022

/test connector=connectors/destination-mongodb

🕑 connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/2161063764
✅ connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/2161063764
No Python unittests run

…are it with actualObject elements. Expected might have empty elements and they are equal missing element in the actual Object. Also, actualObject might contain additional elements which is not an exception.
@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2189031819
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2189031819
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       168     31    82%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1403    520    63%

@DoNotPanicUA
Copy link
Contributor Author

DoNotPanicUA commented Apr 19, 2022

/test connector=connectors/destination-mongodb

🕑 connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/2189032403
✅ connectors/destination-mongodb https://github.com/airbytehq/airbyte/actions/runs/2189032403
No Python unittests run

@DoNotPanicUA DoNotPanicUA self-assigned this Apr 19, 2022
@DoNotPanicUA DoNotPanicUA marked this pull request as ready for review April 19, 2022 10:27
@DoNotPanicUA DoNotPanicUA merged commit 0004cfa into master Apr 19, 2022
@DoNotPanicUA DoNotPanicUA deleted the aleonets/dat-oracle-mongo branch April 19, 2022 10:56
suhomud pushed a commit that referenced this pull request May 23, 2022
* add Boolean, Number, DateTimeWithTZ compare methods

* Improve value comparison

* Compare inherit objects element by element

* Move comparison methods to a new class not to overload the test class.
Basic - old implementation which provides backward compatibility of this PR.
Advanced - comparator required for destinations with enabled DAT tests.

* format

* Move common method to ComparatorUtils. + Review update

* format

* review

* review

* enable DAT tests for destination-oracle

* mark resolveIdentifier method as deprecated

* enable DAT tests for destination-mongodb

* remove size objects validation. We iterate expected elements and compare it with actualObject elements. Expected might have empty elements and they are equal missing element in the actual Object. Also, actualObject might contain additional elements which is not an exception.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Run the new DAT(s) against all destinations and find destinations that are not compliant
2 participants