Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Spark Tests Framework #18551

Merged
merged 7 commits into from
Jan 13, 2021
Merged

Improve Spark Tests Framework #18551

merged 7 commits into from
Jan 13, 2021

Conversation

moderakh
Copy link
Contributor

@moderakh moderakh commented Jan 12, 2021 •
edited
Loading

This PR does not make any changes to the spark source code logic. All changes are scoped to the tests of spark connector.

  1. Improves tests framework.
    -- Provides common test utils for cleanup resources (Cosmos DB database, spark session, etc) after tests
    -- provides framework for sharing CosmosClient, spark session, container in the scope of one test class
    -- improves test framework to rely on scalatest assertions instead of assertj as a pattern

  2. adds a Dev-Readme.md file documenting the test execution instructions.

@ghost ghost added the Cosmos label Jan 12, 2021
@moderakh moderakh changed the title Improve Tests Framework Improve Spark Tests Framework Jan 12, 2021
@moderakh moderakh added the cosmos:spark3 Cosmos DB Spark3 OLTP Connector label Jan 12, 2021
@moderakh moderakh linked an issue Jan 12, 2021 that may be closed by this pull request
test resource management: #18450
Closed
@moderakh moderakh mentioned this pull request Jan 12, 2021
Closed
FabianMeiswinkel
FabianMeiswinkel reviewed Jan 12, 2021
View reviewed changes
...os/azure-cosmos-spark_3-0_2-12/src/test/scala/com/azure/cosmos/spark/CosmosCatalogSpec.scala
spark.sql(s"CREATE DATABASE cosmoscatalog.${databaseName};")
spark.sql(s"CREATE TABLE cosmoscatalog.${databaseName}.${containerName} (word STRING, number INT) using cosmos.items " +
spark.sql(s"CREATE DATABASE testCatalog.${databaseName};")
spark.sql(s"CREATE TABLE testCatalog.${databaseName}.${containerName} (word STRING, number INT) using cosmos.items " +
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated - but I really love that you have wired up the metadata changes already - I am sure Spark developers will love this feature - don't know how many customers I have seen that had taken a dependency to some crappy very old Python SDK just to be able to create/modify containers etc.

FabianMeiswinkel
FabianMeiswinkel approved these changes Jan 12, 2021
View reviewed changes
Copy link
Member

@FabianMeiswinkel FabianMeiswinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! LGTM - except for minor comments. On .gitignore this is mostly a question - if there is a reason why you added .gitignore to the .gitgnore file I missed feel free to ignore

@moderakh
Copy link
Contributor Author

moderakh commented Jan 12, 2021 •
edited
Loading

Thanks! LGTM - except for minor comments. On .gitignore this is mostly a question - if there is a reason why you added .gitignore to the .gitgnore file I missed feel free to ignore

@FabianMeiswinkel
Thanks for catching the .gitignore files. I just removed the .gitignore entry in the .gitignore, that shouldn't be needed.

@moderakh moderakh merged commit 86ac9f5 into Azure:feature/cosmos/spark30 Jan 13, 2021
@moderakh moderakh deleted the users/moderakh/spark-test-improvement branch January 13, 2021 00:12
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-java that referenced this pull request Apr 27, 2022
@sunny27710 @akning-ms
@thisisdevanshu-ms @zhenglaizhang @MicrosoftJim @akankshanb @ArcturusZhang @msyyc
CodeGen from PR 18551 in Azure/azure-rest-api-specs
887d0d7 
[Hub Generated] Publish private branch 'dev-orbital-Microsoft.Orbital-2021-04-04-preview' (Azure#18551)

* New Swagger Spec File

* New Swagger Example Spec File

* New Readme Config File

* New Azure AZ Readme Config File

* New Azure CLI Readme Config File

* New Go Language Readme Config File

* New Python Language Readme Config File

* New Typescript Language Readme Config File

* New C# Language Readme Config File

* Orbital sdk 2021 04 04 preview (#5253)

* New Readme Config File

* New Go Language Readme Config File

* New Typescript Language Readme Config File

* New Python Language Readme Config File

* New C# Language Readme Config File

* New Ruby Language Readme Config File

* New Swagger Spec File

* New Swagger Example Spec File

* Added orbital.json file

* Passed linting using AutoRest for 2020-09-01 Azure Orbital Private Privew

* Updated Operation object under Azure Orbital 2020-09-01 preview.

* Updated Operation object under Azure Orbital 2020-09-01 preview.

* Updated Satellites object for Azure Orbital 2020-09-01 preview

* Updated examples for Azure Orbital Private Preview 2020-09-01

* Updated examples for Azure Orbital Private Preview 2020-09-01

* Updated examples for Azure Orbital Private Preview 2020-09-01

* Addressed feedback from ARMChangesRequested

* Updated the GetAvailableContacts api to add long running operation

* Updated the api version

* Updated the objects based on discussion with PM.

* list available ground station made a POST request (#2521)

* list available ground station made a POST request

* Swagger LintDiff fix

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* swagger specs for version-20210404-preview

* removed cross version references. Copied over unchanged examples from the previous API version 2021-09-01-preview

* removed availableGroundStations API

* New Readme Config File

New Go Language Readme Config File

New Typescript Language Readme Config File

New Python Language Readme Config File

New C# Language Readme Config File

New Ruby Language Readme Config File

New Swagger Spec File

New Swagger Example Spec File

Added orbital.json file

Passed linting using AutoRest for 2020-09-01 Azure Orbital Private Privew

Updated Operation object under Azure Orbital 2020-09-01 preview.

Updated Operation object under Azure Orbital 2020-09-01 preview.

Updated Satellites object for Azure Orbital 2020-09-01 preview

Updated examples for Azure Orbital Private Preview 2020-09-01

Updated examples for Azure Orbital Private Preview 2020-09-01

Updated examples for Azure Orbital Private Preview 2020-09-01

Addressed feedback from ARMChangesRequested

Updated the GetAvailableContacts api to add long running operation

Updated the api version

Updated the objects based on discussion with PM.

list available ground station made a POST request (#2521)

* list available ground station made a POST request

* Swagger LintDiff fix

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* validation check changes. Avocado, SpellCheck and prettier

* validation checks changes

* Modify readme.md for sdk generation

* added system data

* sdk pre namer validation changes

* lintdiff warning changes

* add comms scenario swagger (#3794)

* add comms scenario swagger

* remove unused example

* fix some validation errors

* fix style issues

* update spec based on feedback

* add updated example files

* fix CI errors

* change endpoints/connections list APIs to posts from gets

* connections lists from endpoints return resource IDs rather than full
objects.

* update examples files for endpoints connections list APIs

* fix operationIds and add descriptions

* fix more op ids and add descriptions for real

* fix typos, add constraints, and remove kbps/gbps from bandwidth (#4023)

* Updates to Resource Types (Modifications and Renames) (#4294)

* update orbital.json

* update examples

* prettier fix

* unfix old api version

* fix ci issues

* make more fields immutable

* update provisioning states

* swagger specs for Available groundstation APIs, version 2021-preview (#4920)

* swagger specs for Available groundstation APIs, version 2021-preview

* npm prettier check

* added type object

* removed capability and global comms info from responses

* prettier fix

* changed AP id response

* changes to schema to follow ARM format, resource id update

* Spelling check fix

* Ground Station spell check

* added groundstation to custom words

* Revert "added groundstation to custom words"

This reverts commit 0d7e18217de98836dcb8ba991685e15436004836.

* Revert "Ground Station spell check"

This reverts commit ef85b3b26f9ddf81dc1f50a9b677f9d8f115c4cf.

* Revert "Spelling check fix"

This reverts commit 6bd95c6e013889b6d488703846bbe87bbbb9c494.

* word addition

* type in response

* type definition in swagger

* readonly add

Co-authored-by: Akanksha Bhattacharyya <akankshab@microsoft.com>

* removed comms APIs

* readme updated

* fixed Swagger Avocado errors

* swagger lintDiff fix 1

* swagger lintDiff fix 2

* swagger lintDiff fix 3

Co-authored-by: Sunny Patel <sunny27710@gmail.com>
Co-authored-by: Sunny Patel <sunpatel@microsoft.com>
Co-authored-by: Arthur Ning <57385816+akning-ms@users.noreply.github.com>
Co-authored-by: Devanshu Singh <devasingh@microsoft.com>
Co-authored-by: Zhenglai Zhang <darinzh@microsoft.com>
Co-authored-by: Jim Harris <72156062+MicrosoftJim@users.noreply.github.com>
Co-authored-by: Akanksha Bhattacharyya <akanksha.nb@gmail.com>
Co-authored-by: Akanksha Bhattacharyya <akankshab@microsoft.com>

* operation IDs updated (#5461)

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* Dev orbital microsoft.orbital 2021 04 04 preview (#5491)

* operation IDs updated

* updated OperationListResult definition

* review points implemented

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* Azure Orbital - Review points implemented (#5507)

* operation IDs updated

* updated OperationListResult definition

* review points implemented

* referencing CloudError, ApiVersionParameter, ResourceGroupNameParameter, SubscriptionIdParameter from common-types

* reverting cloudError refs

* prettify json

* reorder some parameters

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>
Co-authored-by: ArcturusZhang <dapzhang@microsoft.com>

* Removed Empty allOf (#5510)

* operation IDs updated

* updated OperationListResult definition

* review points implemented

* referencing CloudError, ApiVersionParameter, ResourceGroupNameParameter, SubscriptionIdParameter from common-types

* reverting cloudError refs

* prettify json

* removed empty allOf

* validation checks

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* readme.go.md fixed (#5512)

* operation IDs updated

* updated OperationListResult definition

* review points implemented

* referencing CloudError, ApiVersionParameter, ResourceGroupNameParameter, SubscriptionIdParameter from common-types

* reverting cloudError refs

* prettify json

* removed empty allOf

* validation checks

* readme.go.md fixed

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* Remove last empty allOf (#5513)

* operation IDs updated

* updated OperationListResult definition

* review points implemented

* referencing CloudError, ApiVersionParameter, ResourceGroupNameParameter, SubscriptionIdParameter from common-types

* reverting cloudError refs

* prettify json

* removed empty allOf

* validation checks

* readme.go.md fixed

* remove one last empty allOf

Co-authored-by: Devanshu Singh <devasingh@microsoft.com>

* stable API version 2022-03-01

* valid json

* lint diff fix

* lint diff fix 2

* avocado fix

* lint diff warninga 1

* lint diff warnings 2

* prettier fix

* readme.me updated 2022-03-01

* Update readme.python.md

Co-authored-by: Sunny Patel <sunny27710@gmail.com>
Co-authored-by: Sunny Patel <sunpatel@microsoft.com>
Co-authored-by: Arthur Ning <57385816+akning-ms@users.noreply.github.com>
Co-authored-by: Devanshu Singh <devasingh@microsoft.com>
Co-authored-by: Zhenglai Zhang <darinzh@microsoft.com>
Co-authored-by: Jim Harris <72156062+MicrosoftJim@users.noreply.github.com>
Co-authored-by: Akanksha Bhattacharyya <akanksha.nb@gmail.com>
Co-authored-by: Akanksha Bhattacharyya <akankshab@microsoft.com>
Co-authored-by: ArcturusZhang <dapzhang@microsoft.com>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FabianMeiswinkel FabianMeiswinkel FabianMeiswinkel approved these changes

@kirankumarkolli kirankumarkolli Awaiting requested review from kirankumarkolli

@kushagraThapar kushagraThapar Awaiting requested review from kushagraThapar

@mbhaskar mbhaskar Awaiting requested review from mbhaskar

@milismsft milismsft Awaiting requested review from milismsft

@simplynaveen20 simplynaveen20 Awaiting requested review from simplynaveen20

@xinlian12 xinlian12 Awaiting requested review from xinlian12

Assignees
No one assigned
Labels
cosmos:spark3 Cosmos DB Spark3 OLTP Connector Cosmos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

test resource management:
2 participants
@moderakh @FabianMeiswinkel