Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools/importer-rest-api-specs: Resource ID parsing now returns map[OperationID]ParsedOperation rather than map[Uri]ParsedOperation #3203

Merged

Conversation

tombuildsstuff
Copy link
Contributor

This PR refactors how Resource ID parsing is done, such that we utilize the Swagger Operation ID as the key for the map rather than the URI, which fixes an issue seen in SQL (#2842) where the same HTTP Path may have different constant values for each of the HTTP Methods.

Currently the URI being the map key means that we have an eventual consistency issue depending on the ordering in which the operations are parsed - however this fixes this by updating the map to use the Operation ID (which has to be unique).

At least through my testing, this is one part to unblocking #2842 for the latest API Version (2023-02-01-preview) - however this also requires a Swagger fix to do so (coming next).

Additionally, we need to confirm if this unblocks the remaining API versions - which is next after the PR.

…2842)

This happens when the same URI/Path contains multiple operations with different HTTP Methods which use different
values for the Constant segments. This means that dependent on the order these are processed, that we can have
eventual consistency/flapping between one constant and another - causing #2842
…e Operation ID rather than URI as the identifier

This works around an issue where one URI can contain multiple values for the constant segments, meaning that because
we were using a map previously, the parsing order of the map (which would be eventually consistent) would cause flapping
on the imported data.
…ogic

This was used to pull data from the Data API to allow overriding existing data, which is no longer
a technique we're planning to use.
@tombuildsstuff
Copy link
Contributor Author

I've gone ahead and imported all of Sql to confirm what's importable and what's not - once Azure/azure-rest-api-specs#26303 has been merged.

With both this PR and Azure/azure-rest-api-specs#26303 - the following API versions can be imported successfully (and do not appear [based on 3 regenerations of the lot] to be eventually consistent):

* 2015-05-01
* 2021-11-01
* 2017-03-01-preview
* 2017-10-01-preview
* 2018-06-01-preview
* 2019-06-01-preview
* 2020-02-02-preview
* 2021-02-01-preview
* 2021-05-01-preview
* 2021-08-01-preview
* 2021-11-01-preview
* 2022-02-01-preview
* 2022-05-01-preview
* 2022-08-01-preview
* 2022-11-01-preview
* 2023-02-01-preview

Whilst the following API versions have issues:

* 2014-04-01
* 2015-05-01-preview
* 2020-08-01-preview
* 2020-11-01-preview

And we do use currently vendor two of these (2014-04-01 and 2020-11-01-preview) - it appears that we should be able to use the latest API version (2023-02-01-preview) instead here.

For the sake of completeness, the regeneration errors for the older API versions are:

2014-04-01

2023-10-19T17:08:08.508+0200 [INFO]  Importer for Service "Sql".Importer for API Version "2014-04-01": ❌ Service "Sql" - Api Version "2014-04-01"
2023-10-19T17:08:08.508+0200 [ERROR] Importer for Service "Sql".Importer for API Version "2014-04-01":      💥 Error: parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/stable/2014-04-01": combining resources for "Sql-2014-04-01": combining operations: duplicate operations named "ListByServer"
2023/10/19 17:08:08 Error: parsing data for Service "Sql": parsing data for Service "Sql" / Version "2014-04-01": parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/stable/2014-04-01": combining resources for "Sql-2014-04-01": combining operations: duplicate operations named "ListByServer"

2015-05-01-preview

2023-10-19T17:06:20.796+0200 [INFO]  Importer for Service "Sql".Importer for API Version "2015-05-01-preview": ❌ Service "Sql" - Api Version "2015-05-01-preview"
2023-10-19T17:06:20.796+0200 [ERROR] Importer for Service "Sql".Importer for API Version "2015-05-01-preview":      💥 Error: parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2015-05-01-preview": combining resources for "Sql-2015-05-01-preview": combining models: duplicate models named "DatabaseBlobAuditingPolicyProperties" with different fields - first 7 - second 9
2023/10/19 17:06:20 Error: parsing data for Service "Sql": parsing data for Service "Sql" / Version "2015-05-01-preview": parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2015-05-01-preview": combining resources for "Sql-2015-05-01-preview": combining models: duplicate models named "DatabaseBlobAuditingPolicyProperties" with different fields - first 7 - second 9

2020-08-01-preview

2023-10-19T17:16:32.916+0200 [INFO]  Importer for Service "Sql".Importer for API Version "2020-08-01-preview": ❌ Service "Sql" - Api Version "2020-08-01-preview"
2023-10-19T17:16:32.916+0200 [ERROR] Importer for Service "Sql".Importer for API Version "2020-08-01-preview":      💥 Error: parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2020-08-01-preview": combining resources for "Sql-2020-08-01-preview": combining operations: duplicate operations named "Failover"
2023/10/19 17:16:32 Error: parsing data for Service "Sql": parsing data for Service "Sql" / Version "2020-08-01-preview": parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2020-08-01-preview": combining resources for "Sql-2020-08-01-preview": combining operations: duplicate operations named "Failover"

2020-11-01-preview

2023-10-19T17:25:42.067+0200 [INFO]  Importer for Service "Sql".Importer for API Version "2020-11-01-preview": ❌ Service "Sql" - Api Version "2020-11-01-preview"
2023-10-19T17:25:42.067+0200 [ERROR] Importer for Service "Sql".Importer for API Version "2020-11-01-preview":      💥 Error: parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2020-11-01-preview": combining resources for "Sql-2020-11-01-preview": combining operations: duplicate operations named "Delete"
2023/10/19 17:25:42 Error: parsing data for Service "Sql": parsing data for Service "Sql" / Version "2020-11-01-preview": parsing Swagger files: parsing files in "../../submodules/rest-api-specs/specification/sql/resource-manager/Microsoft.Sql/preview/2020-11-01-preview": combining resources for "Sql-2020-11-01-preview": combining operations: duplicate operations named "Delete"

Copy link
Contributor

@manicminer manicminer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@tombuildsstuff tombuildsstuff merged commit 263c558 into main Oct 20, 2023
@tombuildsstuff tombuildsstuff deleted the b/arm-importer-different-constants-same-path-2842 branch October 20, 2023 08:58
tombuildsstuff added a commit that referenced this pull request Oct 25, 2023
This is possible now that Azure/azure-rest-api-specs#26303 and
#3203 have been merged

Fixes #2842
Supersedes #2988
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants