tools/importer-rest-api-specs: switching out (most of) the hand-defined Common IDs for the list now available in hashicorp/go-azure-helpers
#4017
Labels
area/resource-ids
Related to Resource IDs, Parsers and Formatters
enhancement
New feature or request
tool/importer-rest-api-specs
Swagger Data Importer issues
Comments
tombuildsstuff
added
enhancement
Merged
4 tasks
tombuildsstuff
added a commit
that referenced
this issue
Jul 12, 2024
…kage There's a bunch of these, and whilst these'll get replaced in time with #4017 but for now this cleans this up
manicminer
pushed a commit
that referenced
this issue
Oct 1, 2024
…kage There's a bunch of these, and whilst these'll get replaced in time with #4017 but for now this cleans this up
stephybun
pushed a commit
that referenced
this issue
Oct 9, 2024
…kage There's a bunch of these, and whilst these'll get replaced in time with #4017 but for now this cleans this up
stephybun
added a commit
that referenced
this issue
Oct 9, 2024
* `tools/importer-rest-api-specs`: refactoring the `commonschema` Matchers out into the new structure * `tools/importer-rest-api-specs`: refactoring the Constant package * `tools/importer-rest-api-specs`: porting the Network workaround into a Data Workaround * `tools/importer-rest-api-specs`: porting the `dataworkarounds` package over to using the SDK types/new path This is a copy and patch up, since I want to leave the existing code path as-is for the moment, to break it apart more effectively * `tools/importer-rest-api-specs`: scaffolding out the logic to parse a Service/APIVersion * `tools/importer-rest-api-specs`: porting over the "remove unused items" logic * `tools/importer-rest-api-specs`: adding a shim to the updated `dataworkarounds` package * dependencies: updating to `v0.69.0` of `github.com/hashicorp/go-azure-helpers` * `tools/importer-rest-api-specs`: refactoring the `normalization` package into the new structure * `tools/importer-rest-api-specs`: moving all of the `cleanup` items into the new package This wants refactoring, but moving it over allows cleaning the rest up * `tools/importer-rest-api-specs`: adding inner most sdk operation object definition * `tools/importer-rest-api-specs`: moving the `commonids` into a subpackage There's a bunch of these, and whilst these'll get replaced in time with #4017 but for now this cleans this up * `tools/data-api-sdk`: removing an empty todo file * `tools/importer-rest-api-specs`: refactoring more logic out to the new package The `parser` package itself needs cleaning, but there's a few things to do first * `tools/importer-rest-api-specs`: moving another `ignore` function * WIP * tools/importer-rest-api-specs: reimplementing #4290 atop the rebase The rebase is messed up, but it's a minor patch so should be fine * `tools/importer-rest-api-specs`: adding a placeholder for parsing an APIResource * `tools/importer-rest-api-specs`: refactoring the parser into smaller packages This leaves the old parser logic in place for now, which'll be cleaned up in the next commit * `tools/importer-rest-api-specs`: fixing an issue where the `validate` command didn't pass the API Definitions Directory correctly * `tools/importer-rest-api-specs`: title-casing the ReferenceName - which ensures that when we compare the ObjectDefinition from a previously parsed/currently parsed dataset we have matching values These are normalised post-parsing, so when comparing against the old data set we need to use TitleCase too, so we might as well do this here * `tools/importer-rest-api-specs`: ensuring that TypeSpec examples are filtered out * `tools/importer-rest-api-specs`: trimming `*.json` from the filename to build the api resource name * `tools/importer-rest-api-specs`: removing (most) of the old code path * `tools/data-api-sdk`: making IsCommonType required so this is more explicit * `tools/importer-rest-api-specs`: fixing the validation * `tools/importer-rest-api-specs`: re-enabling most of the tests * `tools/importer-rest-api-specs`: consolidating/porting over the data tests These ensure that the Configs can be parsed and that the Data from them can be loaded * `tools/importer-rest-api-specs`: fixing the test format * tools/importer-rest-api-specs: skipping the local test * `tools/importer-rest-api-specs`: fixing the non-discriminator tests * `tools/importer-rest-api-specs`: updating/reworking the supplementary data test @manicminer is going to work through fixing this one up, this requires parsing the data identifying any parent types and patching any disciminated implementations across * `importer-rest-api-specs`: fix up discriminator handling for datafactory * `importer-rest-api-specs`: shelve SupplementaryData support for now, since it doesn't seem to be required for DataFactory models as they use external Swagger Refs. * `importer-rest-api-specs`: implement `-services` argument * `importer-rest-api-specs`: operate on maps instead of passing them around * `importer-rest-api-specs`: normalize swagger tag names to prevent duplicates (avoid parsing the same tag twice) * `importer-rest-api-specs`: normalize API resources after parsing all of them * `importer-rest-api-specs`: ensure new APIResource is populated * `importer-rest-api-specs`: maps are pointers * `importer-rest-api-specs`: dereference more maps * `importer-rest-api-specs`: remove use of strings.Title, dereference more maps * limit scope of SDK generation test to the SDK being tested * `importer-rest-api-specs`: two loops become one * `importer-rest-api-specs`: workaround inconsistent casing of "VirtualWANs" * Revert "`importer-rest-api-specs`: two loops become one" This reverts commit 31b45ca. * `importer-rest-api-specs`: normalize "attachednetworks" segment of resource IDs * Revert "`tools/data-api-sdk`: making IsCommonType required so this is more explicit" This reverts commit 71a166f. * importer-rest-api-specs: use updated forks for `go-openapi/analysis`, `go-openapi/jsonreference`, and `go-openapi/spec` * importer-rest-api-specs: remove Supplementary Data parser as is not used/needed * importer-rest-api-specs: rework the swagger spec parser and manually resolve all remote references * importer-rest-api-specs: fix format string, ensure model names are URL-decoded when parsing from the Ref URL fragment * importer-rest-api-specs: updated tests for discriminated models to expect full ancestry * importer-rest-api-specs: remove vestiges of supplementary data spike * importer-rest-api-specs: do not set the description, see #4434 * importer-rest-api-specs: hack for conflicting constant names in the same set of resources * importer-rest-api-specs: disable check for non-default Resource Provider when parsing operations, as this was never utilised and leads to skipping resources/operations * upper case URL in the go sdk --------- Co-authored-by: Tom Bamford <tom@bamford.io> Co-authored-by: Steph <steph@hashicorp.com>
zaowang-ms
pushed a commit
to zaowang-ms/pandora
that referenced
this issue
Oct 22, 2024
…icorp#4307) * `tools/importer-rest-api-specs`: refactoring the `commonschema` Matchers out into the new structure * `tools/importer-rest-api-specs`: refactoring the Constant package * `tools/importer-rest-api-specs`: porting the Network workaround into a Data Workaround * `tools/importer-rest-api-specs`: porting the `dataworkarounds` package over to using the SDK types/new path This is a copy and patch up, since I want to leave the existing code path as-is for the moment, to break it apart more effectively * `tools/importer-rest-api-specs`: scaffolding out the logic to parse a Service/APIVersion * `tools/importer-rest-api-specs`: porting over the "remove unused items" logic * `tools/importer-rest-api-specs`: adding a shim to the updated `dataworkarounds` package * dependencies: updating to `v0.69.0` of `github.com/hashicorp/go-azure-helpers` * `tools/importer-rest-api-specs`: refactoring the `normalization` package into the new structure * `tools/importer-rest-api-specs`: moving all of the `cleanup` items into the new package This wants refactoring, but moving it over allows cleaning the rest up * `tools/importer-rest-api-specs`: adding inner most sdk operation object definition * `tools/importer-rest-api-specs`: moving the `commonids` into a subpackage There's a bunch of these, and whilst these'll get replaced in time with hashicorp#4017 but for now this cleans this up * `tools/data-api-sdk`: removing an empty todo file * `tools/importer-rest-api-specs`: refactoring more logic out to the new package The `parser` package itself needs cleaning, but there's a few things to do first * `tools/importer-rest-api-specs`: moving another `ignore` function * WIP * tools/importer-rest-api-specs: reimplementing hashicorp#4290 atop the rebase The rebase is messed up, but it's a minor patch so should be fine * `tools/importer-rest-api-specs`: adding a placeholder for parsing an APIResource * `tools/importer-rest-api-specs`: refactoring the parser into smaller packages This leaves the old parser logic in place for now, which'll be cleaned up in the next commit * `tools/importer-rest-api-specs`: fixing an issue where the `validate` command didn't pass the API Definitions Directory correctly * `tools/importer-rest-api-specs`: title-casing the ReferenceName - which ensures that when we compare the ObjectDefinition from a previously parsed/currently parsed dataset we have matching values These are normalised post-parsing, so when comparing against the old data set we need to use TitleCase too, so we might as well do this here * `tools/importer-rest-api-specs`: ensuring that TypeSpec examples are filtered out * `tools/importer-rest-api-specs`: trimming `*.json` from the filename to build the api resource name * `tools/importer-rest-api-specs`: removing (most) of the old code path * `tools/data-api-sdk`: making IsCommonType required so this is more explicit * `tools/importer-rest-api-specs`: fixing the validation * `tools/importer-rest-api-specs`: re-enabling most of the tests * `tools/importer-rest-api-specs`: consolidating/porting over the data tests These ensure that the Configs can be parsed and that the Data from them can be loaded * `tools/importer-rest-api-specs`: fixing the test format * tools/importer-rest-api-specs: skipping the local test * `tools/importer-rest-api-specs`: fixing the non-discriminator tests * `tools/importer-rest-api-specs`: updating/reworking the supplementary data test @manicminer is going to work through fixing this one up, this requires parsing the data identifying any parent types and patching any disciminated implementations across * `importer-rest-api-specs`: fix up discriminator handling for datafactory * `importer-rest-api-specs`: shelve SupplementaryData support for now, since it doesn't seem to be required for DataFactory models as they use external Swagger Refs. * `importer-rest-api-specs`: implement `-services` argument * `importer-rest-api-specs`: operate on maps instead of passing them around * `importer-rest-api-specs`: normalize swagger tag names to prevent duplicates (avoid parsing the same tag twice) * `importer-rest-api-specs`: normalize API resources after parsing all of them * `importer-rest-api-specs`: ensure new APIResource is populated * `importer-rest-api-specs`: maps are pointers * `importer-rest-api-specs`: dereference more maps * `importer-rest-api-specs`: remove use of strings.Title, dereference more maps * limit scope of SDK generation test to the SDK being tested * `importer-rest-api-specs`: two loops become one * `importer-rest-api-specs`: workaround inconsistent casing of "VirtualWANs" * Revert "`importer-rest-api-specs`: two loops become one" This reverts commit 31b45ca. * `importer-rest-api-specs`: normalize "attachednetworks" segment of resource IDs * Revert "`tools/data-api-sdk`: making IsCommonType required so this is more explicit" This reverts commit 71a166f. * importer-rest-api-specs: use updated forks for `go-openapi/analysis`, `go-openapi/jsonreference`, and `go-openapi/spec` * importer-rest-api-specs: remove Supplementary Data parser as is not used/needed * importer-rest-api-specs: rework the swagger spec parser and manually resolve all remote references * importer-rest-api-specs: fix format string, ensure model names are URL-decoded when parsing from the Ref URL fragment * importer-rest-api-specs: updated tests for discriminated models to expect full ancestry * importer-rest-api-specs: remove vestiges of supplementary data spike * importer-rest-api-specs: do not set the description, see hashicorp#4434 * importer-rest-api-specs: hack for conflicting constant names in the same set of resources * importer-rest-api-specs: disable check for non-default Resource Provider when parsing operations, as this was never utilised and leads to skipping resources/operations * upper case URL in the go sdk --------- Co-authored-by: Tom Bamford <tom@bamford.io> Co-authored-by: Steph <steph@hashicorp.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Today to add a new Common ID we need to send a PR to
hashicorp/go-azure-helpers, merge and release that, vendor that into this repository (and downstream dependencies) - and then configure it for detection in this repository.As of
v0.67.0a newcommonids.CommonIds()function is available which returns all of the Common ID types defined withinhashicorp/go-azure-sdk- meaning that the majority of the Common ID functions defined here) should be able to replaced by this new function - meaning that this function will require updatingThis would mean that we no longer need to define the Common IDs twice - and whilst the other steps would still apply (so we'd need to ensure
hashicorp/terraform-provider-azurermandhashicorp/go-azure-sdkhad theirhashicorp/go-azure-helpersdependency updated before this repository) - this'd remove a bunch of duplicated logic.This isn't urgent, and I suspect that we may need to leave a couple of them defined in
importer-rest-api-specs- but it'd be worthwhile to reduce the duplication.The text was updated successfully, but these errors were encountered: