Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tfsdk: Consider Caching Provider Defined Resources/Data Sources in Server #299

Closed
bflad opened this issue Apr 22, 2022 · 3 comments · Fixed by #333
Closed

tfsdk: Consider Caching Provider Defined Resources/Data Sources in Server #299

bflad opened this issue Apr 22, 2022 · 3 comments · Fixed by #333
Assignees
@bflad
Labels
enhancement New feature or request

Comments

@bflad
Copy link
Contributor

bflad commented Apr 22, 2022

Module version

v0.6.1

Use-cases

The current provider server logic always calls back into the provider defined GetResources() and GetDataSources() methods when trying to find each ResourceType or DataSourceType associated with every RPC. While not a biggest performance deal, it does mean that it is repeatedly fetching the maps and executing any provider logic (if any). Terraform CLI and the framework currently expect the type mapping to be consistent for providers after GetProviderSchema. The server could cache these maps on first use.

Proposal

  • Add dataSourceTypes map[string]DataSourceType and resourceTypes map[string]ResourceType fields to the server type
  • In (*server).GetProviderSchema(), set s.dataSourceTypes = dataSourceSchemas and s.resourceTypes = resourceSchemas
  • Update (*server).getDataSourceType() and (*server).getResourceType() methods to check s.dataSourceTypes and s.resourceTypes fields instead of calling the provider defined GetDataSources() and GetResources() methods
@bflad bflad added the enhancement New feature or request label Apr 22, 2022
@bflad
Copy link
Contributor Author

bflad commented Apr 22, 2022 •
edited
Loading

Similar caching could also be done for Provider, DataSourceType, and ResourceType GetSchema calls.

bflad added a commit that referenced this issue May 9, 2022
@bflad
internal: Migrate ValidateProviderConfig RPC to fromproto6, fwserver,…
0c04137 
… and toproto6

Reference: #215

One particular quirk that this implementation shows is that it is not entirely possible to avoid working with `tfsdk`/`fwserver` bits in `proto6server` as the current data types within the framework (`Config`, `Plan`, `State`) require the `Schema` so data can be appropriately converted from the protocol to the framework. To support this effort and prevent repetitively executing the provider defined `GetSchema` method, a `fwserver` method is introduced that caches the `Schema` and `Diagnostics` results on first use. This methodology can also be applied in the future to resolve #299.
@bflad bflad mentioned this issue May 9, 2022
Merged
@bflad bflad closed this as completed in 25f29d3 May 11, 2022
@bflad bflad reopened this May 11, 2022
@bflad
Copy link
Contributor Author

bflad commented May 11, 2022

(Commit message wording accidentally closed this)

@bflad bflad self-assigned this May 11, 2022
bflad added a commit that referenced this issue May 11, 2022
@bflad
internal: Migrate ValidateDataSourceConfig RPC to fromproto6, fwserve…
9ece427 
…r, and toproto6

Reference: #215
Reference: #299

This introduces framework server caching for data source types and schemas, similar to the provider schema cache, which is needed with other data source RPCs. The same methodology will be applied for resource types and schemas, when the `ValidateResourceConfig` RPC is migrated.
@bflad bflad mentioned this issue May 11, 2022
Merged
bflad added a commit that referenced this issue May 12, 2022
@bflad
internal: Migrate ValidateDataSourceConfig RPC to fromproto6, fwserve…
d66b674 
…r, and toproto6 (#324)

Reference: #215
Reference: #299

This introduces framework server caching for data source types and schemas, similar to the provider schema cache, which is needed with other data source RPCs. The same methodology will be applied for resource types and schemas, when the `ValidateResourceConfig` RPC is migrated.
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal: Migrate ValidateResourceConfig RPC to fromproto6, fwserver,…
65a9bf0 
… and toproto6

Reference: #215
Reference: #299

This introduces framework server caching for resource types and schemas, similar to the data source schema, data source type, and provider schema caches, which is needed with other resource RPCs.
@bflad bflad mentioned this issue May 19, 2022
Merged
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal: Migrate ValidateResourceConfig RPC to fromproto6, fwserver,…
004e1b0 
… and toproto6 (#331)

Reference: #215
Reference: #299

This introduces framework server caching for resource types and schemas, similar to the data source schema, data source type, and provider schema caches, which is needed with other resource RPCs.
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal/proto6server: Always use data source and resource caches, re…
1a9057b 
…move intermediate Provider field

Reference: #215
Reference: #299

This completes the migration to the cached data source and resource schema/type handling in the new `internal/fwserver` package. This also removes the previous `Provider` field so that implementation detail is now wholly within the `internal/fwserver` to remove any confusion.
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal/proto6server: Always use data source and resource caches, re…
b1ffc22 
…move intermediate Provider field

Reference: #215
Reference: #299

This completes the migration to the cached data source and resource schema/type handling in the new `internal/fwserver` package. This also removes the previous `Provider` field so that implementation detail is now wholly within the `internal/fwserver` to remove any confusion.
@bflad bflad mentioned this issue May 19, 2022
Merged
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal/proto6server: Always use data source and resource caches, re…
cf2868d 
…move intermediate Provider field (#333)

Reference: #215
Reference: #299

This completes the migration to the cached data source and resource schema/type handling in the new `internal/fwserver` package. This also removes the previous `Provider` field so that implementation detail is now wholly within the `internal/fwserver` to remove any confusion.
bflad added a commit that referenced this issue May 19, 2022
@bflad
internal: Migrate ImportResourceState RPC to fromproto6, fwserver, an…
fe2a56b 
…d toproto6

Reference: #299
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@bflad bflad

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
1 participant
@bflad