pageserver/storcon: add patch endpoints for tenant config metrics

[VladLazar]

Problem

Cplane and storage controller tenant config changes are not additive. Any change overrides all existing tenant configs. This would be fine if both did client side patching, but that's not the case.

Once this merges, we must update cplane to use the PATCH endpoint.

Summary of changes

High Level

Allow for patching of tenant configuration with a PATCH /v1/tenant/config endpoint.
It takes the same data as it's PUT counterpart. For example the payload below will update gc_period and unset compaction_period. All other fields are left in their original state.

{
  "tenant_id": "1234",
  "gc_period": "10s",
  "compaction_period": null
}

Low Level

• PS and storcon gain PATCH /v1/tenant/config endpoints. PS endpoint is only used for cplane managed instances.
• storcon_cli is updated to have separate commands for set-tenant-config and patch-tenant-config

Related https://github.com/neondatabase/cloud/issues/21043

[github-actions]

7792 tests run: 7461 passed, 0 failed, 331 skipped (full report)

---

Flaky tests (10)

Postgres 17

• test_compute_pageserver_connection_stress: debug-x86-64
• test_lr_with_slow_safekeeper: release-arm64
• test_pageserver_small_inmemory_layers[False]: debug-x86-64
• test_idle_checkpoints: debug-x86-64
• test_physical_replication_config_mismatch_too_many_known_xids: debug-x86-64
• test_check_visibility_map: debug-x86-64, debug-x86-64

Postgres 15

• test_parallel_copy: release-arm64

Postgres 14

• test_pgdata_import_smoke[8-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64
• test_pgdata_import_smoke[None-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: release-arm64

Code coverage* (full report)

• functions: 31.4% (8382 of 26736 functions)
• lines: 47.7% (65943 of 138163 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
c1115bf at 2024-12-11T19:25:57.851Z :recycle:}

[problame]

In principle, I applaud this.

I don't know if inventing our own patching format is the way to go, as opposed to using sth like https://jsonpatch.com/.

But, not asking to do it here, and AFAICT this will be only used internally, right?

Or are we planning for cplane to use this new API to set pitr_interval?

We could offer a dedicated pitr endpoint for cplane to sidestep the issue.

[VladLazar]

Or are we planning for cplane to use this new API to set pitr_interval?

Yeah. That's the idea.

We could offer a dedicated pitr endpoint for cplane to sidestep the issue.

I'd rather offer something generic so we don't have to do this again when cplane wants to set a new field.
In any case, the pageserver stuff goes away when we finish the migration to storcon, so perhaps not
super relevant either way.

[problame]

I'd rather offer something generic so we don't have to do this again when cplane wants to set a new field.
In any case, the pageserver stuff goes away when we finish the migration to storcon, so perhaps not
super relevant either way.

The pageserver-side yes, but the API will continue to exist, cplane will simply call storcon instead of PS.

Won't block the PR on this though as it's a lateral move in terms of API surface towards cplane.

[VladLazar]

I'd rather offer something generic so we don't have to do this again when cplane wants to set a new field.
In any case, the pageserver stuff goes away when we finish the migration to storcon, so perhaps not
super relevant either way.

The pageserver-side yes, but the API will continue to exist, cplane will simply call storcon instead of PS.

Which API will continue to exist? The pageserver tenant config APIs can be removed once the migration is complete.
On storcon they will still exist and interface with the pageservers via location_config.

[problame]

Right, but cplane will still call storcon with an API to set (or eventually, patch) the tenant config directly, instead of a well-defined API to set just the pitr interval and leave tenant config be a storage-internal concept.

[VladLazar]

Right, but cplane will still call storcon with an API to set (or eventually, patch) the tenant config directly, instead of a well-defined API to set just the pitr interval and leave tenant config be a storage-internal concept.

Okay, I see your point now. I don't think it makes much difference. We should allow cplane to set mutiple "project configs". I agree that project configs are a subset of tenant configs and we could model it as such, but I don't see a huge benefit in doing that.