pageserver: do not read redundant timeline_layers from IndexPart, so that we can remove it later

[jcsp]

Problem

IndexPart contains two redundant lists of layer names: a set of the names, and then a map of name to metadata.

We already required that all the layers in timeline_layers are also in layers_metadata, in initialize_with_current_remote_index_part, so if there were any index_part.json files in the field that relied on these sets being different, they would already be broken.

Summary of changes

timeline_layers is made private and no longer read at runtime. It is still serialized, but not deserialized.

disk_consistent_lsn is also made private, as this field only exists for convenience of humans reading the serialized JSON.

This prepares us to entirely remove timeline_layers in a future release, once this change is fully deployed, and therefore no pageservers are trying to read the field.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

1624 tests run: 1550 passed, 0 failed, 74 skipped (full report)

---

_{The comment gets automatically updated with the latest test results
0657362 at 2023-08-21T11:10:48.836Z :recycle:}

[arpad-m]

Yeah the two lists match, only one has values in addition. It's thus redundant and can be removed.

[koivunej]

I think we should bump the version (cannot add comment) because this is a meaningful change overall. It'll all contribute to us being able to spot if some old version is surprisingly still lurking around when doing cleanups.

No other issues.

[problame]

Maybe worth noting in this PR the earlier commit that started requiring layer_metadata. It is dd22c87 .

By the way, in theory, there is no guarantee that Control Plane has attached all tenants all the time.
So, in theory, we can't be certain that all index_part.json's have been migrated.

I guess it's fine though.

[arpad-m]

there is no guarantee that Control Plane has attached all tenants all the time.
So, in theory, we can't be certain that all index_part.json's have been migrated.

That's important to keep in mind. I think right now we are still writing out the old format, we just don't read the field any more, basically allowing for index_part.json's that don't have it. This ensures that we can still migrate tenants from an old pageserver version to a newer pageserver version, i.e. when someone uses the migration button during deployment. I think it's quite rare to have this, but it's good that @jcsp is so careful about it.

[jcsp]

I checked that we didn't have any indices in the field with timeline_layers that didn't match layer_metadata, and defensively bumped the IndexPart version: even though the serialized data should be the same, the version will make it obvious which version wrote the file.