Support persisting TableMetadata in the metastore

[eric-maynard]

Description

This adds a new flag METADATA_CACHE_MAX_BYTES which allows the catalog to store table metadata in the metastore and vend it from there when loadTable is called.

Entries are cached based on the metadata location. Currently, the entire metadata.json content is cached.

---

Features not included in this PR:

1. Support for updating the cache when a table is updated
2. Support for invalidating cache entries in the background, rather than waiting for loadTable to be called
3. Structured storage for table metadata

There is partial support for (1) here and I want to extend it, but the goal is to structure things in a way that will allow us to implement (2) and (3) in the future as well.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Documentation update
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Existing tests vend table metadata correctly when caching is enabled.

Added a small test in BasePolarisCatalogTest to cover the basic semantics of caching

Manual testing with eclipselink -- I observed the entities getting created in Postgres and saw large metadata being cached:

db=# select length(internalproperties), substring(internalproperties, 1, 1000) from entities where id = 152;
...
 768691 | {"metadata_location":"file:/tmp/quickstart/ns/tn1731005976265/metadata/00000-e77a2576-5efa-4b7a-b948-121813d713f8.metadata.json","content":"{\"format ...

With MySQL, small metadata is persisted:

mysql> SELECT length(internalproperties), substring(internalproperties, 1, 1000) FROM ENTITIES WHERE id = (SELECT MAX(id) FROM ENTITIES WHERE typecode = 10);
. . .
8159 | {"metadata_location":"file:/tmp/quickstart/ns/t2/metadata/00000-64f975bd-c3a8-4069-bb56-f282003e9157.metadata.json","content":"{\"format-version\"

However large metadata may cause internalproperties to exceed the size limit and nothing will be cached. Calls still return safely.

[dennishuo]

LGTM, suggestion to add comment for readers of the code to more easily understand why we might expect a stale cached metadata location

[snazy]

Thanks for tackling this effort!

However, I think the efficiency of this approach wrt resource usage (heap/network/database pressure) could be drastically improved.

There seems to be a potential inconsistency w/ the cached metadata, which shouldn't happen at all IMO.

[snazy]

This is rather a collection of static utility methods than a "cache manager"? Maybe give it a better name.
BTW: If it only has utility methods, add a private no-arg constructor.

[eric-maynard]

Sounds good, do you have an alternative name you prefer?

Since we are treating the metastore as a cache for the metadata in object storage, and this class manages that cache, I chose the current name.

[snazy]

Storing JSON uncompressed and as a string is a waste of resources on the network, on heap and in the database.
I'd suggest to use Smile serialization format and likely compress that as well.

[eric-maynard]

I think this is probably not a blocker for this initial PR; we can always add another format later on. Personally, I would rather us use a more structured schema (e.g. a more relational one) that supports partial metadata loading rather than just compress the current JSON-based approach.

Storing JSON uncompressed and as a string

Notably, the metastore does exactly this today for properties and internal properties

[snazy]

TBH, this requires a lot of duplicate expensive operations:

1. it loads the metadata JSON from the object store
2. parses JSON into a TableMetadata
3. serializes the TableMetadata to the same string representation as in step 1
4. converts the String into a byte[] representation - just to check the length

The latter steps are likely performed for every access to a table-metadata that exceeds the METADATA_CACHE_MAX_BYTES_NO_CACHING setting, adding a lot of new heap pressure.

[eric-maynard]

I'm concerned about this too, but it's only there for users who have a non-infinite and non-zero METADATA_CACHE_MAX_BYTES. Even still, the overhead is not small. Do you think just checking the string length is sufficient?

[snazy]

What's the reason why view metadata isn't handled as well?

[eric-maynard]

My hope is to limit the scope of the initial PR and expand from there. Do you think view support should be included in the initial change though?