[improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability #41510
Conversation
…ty (apache#40891) This PR makes the following changes to the uppercase and lowercase mapping of JdbcCatalog 1. The identifierMapping is managed by JdbcExternalCatalog instead of JdbcClient to better control its lifecycle 2. The identifierMapping no longer loads remoteName alone, but Catalog controls the loading uniformly 3. The identifierMapping will be loaded when each FE performs makeSureInitialized() to ensure that each FE has a mapping 4. The initialization of mapping will only be performed once in makeSureInitialized(), which means that even if you use metaCache, if your source data is updated when identifierMapping is enabled, you must refresh the catalog to query normally. 5. The identifierMapping is only responsible for the properties of the Catalog and is no longer affected by the fe config, simplifying the processing logic 6. If lower_case_mete_names is false and meta_names_mapping is empty in the catalog properties, the identifierMapping will no longer take effect, further enhancing the stability of the default settings 7. The JdbcClient is no longer closed during onRefreshCache, reducing the repeated creation of resources, improving reuse, and reducing the leakage of some global shared threads
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
| String mappedName = nameMapping.getOrDefault(name, name); | ||
| String localName = isLowerCaseMetaNames ? mappedName.toLowerCase() : mappedName; | ||
| public interface IdentifierMapping { | ||
| List<String> fromRemoteDatabaseName(List<String> remoteDatabaseNames); |
There was a problem hiding this comment.
Add comment for all these methods
| localNameToRemoteName.computeIfAbsent(localName, k -> name); | ||
|
|
||
| if (isLowerCaseMetaNames && !lowerCaseNames.add(localName)) { | ||
| if (nameMap.containsKey(localName)) { |
There was a problem hiding this comment.
Looks like the nameMap must contains localName if goes here?
| + " use meta_name_mapping to specify the column names."); | ||
| } | ||
| for (int i = 0; i < remoteColumns.size(); i++) { | ||
| remoteColumns.get(i).setName(localColumnNames.get(i)); |
There was a problem hiding this comment.
Do not modify the input parameter
There was a problem hiding this comment.
And I think, the method fromRemoteColumnName should be act same as fromRemoteDbName and fromRemoteTableName, to return List<String> instead of List<Column>.
Which means these method only handle Names, you can wrap it and generate Column object outside
| return columnMap; | ||
| } | ||
|
|
||
| private <K, V> V getRequiredMapping(Map<K, V> map, K key, String typeName, String entityName) { |
There was a problem hiding this comment.
The entityName seems unnecessary.
| private <K, V> V getRequiredMapping(Map<K, V> map, K key, String typeName, String entityName) { | ||
| V value = map.get(key); | ||
| if (value == null) { | ||
| LOG.warn("No remote {} found for {}: {}. Please refresh this catalog.", typeName, typeName, entityName); |
There was a problem hiding this comment.
| LOG.warn("No remote {} found for {}: {}. Please refresh this catalog.", typeName, typeName, entityName); | |
| LOG.warn("No remote {} found for: {}. Please refresh this catalog.", typeName, key); |
| } | ||
|
|
||
| @Override | ||
| protected void buildDatabaseMapping() { |
There was a problem hiding this comment.
this method's logic is same as listDatabaseNames(), extract one
| @@ -149,6 +150,9 @@ public abstract class ExternalCatalog | |||
| protected Optional<Boolean> useMetaCache = Optional.empty(); | |||
| protected MetaCache<ExternalDatabase<? extends ExternalTable>> metaCache; | |||
|
|
|||
| protected IdentifierMapping identifierMapping; | |||
| private boolean mappingsInitialized = false; | |||
There was a problem hiding this comment.
Why using a separator flag mappingsInitialized?
Why not just using initialized?
| public List<String> fromRemoteTableName(String remoteDbName, List<String> remoteTableNames) { | ||
| // If mapping is not required, return the original input | ||
| if (!isLowerCaseMetaNames && isMappingInvalid()) { | ||
| return remoteTableNames; |
There was a problem hiding this comment.
If user set Config.lower_case_table_names = 1,
I think we also need to handle the name mapping?
This PR makes the following changes to the uppercase and lowercase mapping of JdbcCatalog