Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tesing WD with CDH 3.x version #284

Open
sivaponting opened this issue Aug 18, 2023 · 2 comments
Open

Tesing WD with CDH 3.x version #284

sivaponting opened this issue Aug 18, 2023 · 2 comments

Comments

@sivaponting
Copy link

Hi Team,
Started testing WD with CDH (Hive 3.1.300.7.1.7.0-551 & Hadoop 3.1.1.7.1.7.0-551). Some of the integration tests are failing.

Integration Test name: WaggleDanceIntegrationTest.prefixedModeDatabaseNameMapping
Issue: Both the local & remote metastore retrieving same databases

Added the console log here. Highlighted some of the logs. Suspect both the local & remote are getting the database name from same metastore.

.  .

' '
' ,-. ' _ __ __ ____
' \ / ' | | / /___ _____ _____ / /_ / __ ____ _____ ________
'. {|||)< ' . | | /| / / __ / __ / __ / / _ \/ / / / __ / __ / / _
. / \ . | |/ |/ / // / // / // / / / // / // / / / / // _/
. -' . |__/|__/\__,_/\__, /\__, /_/\___/_____/\__,_/_/ /_/\___/\___/ . . /____//____/ ' '
2023-08-18T09:40:04,512 INFO org.hibernate.validator.internal.util.Version:21 - HV000001: Hibernate Validator 6.2.5.Final
2023-08-18T09:40:04,623 INFO com.hotels.bdp.waggledance.WaggleDance:55 - Starting WaggleDance using Java 1.8.0_252 on M01GMBFYC2M42 with PID 53209 (/Users/sivakumard/DBS/Technology/Hadoop_EcoSystem/waggle-dance-hive-3.x/waggle-dance-core/target/classes started by sivakumard in /Users/sivakumard/DBS/Technology/Hadoop_EcoSystem/waggle-dance-hive-3.x/waggle-dance-integration-tests)
2023-08-18T09:40:04,629 DEBUG com.hotels.bdp.waggledance.WaggleDance:56 - Running with Spring Boot v2.7.13, Spring v5.3.28
2023-08-18T09:40:04,630 INFO com.hotels.bdp.waggledance.WaggleDance:631 - No active profile set, falling back to 1 default profile: "default"
2023-08-18T09:40:07,402 INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer:108 - Tomcat initialized with port(s): 61256 (http)
2023-08-18T09:40:07,416 INFO org.apache.coyote.http11.Http11NioProtocol:173 - Initializing ProtocolHandler ["http-nio-61256"]
2023-08-18T09:40:07,417 INFO org.apache.catalina.core.StandardService:173 - Starting service [Tomcat]
2023-08-18T09:40:07,418 INFO org.apache.catalina.core.StandardEngine:173 - Starting Servlet engine: [Apache Tomcat/9.0.76]
2023-08-18T09:40:07,835 INFO org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]:173 - Initializing Spring embedded WebApplicationContext
2023-08-18T09:40:07,837 INFO org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext:292 - Root WebApplicationContext: initialization completed in 2885 ms
2023-08-18T09:40:08,384 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:85 - Manifest location in JARs is null
2023-08-18T09:40:08,386 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:134 - Could not find manifest in location file:/Users/sivakumard/DBS/Technology/Hadoop_EcoSystem/waggle-dance-hive-3.x/waggle-dance-core/target/classes//META-INF/MANIFEST.MF
2023-08-18T09:40:08,386 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:91 - Manifest location on disk is null
2023-08-18T09:40:08,386 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:97 - Manifest location via getResource() is jar:file:/Users/sivakumard/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre/lib/ext/openjsse.jar!/META-INF/MANIFEST.MF
2023-08-18T09:40:08,387 INFO com.hotels.bdp.waggledance.WaggleDance:134 - Implementation-Title=openjsse; Implementation-Version=1.1.3; Specification-Title=openjsse; Build-Jdk-Spec=1.8; Created-By=Maven Archiver 3.4.0; Specification-Version=1.1; Manifest-Version=1.0;
2023-08-18T09:40:08,455 INFO com.hotels.bdp.waggledance.mapping.service.impl.YamlFederatedMetaStoreStorage:161 - Loading federations from /var/folders/fl/2060hb1n6fx0cn0bjpmj4bjr0000gq/T/junit2886136884324204085/config/federation-config.yml
2023-08-18T09:40:08,828 INFO com.hotels.bdp.waggledance.mapping.service.impl.YamlFederatedMetaStoreStorage:122 - Adding federation PrimaryMetaStore{name=primary, databasePrefix=, federationType=PRIMARY, remoteMetaStoreUris=thrift://localhost:61222, metastoreTunnel=null, accessControlType=READ_ONLY, writableDatabaseWhiteList=null, status=UNKNOWN}
2023-08-18T09:40:08,832 INFO com.hotels.bdp.waggledance.mapping.service.impl.YamlFederatedMetaStoreStorage:122 - Adding federation FederatedMetaStore{name=waggle_remote, databasePrefix=waggle_remote, federationType=FEDERATED, remoteMetaStoreUris=thrift://localhost:61167, metastoreTunnel=null, accessControlType=READ_ONLY, writableDatabaseWhiteList=null, status=UNKNOWN}
2023-08-18T09:40:08,832 INFO com.hotels.bdp.waggledance.mapping.service.impl.YamlFederatedMetaStoreStorage:180 - Loaded 2 federations
2023-08-18T09:40:10,816 INFO org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver:58 - Exposing 1 endpoint(s) beneath base path '/actuator'
2023-08-18T09:40:11,094 INFO org.apache.coyote.http11.Http11NioProtocol:173 - Starting ProtocolHandler ["http-nio-61256"]
2023-08-18T09:40:11,125 INFO org.springframework.boot.web.embedded.tomcat.TomcatWebServer:220 - Tomcat started on port(s): 61256 (http) with context path ''
2023-08-18T09:40:11,151 INFO org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor:315 - No TaskScheduler/ScheduledExecutorService bean found for scheduled processing
2023-08-18T09:40:11,168 DEBUG com.hotels.bdp.waggledance.mapping.service.impl.PollingFederationService:54 - polling status
2023-08-18T09:40:11,172 INFO com.hotels.bdp.waggledance.WaggleDance:61 - Started WaggleDance in 7.241 seconds (JVM running for 32.23)
2023-08-18T09:40:11,174 INFO com.hotels.bdp.waggledance.mapping.model.MetaStoreMappingFactoryImpl:76 - Mapping databases with name 'waggle_remote' to metastore: thrift://localhost:61167
2023-08-18T09:40:11,174 INFO com.hotels.bdp.waggledance.mapping.model.MetaStoreMappingFactoryImpl:76 - Mapping databases with name 'primary' to metastore: thrift://localhost:61222
2023-08-18T09:40:11,208 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:120 - Starting WaggleDance on port 48869
2023-08-18T09:40:11,213 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:182 - Starting WaggleDance Server
2023-08-18T09:40:11,213 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:217 - Started the new WaggleDance on port [48869]...
2023-08-18T09:40:11,216 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:218 - Options.minWorkerThreads = 200
2023-08-18T09:40:11,216 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:219 - Options.maxWorkerThreads = 1000
2023-08-18T09:40:11,216 INFO com.hotels.bdp.waggledance.server.MetaStoreProxyServer:220 - TCP keepalive = true
2023-08-18T09:40:11,273 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:124 - useSasl=false, useSsl=false, useFramedTransport=false, useCompactProtocol=false, clientSocketTimeout=600000
2023-08-18T09:40:11,273 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:124 - useSasl=false, useSsl=false, useFramedTransport=false, useCompactProtocol=false, clientSocketTimeout=600000
2023-08-18T09:40:11,274 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:126 - metastoreUris=[thrift://localhost:61167]
2023-08-18T09:40:11,274 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:126 - metastoreUris=[thrift://localhost:61222]
2023-08-18T09:40:11,275 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:130 - Trying to connect to metastore with URI thrift://localhost:61167
2023-08-18T09:40:11,275 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:130 - Trying to connect to metastore with URI thrift://localhost:61222
2023-08-18T09:40:11,275 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:172 - Entering TBinaryProtocol
2023-08-18T09:40:11,275 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:172 - Entering TBinaryProtocol
2023-08-18T09:40:11,300 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:179 - Opened a connection to metastore 'thrift://localhost:61222', total current connections to all metastores: 1
2023-08-18T09:40:11,300 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:179 - Opened a connection to metastore 'thrift://localhost:61167', total current connections to all metastores: 2
2023-08-18T09:40:11,301 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:189 - Connection opened with out #set_ugi call', on URI thrift://localhost:61167
2023-08-18T09:40:11,302 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:189 - Connection opened with out #set_ugi call', on URI thrift://localhost:61222
2023-08-18T09:40:11,302 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:222 - Connected to metastore.
2023-08-18T09:40:11,302 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====shutdown
2023-08-18T09:40:11,302 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:222 - Connected to metastore.
2023-08-18T09:40:11,303 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====shutdown
2023-08-18T09:40:11,304 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====null
2023-08-18T09:40:11,304 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====null
2023-08-18T09:40:11,305 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:273 - Closed a connection to metastore, current connections: 1
2023-08-18T09:40:11,305 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:273 - Closed a connection to metastore, current connections: 0
2023-08-18T09:40:11,314 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:1053 - 9: Done cleaning up thread local RawStore
2023-08-18T09:40:11,314 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:1053 - 10: Done cleaning up thread local RawStore
2023-08-18T09:40:12,816 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient:247 - HMS client filtering is enabled.
2023-08-18T09:40:12,820 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient:526 - Trying to connect to metastore with URI thrift://localhost:48869
2023-08-18T09:40:12,832 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient:636 - Opened a connection to metastore, current connections: 4
2023-08-18T09:40:12,833 INFO com.hotels.bdp.waggledance.server.TSetIpAddressProcessorFactory:57 - Received a connection from ip: 127.0.0.1
2023-08-18T09:40:12,842 INFO com.hotels.bdp.waggledance.mapping.model.MetaStoreMappingFactoryImpl:76 - Mapping databases with name 'primary' to metastore: thrift://localhost:61222
2023-08-18T09:40:12,879 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:124 - useSasl=false, useSsl=false, useFramedTransport=false, useCompactProtocol=false, clientSocketTimeout=600000
2023-08-18T09:40:12,880 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:126 - metastoreUris=[thrift://localhost:61222]
2023-08-18T09:40:12,880 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:130 - Trying to connect to metastore with URI thrift://localhost:61222
2023-08-18T09:40:12,880 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:172 - Entering TBinaryProtocol
2023-08-18T09:40:12,888 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:179 - Opened a connection to metastore 'thrift://localhost:61222', total current connections to all metastores: 1
2023-08-18T09:40:12,889 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:189 - Connection opened with out #set_ugi call', on URI thrift://localhost:61222
2023-08-18T09:40:12,890 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:222 - Connected to metastore.
2023-08-18T09:40:12,892 INFO com.hotels.bdp.waggledance.mapping.model.MetaStoreMappingFactoryImpl:76 - Mapping databases with name 'waggle_remote' to metastore: thrift://localhost:61167
2023-08-18T09:40:13,013 INFO com.jcabi.aspects.aj.NamedThreads:241 - jcabi-aspects 0.25.1/613e700 started new daemon thread jcabi-loggable for watching of @loggable annotated methods
2023-08-18T09:40:13,025 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #setConf('Configuration: HiveConf.java'): in 2.75ms
2023-08-18T09:40:13,025 INFO com.hotels.bdp.waggledance.server.TSetIpAddressProcessorFactory:62 - useSASL=false
2023-08-18T09:40:13,030 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #getConf(): ... in 2.52ms
2023-08-18T09:40:13,035 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #getThreadId(): ... in 1.59ms
2023-08-18T09:40:13,039 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #getThreadId(): ... in 282.04µs
2023-08-18T09:40:13,045 INFO com.hotels.bdp.waggledance.mapping.service.requests.SetUgiRequest:38 - user=[sivakumard]
2023-08-18T09:40:13,045 INFO com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory.ReconnectingMetastoreClientInvocationHandler:86 - calling #set_ugi (on already open client) for user 'sivakumard', on metastore waggledance-primary
2023-08-18T09:40:13,047 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====set_ugi
2023-08-18T09:40:13,051 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====[staff, everyone, localaccounts, _developer, com.apple.sharepoint.group.1, _lpoperator, com.apple.sharepoint.group.2]
2023-08-18T09:40:13,052 INFO com.hotels.bdp.waggledance.mapping.service.requests.SetUgiRequest:38 - user=[staff, everyone, localaccounts, _developer, com.apple.sharepoint.group.1, _lpoperator, com.apple.sharepoint.group.2]
2023-08-18T09:40:13,052 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:70 - RESULT for :[staff, everyone, localaccounts, _developer, com.apple.sharepoint.group.1, _lpoperator, com.apple.sharepoint.group.2]
2023-08-18T09:40:13,052 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:70 - RESULT for :[sivakumard]
2023-08-18T09:40:13,052 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:73 - ALL RESULT:[staff, everyone, localaccounts, _developer, com.apple.sharepoint.group.1, _lpoperator, com.apple.sharepoint.group.2, sivakumard]
2023-08-18T09:40:13,059 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #set_ugi('sivakumard', '[staff, everyone, localaccounts, _developer, com...19.. _lpoperator, com.apple.sharepoint.group.2]'): ... in 16.75ms
2023-08-18T09:40:13,060 INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient:704 - Connected to metastore.
2023-08-18T09:40:13,066 DEBUG com.hotels.bdp.waggledance.server.invocation-log:202 - #getThreadId(): ... in 240.50µs
Entering get_databases
2023-08-18T09:40:13,071 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:124 - useSasl=false, useSsl=false, useFramedTransport=false, useCompactProtocol=false, clientSocketTimeout=600000
2023-08-18T09:40:13,072 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:126 - metastoreUris=[thrift://localhost:61167]
2023-08-18T09:40:13,072 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:130 - Trying to connect to metastore with URI thrift://localhost:61167
2023-08-18T09:40:13,072 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:172 - Entering TBinaryProtocol
2023-08-18T09:40:13,083 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:179 - Opened a connection to metastore 'thrift://localhost:61167', total current connections to all metastores: 2
2023-08-18T09:40:13,083 INFO com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:186 - calling #set_ugi for user 'sivakumard', on URI thrift://localhost:61167
2023-08-18T09:40:13,083 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====set_ugi
2023-08-18T09:40:13,084 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====[staff, everyone, localaccounts, _developer, com.apple.sharepoint.group.1, lpoperator, com.apple.sharepoint.group.2]
2023-08-18T09:40:13,084 DEBUG com.hotels.bdp.waggledance.client.ThriftMetastoreClientManager:222 - Connected to metastore.
2023-08-18T09:40:13,085 INFO com.hotels.bdp.waggledance.mapping.service.PanopticOperationHandler:75 - Metastore-MappingName=primary, DatabasePrefix=, DatabasePattern=@hive#
2023-08-18T09:40:13,087 INFO com.hotels.bdp.waggledance.mapping.service.PanopticOperationHandler:75 - Metastore-MappingName=waggle_remote, DatabasePrefix=waggle_remote, DatabasePattern=@hive#
2023-08-18T09:40:13,087 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:41 - Metastore-MappingName=primary, DbPattern=@hive#
2023-08-18T09:40:13,087 INFO com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory.ReconnectingMetastoreClientInvocationHandler:95 - cachedUgi=com.hotels.bdp.waggledance.client.HiveUgiArgs@b7fc5832
2023-08-18T09:40:13,088 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====get_databases
2023-08-18T09:40:13,087 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:41 - Metastore-MappingName=waggle_remote, DbPattern=@hive#
2023-08-18T09:40:13,090 INFO com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory.ReconnectingMetastoreClientInvocationHandler:95 - cachedUgi=com.hotels.bdp.waggledance.client.HiveUgiArgs@b7fc5832
2023-08-18T09:40:13,092 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:1053 - 12: source:127.0.0.1 get_databases: @hive#
2023-08-18T09:40:13,092 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:43 - Method=====get_databases
2023-08-18T09:40:13,097 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:1053 - 13: source:127.0.0.1 get_databases: @hive#
2023-08-18T09:40:13,098 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit:355 - ugi=sivakumard ip=127.0.0.1 cmd=source:127.0.0.1 get_databases: @hive#
2023-08-18T09:40:13,098 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit:355 - ugi=sivakumard ip=127.0.0.1 cmd=source:127.0.0.1 get_databases: @hive#
2023-08-18T09:40:13,099 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:869 - 12: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2023-08-18T09:40:13,099 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:869 - 13: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2023-08-18T09:40:13,103 INFO org.apache.hadoop.hive.metastore.PersistenceManagerProvider:170 - Updating the pmf due to property change
2023-08-18T09:40:13,103 INFO org.apache.hadoop.hive.metastore.ObjectStore:627 - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@6f62dbd4, with PersistenceManager: null will be shutdown
2023-08-18T09:40:14,738 INFO org.apache.hadoop.hive.metastore.PersistenceManagerProvider:253 - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2023-08-18T09:40:14,741 INFO org.apache.hadoop.hive.metastore.ObjectStore:627 - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@5bca5fa2, with PersistenceManager: null will be shutdown
2023-08-18T09:40:14,749 INFO org.apache.hadoop.hive.metastore.ObjectStore:474 - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@6f62dbd4, with PersistenceManager: org.datanucleus.api.jdo.JDOPersistenceManager@4a4fa713 created in the thread with id: 92
2023-08-18T09:40:14,749 INFO org.apache.hadoop.hive.metastore.ObjectStore:474 - RawStore: org.apache.hadoop.hive.metastore.ObjectStore@5bca5fa2, with PersistenceManager: org.datanucleus.api.jdo.JDOPersistenceManager@7db02bec created in the thread with id: 88
2023-08-18T09:40:14,781 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:847 - Created RawStore: org.apache.hadoop.hive.metastore.ObjectStore@6f62dbd4 from thread id: 92
2023-08-18T09:40:14,781 INFO org.apache.hadoop.hive.metastore.HiveMetaStore:847 - Created RawStore: org.apache.hadoop.hive.metastore.ObjectStore@5bca5fa2 from thread id: 88
2023-08-18T09:40:15,727 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====[default, local_database]
2023-08-18T09:40:15,727 INFO com.hotels.bdp.waggledance.client.compatibility.HiveCompatibleThriftHiveMetastoreIfaceFactory:46 - Object=====[default, local_database]
2023-08-18T09:40:15,729 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:43 - Before Filter - Metastore-MappingName=primary, Databases=[default, local_database]
2023-08-18T09:40:15,729 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:43 - Before Filter - Metastore-MappingName=waggle_remote, Databases=[default, local_database]
2023-08-18T09:40:15,730 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:51 - After Filter - Metastore-MappingName=waggle_remote, Mapped Databases=[]
2023-08-18T09:40:15,730 DEBUG com.hotels.bdp.waggledance.mapping.model.DatabaseNameMapping:56 - transformOutboundDatabaseName 'local_database' to '[abc]'
2023-08-18T09:40:15,730 INFO com.hotels.bdp.waggledance.mapping.service.requests.GetAllDatabasesByPatternRequest:51 - After Filter - Metastore-MappingName=primary, Mapped Databases=[default, local_database, abc]
2023-08-18T09:40:15,731 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:70 - RESULT for :[default, local_database, abc]
2023-08-18T09:40:15,731 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:70 - RESULT for :[]
2023-08-18T09:40:15,731 INFO com.hotels.bdp.waggledance.mapping.service.PanopticConcurrentOperationExecutor:73 - ALL RESULT:[default, local_database, abc]
2023-08-18T09:40:15,731 INFO com.hotels.bdp.waggledance.mapping.service.PanopticOperationHandler:83 - All Databases Result=[default, local_database, abc] `
@patduin
Copy link
Contributor

patduin commented Aug 18, 2023

What version of WD are you using?

The Integration Tests are generally a bit flaky I've noticed. Try to run the test as a single test and see if they succeed.

@sivaponting
Copy link
Author

What version of WD are you using?

The Integration Tests are generally a bit flaky I've noticed. Try to run the test as a single test and see if they succeed.

Hi @patduin ,
I'm using Hive-3.x branch. After changing the PanopticOperationHandler code highlighted below, it's working as expected. But we are calling the PanopticConcurrentExecutor for each database mapping.

` protected List getAllDatabases(
Map<DatabaseMapping, String> databaseMappingsForPattern,
BiFunction<String, DatabaseMapping, Boolean> filter) {

List<String> result = new ArrayList<>();
for (Entry<DatabaseMapping, String> mappingWithPattern : databaseMappingsForPattern.entrySet()) {
  DatabaseMapping mapping = mappingWithPattern.getKey();
  log.info("Metastore-MappingName={}, DatabasePrefix={}, DatabasePattern={}",
          mapping.getMetastoreMappingName(), mapping.getDatabasePrefix(), mappingWithPattern.getValue());
  GetAllDatabasesByPatternRequest databasesByPatternRequest = new GetAllDatabasesByPatternRequest(mapping,
      mappingWithPattern.getValue(), filter);
  List<GetAllDatabasesByPatternRequest> allRequests = new ArrayList<>();
  allRequests.add(databasesByPatternRequest);
  **result.addAll(getPanopticOperationExecutor()
          .executeRequests(allRequests, GET_DATABASES_TIMEOUT, "Can't fetch databases by pattern: {}"));**

}
/*List<String> result = getPanopticOperationExecutor()
    .executeRequests(allRequests, GET_DATABASES_TIMEOUT, "Can't fetch databases by pattern: {}");*/
log.info("All Databases Result={}", result);
return result;`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@patduin @sivaponting