[HUDI-3654] Add new module `hudi-metaserver` #5064

minihippo · 2022-03-18T01:02:34Z

Change Logs

Add a new module hudi-metastore

Impact

HoodieMetastoreBasedTimeline
HoodieMetastoreFileSystemView
MetaStore has three parts：

client, connects with server by Thrift
service, is divided into tableService, partitionService, timelineService and snapshotService
store, has a relation db based implementation with the power of MyBatis

Writing a commit/deltacommit is available, but read is not ready.

Risk level

medium

For test, metastore will start up with an embedded one.

This change added tests and can be verified as follows:

Add metastore client ut
Add metastore store ut
Add a case of cow writing based on metastore

Documentation Update

add a new section on website for introduction

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

melin · 2022-03-18T05:48:25Z

Add catalogName parameter to MetadataStore interface method

xiarixiaoyao · 2022-03-18T08:21:22Z

great feature !!!

nsivabalan · 2022-04-15T20:17:25Z

@xiarixiaoyao : can you review this when you get a chance. I have assigned it to myself as well. So, will try to review in a weeks time.

xiarixiaoyao · 2022-04-16T02:08:39Z

@minihippo could you pls rebase the code and run azure again, thanks

minihippo · 2022-05-18T14:55:58Z

The CI environment lacks thrift under /usr/local/bin/thrift, so that hudi-metastore can't be compiled. I push the compiled thrift classes as a temporary way to pass CI.

hudi-metastore/src/main/java/org/apache/hudi/common/table/HoodieTableMetastoreClient.java

...astore/src/main/java/org/apache/hudi/common/table/timeline/HoodieMetastoreBasedTimeline.java

minihippo · 2022-05-30T07:25:12Z

@xiarixiaoyao It seems that the code review doesn't finish, right?

hudi-metastore/src/main/java/org/apache/hudi/metastore/HoodieMetastoreServer.java

xiarixiaoyao · 2022-05-31T07:01:13Z

hudi-metastore/src/main/java/org/apache/hudi/metastore/HoodieMetastoreServer.java

+          .newProxyInstance(HoodieMetaStoreProxyHandler.class.getClassLoader(),
+              new Class[]{ThriftHoodieMetastore.Iface.class}, proxyHandler);
+      ThriftHoodieMetastore.Processor processor = new ThriftHoodieMetastore.Processor(proxy);
+      TServerTransport serverTransport = new TServerSocketWrapper(9090);


Let's make the port configurable

xiarixiaoyao · 2022-05-31T07:01:13Z

hudi-metastore/src/main/java/org/apache/hudi/metastore/HoodieMetastoreServer.java

+          .newProxyInstance(HoodieMetaStoreProxyHandler.class.getClassLoader(),
+              new Class[]{ThriftHoodieMetastore.Iface.class}, proxyHandler);
+      ThriftHoodieMetastore.Processor processor = new ThriftHoodieMetastore.Processor(proxy);
+      TServerTransport serverTransport = new TServerSocketWrapper(9090);


Let's make the port configurable

We need a. jcommander to do it, will fix later

hudi-metastore/src/main/java/org/apache/hudi/metastore/client/HoodieMetastoreClientImp.java

hudi-metastore/src/main/java/org/apache/hudi/metastore/HoodieMetastoreServer.java

xiarixiaoyao · 2022-05-31T07:40:15Z

...-metastore/src/main/java/org/apache/hudi/metastore/client/RetryingHoodieMetastoreClient.java

+    Throwable err = null;
+    do {
+      try {
+        Object res = method.invoke(client, args);


it will be better to log time cost

Should us involve metrics to do this or just log?

xiarixiaoyao · 2022-05-31T07:40:15Z

...-metastore/src/main/java/org/apache/hudi/metastore/client/RetryingHoodieMetastoreClient.java

+    Throwable err = null;
+    do {
+      try {
+        Object res = method.invoke(client, args);


it will be better to log time cost

hudi-metastore/src/main/java/org/apache/hudi/metastore/service/HoodieMetastoreService.java

xiarixiaoyao · 2022-05-31T07:44:12Z

hudi-metastore/src/main/java/org/apache/hudi/metastore/service/HoodieMetastoreService.java

+import java.nio.ByteBuffer;
+import java.util.List;
+
+public class HoodieMetastoreService implements ThriftHoodieMetastore.Iface, Serializable {


pls add doc for all new class

XuQianJin-Stars · 2022-06-07T08:07:40Z

hudi-metastore/src/main/resources/mybatis/DDLMapper.xml

+            location_uri VARCHAR(512) COMMENT 'database storage path',
+            name VARCHAR(512) UNIQUE COMMENT 'database name',
+            owner_name VARCHAR(512) COMMENT 'database owner' ,
+            owner_type VARCHAR(512) COMMENT 'database owner'


All tables here are added with create_time and update_time for easy problem analysis

XuQianJin-Stars · 2022-06-07T08:54:08Z

...src/test/java/org/apache/hudi/table/action/commit/TestCopyOnWriteActionBasedOnMetastore.java

+
+  @Override
+  protected void initMetaClient(HoodieTableType tableType, Properties properties) throws IOException {
+    properties.put(HoodieTableConfig.NAME.key(), "test");


Redundant, repeat with initMetastoreConfig below.

…ault

minihippo · 2023-01-09T03:33:25Z

@hudi-bot run azure

xushiyan

lgtm overall. suggest some minor changes. pls follow up the comments in separate PRs

xushiyan · 2023-01-16T04:08:21Z

...etaserver-client/src/main/java/org/apache/hudi/common/table/HoodieTableMetaserverClient.java

+        }
+        LOG.info(String.format("Table %s.%s doesn't exist, will create it.", db, tb));
+        table = new Table();
+        table.setDbName(db);


setDatabaseName()

xushiyan · 2023-01-16T04:42:10Z

...aserver-server/src/main/java/org/apache/hudi/metaserver/service/HoodieMetaserverService.java

+/**
+ * A proxy for meta server, accepts all thrift calls and routes them to the corresponding service.
+ */
+public class HoodieMetaserverService implements ThriftHoodieMetaserver.Iface, Serializable {


MetaserverGateway sounds better

xushiyan · 2023-01-16T04:46:19Z

...taserver-server/src/main/java/org/apache/hudi/metaserver/store/RelationalDBBasedStorage.java

+    // todo: support SSS
+    SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss");


should actually reuse the format from the existing constant. And make millisecond first supported.

xushiyan · 2023-01-16T04:48:07Z

...taserver-server/src/main/java/org/apache/hudi/metaserver/store/RelationalDBBasedStorage.java

+  }
+
+  @Override
+  public byte[] getInstantMetadata(long tableId, THoodieInstant instant) throws MetaserverStorageException {


the other interface is using Option<byte[]>. we should align them for consistency

xushiyan · 2023-01-16T04:49:04Z

...er/hudi-metaserver-server/src/main/java/org/apache/hudi/metaserver/store/bean/TableBean.java

+ * Table entity for store.
+ */
+public class TableBean {
+  private String dbName;


use databaseName; pls review all occurrences

xushiyan · 2023-01-16T04:51:24Z

...metaserver-server/src/main/java/org/apache/hudi/metaserver/store/jdbc/BatchDaoOperation.java

+  }
+
+  public BatchDaoOperation(String sqlID, Object parameter, String operationType) {
+    this(null, sqlID, parameter, operationType);


namespace is nullable? let's avoid passing null around by making Option namespace

delete the useless code

xushiyan · 2023-01-16T04:52:49Z

hudi-platform-service/hudi-metaserver/src/main/thrift/hudi-metaserver.thrift

+ // table related
+ struct Table {
+   1: string tableName,
+   2: string dbName,


databaseName

comments addressed and planned for follow up changes

xushiyan · 2023-01-16T05:44:28Z

CI passed

Co-authored-by: gengxiaoyu <gengxiaoyu@bytedance.com> Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>

minihippo changed the title ~~[HUDI-3654] Initialize hudi metastore module.~~ [WIP][HUDI-3654] Initialize hudi metastore module. Mar 18, 2022

minihippo changed the title ~~[WIP][HUDI-3654] Initialize hudi metastore module.~~ [HUDI-3654] Initialize hudi metastore module. Apr 1, 2022

nsivabalan assigned nsivabalan and xiarixiaoyao Apr 15, 2022

minihippo force-pushed the rfc-36-pr branch from e068817 to 2aa700c Compare May 13, 2022 02:43

minihippo mentioned this pull request May 13, 2022

[HUDI-3654] Preparations for hudi metastore. #5572

Merged

5 tasks

minihippo force-pushed the rfc-36-pr branch 3 times, most recently from 2dfad79 to 51c78ac Compare May 18, 2022 14:45

XuQianJin-Stars requested review from xushiyan, nsivabalan, XuQianJin-Stars and xiarixiaoyao May 19, 2022 01:47

xushiyan assigned vinothchandar May 19, 2022

xushiyan added big-needle-movers rfc and removed rfc labels May 19, 2022

xiarixiaoyao reviewed May 22, 2022

View reviewed changes

hudi-metastore/src/main/java/org/apache/hudi/common/table/HoodieTableMetastoreClient.java Outdated Show resolved Hide resolved

xiarixiaoyao reviewed May 22, 2022

View reviewed changes

hudi-metastore/src/main/java/org/apache/hudi/common/table/HoodieTableMetastoreClient.java Outdated Show resolved Hide resolved

xiarixiaoyao reviewed May 22, 2022

View reviewed changes

...astore/src/main/java/org/apache/hudi/common/table/timeline/HoodieMetastoreBasedTimeline.java Outdated Show resolved Hide resolved

xushiyan assigned bvaradar May 25, 2022

xiarixiaoyao requested changes May 31, 2022

View reviewed changes

XuQianJin-Stars reviewed Jun 7, 2022

View reviewed changes

minihippo force-pushed the rfc-36-pr branch from 53aa21b to f36e898 Compare June 7, 2022 09:17

minihippo and others added 18 commits January 6, 2023 02:50

[HUDI-3654] Initialize hudi metastore module.

748091a

add thrift compile files to pass CI.

b89ced3

fix comments

77b2b26

revert thrift source code

0b88b63

add docker to compile thrift

e08231f

revert spark metaserver ut

c4298ee

add metaserver profile that will not compile metaserver module as def…

c894419

…ault

rename metaServer to metaserver

6a317f6

fix comments

92286a1

config CI

d09537a

code cleanups and fixes

0eed6a6

update pom configs for nested modules

6501518

modify thrift

48b2771

split module into client and server

634283d

fix comments

c6b2553

fix comments

871f2a2

not compile the service module except using profile.

e5bcec0

fix comments

7280a57

minihippo force-pushed the rfc-36-pr branch from dc4b3dd to 7280a57 Compare January 6, 2023 07:05

gengxiaoyu added 3 commits January 9, 2023 23:27

bytebuffer & timestamp

140a564

fix code style

60e2505

bugfix

e6ad52e

apache deleted a comment from hudi-bot Jan 16, 2023

xushiyan approved these changes Jan 16, 2023

View reviewed changes

xushiyan merged commit 16d33ba into apache:master Jan 16, 2023

fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Jan 31, 2023

[HUDI-3654] Add new module hudi-metaserver (apache#5064)

d1353d4

Co-authored-by: gengxiaoyu <gengxiaoyu@bytedance.com> Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>

fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023

[HUDI-3654] Add new module hudi-metaserver (apache#5064)

1a1eef5

Co-authored-by: gengxiaoyu <gengxiaoyu@bytedance.com> Co-authored-by: Raymond Xu <2701446+xushiyan@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-3654] Add new module `hudi-metaserver` #5064

[HUDI-3654] Add new module `hudi-metaserver` #5064

minihippo commented Mar 18, 2022 •

edited by xushiyan

Loading

melin commented Mar 18, 2022

xiarixiaoyao commented Mar 18, 2022

nsivabalan commented Apr 15, 2022

xiarixiaoyao commented Apr 16, 2022

minihippo commented May 18, 2022

minihippo commented May 30, 2022

xiarixiaoyao May 31, 2022

xiarixiaoyao May 31, 2022

minihippo Jun 7, 2022

xiarixiaoyao May 31, 2022

minihippo Jun 7, 2022

xiarixiaoyao May 31, 2022

xiarixiaoyao May 31, 2022

XuQianJin-Stars Jun 7, 2022 •

edited

Loading

minihippo Jun 13, 2022

XuQianJin-Stars Jun 7, 2022

minihippo Jun 13, 2022

vinothchandar Jul 13, 2022

minihippo commented Jan 9, 2023

xushiyan left a comment

xushiyan Jan 16, 2023

minihippo Jan 20, 2023

xushiyan Jan 16, 2023

minihippo Jan 20, 2023

xushiyan Jan 16, 2023

minihippo Jan 21, 2023

xushiyan Jan 16, 2023

minihippo Jan 20, 2023

xushiyan Jan 16, 2023

minihippo Jan 20, 2023

xushiyan Jan 16, 2023

minihippo Jan 20, 2023

xushiyan Jan 16, 2023

xushiyan commented Jan 16, 2023

		// todo: support SSS
		SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss");

[HUDI-3654] Add new module hudi-metaserver #5064

[HUDI-3654] Add new module hudi-metaserver #5064

Conversation

minihippo commented Mar 18, 2022 • edited by xushiyan Loading

Change Logs

Impact

Risk level

Documentation Update

Contributor's checklist

melin commented Mar 18, 2022

xiarixiaoyao commented Mar 18, 2022

nsivabalan commented Apr 15, 2022

xiarixiaoyao commented Apr 16, 2022

minihippo commented May 18, 2022

minihippo commented May 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuQianJin-Stars Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minihippo commented Jan 9, 2023

xushiyan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xushiyan commented Jan 16, 2023

[HUDI-3654] Add new module `hudi-metaserver` #5064

[HUDI-3654] Add new module `hudi-metaserver` #5064

minihippo commented Mar 18, 2022 •

edited by xushiyan

Loading

XuQianJin-Stars Jun 7, 2022 •

edited

Loading