Add Alluxio file system #21603

JiamingMai · 2024-04-18T07:46:16Z

Description

Add trino-filesystem-alluxio module to implement the new interfaces (TrinoFileSystem, TrinoInput, TrinoInputFile, TrinoOutputFile ) and support native Alluxio.

Additional context and related issues

Docs in #23269

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

cla-bot · 2024-04-22T19:49:09Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-23T13:09:19Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-23T13:34:46Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-23T17:01:06Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-23T23:39:44Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T07:10:20Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T07:22:12Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T07:24:48Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T08:04:29Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T08:35:32Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T08:58:54Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-04-24T08:59:26Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

lib/trino-filesystem-alluxio/pom.xml

lib/trino-filesystem-cache-alluxio/pom.xml

pom.xml

jja725 · 2024-09-25T22:33:44Z

@electrum All comments resolved, I can rebase the commits if everything looks good.

raunaqmorarka

Please squash commits and rebase to latest master.
Also fix the failure in check-commit

jja725 · 2024-09-26T20:32:15Z

@mosabua I believe the test failure(trino-google-sheets) is not related to the change. Do you mind helping me rerun the test ?

mosabua · 2024-09-26T21:18:16Z

No need to rerun the build ... the google sheets failure is definitely a false alarm.

wendigo · 2024-09-26T21:39:46Z

@mosabua yeah, it's super flaky recently

wendigo · 2024-09-27T15:36:55Z

Thanks @JiamingMai for a great contribution!

mosabua · 2024-09-27T15:56:36Z

Docs are in as well now

ebyhr · 2024-09-28T04:14:32Z

...esystem-alluxio/src/test/java/io/trino/filesystem/alluxio/AbstractTestAlluxioFileSystem.java

+import static org.assertj.core.api.Assertions.assertThat;
+
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
+public abstract class AbstractTestAlluxioFileSystem


Why do we need an abstract class? Can we merge into TestAlluxioFileSystem?

ebyhr · 2024-09-28T04:19:07Z

lib/trino-filesystem-alluxio/src/test/java/io/trino/filesystem/alluxio/TestAlluxioUtils.java

+    @Test
+    public void testSimplifyPath()
+    {
+        String path = "test/level0-file0";


Inline this variable.

ebyhr · 2024-09-28T04:20:21Z

lib/trino-filesystem-alluxio/src/test/java/io/trino/filesystem/alluxio/TestAlluxioUtils.java

+        String path = "test/level0-file0";
+        assertThat(path).isEqualTo(AlluxioUtils.simplifyPath(path));
+        path = "a/./b/../../c/";
+        assertThat("c/").isEqualTo(AlluxioUtils.simplifyPath(path));


The call order is wrong. It should be

assertThat(AlluxioUtils.simplifyPath("a/./b/../../c/")).isEqualTo("c/");

Same for others.

raunaqmorarka · 2024-09-28T04:59:47Z

Looks like this PR was merged with red CI. https://github.com/trinodb/trino/actions/runs/11057611394/job/30721797373

The only failing test yesterday was the google sheets one, which is unrelated
https://github.com/trinodb/trino/actions/runs/11058479149/job/30724603118

ebyhr · 2024-09-28T05:47:32Z

...rino-filesystem-alluxio/src/test/java/io/trino/filesystem/alluxio/TestAlluxioFileSystem.java

+    void testContainer()
+    {
+        assertThat(ALLUXIO_MASTER_CONTAINER.isRunning()).isTrue();
+        assertThat(ALLUXIO_WORKER_CONTAINER.isRunning()).isTrue();


What's the motivation to verify containers' status here?

We can remove it in a later PR. it's mostly for verify the docker status during the test

ebyhr · 2024-09-28T05:48:50Z

lib/trino-filesystem-alluxio/src/test/java/io/trino/filesystem/alluxio/TestAlluxioUtils.java

+    }
+
+    @Test
+    public void convertToLocation()


test prefix is missing.

Please follow https://trino.io/docs/current/develop/tests.html

Test classes should be defined as package-private and final.

Test method must start with test, for example testExplain()

Test methods should be defined as package-private.

Thanks for fixing this

cla-bot bot added the cla-signed label Apr 18, 2024

github-actions bot added the tests:hive label Apr 18, 2024

JiamingMai changed the title ~~Add alluxio file system~~ Add Alluxio filesystem Apr 18, 2024

JiamingMai force-pushed the add-alluxio-file-system branch from 1aecc4c to 7e84237 Compare April 18, 2024 16:50

findepi removed the tests:hive label Apr 18, 2024

cla-bot bot removed the cla-signed label Apr 22, 2024

mosabua requested a review from electrum April 22, 2024 20:43

JiamingMai self-assigned this Apr 24, 2024

JiamingMai force-pushed the add-alluxio-file-system branch from b724c3d to ae34ab1 Compare April 24, 2024 08:59

JiamingMai force-pushed the add-alluxio-file-system branch from ae34ab1 to 1e86c3c Compare April 24, 2024 09:32

cla-bot bot added the cla-signed label Apr 24, 2024

github-actions bot added docs hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector bigquery BigQuery connector mongodb MongoDB connector labels Apr 24, 2024