Add support for `CREATE OR REPLACE TABLE` statement #13681

mdesmet · 2022-08-15T20:33:38Z

Support CREATE OR REPLACE syntax in engine and Iceberg connector.

Description

Fixes #13180

Is this change a fix, improvement, new feature, refactoring, or other?

new Feature

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Change to core query engine, SPI and connectors

How would you describe this change to a non-technical end user or system administrator?

Related issues, pull requests, and links

Fixes CREATE [OR REPLACE] TABLE support for Iceberg and Delta #13180

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

findepi

On hold until consensus whether we want the feature #13180 is reached. Let's discuss there.

wrb2 · 2022-11-08T19:03:46Z

Hi, I think this could be unblocked, no? Or am I missing something?

core/trino-spi/src/main/java/io/trino/spi/connector/ConnectorMetadata.java

core/trino-main/src/main/java/io/trino/execution/CreateTableTask.java

stale

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractTrinoCatalog.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

findepi · 2023-03-31T08:26:17Z

removed the syntax-needs-review label as it got removed from #13180.

Now that we have directional agreement, let's get this moving.
@electrum @mdesmet would it be OK if we review commits incrementally? For example, we could move first (grammar changes) commit to its own PR.

mdesmet · 2023-04-04T20:22:02Z

Now that we have directional agreement, let's get this moving. @electrum @mdesmet would it be OK if we review commits incrementally? For example, we could move first (grammar changes) commit to its own PR.

Please confirm this is the way to go. If not please start reviewing this PR.

jkylling · 2023-04-17T19:01:19Z

We are interested in support for CREATE OR REPLACE in trino-delta-lake and have started on an implementation on top of this PR here. Thank you for the very useful PR!

Praveen2112

LGTM. After applying @findinpath and @mosabua 's comment

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

testing/trino-testing/src/main/java/io/trino/testing/BaseConnectorTest.java

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java

Praveen2112 · 2023-10-16T16:18:34Z

@mosabua Can you please take a look at the docs once again.

docs/src/main/sphinx/sql/create-table-as.md

docs/src/main/sphinx/sql/create-table.md

docs/src/main/sphinx/connector/iceberg.md

findinpath · 2023-10-22T20:51:55Z

@mdesmet Build is 🔴

mosabua

Docs look good now.

findinpath · 2023-10-23T18:22:43Z

Unrelated CI failure
#19446

findinpath · 2023-10-23T18:29:05Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

+                verifyTableVersionForUpdate(table);
+                Table icebergTable = catalog.loadTable(session, table.getSchemaTableName());
+                validateNotModifyingOldSnapshot(table, icebergTable);
+                getTableMetadata(session, table);


Is this call a left-over?
Why do we need to do the call ? If there is a reasoning behind it, pls document it.

Praveen2112 · 2023-10-25T07:01:37Z

Thanks a lot @mdesmet for working on this amazing feature

findinpath · 2023-10-25T07:03:40Z

@jkylling road is open for Delta Lake PR

findepi · 2023-10-25T08:22:33Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

        Location location = Location.of(transaction.table().location());
        TrinoFileSystem fileSystem = fileSystemFactory.create(session);
        try {
-            if (fileSystem.listFiles(location).hasNext()) {
+            if (!replace && fileSystem.listFiles(location).hasNext()) {


This should do the check also for replace case, when table handle was not found.

findepi · 2024-02-19T14:11:53Z

core/trino-spi/src/main/java/io/trino/spi/connector/ConnectorMetadata.java

     */
+    @Deprecated


It is time to remove this deprecated method. Can you please follow-up?

OK. will have a look at this.

cla-bot bot added the cla-signed label Aug 15, 2022

mdesmet marked this pull request as draft August 15, 2022 20:33

ebyhr added the syntax-needs-review label Aug 15, 2022

ebyhr changed the title ~~Feature/create or replace~~ Add support for CREATE OR REPLACE statement Aug 15, 2022

mdesmet force-pushed the feature/create-or-replace branch 4 times, most recently from cb2ce4f to 6c8e05c Compare August 18, 2022 22:45

mdesmet marked this pull request as ready for review August 19, 2022 06:56

findepi changed the title ~~Add support for CREATE OR REPLACE statement~~ Add support for CREATE OR REPLACE TABLE statement Aug 23, 2022

findepi previously requested changes Aug 23, 2022

View reviewed changes

mdesmet mentioned this pull request Aug 25, 2022

CREATE [OR REPLACE] TABLE support for Iceberg and Delta #13180

Closed

electrum reviewed Feb 23, 2023

View reviewed changes

core/trino-spi/src/main/java/io/trino/spi/connector/ConnectorMetadata.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/execution/CreateTableTask.java Outdated Show resolved Hide resolved

findepi reviewed Feb 27, 2023

View reviewed changes

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

mosabua requested a review from martint March 17, 2023 17:20

mdesmet force-pushed the feature/create-or-replace branch 2 times, most recently from 856ab24 to 9958bdd Compare March 30, 2023 12:43

github-actions bot added the iceberg Iceberg connector label Mar 30, 2023

mdesmet force-pushed the feature/create-or-replace branch from 9958bdd to 0dc5e09 Compare March 31, 2023 07:15

findepi removed the syntax-needs-review label Mar 31, 2023

jkylling mentioned this pull request Apr 17, 2023

Implement CREATE OR REPLACE in trino-delta-lake mdesmet/trino#13

Closed

mdesmet self-assigned this Jul 12, 2023

mdesmet force-pushed the feature/create-or-replace branch 2 times, most recently from 9404f7c to 0a814fc Compare July 29, 2023 09:57

Praveen2112 approved these changes Oct 16, 2023

View reviewed changes

mosabua requested changes Oct 16, 2023

View reviewed changes

marcinsbd approved these changes Oct 18, 2023

View reviewed changes

mdesmet force-pushed the feature/create-or-replace branch from 2add4d0 to f72ab79 Compare October 21, 2023 12:58

Extract table properties creation for Iceberg as a separate method

b62a0cb

mdesmet force-pushed the feature/create-or-replace branch from f72ab79 to fe8ca65 Compare October 23, 2023 13:09

findinpath requested a review from Praveen2112 October 23, 2023 13:24

mdesmet added 2 commits October 23, 2023 16:52

Remove obsolete schema check

937dac1

Implement engine support for CREATE OR REPLACE TABLE

ccb01a2

mdesmet force-pushed the feature/create-or-replace branch from fe8ca65 to bec23c3 Compare October 23, 2023 14:53

mosabua approved these changes Oct 23, 2023

View reviewed changes

findinpath mentioned this pull request Oct 23, 2023

Flaky TestMinWorkerRequirement.testMultipleRequiredWorkerNodesSessionOverride #19446

Closed

findinpath reviewed Oct 23, 2023

View reviewed changes

mdesmet added 2 commits October 24, 2023 13:44

Implement CREATE OR REPLACE TABLE in trino-iceberg

cc27eaf

Add documentation for CREATE OR REPLACE

1a0eec6

mdesmet force-pushed the feature/create-or-replace branch from bec23c3 to 1a0eec6 Compare October 24, 2023 11:45

findinpath approved these changes Oct 24, 2023

View reviewed changes

Praveen2112 approved these changes Oct 25, 2023

View reviewed changes

Praveen2112 merged commit 2771775 into trinodb:master Oct 25, 2023

github-actions bot added this to the 431 milestone Oct 25, 2023

findepi reviewed Oct 25, 2023

View reviewed changes

mosabua mentioned this pull request Oct 25, 2023

Add Trino 431 release notes #19497

Merged

findepi reviewed Feb 19, 2024

View reviewed changes

This was referenced Apr 4, 2024

Implement CREATE OR REPLACE TABLE for delta lake connector #19991

Merged

CREATE [OR REPLACE] TABLE support for Hive #21461

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for `CREATE OR REPLACE TABLE` statement #13681

Add support for `CREATE OR REPLACE TABLE` statement #13681

mdesmet commented Aug 15, 2022 •

edited

Loading

findepi left a comment

wrb2 commented Nov 8, 2022

findepi commented Mar 31, 2023

mdesmet commented Apr 4, 2023

jkylling commented Apr 17, 2023

Praveen2112 left a comment

Praveen2112 commented Oct 16, 2023

findinpath commented Oct 22, 2023 •

edited

Loading

mosabua left a comment

findinpath commented Oct 23, 2023

findinpath Oct 23, 2023

Praveen2112 commented Oct 25, 2023

findinpath commented Oct 25, 2023

findepi Oct 25, 2023

findepi Feb 19, 2024

mdesmet Feb 20, 2024

Add support for CREATE OR REPLACE TABLE statement #13681

Add support for CREATE OR REPLACE TABLE statement #13681

Conversation

mdesmet commented Aug 15, 2022 • edited Loading

Description

Related issues, pull requests, and links

Documentation

Release notes

findepi left a comment

Choose a reason for hiding this comment

wrb2 commented Nov 8, 2022

findepi commented Mar 31, 2023

mdesmet commented Apr 4, 2023

jkylling commented Apr 17, 2023

Praveen2112 left a comment

Choose a reason for hiding this comment

Praveen2112 commented Oct 16, 2023

findinpath commented Oct 22, 2023 • edited Loading

mosabua left a comment

Choose a reason for hiding this comment

findinpath commented Oct 23, 2023

findinpath Oct 23, 2023

Choose a reason for hiding this comment

Praveen2112 commented Oct 25, 2023

findinpath commented Oct 25, 2023

findepi Oct 25, 2023

Choose a reason for hiding this comment

findepi Feb 19, 2024

Choose a reason for hiding this comment

mdesmet Feb 20, 2024

Choose a reason for hiding this comment

Add support for `CREATE OR REPLACE TABLE` statement #13681

Add support for `CREATE OR REPLACE TABLE` statement #13681

mdesmet commented Aug 15, 2022 •

edited

Loading

findinpath commented Oct 22, 2023 •

edited

Loading