-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Defer file creation to write #8539
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -57,7 +57,7 @@ CREATE EXTERNAL TABLE dictionary_encoded_parquet_partitioned( | |
b varchar, | ||
) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/parquet_types_partitioned' | ||
LOCATION 'test_files/scratch/insert_to_external/parquet_types_partitioned/' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is necessary because now that the directories no longer exist it falls back to using the trailing / to determining if a directory or file is desired. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now that inserting (appending) to an individual file is no longer a thing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, this is definitely the eventual goal, is this something you would be interested in working on? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The usecase of writing to I think that would still be achievable with the proposal above, I just wanted to point it out There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, COPY has a SINGLE_FILE_OUTPUT option that gates this, but for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ticket for this #8548 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm happy to take a stab at removing the single file table option next week, when I also plan to look at adding support for writing to arrow files. |
||
PARTITIONED BY (b) | ||
OPTIONS( | ||
create_local_path 'true', | ||
|
@@ -292,7 +292,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
directory_test(a bigint, b bigint) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q0' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q0/' | ||
OPTIONS( | ||
create_local_path 'true', | ||
); | ||
|
@@ -312,7 +312,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
table_without_values(field1 BIGINT NULL, field2 BIGINT NULL) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q1' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q1/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
query TT | ||
|
@@ -378,7 +378,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
table_without_values(field1 BIGINT NULL, field2 BIGINT NULL) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q2' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q2/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
query TT | ||
|
@@ -423,7 +423,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
table_without_values(c1 varchar NULL) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q3' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q3/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
# verify that the sort order of the insert query is maintained into the | ||
|
@@ -462,7 +462,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
table_without_values(id BIGINT, name varchar) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q4' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q4/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
query IT | ||
|
@@ -505,7 +505,7 @@ statement ok | |
CREATE EXTERNAL TABLE | ||
table_without_values(field1 BIGINT NOT NULL, field2 BIGINT NULL) | ||
STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q5' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q5/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
query II | ||
|
@@ -555,7 +555,7 @@ CREATE EXTERNAL TABLE test_column_defaults( | |
d text default lower('DEFAULT_TEXT'), | ||
e timestamp default now() | ||
) STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q6' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q6/' | ||
OPTIONS (create_local_path 'true'); | ||
|
||
# fill in all column values | ||
|
@@ -608,5 +608,5 @@ CREATE EXTERNAL TABLE test_column_defaults( | |
a int, | ||
b int default a+1 | ||
) STORED AS parquet | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q7' | ||
LOCATION 'test_files/scratch/insert_to_external/external_parquet_table_q7/' | ||
OPTIONS (create_local_path 'true'); |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -78,11 +78,9 @@ The following special options are specific to the `COPY` command. | |
|
||
The following special options are specific to creating an external table. | ||
|
||
| Option | Description | Default Value | | ||
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | ||
| SINGLE_FILE | If true, indicates that this external table is backed by a single file. INSERT INTO queries will append to this file. | false | | ||
| CREATE_LOCAL_PATH | If true, the folder or file backing this table will be created on the local file system if it does not already exist when running INSERT INTO queries. | false | | ||
| INSERT_MODE | Determines if INSERT INTO queries should append to existing files or append new files to an existing directory. Valid values are append_to_file, append_new_files, and error. Note that "error" will block inserting data into this table. | CSV and JSON default to append_to_file. Parquet defaults to append_new_files | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The only valid INSERT_MODE is append_new_files so we might as well remove this from the documentation |
||
| Option | Description | Default Value | | ||
| ----------- | --------------------------------------------------------------------------------------------------------------------- | ------------- | | ||
| SINGLE_FILE | If true, indicates that this external table is backed by a single file. INSERT INTO queries will append to this file. | false | | ||
|
||
### JSON Format Specific Options | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we please leave an comment here explaining why this is being ignored? Maybe even with a ticket reference to a ticket tracking removing it?
I can file such a ticket if it would be helpful
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created #8547