Make `CREATE EXTERNAL TABLE` format options consistent, remove special syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` #10404

berkaysynnada · 2024-05-07T11:02:20Z

Which issue does this PR close?

Rationale for this change

datafusion/datafusion/core/src/datasource/listing_table_factory.rs

Lines 67 to 73 in 4bd7c13

    
           let file_format: Arc<dyn FileFormat> = match file_type { 
        
               FileType::CSV => { 
        
                   let mut csv_options = table_options.csv; 
        
                   csv_options.has_header = cmd.has_header; 
        
                   csv_options.delimiter = cmd.delimiter as u8; 
        
                   csv_options.compression = cmd.file_compression_type; 
        
                   Arc::new(CsvFormat::default().with_options(csv_options))

These lines of ListingTableFactory.create() overwrites the config options by cmd (CreateExternalTable options). Configs coming from OPTIONS('format... are discarded silently.

statement ok
CREATE EXTERNAL TABLE aggregate_simple (
  c1 FLOAT NOT NULL,
  c2 DOUBLE NOT NULL,
  c3 BOOLEAN NOT NULL
)
STORED AS CSV
LOCATION '../core/tests/data/aggregate_simple.csv'
WITH HEADER ROW

query RRB
SELECT * FROM aggregate_simple LIMIT 3;
----

works as expected, but

statement ok
CREATE EXTERNAL TABLE aggregate_simple (
  c1 FLOAT NOT NULL,
  c2 DOUBLE NOT NULL,
  c3 BOOLEAN NOT NULL
)
STORED AS CSV
LOCATION '../core/tests/data/aggregate_simple.csv'
OPTIONS('format.has_header' 'true')

query RRB
SELECT * FROM aggregate_simple LIMIT 3;
----

does not, since CREATE EXTERNAL TABLE does not have WITH HEADER ROW, which overwrites csv.has_header to false.

To summarize:
CreateExternalTable has 3 format specific fields:

has_header
delimiter
file_compression_type
After Systematic Configuration in 'Create External Table' and 'Copy To' Options #9382, CreateExternalTable.options does store these format specific configurations.

What changes are included in this PR?

CreateExternalTable.delimiter and CreateExternalTable.file_compression_type are removed, and they are needed to be given from options, such as

OPTIONS(
    'format.delimiter' ',',
    'format.compression' 'gzip')

~~For the has_header config, as @alamb #9945 (comment) suggests, we can sustain its usage as WITH HEADER ROW from CreateExternalTable's scope. I refactor the code such that~~

(WITH HEADER ROW) + (format.has_header = true) => OK with header
(WITH HEADER ROW) + (format.has_header = false) => Error
(WITH HEADER ROW) + (format.has_header = not specified) => OK with header
() + (format.has_header = true) => OK with header
() + (format.has_header = false) => Ok without header
() + (format.has_header = not_specified) => Ok without header

~~However in the long term, I think removing also WITH HEADER ROW option will be a better design and help having more clear UX.~~

WITH HEADER ROW is also removed to OPTIONS tab.

Are these changes tested?

Yes.

Are there any user-facing changes?

Users access format specific configurations (delimiter and compression type) via OPTIONS() command.

ozankabak

Thank you for this clean-up PR @berkaysynnada. This almost closes the loop on getting us to a consistent API w.r.t. I/O option handling (e.g. format options). With this PR, the only "weird" thing remaining in the API will be the WITH HEADER ROW top-level specifier (which it shouldn't be, as it is a format-specific option (CSV)).

We organized this PR to keep WITH HEADER ROW and gave it consistent semantics w.r.t. its interaction with format options. Maybe can add a deprecation warning in a follow-on PR and remove it entirely in the future.

@alamb, it'd be great if you can take a look as we discussed this before. Thanks

alamb

Thank you @berkaysynnada and @ozankabak

This PR looks great to me other than removing support for DELIMTER and COMRESSION in the CREATE EXTERNAL TABLE syntax (and I almost missed that change when skimming this PR initially)

What I suggest is:

Update this PR to just fix the has_header bug in Overwritten Format Configs by CreateExternalTable Options #9945
Start a new discussion / ticket about removing / deprecating the WITH HEADER ROW / COMPRESSION, DELIMITER syntax support.

However in the long term, I think removing also WITH HEADER ROW option will be a better design and help having more clear UX.

I don't really have a strong preference myself, but given how long the other syntax has been around I think other people might have

datafusion/common/src/config.rs

datafusion/core/src/datasource/file_format/csv.rs

datafusion/core/src/datasource/listing_table_factory.rs

datafusion/sql/src/parser.rs

ozankabak · 2024-05-07T19:17:08Z

Thanks for taking a look. Do you suggest having a similar mechanism for DELIMITER and COMPRESSION like we did for WITH HEADER ROW? We can do that, but it will increase code complexity and we will need maintain that code until we finally remove them to arrive at the consistent syntax we aim.

I fear the longer we support the current syntax, the harder it will be to fix it. We may end up being stuck with CSV-related options living at the top level.

Start a new discussion / ticket about removing / deprecating the WITH HEADER ROW / COMPRESSION, DELIMITER syntax support.

Makes sense, let's do this

alamb · 2024-05-07T19:33:19Z

Thanks for taking a look. Do you suggest having a similar mechanism for DELIMITER and COMPRESSION like we did for WITH HEADER ROW?

Yes, that is my suggestion

We can do that, but it will increase code complexity and we will need maintain that code until we finally remove them to arrive at the consistent syntax we aim.

I agree it will make the code more complex

I fear the longer we support the current syntax, the harder it will be to fix it. We may end up being stuck with CSV-related options living at the top level.

I agree -- I will file a follow on ticket discuss removing the options

alamb · 2024-05-07T19:43:10Z

I agree -- I will file a follow on ticket discuss removing the options

Filed #10414 to discus removing these options

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…i/datafusion-upstream into feature/format-options

This reverts commit 9d81601.

berkaysynnada · 2024-05-09T14:57:43Z

datafusion/sqllogictest/test_files/parquet.slt

@@ -66,7 +66,6 @@ CREATE EXTERNAL TABLE test_table (
  date_col DATE
 )
 STORED AS PARQUET
-WITH HEADER ROW


No need to specify header rows for parquet

berkaysynnada · 2024-05-09T15:04:53Z

This PR is now ready for review. By employing that method, users can choose to maintain the existing behavior if desired.

berkaysynnada · 2024-05-09T15:10:02Z

datafusion/sqllogictest/test_files/create_external_table.slt

@@ -68,18 +60,6 @@ CREATE EXTERNAL TABLE t STORED AS CSV STORED AS PARQUET LOCATION 'foo.parquet'
 statement error DataFusion error: SQL error: ParserError\("LOCATION specified more than once"\)
 CREATE EXTERNAL TABLE t STORED AS CSV LOCATION 'foo.csv' LOCATION 'bar.csv'

-# Duplicate `WITH HEADER ROW` clause


Not applicable anymore

berkaysynnada · 2024-05-09T15:10:09Z

datafusion/sqllogictest/test_files/create_external_table.slt

@@ -52,14 +52,6 @@ CREATE EXTERNAL TABLE t STORED AS CSV WITH HEADER LOCATION 'abc'
 statement error DataFusion error: SQL error: ParserError\("Expected BY, found: LOCATION"\)
 CREATE EXTERNAL TABLE t STORED AS CSV PARTITIONED LOCATION 'abc'

-# Missing `TYPE` in COMPRESSION clause


Not applicable anymore

berkaysynnada · 2024-05-09T15:10:36Z

datafusion/sql/src/parser.rs

            constraints: vec![],
        });
        expect_parse_ok(sql, expected)?;

-        // positive case: it is ok for case insensitive sql stmt with `WITH HEADER ROW` tokens


Not applicable

ozankabak

I did one more round of review and it looks good to me. Seems like @berkaysynnada threw in a minor improvement of being able to use non-quoted strings in the OPTIONS clause as well, which is great. Creating warnings when WITH HEADER ROW is used and directing users to the OPTIONS clause is also great. Great work @berkaysynnada 🚀

@alamb, feel free to merge once you take a look and all looks good.

alamb · 2024-05-10T14:06:37Z

I plan to review this again later today. Thanks @berkaysynnada and @ozankabak

alamb

Thank you @berkaysynnada and @ozankabak -- i looked over this PR and I think the code looks really nice 👏

From my perspective this PR also now implements the consensus from #10414 and everyone seems happy with the overall increase in consistency (and there is a workaround for anyone who needs the old special syntax -- kudos to @phillipleblanc 🙏 ).

Something else I noticed while reviewing this PR is that the available options (e.g. format.has_header, format.delimiter, etc do not appear to be documented anywhere -- I filed #10451 to track that

docs/source/user-guide/sql/ddl.md

datafusion/core/src/datasource/file_format/csv.rs

alamb · 2024-05-10T19:49:01Z

datafusion/core/src/datasource/stream.rs

+        {
+            opt.unwrap_or(false)
+        } else {
+            return config_err!("format.has_header can either be true or false");


I am not quite sure what this error message means. Does it mean that there are inconsistent settings somehow? Perhaps we could improve the text to give some hint about how the user can fix whatever the problem is 🤔

An error occurs here if the user would set such an option: ('format.has_header' 'x') where x is anything other than 'true' or 'false'. I'd like to emphasize that. But as you said, the message may direct the user as if a conflicting options are set. Thanks for the feedback, I am updating it now.

datafusion/sql/src/statement.rs

datafusion/sql/src/parser.rs

datafusion/sql/src/statement.rs

docs/source/user-guide/cli/datasources.md

alamb · 2024-05-10T19:56:01Z

I also updated this PR's title to reflect the user visible changes and added the api-change label

ozankabak · 2024-05-11T16:09:51Z

Thank you. We will address your review feedback and then merge afterwards. 🚀

alamb · 2024-05-13T10:38:54Z

Thanks again for this work @berkaysynnada and the guidance @ozankabak

Blizzara · 2024-05-31T13:29:35Z

Hey, this changed the behavior of writing and reading CSVs to be slightly inconsistent: now writing defaults to not writing a header, while reading defaults to having a header. Previously both defaulted to having a header, ie. the write side has flipped.

I noticed as our test started failing - a simplified version below, the following worked previously, after this commit it fails:

#[tokio::test]
async fn write_then_read_csv() -> Result<()> {
    let test_path = test_file_path("csv/");

    let ctx = SessionContext::new();

    let df = ctx.sql("SELECT 1 AS col").await?;
    assert_eq!(df.to_owned().count().await?, 1);
    df.write_csv(&test_path, DataFrameWriteOptions::default(), None).await?;

    let df = ctx.read_csv(&test_path, CsvReadOptions::default()).await?;
    assert_eq!(df.count().await?, 1);

    Ok(())
}

but can be fixed by adding Some(CsvOptions::default().with_has_header(true)) to the write_csv (restoring old behavior), or alternatively changing read_csv to have CsvReadOptions::default().has_header(false) to make it match the write.

Doesn't really matter to me how it works, though I find the mismatch somewhat surprising - but mainly I just wanted to flag this in case this was not expected/intentional change.

ozankabak · 2024-05-31T13:49:13Z

Hmm, this shouldn't be the case. The logic is supposed to be the following for both reads and writes:

Use the explicitly specified header flag, if it exists.
Otherwise, consult the session config and use the default value there.

If that is not so, we should open and issue and fix. Thanks for letting us know!

berkaysynnada · 2024-05-31T13:49:39Z

Doesn't really matter to me how it works, though I find the mismatch somewhat surprising - but mainly I just wanted to flag this in case this was not expected/intentional change.

Thanks for reporting that. It's surprising that this issue hasn't emerged before. CatalogOptions's has_header value was false as default from the beginning, which is the write options in that case. However, reader's is true:

datafusion/datafusion/core/src/datasource/file_format/options.rs

Line 90 in 09dde27

has_header: true,

I believe we should also set it as false while creating it.

…l syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` (apache#10404) * Simplify format options * Keep PG copy from tests same * Update datafusion/common/src/config.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/core/src/datasource/file_format/csv.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Remove WITH HEADER ROW * Review Part 1 * . * Fix failing tests * Revert "Fix failing tests" This reverts commit 9d81601. * Final commit * Minor * Review * Update avro.slt * Apply suggestions * Fix imports --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>

Simplify format options

ec38761

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels May 7, 2024

Keep PG copy from tests same

d1611a9

ozankabak approved these changes May 7, 2024

View reviewed changes

alamb mentioned this pull request May 7, 2024

DataFusion weekly project plan (Andrew Lamb) - May 6, 2024 #10395

Closed

7 tasks

alamb reviewed May 7, 2024

View reviewed changes

alamb mentioned this pull request May 7, 2024

DISCUSSION: remove CREATE EXTERNAL TABLE syntax: DELIMITER, WITH HEADER ROW and COMPRESSION #10414

Closed

berkaysynnada and others added 8 commits May 8, 2024 09:47

Update datafusion/common/src/config.rs

99fae81

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Update datafusion/core/src/datasource/file_format/csv.rs

13c4db6

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Merge branch 'apache_main' into feature/format-options

7b86b11

Remove WITH HEADER ROW

c613a33

Review Part 1

4873d61

.

f0f6f09

Merge branch 'feature/format-options' of https://github.com/synnada-a…

d64de7e

…i/datafusion-upstream into feature/format-options

Fix failing tests

9d81601

berkaysynnada force-pushed the feature/format-options branch from d751320 to 9d81601 Compare May 9, 2024 12:13

berkaysynnada added 2 commits May 9, 2024 15:33

Revert "Fix failing tests"

bb61cc1

This reverts commit 9d81601.

Final commit

43be99b

berkaysynnada commented May 9, 2024

View reviewed changes

Minor

fe505a2

berkaysynnada force-pushed the feature/format-options branch from fec33b7 to fe505a2 Compare May 9, 2024 15:04

berkaysynnada commented May 9, 2024

View reviewed changes

Review

8ab014c

ozankabak approved these changes May 10, 2024

View reviewed changes

berkaysynnada added 2 commits May 10, 2024 09:27

Merge branch 'apache_main' into feature/format-options

9a5e44b

Update avro.slt

86e473e

alamb changed the title ~~Simplify Format Options~~ Make CREATE EXTERNAL TABLE format options consistent, remove special syntax for HEADER ROW, DELIMITER and COMPRESSION May 10, 2024

alamb added the api change Changes the API exposed to users of the crate label May 10, 2024

alamb mentioned this pull request May 10, 2024

Minor: Improve documentation for catalog.has_header config option #10452

Merged

alamb approved these changes May 10, 2024

View reviewed changes

berkaysynnada added 3 commits May 13, 2024 10:26

Apply suggestions

3af0bfe

Merge branch 'apache_main' into feature/format-options

5d232e6

Fix imports

a817dc8

berkaysynnada force-pushed the feature/format-options branch from 08237ea to a817dc8 Compare May 13, 2024 08:30

alamb merged commit 58cc4e1 into apache:main May 13, 2024
25 checks passed

alamb mentioned this pull request May 13, 2024

DataFusion weekly project plan (Andrew Lamb) - May 13, 2024 #10482

Closed

8 tasks

ozankabak mentioned this pull request May 13, 2024

Improved ergonomy for CREATE EXTERNAL TABLE OPTIONS: Don't require quotations for simple namespaced keys like foo.bar #10483

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `CREATE EXTERNAL TABLE` format options consistent, remove special syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` #10404

Make `CREATE EXTERNAL TABLE` format options consistent, remove special syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` #10404

berkaysynnada commented May 7, 2024 •

edited by alamb

Loading

ozankabak left a comment •

edited

Loading

alamb left a comment

ozankabak commented May 7, 2024

alamb commented May 7, 2024

alamb commented May 7, 2024

berkaysynnada May 9, 2024

berkaysynnada commented May 9, 2024

berkaysynnada May 9, 2024

berkaysynnada May 9, 2024

berkaysynnada May 9, 2024

ozankabak left a comment

alamb commented May 10, 2024

alamb left a comment

alamb May 10, 2024

berkaysynnada May 13, 2024

alamb commented May 10, 2024

ozankabak commented May 11, 2024

alamb commented May 13, 2024

Blizzara commented May 31, 2024

ozankabak commented May 31, 2024 •

edited

Loading

berkaysynnada commented May 31, 2024

	let file_format: Arc<dyn FileFormat> = match file_type {
	FileType::CSV => {
	let mut csv_options = table_options.csv;
	csv_options.has_header = cmd.has_header;
	csv_options.delimiter = cmd.delimiter as u8;
	csv_options.compression = cmd.file_compression_type;
	Arc::new(CsvFormat::default().with_options(csv_options))

Make CREATE EXTERNAL TABLE format options consistent, remove special syntax for HEADER ROW, DELIMITER and COMPRESSION #10404

Make CREATE EXTERNAL TABLE format options consistent, remove special syntax for HEADER ROW, DELIMITER and COMPRESSION #10404

Conversation

berkaysynnada commented May 7, 2024 • edited by alamb Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

ozankabak left a comment • edited Loading

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

ozankabak commented May 7, 2024

alamb commented May 7, 2024

alamb commented May 7, 2024

berkaysynnada May 9, 2024

Choose a reason for hiding this comment

berkaysynnada commented May 9, 2024

berkaysynnada May 9, 2024

Choose a reason for hiding this comment

berkaysynnada May 9, 2024

Choose a reason for hiding this comment

berkaysynnada May 9, 2024

Choose a reason for hiding this comment

ozankabak left a comment

Choose a reason for hiding this comment

alamb commented May 10, 2024

alamb left a comment

Choose a reason for hiding this comment

alamb May 10, 2024

Choose a reason for hiding this comment

berkaysynnada May 13, 2024

Choose a reason for hiding this comment

alamb commented May 10, 2024

ozankabak commented May 11, 2024

alamb commented May 13, 2024

Blizzara commented May 31, 2024

ozankabak commented May 31, 2024 • edited Loading

berkaysynnada commented May 31, 2024

Make `CREATE EXTERNAL TABLE` format options consistent, remove special syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` #10404

Make `CREATE EXTERNAL TABLE` format options consistent, remove special syntax for `HEADER ROW`, `DELIMITER` and `COMPRESSION` #10404

berkaysynnada commented May 7, 2024 •

edited by alamb

Loading

ozankabak left a comment •

edited

Loading

ozankabak commented May 31, 2024 •

edited

Loading