Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(file format): unify format settings/options. #8566

Merged
merged 43 commits into from
Nov 2, 2022

Conversation

youngsofun
Copy link
Member

@youngsofun youngsofun commented Nov 1, 2022

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

same options used for:

  1. input and output
  2. copy, clickhouse Format, streaming load.

todo

  • add NestedFormatSettings, used for struct/array
    • used in output formats
    • used in input formats (postpone to next pr if no big error)
  • use "" for escape by default in CSV output
  • unify option processing and parse and use defaults by format
    • map clickhouse type to (Type + Options) internally
    • let options = fmt.f1(settings);
    • let format_setting = fmt.f2(options)
  • use nan instead of NaN
  • stateful test test_load_unloaded
    • csv
    • tsv ndjson (next pr, need used NestedFormatSettings in input formats )

Fixes #8491

@vercel
Copy link

vercel bot commented Nov 1, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Nov 2, 2022 at 0:58AM (UTC)

@mergify
Copy link
Contributor

mergify bot commented Nov 1, 2022

This pull request's title is not fulfill the requirements. @youngsofun please update it 🙏.

Valid format:

fix(query): fix group by string bug
  ^         ^---------------------^
  |         |
  |         +-> Summary in present tense.
  |
  +-------> Type: rfc, feat, fix, refactor, ci, docs, chore

Valid types:

  • rfc: this PR proposes a new RFC
  • feat: this PR introduces a new feature to the codebase
  • fix: this PR patches a bug in codebase
  • refactor: this PR changes the code base without new features or bugfix
  • ci: this PR changes build/testing/ci steps
  • docs: this PR changes the documents or websites
  • chore: this PR only has small changes that no need to record

@youngsofun youngsofun changed the title Feat: unify format settings/options. feat(file format): unify format settings/options. Nov 1, 2022
@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Nov 1, 2022
@youngsofun youngsofun marked this pull request as draft November 1, 2022 05:09
@youngsofun youngsofun force-pushed the fix branch 2 times, most recently from d3495dc to 0f5e190 Compare November 1, 2022 12:18
error message: Cast error happens in casting from Array(Nullable(UInt8)) to Array(Nullable(Int64))
@youngsofun youngsofun marked this pull request as ready for review November 2, 2022 01:53
@youngsofun youngsofun requested a review from zhang2014 November 2, 2022 02:38
@youngsofun
Copy link
Member Author

The merge-queue pull request can't be update
Details: `Base branch update has failed

refusing to allow a GitHub App to create or update workflow .github/workflows/dev-linux.yml without workflows permission
err-code: 112C8`

@BohuTANG BohuTANG merged commit 2686a7b into databendlabs:main Nov 2, 2022
@youngsofun youngsofun mentioned this pull request Nov 8, 2022
58 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

refactor format_setting and options
4 participants