feat: output of Copy.

[youngsofun]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

• Closes feat: COPY INTO returns more status #7730

columns:

• file
• rows_loaded
• errors_seen
• first_error
• first_error_line

candidites

• status = loaded, load failed or partially loaded. (can deduced from rows_loaded and errors_seen)
• FIRST_ERROR_COLUMN_NAME ( helpful, try add it in next pr?)
• rows_parsed (redundant for now: where is not supported, each rows_parsed is counted either in rows_loaded or errors_seen)

files skipped for already loaded are not reported.

file_status is stored to table_ctx each time a chuck of data is deserialized, these infos is convert to DataBlock as result after pipeline is successful finished. in distributed copy, slave send status to master before shut_down, reusing the channel for progressInfo.

tasks

• text files: not summery error by error code, because current error code is not. first_error may be more useful.
• copy parquet report status too (but no error).
• ndjson not split file to get first error line.
• sort results by path to ease tests
• tests

todo but not in pr:

• docs docs: output of COPY INTO TABLE. #12605
• distributed copy abort_n where n> 1 abort quickly
• add column FIRST_ERROR_COLUMN_NAME

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
databend	⬜️ Ignored (Inspect)	Visit Preview		Aug 29, 2023 1:02pm