Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support copy from stage #4437

Merged
merged 20 commits into from
Mar 15, 2022
Merged

Support copy from stage #4437

merged 20 commits into from
Mar 15, 2022

Conversation

sundy-li
Copy link
Member

@sundy-li sundy-li commented Mar 14, 2022

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

  • Support create/desc/drop internal/external
CREATE STAGE named_external_stage url = 's3://testbucket/admin/data/' credentials=(aws_key_id='minioadmin' aws_secret_key='minioadmin');
  • Support copy from stage
copy into ontime200 from '@named_external_stage'  PATTERN = 'ontime.*parquet' FILE_FORMAT = (type = 'PARQUET')

Changelog

  • Improvement

Related Issues

Part of #4104

Test Plan

Unit Tests

Stateless Tests

@vercel
Copy link

vercel bot commented Mar 14, 2022

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/databend/databend/9PppuBU3KfXkqCRABxtn7NWCvWWV
✅ Preview: https://databend-git-fork-sundy-li-stage-copy2-databend.vercel.app

[Deployment for 09b726e canceled]

@mergify
Copy link
Contributor

mergify bot commented Mar 14, 2022

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@sundy-li sundy-li marked this pull request as ready for review March 14, 2022 14:40
@sundy-li sundy-li requested a review from BohuTANG as a code owner March 14, 2022 14:40
@sundy-li
Copy link
Member Author

Copy from internal stage test is skipped because #4442

@BohuTANG
Copy link
Member

There are some errors in the stateless test:

2022-03-15T00:58:18.476058Z ERROR opendal::services::fs::backend: object /home/runner/work/databend/databend/stateless_test_data/31/73/_ss/f44cf14595dc436aa141721f6f7b4fd7 stat: Object { kind: ObjectNotExist, op: "stat", path: "/home/runner/work/databend/databend/stateless_test_data/31/73/_ss/f44cf14595dc436aa141721f6f7b4fd7", source: No such file or directory (os error 2) }   

Not sure this is expected or not?

@sundy-li sundy-li marked this pull request as ready for review March 15, 2022 05:36
Copy link
Member

@BohuTANG BohuTANG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, we have the named external stage!

@BohuTANG BohuTANG merged commit eed05a5 into databendlabs:main Mar 15, 2022
@sundy-li sundy-li mentioned this pull request Mar 17, 2022
4 tasks
@BohuTANG BohuTANG mentioned this pull request Mar 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants