-
Notifications
You must be signed in to change notification settings - Fork 6.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3-plain based disk supporting directory rename #61116
S3-plain based disk supporting directory rename #61116
Conversation
This is an automated comment for commit dc95558 with description of existing statuses. It's updated for the latest CI running ❌ Click here to open a full report in a separate page
Successful checks
|
e13a6b4
to
b740637
Compare
e198384
to
84c645b
Compare
7d6a514
to
8e45f6f
Compare
src/Disks/IDisk.h
Outdated
/// Whether this disk support mutations. | ||
virtual bool isMutable() const { return true; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May be better to explicitly name it supportsHardlinks
? Because isMutable
is kind of the opposite of IDisk::isReadOnly
, so might be confusing
|
||
drop table if exists test_mt; | ||
create table test_mt (a Int32, b Int64) engine = MergeTree() order by a | ||
settings disk = disk( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_log system tables defined for CI are in ClickHouse/tests/config/config.d/
, for example, https://github.com/ClickHouse/ClickHouse/blob/master/tests/config/config.d/blob_storage_log.xml. It is possible to define engine
parameter in this config, where we can also add storage_policy
, example:
ClickHouse/tests/integration/test_system_logs/configs/config.d/system_logs_engine.xml
Line 3 in 5139ef7
<engine>Engine = MergeTree PARTITION BY event_date ORDER BY event_time TTL event_date + INTERVAL 30 day SETTINGS storage_policy='policy2', ttl_only_drop_parts=1</engine> |
We could add
s3_plain_rewritable
for blob_storage_log
from the example above. Possibly it will be better to do this only for a tests run which runs all tests on top of s3, this can be checked here ClickHouse/tests/config/install.sh
Line 159 in 5139ef7
if [[ "$USE_S3_STORAGE_FOR_MERGE_TREE" == "1" ]]; then |
if
branch we could substitute blob_storage_log.xml to a different one (with a different storage policy) instead of what is added here ClickHouse/tests/config/install.sh
Line 22 in 5139ef7
ln -sf $SRC_PATH/config.d/blob_storage_log.xml $DEST_SERVER_PATH/config.d/ |
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
…Operations.cpp Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
non-functional changes
Make PlainRewritableObjectStorage generic; Support config type ``` <s3_plain_rewritable> <type>object_storage</type> <object_storage_type>s3</object_storage_type> <metadata_type>plain_rewritable</metadata_type> <endpoint>https://s3.eu-west-1.amazonaws.com/clickhouse-eu-west-1.clickhouse.com/data/</endpoint> <use_environment_credentials>1</use_environment_credentials> </s3_plain_rewritable> ```
ce79982
to
0c62705
Compare
Getting an error when trying to specify
I'll remove the commit with |
0c62705
to
dc95558
Compare
Fixes: #58347
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added a rewritable S3 disk which supports INSERT operations and does not require locally stored metadata.
Documentation entry for user-facing changes