Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--tables parameter doesn't know about .inner* table when try to create backup for materialized view #613

Closed
lavanyachennupati opened this issue Feb 25, 2023 · 1 comment
Assignees
Milestone

Comments

@lavanyachennupati
Copy link

Is it possible to backup and restores for materialized view tables with clickhouse-backup?

Details:
I have a materialized view with SummingMergeTree engine from a MergeTree as the following:

 CREATE MATERIALIZED VIEW IF NOT EXISTS <materialized_view>
ENGINE = SummingMergeTree
PARTITION BY <key> ORDER BY (....)
AS SELECT ...
FROM <MergeTree>
GROUP BY ...

When i try to create a backup of this table (which has data) in the following way:
/bin/clickhouse-backup create_remote --tables=materialized_view --config=/etc/clickhouse-backup/aggregates-config.yml
I get the following output

2023/02/25 00:59:11.699048  info SELECT name, engine FROM system.databases WHERE name NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
2023/02/25 00:59:11.703321  info SHOW CREATE DATABASE `default`
2023/02/25 00:59:11.708491  info SELECT count() FROM system.settings WHERE name = 'show_table_uuid_in_table_create_query_if_not_nil'
2023/02/25 00:59:11.712561  info SELECT name FROM system.databases WHERE engine IN ('MySQL','PostgreSQL')
2023/02/25 00:59:11.714604  info    SELECT     countIf(name='data_path') is_data_path_present,     countIf(name='data_paths') is_data_paths_present,     countIf(name='uuid') is_uuid_present,     countIf(name='create_table_query') is_create_table_query_present,     countIf(name='total_bytes') is_total_bytes_present    FROM system.columns WHERE database='system' AND table='tables'
2023/02/25 00:59:11.718679  info SELECT database, name, engine , data_paths , uuid , create_table_query , coalesce(total_bytes, 0) AS total_bytes   FROM system.tables WHERE is_temporary = 0 AND match(concat(database,'.',name),'default\.insight_logs_aggregates_test_dev')  SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1
2023/02/25 00:59:11.735394  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2023/02/25 00:59:11.738775  info SELECT * FROM system.disks;
2023/02/25 00:59:11.743293  info done                      backup=2023-02-25T00-59-11 operation=create table=default.insight_logs_aggregates_test_dev
2023/02/25 00:59:11.743335  info SELECT toUInt8(count()) udf_presents FROM system.columns WHERE database='system' AND table='functions' AND name='create_query'
2023/02/25 00:59:11.747254  info SELECT name, create_query FROM system.functions WHERE create_query!=''
2023/02/25 00:59:11.751183  info SELECT value FROM `system`.`build_options` where name='VERSION_DESCRIBE'
2023/02/25 00:59:11.754375  info done                      backup=2023-02-25T00-59-11 duration=58ms operation=create
2023/02/25 00:59:11.760799  info done                      backup=2023-02-25T00-41-31 duration=4ms location=local operation=delete
2023/02/25 00:59:11.764939  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2023/02/25 00:59:11.767568  info SELECT * FROM system.disks;
2023/02/25 00:59:11.772210  info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts
2023/02/25 00:59:11.777454  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND table='macros'
2023/02/25 00:59:11.780810  info SELECT * FROM system.macros
2023/02/25 00:59:11.872699  info done                      backup=2023-02-25T00-59-11 duration=37ms operation=upload size=1.58KiB table=default.insight_logs_aggregates_test_dev
2023/02/25 00:59:11.919255  info done                      backup=2023-02-25T00-59-11 duration=158ms operation=upload size=2.11KiB
2023/02/25 00:59:11.923501  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER'
2023/02/25 00:59:11.927337  info SELECT * FROM system.disks;

But there's no data in the /shadow folder of the backup in s3. So I'm wondering if it's even possible to backup and restore data for materialized views?

@lavanyachennupati lavanyachennupati changed the title backup and restores for materialized view tables. is it possible to backup and restore data for materialized view tables. Feb 25, 2023
@Slach
Copy link
Collaborator

Slach commented Feb 25, 2023

when you create materialized view without TO table_name clause, then clickhouse will create separate
.inner_id.XXX table where XXX UUID of materialized view

as quick workardound use --tables=default.materialized_view,default..inner.*

or better use

CREATE TABLE dst_table_name (
) ENGINE = SummingMergeTree
PARTITION BY <key> ORDER BY (....);

CREATE MATERIALIZED VIEW IF NOT EXISTS materialized_view
TO dst_table_name
AS SELECT ...
FROM ...
GROUP BY ...;

and
--tables=default.materialized_view,default.dst_table_name

@Slach Slach changed the title is it possible to backup and restore data for materialized view tables. --tables parameter doesn't know about .inner* table when try to create backup for materialized view Feb 25, 2023
@Slach Slach self-assigned this Feb 25, 2023
@Slach Slach added this to the 2.3.0 milestone Feb 25, 2023
@Slach Slach closed this as completed in ab4011a Jun 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants