Incremental backup failing due to missing part error #462
Comments
|
Same issue |
|
One more error during that: |
|
@piyushsriv in my environment an issue is reproduced only when trying to download backup with parent with non exist in local, so pulling the parent backups first could bypass an error. |
Yeah, right. When we are retrying it after deleting the folder. We are also getting this error.
In our case, we daily sync our backup. So, both parent backups (28 and 29) are already present locally when we started downloading 30 one. |
Did you pull them manually? I mean try to download them without auto tree pass: |
|
We can do that and it may solve the problem but then you have to download the full backup again which takes long hours. This problem shouldn't happen in the first place. What if this happen just after you download all the data from S3 to restore the cluster in an outage situation then you have to do it again. :( |
|
@piyushsriv I tried to reproduce CREATE TABLE t1(id UInt64) ENGINE=MergeTree() PARTITION BY id ORDER BY id;
INSERT INTO t1 SELECT number FROM numbers(100);source server clickhouse-backup create full
clickhouse-backup create increment1
clickhouse-backup create increment2
clickhouse-backup upload full
clickhouse-backup upload increment1 --diff-from=full
clickhouse-backup create increment2 --diff-from=increment1destination server clickhouse-backup download increment2 all 100 data parts was download from could you share from your S3 following files: need ensure moreover, could you share |
Currently, only filter |
@k0t3n |
I also tried reproduce this case CREATE TABLE t1(id UInt64) ENGINE=MergeTree() PARTITION BY id ORDER BY id;
INSERT INTO t1 SELECT number FROM numbers(100);clickhouse-backup create full
clickhouse-backup create increment1
clickhouse-backup create increment2
clickhouse-backup upload full
clickhouse-backup upload increment1 --diff-from=full
clickhouse-backup upload increment2 --diff-from=increment1
clickhouse-backup delete local increment2
clickhouse-backup download increment2
in logs it expected, cause all 100 data parts already present on in other backup during download are you sure |
Could you please clarify, which exactly folder do you mean? Moreover, could you share results for the following command |
As I mentioned in the issue we do regular downloads (each day) on different dev cluster. So, we take backup everyday on production cluster, upload it to S3, and then download it to dev cluster.
please find below
The local backup folder i.e. daily-backup-v2-data-incr-2022-06-30 (in var/lib/clickhouse/backup) Unfortunately, I can't confirm this now. Every month we delete all old backups and start with new so we have deleted those backup folders. |
OK. |
Is it possible that due to any reason if the file becomes missing then the download will break? |
|
@piyushsriv
|
looks weird table_2022-06-28.txt contains "files": {
"default": [
"default_72ea255cc4b9d84bc294ae4ac6daf1b7_2_2_0.tar",
...
]
},
"table": "<MASKED>",
"database": "<MASKED>",
"parts": {
"default": [
...
{
"name": "72ea255cc4b9d84bc294ae4ac6daf1b7_2_2_0"
},
...
]
},it expected for full backup table_2022-06-29.txt and table_2022-06-30.txt {
"table": "<MASKED>",
"database": "<MASKED>",
"parts": {
"default": [
...
{
"name": "72ea255cc4b9d84bc294ae4ac6daf1b7_2_2_0",
"required": true
},
...
]
},
...
}and part properly marks as required
so, if 30 full backup downloads was break but even for this corner case clickhouse-backup should download part |
|
@Slach still reproduces on v1.4.6 S3 download debug log: error: |
|
@k0t3n please remove s3:
debug: trueand setup general:
log_level: debugafter it, please try to share full log for |
|
@k0t3n is your S3_DEBUG log contains ? |
|
@Slach sorry for long response. Yes, my log contains So url encoding was missed. An error is reproduced only with incremental backup download and only in |
|
@k0t3n look like gist with logs is private, could you make it public? |
|
@Slach sorry, fixed |
I was looking at the code, to implement I tested it and it is working. Earlier when it was not working for me was an old version. It seems the latest version has this capability. |
|
@k0t3n could you share SHOW CREATE TABLE secret.eventlogtable;need to know PARTITION BY clause |
|
|
@k0t3n reproduced on my side |
|
@Slach thank you! Looking forward for the fix 🚀 |
Hi Team,
We are facing this issue with our incremental backup.
We have taken backups which are like below
We also sync our daily backup on our dev cluster (restore them when we need it) like below
Now, we are facing an issue in download where it throws an error like below
So, this table exists in our full backup. We confirmed the same part is present in the full backup. This table didn't change since the full backup. This problem was not there in
daily-backup-v2-data-incr-2022-06-29but coming indaily-backup-v2-data-incr-2022-06-30What could be the reason and how can we fix it?
Currently, the implementation is that if we have a problem in one table then the whole process stop. Can we have some exclusion table list mechanism in such a process that skips those tables?
Like we have for backup where we can skip tables using
skip_tableswhich I found only works for backup and not for restore like operation.The text was updated successfully, but these errors were encountered: