Upload backup to S3 stuck in case of error

[dima-altinity]

Hi Alex

just found small bug related that chbackup stucks after successfully uploading backup. No error or something in logs. Its just waiting forever.
I tried to enable "S3_DEBUG" and see in logs that it successfully uploaded all tar files and then failed to delete file on S3. After that no messages in log.

---[ REQUEST POST-SIGN ]-----------------------------
DELETE /PATH_HERE/BACKUP_NAME_HERE/metadata.json HTTP/1.1

---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden

I added permissions to DELETE files in bucket and it works fine.

It would be good to catch this situation, output message into log and finish backup procedure with exit code -1.

How to reproduce:

1. use S3 account that doesnt have permission to DELETE file
2. insert record into " system.backup_actions" to initiate and upload backup to S3
3. wait for uploading backup complete
4. See that it doesnt exit ( waiting) and no info in log

Thanks

[yangtian9999]

Yes, we encounted the same issue.

create local action is normal.
upload to minio is waiting and no response.

We currently run linux crontab every day at 2:00 am, and we found upload fail every day. Not all tables upload successfully, and no error code at crontab job.

Very difficult to find out the root cause.

[Slach]

@yangtian9999 try to your cron with LOG_LEVEL=debug S3_DEBUG=true and share results
and please share your config.yaml (without credentials)

[yangtian9999]

export CLICKHOUSE_PASSWORD="aaa"
export CLICKHOUSE_FREEZE_BY_PART="false"
export BACKUPS_TO_KEEP_LOCAL="1"
export BACKUPS_TO_KEEP_REMOTE="7"
export S3_ENDPOINT="1.1.1.1:9000"
export S3_BUCKET="clickhouse-backup"
export S3_ACCESS_KEY="11111"
export S3_SECRET_KEY="11111"
export S3_DISABLE_SSL="true"
export REMOTE_STORAGE="s3"
export S3_ACL="private"
export S3_FORCE_PATH_STYLE="true"
export S3_SSE=""
export S3_PATH=""
export S3_DISABLE_CERT_VERIFICATION="true"
/opt/clickhouse-backup/clickhouse-backup create ckbackup-$(hostname)-$(date +%Y%m%d)
/opt/clickhouse-backup/clickhouse-backup upload ckbackup-$(hostname)-$(date +%Y%m%d)

will provide crontab log tomorrow. Thanks.

[Slach]

@yangtian9999 add export S3_DEBUG=true
and share results

is currently all works fine?

[yangtian9999]

Thanks. @Slach , I prefer send you real crontab log with new adding config tomorrow, and check if upload normally
adding
export S3_DEBUG="true"
export LOG_LEVEL="debug"

[yangtian9999]

@Slach
*** this is today's command, and hang for 2 hours *********
[root@ck-2 ~]# ps -ef | grep clickhouse-backup
root 43895 43873 0 12:36 ? 00:00:40 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220112

*** the hang table size is around 11G.

2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYORDERNUM.bin to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYORDERNUM.mrk2 to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYPASSCOUNT.bin to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYPASSCOUNT.mrk2 to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 info DEBUG: Request s3/AbortMultipartUpload Details:
---[ REQUEST POST-SIGN ]-----------------------------
DELETE /clickhouse-backup-2/ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar?uploadId=48f8cd5c-1b45-4a1c-934f-1970c8007786 HTTP/1.1
Host: 10.47.193.181:9000
User-Agent: aws-sdk-go/1.40.31 (go1.17.3; linux; amd64) S3Manager
Authorization: AWS4-HMAC-SHA256 Credential=admin/20220112/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=84272e65c07c43ec1312b0c3f9e1b0795d126ad4200396f354395172b2a8d248
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220112T043651Z
Accept-Encoding: gzip

---

2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYPASSCOUNTBEFORE.bin to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 debug compress /var/lib/clickhouse/backup/ckbackup-ck-2-20220112/shadow/database/issue_table/default/0_108354_137655_58/GANTRYPASSCOUNTBEFORE.mrk2 to ckbackup-ck-2-20220112/shadow/database/issue_table/default_1.tar
2022/01/12 12:36:51 info DEBUG: Response s3/AbortMultipartUpload Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Wed, 12 Jan 2022 04:37:14 GMT
Server: MinIO
Vary: Origin
X-Amz-Request-Id: 16C96BF3F9C57830
X-Xss-Protection: 1; mode=block

---

[yangtian9999]

@Slach , also check this,
*** this is a whole weeks' command, and still hang *********
[root@ck-2 home]# ps -ef | grep clickhouse-backup
root 35849 35829 0 Jan06 ? 00:07:23 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220106
root 36936 36918 0 Jan07 ? 00:05:08 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220107
root 38041 38022 0 Jan08 ? 00:04:01 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220108
root 39840 39821 0 Jan09 ? 00:04:23 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220109
root 40793 40774 0 Jan10 ? 00:02:45 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220110
root 41752 41733 0 Jan11 ? 00:01:43 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220111
root 42795 42777 0 02:00 ? 00:00:55 /opt/clickhouse-backup/clickhouse-backup upload ckbackup-ck-2-20220112

[yangtian9999]

@Slach would you pls help to check the above? Thanks.

[Slach]

@yangtian9999 do you have /etc/clickhouse-backup/config.yml?
I need to ensure we know full context and settings for your error
also, could you share clickhouse-backup version results?

[yangtian9999]

hello @Slach ,
clickhouse-backup version is 1.2.2

we pasted these into a shell file and used sh command to excute this. we did not edit config.yml.

export CLICKHOUSE_PASSWORD="111"
export CLICKHOUSE_FREEZE_BY_PART="false"
export BACKUPS_TO_KEEP_LOCAL="1"
export BACKUPS_TO_KEEP_REMOTE="7"
export S3_ENDPOINT="1.1.1.1:9000"
export S3_BUCKET="clickhouse-backup-4"
export S3_ACCESS_KEY="111"
export S3_SECRET_KEY="1111"
export S3_DISABLE_SSL="true"
export REMOTE_STORAGE="s3"
export S3_ACL="private"
export S3_FORCE_PATH_STYLE="true"
export S3_SSE=""
export S3_PATH=""
export S3_DISABLE_CERT_VERIFICATION="true"
export S3_DEBUG="true"
export LOG_LEVEL="debug"
/opt/clickhouse-backup/clickhouse-backup create ckbackup-$(hostname)-$(date +%Y%m%d)
/opt/clickhouse-backup/clickhouse-backup upload ckbackup-$(hostname)-$(date +%Y%m%d)

Thanks for your help.