S3 upload EntityTooSmall: Your proposed upload is smaller than the minimum allowed size

[nerijus]

Trying to upload to wasabi:

$ AWS_ACCESS_KEY_ID=Qw2.... AWS_SECRET_ACCESS_KEY=wGh3... dumpling -c gzip -o s3://bucket/prefix/ --s3.endpoint https://s3.eu-central-1.wasabisys.com
Release version: v5.2.0-alpha-dev
Git commit hash: 292cbe6c9da0f53a262332d7711dd6ba96567411
Git branch:      master
Build timestamp: 2021-06-21 06:51:33Z
Go version:      go version go1.16.5 linux/amd64

[2021/06/22 19:06:00.588 +00:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=v5.2.0-alpha-dev] ["Git Commit Hash"=292cbe6c9da0f53a262332d7711dd6ba96567411] ["Git Branch"=master] ["Build timestamp"="2021-06-21 06:51:33"] ["Go Version"="go version go1.16.5 linux/amd64"]
[2021/06/22 19:06:00.592 +00:00] [INFO] [config.go:602] ["detect server type"] [type=MariaDB]
[2021/06/22 19:06:00.592 +00:00] [INFO] [config.go:621] ["detect server version"] [version=10.4.19-MariaDB-log]
[2021/06/22 19:06:00.592 +00:00] [INFO] [dump.go:83] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"https://s3.eu-central-1.wasabisys.com\",\"region\":\"us-east-1\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":1,\"Host\":\"127.0.0.1\",\"Port\":4000,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"s3://bucket/prefix/\",\"StatusAddr\":\":8281\",\"Snapshot\":\"\",\"Consistency\":\"flush\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[],\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"HasTiKV\":false,\"ServerType\":2,\"ServerVersion\":\"10.4.19-MariaDB-log\"},\"Rows\":0,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]
[2021/06/22 19:06:00.673 +00:00] [INFO] [dump.go:178] ["All the dumping transactions have started. Start to unlock tables"]
[2021/06/22 19:06:01.667 +00:00] [WARN] [writer.go:230] ["no data written in table chunk"] [database=prematch33517] [table=config] [chunkIdx=0]
[2021/06/22 19:06:04.653 +00:00] [WARN] [writer.go:230] ["no data written in table chunk"] [database=prematch33517] [table=temp] [chunkIdx=0]
[2021/06/22 19:06:30.866 +00:00] [ERROR] [writer_util.go:459] ["close file failed"] [path=s3:/bucket/prefix/prematch33517.active_event.000000000.sql.gz] [error="EntityTooSmall: Your proposed upload is smaller than the minimum allowed size\n\tstatus code: 400, request id: 6257ACF0DE551E45, host id: o+RQqzyBp+iE+mUBqp6nlSDsWiz6Axor2rMImRsKZ6L5ma04hywAOB84xyQP1t2Ripa76sRJJB6S"] [errorVerbose="EntityTooSmall: Your proposed upload is smaller than the minimum allowed size\n\tstatus code: 400, request id: 6257ACF0DE551E45, host id: o+RQqzyBp+iE+mUBqp6nlSDsWiz6Axor2rMImRsKZ6L5ma04hywAOB84xyQP1t2Ripa76sRJJB6S\ngit.luolix.top/pingcap/errors.AddStack\n\tgit.luolix.top/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngit.luolix.top/pingcap/errors.Trace\n\tgit.luolix.top/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngit.luolix.top/pingcap/br/pkg/storage.(*S3Uploader).Close\n\tgit.luolix.top/pingcap/br@v5.1.0-alpha.0.20210601094737-6cb0c4abc210+incompatible/pkg/storage/s3.go:104\ngit.luolix.top/pingcap/br/pkg/storage.(*bufferedWriter).Close\n\tgit.luolix.top/pingcap/br@v5.1.0-alpha.0.20210601094737-6cb0c4abc210+incompatible/pkg/storage/writer.go:180\ngit.luolix.top/pingcap/dumpling/v4/export.buildInterceptFileWriter.func2\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer_util.go:457\ngit.luolix.top/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer.go:205\ngit.luolix.top/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer.go:189\ngit.luolix.top/pingcap/br/pkg/utils.WithRetry\n\tgit.luolix.top/pingcap/br@v5.1.0-alpha.0.20210601094737-6cb0c4abc210+incompatible/pkg/utils/retry.go:46\ngit.luolix.top/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer.go:160\ngit.luolix.top/pingcap/dumpling/v4/export.(*Writer).handleTask\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer.go:103\ngit.luolix.top/pingcap/dumpling/v4/export.(*Writer).run\n\tgit.luolix.top/pingcap/dumpling/v4/export/writer.go:85\ngit.luolix.top/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\tgit.luolix.top/pingcap/dumpling/v4/export/dump.go:273\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1371"]

The file prematch33517.active_event.000000000.sql.gz which fails to close is about 35 MB size. 2 other files (sizes 2.7 GB and 379 MB) fail too.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html says "Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload."
This error occurs when one of the parts is less than 5MB in size and isn't the last part (the last part can be any size).

[lichunzhu]

We will check this soon. You can try without -c gzip which should be fine.

[nerijus]

Yes, without -c gzip it is OK.

[glkappe]

@lichunzhu How to understand this error

[lichunzhu]

@lichunzhu How to understand this error

External storage didn't calculate the compressed data's length correctly.

[fubinzh]

/type bug

[fubinzh]

/severity major

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.