Pump deployment fails: silently failing due to write binlog error no available space #656

sokada1221 · 2019-07-14T05:42:07Z

Bug Report

What version of Kubernetes are you using?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-eks-c57ff8", GitCommit:"c57ff8e35590932c652433fab07988da79265d5b", GitTreeState:"clean", BuildDate:"2019-06-07T20:43:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

$ kubectl exec -n tidb-admin tidb-controller-manager-7b998f5747-fsktc -- tidb-controller-manager -V
TiDB Operator Version: version.Info{TiDBVersion:"2.1.0", GitVersion:"v1.0.0-beta.3", GitCommit:"6257dfaad68f55f745f20f6f5d19b10bea2b0bea", GitTreeState:"clean", BuildDate:"2019-06-06T09:51:04Z", GoVersion:"go1.12", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

$ kubectl get sc
NAME            PROVISIONER                    AGE
ebs-gp2         kubernetes.io/aws-ebs          19h
gp2 (default)   kubernetes.io/aws-ebs          19h
local-storage   kubernetes.io/no-provisioner   19h
$ kubectl get pvc -n stg4tidb
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-stg4tidb-pump-0   Bound    pvc-57fb40ee-a5ef-11e9-9131-060d2ce056e4   10Gi       RWO            ebs-gp2         60m
data-stg4tidb-pump-1   Bound    pvc-6412c9d2-a5ef-11e9-9131-060d2ce056e4   10Gi       RWO            ebs-gp2         60m
pd-stg4tidb-pd-0       Bound    pvc-c2367f92-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
pd-stg4tidb-pd-1       Bound    pvc-c23b2325-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
pd-stg4tidb-pd-2       Bound    pvc-c23f4d3b-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
stg4tidb-monitor       Bound    pvc-c2069a71-a55d-11e9-8a1c-0a3a29ff5ab2   100Gi      RWO            ebs-gp2         18h
tikv-stg4tidb-tikv-0   Bound    local-pv-7ef562a2                          1741Gi     RWO            local-storage   18h
tikv-stg4tidb-tikv-1   Bound    local-pv-dfc17131                          1741Gi     RWO            local-storage   18h
tikv-stg4tidb-tikv-2   Bound    local-pv-a116ff25                          1741Gi     RWO            local-storage   18h

What's the status of the TiDB cluster pods?

$ kubectl get po -n stg4tidb -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
stg4tidb-discovery-c4c55c845-sp4ls   1/1     Running   0          18h   10.220.18.55    ip-10-220-18-177.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-monitor-d6bfd6865-9hnh7     2/2     Running   0          58m   10.220.18.246   ip-10-220-18-177.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-0                        1/1     Running   2          18h   10.220.19.87    ip-10-220-19-116.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-1                        1/1     Running   0          18h   10.220.17.208   ip-10-220-17-112.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-2                        1/1     Running   0          18h   10.220.18.101   ip-10-220-18-5.ap-northeast-1.compute.internal     <none>           <none>
stg4tidb-pump-0                      1/1     Running   0          80s   10.220.19.223   ip-10-220-19-150.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pump-1                      1/1     Running   0          67s   10.220.18.54    ip-10-220-18-29.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tidb-0                      1/1     Running   0          10m   10.220.18.45    ip-10-220-18-150.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-tidb-1                      1/1     Running   0          48s   10.220.19.239   ip-10-220-19-133.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-tikv-0                      1/1     Running   0          18h   10.220.17.190   ip-10-220-17-50.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tikv-1                      1/1     Running   0          18h   10.220.18.36    ip-10-220-18-49.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tikv-2                      1/1     Running   0          18h   10.220.19.29    ip-10-220-19-246.ap-northeast-1.compute.internal   <none>           <none>

What did you do?

Deploy TiDB operator and TiDB cluster with pump enabled
Relevant values used for installing TiDB cluster:

binlog:
  pump:
    create: false
    replicas: 2
    image: pingcap/tidb-binlog:v2.1.14
    imagePullPolicy: IfNotPresent
    logLevel: info
    # storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
    # different classes might map to quality-of-service levels, or to backup policies,
    # or to arbitrary policies determined by the cluster administrators.
    # refer to https://kubernetes.io/docs/concepts/storage/storage-classes
    storageClassName: ebs-gp2
    storage: 10Gi
    syncLog: true
    # a integer value to control expiry date of the binlog data, indicates for how long (in days) the binlog data would be stored.
    # must bigger than 0
    gc: 7
    # number of seconds between heartbeat ticks (in 2 seconds)
    heartbeatInterval: 2

Pump and TiDB pods are both running okay but TiDB becomes inaccessible due to pump

What did you expect to see?
Successful deployment with correctly functioning pump

What did you see instead?
Pump log is filled with these error messages:

runtime.main
	/usr/local/go/src/runtime/proc.go:188
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337�[0m
2019/07/14 05:13:17 server.go:239: �[0;31m[error] write binlog error no available space
github.com/pingcap/tidb-binlog/pump/storage.init.ializers
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb-binlog/pump/storage/errors.go:10
runtime.main
	/usr/local/go/src/runtime/proc.go:188
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337�[0m
2019/07/14 05:13:17 server.go:239: �[0;31m[error] write binlog error no available space
github.com/pingcap/tidb-binlog/pump/storage.init.ializers
	/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb-binlog/pump/storage/errors.go:10
runtime.main
	/usr/local/go/src/runtime/proc.go:188
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1337�[0m

Eventually, it starts printing the following info messages:

2019/07/14 05:13:21 storage.go:356: �[0;37m[info] DBStats: {WriteDelayCount:0 WriteDelayDuration:0s WritePaused:false AliveSnapshots:0 AliveIterators:0 IOWrite:38920 IORead:144245 BlockCacheSize:13003 OpenedTablesCount:1 LevelSizes:[36962] LevelTablesCounts:[1] LevelRead:[0] LevelWrite:[0] LevelDurations:[0s]}�[0m
2019/07/14 05:13:21 server.go:498: �[0;37m[info] writeBinlogCount: 26641, alivePullerCount: 0,  maxCommitTS: 409752353337508548�[0m
2019/07/14 05:13:31 storage.go:356: �[0;37m[info] DBStats: {WriteDelayCount:0 WriteDelayDuration:0s WritePaused:false AliveSnapshots:0 AliveIterators:0 IOWrite:39412 IORead:144245 BlockCacheSize:13003 OpenedTablesCount:1 LevelSizes:[36962] LevelTablesCounts:[1] LevelRead:[0] LevelWrite:[0] LevelDurations:[0s]}�[0m
2019/07/14 05:13:31 server.go:498: �[0;37m[info] writeBinlogCount: 26641, alivePullerCount: 0,  maxCommitTS: 409752360415133697�[0m

TiDB log is filled with these warning messages:

2019/07/14 05:27:30.499 client.go:288: [warning] [pumps client] write binlog to pump stg4tidb-pump-1:8250 (type: Prewrite, start ts: 409752581022679048, commit ts: 0, length: 63077) error rpc error: code = Unknown desc = no available space

Eventually TiDB stops responding as it fails to communicate with pump:

[2019/07/14 05:27:45.538 +00:00] [INFO] [2pc.go:635] ["2PC clean up done"] [conn=27] [txnStartTS=409752581704253441]
2019/07/14 05:27:45.542 client.go:321: [info] [pumps client] write binlog to avaliable pumps all failed, will try unavaliable pumps
[2019/07/14 05:27:45.547 +00:00] [ERROR] [binloginfo.go:113] ["write binlog failed"] [error="write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="write binlog failed, the last error no avaliable pump to write binlog\ngit.luolix.top/pingcap/tidb-tools/tidb-binlog/pump_client.(*PumpsClient).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190116051332-34c808eef588+incompatible/tidb-binlog/pump_client/client.go:328\ngit.luolix.top/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:111\ngit.luolix.top/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [stack="github.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:113\ngit.luolix.top/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743"]
[2019/07/14 05:27:45.547 +00:00] [WARN] [session.go:382] ["commit failed"] [conn=20] ["finished txn"="Txn{state=invalid}"] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngit.luolix.top/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngit.luolix.top/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngit.luolix.top/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"]
2019/07/14 05:27:45.548 client.go:321: [info] [pumps client] write binlog to avaliable pumps all failed, will try unavaliable pumps
2019/07/14 05:27:45.548 adapter.go:413: [warning] # Txn_start_ts: 409752581560074260
# User: root@10.220.19.146
# Conn_ID: 20
# Query_time: 13.053171456
# Prewrite_time: 0.028558435 Write_keys: 3000 Write_size: 156000 Prewrite_region: 3
# DB: paylite_payment_migration
# Is_internal: false
# Digest: cac5c7d1c1782e36215435bcbec6c1be82b46486325beb8e04895bef8a620c16
# Num_cop_tasks: 0
# Cop_proc_avg: 0 Cop_proc_p90: 0 Cop_proc_max: 0
# Cop_wait_avg: 0 Cop_wait_p90: 0 Cop_wait_max: 0
"insert ignore into  acquiring_migration_status (orderId,status,version) values   (  3024886118766886912,'PUBLISHING',1 ),( 3024886118766895104,'PUBLISHING',1 ),( 3024886118766903296,'PUBLISHING',1 ),( 3024886118766919680,'PUBLISHING',1 ),( 3024886118766919681,'PUBLISHING',1 ),( 3024886118766919682,'PUBLISHING',1 ),( 3024886118766927872,'PUBLISHING',1 ),( 3024886118766944256,'PUBLISHING',1 ),( 3024886118766944257,'PUBLISHING',1 ),( 3024886118766944258,'PUBLISHING',1 ),( 3024886118766952448,'PUBLISHING',1 ),( 3024886118766952449,'PUBLISHING',1 ),( 3024886118766960640,'PUBLISHING',1 ),( 3024886118766968832,'PUBLISHING',1 ),( 3024886118766977024,'PUBLISHING',1 ),( 3024886118766993408,'PUBLISHING',1 ),( 3024886118767001600,'PUBLISHING',1 ),( 3024886118767009792,'PUBLISHING',1 ),( 3024886118767042560,'PUBLISHING',1 ),( 3024886118767042561,'PUBLISHING',1 ),( 3024886118767058944,'PUBLISHING',1 ),( 3024886118767058945,'PUBLISHING',1 ),( 3024886118767067136,'PUBLISHING',1 ),( 3024886118767075328,'PUBLISHING',1 ),( 3024886118767075329,'PUBLISHING',1 ),( 3024886118767091712,'PUBLISHING',1 ),( 3024886118767099904,'PUBLISHING',1 ),( 3024886118767108096,'PUBLISHING',1 ),( 3024886118767116288,'PUBLISHING',1 ),( 3024886118767132672,'PUBLISHING',1 ),( 3024886118767140864,'PUBLISHING',1 ),( 3024886127356755968,'PUBLISHING',1 ),( 3024886127356764160,'PUBLISHING',1 ),( 3024886127356788736,'PUBLISHING',1 ),( 3024886127356788737,'PUBLISHING',1 ),( 3024886127356846080,'PUBLISHING',1 ),( 3024886127356854272,'PUBLISHING',1 ),( 3024886127356854273,'PUBLISHING',1 ),( 3024886127356862464,'PUBLISHING',1 ),( 3024886127356878848,'PUBLISHING',1 ),( 3024886127356895232,'PUBLISHING',1 ),( 3024886127356903424,'PUBLISHING',1 ),( 3024886127356911616,'PUBLISHING',1 ),( 3024886127356928000,'PUBLISHING',1 ),( 3024886127356928001,'PUBLISHING',1 ),( 3024886127356928002,'PUBLISHING',1 ),( 3024886127356960768,'PUBLISHING',1 ),( 3024886127356985344,'PUBLISHING',1 ),( 3024886127356993536,'PUBLISHING',1 ),( 3024886127357009920,'PUBLISHING',1 ),( 30248861273570"(len:39081);
[2019/07/14 05:27:45.548 +00:00] [WARN] [session.go:834] ["run statement error"] [conn=20] [schemaVersion=72] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngit.luolix.top/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngit.luolix.top/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngit.luolix.top/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [session="{\n  \"currDBName\": \"paylite_payment_migration\",\n  \"id\": 20,\n  \"lastInsertID\": 47452074,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.220.19.146\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"\",\n    \"AuthHostname\": \"\"\n  }\n}"]
[2019/07/14 05:27:45.548 +00:00] [ERROR] [conn.go:512] ["critical error, stop the server listener"] [conn=20] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngit.luolix.top/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngit.luolix.top/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngit.luolix.top/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngit.luolix.top/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:512\ngit.luolix.top/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:383"]
[2019/07/14 05:27:45.548 +00:00] [INFO] [server.go:362] ["close connection"] [conn=20]

[2019/07/14 05:28:45.453 +00:00] [ERROR] [server.go:317] ["listener stopped, waiting for manual kill."] [stack="github.com/pingcap/tidb/server.(*Server).Run\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:317\nmain.runServer\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:552\nmain.main\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:164\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"]

But pump has 10GiB of space configured and allocated. Disk space availability is correctly reported on Grafana, and it can also be confirmed inside the pod:

$ kubectl exec -n stg4tidb -it stg4tidb-pump-0 -- /bin/sh
/ # ps -ef | grep pump
    1 root      0:01 /pump -L=info -advertise-addr=stg4tidb-pump-0.stg4tidb-pump:8250 -config=/etc/pump/pump.toml -log-file=
   29 root      0:00 grep pump
/ # cat /etc/pump/pump.toml
# pump Configuration.

# addr(i.e. 'host:port') to listen on for client traffic
addr = "0.0.0.0:8250"

# addr(i.e. 'host:port') to advertise to the public
advertise-addr = ""

# a integer value to control expiry date of the binlog data, indicates for how long (in days) the binlog data would be stored.
# must bigger than 0
gc = 7

# path to the data directory of pump's data
data-dir = "/data"

# number of seconds between heartbeat ticks (in 2 seconds)
heartbeat-interval = 2

# a comma separated list of PD endpoints
pd-urls = "http://stg4tidb-pd:2379"

#[security]
# Path of file that contains list of trusted SSL CAs for connection with cluster components.
# ssl-ca = "/path/to/ca.pem"
# Path of file that contains X509 certificate in PEM format for connection with cluster components.
# ssl-cert = "/path/to/drainer.pem"
# Path of file that contains X509 key in PEM format for connection with cluster components.
# ssl-key = "/path/to/drainer-key.pem"
#
[storage]
# Set to `true` (default) for best reliability, which prevents data loss when there is a power failure.
sync-log = true
#
# we suggest using the default config of the embedded LSM DB now, do not change it unless you know what you are doing
# [storage.kv]
# block-cache-capacity = 8388608
# block-restart-interval = 16
# block-size = 4096
# compaction-L0-trigger = 8
# compaction-table-size = 67108864
# compaction-total-size = 536870912
# compaction-total-size-multiplier = 8.0
# write-buffer = 67108864
# write-L0-pause-trigger = 24
# write-L0-slowdown-trigger = 17
/ # cd /data
/data # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme1n1              9.7G     36.2M      9.7G   0% /data

The text was updated successfully, but these errors were encountered:

xiaojingchen · 2019-07-14T06:14:41Z

@shinnosuke-okada what version of the pump?

sokada1221 · 2019-07-14T06:17:32Z

@xiaojingchen

    image: pingcap/tidb-binlog:v2.1.14

xiaojingchen · 2019-07-14T06:28:05Z

@shinnosuke-okada the new version pump config have a parameter stop-write-at-available-space,

# stop write when disk available space less then the configured size
# 42 MB -> 42000000, 42 mib -> 44040192
# default: 10 gib
# stop-write-at-available-space = "10 gib"

so if the disk space is less than 10G, the pump will stop writing. you need to use a larger disk or change the parameter.
But current operator can not support to change this parameter directly.

xiaojingchen · 2019-07-14T06:38:55Z

In production environments, it is recommended to allocate larger disks for Pump; In the other hand, we will improve Operator's configuration modification to make it more flexible and easy to use.
this is a relate PR: #638

sokada1221 · 2019-07-14T06:49:48Z

@xiaojingchen Thanks a lot. These are very helpful! Let me see if I can unblock myself based on your information.

sokada1221 · 2019-07-14T07:28:27Z

@xiaojingchen I worked around by allocating more disk space for pump services. I think we have 2 issues here:

tidb-cluster's default pump config shouldn't come with 10GiB
pump should print a better error message for the case when stop-write-at-available-space is causing the no available space error

I'll make a PR for issue 1 immediately, and consider that as a solution to this issue. Planning to increase default disk size for pumps from 10GiB to 20GiB.

xiaojingchen · 2019-07-14T07:53:41Z

@shinnosuke-okada thanks for your feedback, that is very useful to us, and please create an new issue if it's convenient for you.

sokada1221 · 2019-07-15T16:23:51Z

Logged pingcap/tidb-binlog#679 to address the error message issue. As the problem on TiDB-operator side is addressed, closing this ticket.

…restore db accounts (#656) * bump tidb version to v4.0.4 * add database privilege demands * fix headings * address comments

sokada1221 changed the title ~~Pump is silently failing due to write binlog error no available space~~ Pump deployment fails: silently failing due to write binlog error no available space Jul 14, 2019

This was referenced Jul 14, 2019

Increase the default storage size for pump from 10Gi to 20Gi #657

Merged

Need a better error message when pump is running out of storage pingcap/tidb-binlog#679

Closed

sokada1221 closed this as completed Jul 15, 2019

yahonda pushed a commit that referenced this issue Dec 27, 2021

zh: bump tidb version to v4.0.4 and add privilege demands for backup/…

a33a858

…restore db accounts (#656) * bump tidb version to v4.0.4 * add database privilege demands * fix headings * address comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pump deployment fails: silently failing due to write binlog error no available space #656

Pump deployment fails: silently failing due to write binlog error no available space #656

sokada1221 commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 15, 2019

Pump deployment fails: silently failing due to write binlog error no available space #656

Pump deployment fails: silently failing due to write binlog error no available space #656

Comments

sokada1221 commented Jul 14, 2019

Bug Report

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

sokada1221 commented Jul 14, 2019

xiaojingchen commented Jul 14, 2019

sokada1221 commented Jul 15, 2019