Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support ticdc data dir #1372

Merged
merged 42 commits into from
Jun 1, 2021
Merged

support ticdc data dir #1372

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
0e0dbc2
add a data dir field in cdc spec.
3AceShowHand May 19, 2021
d33d050
update go mod file.
3AceShowHand May 19, 2021
6b79d4e
add data dir into ticdc.
3AceShowHand May 19, 2021
1421709
add debug log.
3AceShowHand May 19, 2021
b49a641
remove debug.
3AceShowHand May 20, 2021
d77600c
add some testcase for ticdc data_dir.
3AceShowHand May 20, 2021
6a7c430
add data_dir by version.
3AceShowHand May 21, 2021
6439c29
add comment about CDCScript.WithDataDir.
3AceShowHand May 22, 2021
106d7d1
set data dir to '', if cluster version lower than 4.0.13
3AceShowHand May 24, 2021
32af2e9
fix panic on cdc data dir.
3AceShowHand May 24, 2021
9b6294e
add debug log .
3AceShowHand May 24, 2021
edb5736
tiny fix on adjust.
3AceShowHand May 24, 2021
6704135
check version on scale out.
3AceShowHand May 24, 2021
dabb1a3
fix conflicts at scale-out
3AceShowHand May 24, 2021
fed2886
fix scale_out issue.
3AceShowHand May 24, 2021
d98c095
fix cdc data dir is empty when display in confirm.
3AceShowHand May 24, 2021
122f040
do not throw error if data dir set in cdc, just ignore it.
3AceShowHand May 24, 2021
157291f
fix confirm
3AceShowHand May 24, 2021
e3db218
one more try.
3AceShowHand May 25, 2021
4f29ae0
try to fix rename.
3AceShowHand May 25, 2021
4bfb002
try to fix scale-out issue.
3AceShowHand May 25, 2021
76e1caa
warn for wrong data-dir use in cdc.
3AceShowHand May 25, 2021
cd32e8f
move warn to init config.
3AceShowHand May 25, 2021
1d26c06
refine log warn message.
3AceShowHand May 25, 2021
3c1938e
rename cdc start script folder.
3AceShowHand May 25, 2021
8312c3e
move warn to adjust..
3AceShowHand May 25, 2021
bd9b47a
tiny adjust.
3AceShowHand May 25, 2021
c2919f9
make warn red.
3AceShowHand May 25, 2021
ea70a06
move warn out of loop.
3AceShowHand May 25, 2021
d9ffcc4
remove cdc data dir in check cluster.
3AceShowHand May 25, 2021
45191d1
add mkdir.
3AceShowHand May 26, 2021
047a073
remove log.
3AceShowHand May 26, 2021
99dc83d
Merge branch 'master' into ling.jin/support-ticdc-data-dir
AstroProfundis May 26, 2021
0691dc3
Merge branch 'master' into ling.jin/support-ticdc-data-dir
3AceShowHand May 26, 2021
34e2030
Merge branch 'ling.jin/support-ticdc-data-dir' of https://github.com/…
3AceShowHand May 26, 2021
69a1116
support data-dir .
3AceShowHand May 27, 2021
102370e
add testcase for cdc script initialization by version.
3AceShowHand May 27, 2021
25d4b9c
tiny change.
3AceShowHand May 27, 2021
940aa46
fix.
3AceShowHand May 27, 2021
1a2bd76
add ignore .
3AceShowHand May 28, 2021
476a3c5
Merge branch 'master' into ling.jin/support-ticdc-data-dir
3AceShowHand May 28, 2021
c2ec4db
Merge branch 'master' into ling.jin/support-ticdc-data-dir
3AceShowHand May 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions components/client/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ go 1.16
require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/gizak/termui/v3 v3.1.0
github.com/google/flatbuffers v2.0.0+incompatible // indirect
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved
github.com/mattn/go-sqlite3 v2.0.2+incompatible // indirect
github.com/mitchellh/go-wordwrap v1.0.0 // indirect
github.com/sergi/go-diff v1.1.0 // indirect
Expand Down
3 changes: 2 additions & 1 deletion components/client/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,8 +499,9 @@ github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ
github.com/google/btree v1.0.1 h1:gK4Kx5IaGY9CD5sPJ36FHiBJ6ZXl0kilRiiCj+jdYp4=
github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA=
github.com/google/flatbuffers v1.11.0/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v1.12.0 h1:/PtAHvnBY4Kqnx/xCQ3OIV9uYcSFGScBsWI3Oogeh6w=
github.com/google/flatbuffers v1.12.0/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v2.0.0+incompatible h1:dicJ2oXwypfwUGnB2/TYWYEKiuk9eYQlQO/AnOHl5mI=
github.com/google/flatbuffers v2.0.0+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
Expand Down
1 change: 1 addition & 0 deletions embed/templates/scripts/run_cdc.sh.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ exec bin/cdc server \
--addr "0.0.0.0:{{.Port}}" \
--advertise-addr "{{.IP}}:{{.Port}}" \
--pd "{{template "PDList" .Endpoints}}" \
--sort-dir="{{.DataDir}}" \
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved
{{- if .TLSEnabled}}
--ca tls/ca.crt \
--cert tls/cdc.crt \
Expand Down
4 changes: 4 additions & 0 deletions pkg/cluster/manager/deploy.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,10 @@ func (m *Manager) Deploy(
deployDir := spec.Abs(globalOptions.User, inst.DeployDir())
// data dir would be empty for components which don't need it
dataDirs := spec.MultiDirAbs(globalOptions.User, inst.DataDir())

log.Debugf("inst.ComponentName=%+v, inst.DataDir=%+v, dataDirs=%+v\n",
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved
inst.ComponentName(), inst.DataDir(), dataDirs)

// log dir will always be with values, but might not used by the component
logDir := spec.Abs(globalOptions.User, inst.LogDir())
// Deploy component
Expand Down
3 changes: 3 additions & 0 deletions pkg/cluster/spec/cdc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ type CDCSpec struct {
Patched bool `yaml:"patched,omitempty"`
Port int `yaml:"port" default:"8300"`
DeployDir string `yaml:"deploy_dir,omitempty"`
DataDir string `yaml:"data_dir,omitempty"`
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved
LogDir string `yaml:"log_dir,omitempty"`
Offline bool `yaml:"offline,omitempty"`
GCTTL int64 `yaml:"gc-ttl,omitempty"`
Expand Down Expand Up @@ -96,6 +97,7 @@ func (c *CDCComponent) Instances() []Instance {
},
Dirs: []string{
s.DeployDir,
s.DataDir,
},
StatusFn: func(tlsCfg *tls.Config, _ ...string) string {
return statusByHost(s.Host, s.Port, "/status", tlsCfg)
Expand Down Expand Up @@ -161,6 +163,7 @@ func (i *CDCInstance) InitConfig(
cfg := scripts.NewCDCScript(
i.GetHost(),
paths.Deploy,
paths.Data[0],
paths.Log,
enableTLS,
spec.GCTTL,
Expand Down
2 changes: 2 additions & 0 deletions pkg/cluster/spec/instance.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import (
"github.com/pingcap/tiup/pkg/cluster/ctxt"
"github.com/pingcap/tiup/pkg/cluster/module"
system "github.com/pingcap/tiup/pkg/cluster/template/systemd"
"github.com/pingcap/tiup/pkg/logger/log"
"github.com/pingcap/tiup/pkg/meta"
"go.uber.org/zap"
)
Expand Down Expand Up @@ -334,6 +335,7 @@ func (i *BaseInstance) DeployDir() string {
func (i *BaseInstance) DataDir() string {
dataDir := reflect.Indirect(reflect.ValueOf(i.InstanceSpec)).FieldByName("DataDir")
if !dataDir.IsValid() {
log.Debugf("i.ComponentName=%+v, dataDir is invalid")
return ""
}

Expand Down
4 changes: 3 additions & 1 deletion pkg/cluster/template/scripts/cdc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ type CDCScript struct {
Port int
DeployDir string
LogDir string
DataDir string
NumaNode string
GCTTL int64
TZ string
Expand All @@ -37,12 +38,13 @@ type CDCScript struct {
}

// NewCDCScript returns a CDCScript with given arguments
func NewCDCScript(ip, deployDir, logDir string, enableTLS bool, gcTTL int64, tz string) *CDCScript {
func NewCDCScript(ip, deployDir, dataDir, logDir string, enableTLS bool, gcTTL int64, tz string) *CDCScript {
return &CDCScript{
IP: ip,
Port: 8300,
DeployDir: deployDir,
LogDir: logDir,
DataDir: dataDir,
TLSEnabled: enableTLS,
GCTTL: gcTTL,
TZ: tz,
Expand Down