This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* restore: merge tidb-tools/pkg/restore-util (#146) * restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (#101) * backup: add raw backup command * restore: speed up retry on not leader (#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (#176) * *: add license header (#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (#174) * Fix summary log (#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * restore: remove tiflash replica before restore (#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <ArGregoryIan@gmail.com> * *: refline logs (#189) * tests: disable TLS test (#204) Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add S3 quick start and few enhancement of log (#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <overvenus@gmail.com> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <overvenus@gmail.com> * Update pkg/task/common.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: cat log if br fails Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * restore: add error field to `DownloadResponse` (#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore: populate restore cancel error (#207) Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * enhance usability of br (#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <kennytm@gmail.com> * task: do not run checksum if restore failed (#209) * fix incremental bug in llroad test (#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (#211) * Revert "tests: disable TLS test (#204)" (#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <overvenus@gmail.com> * move waiting reject stores in import file (#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix ci Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Max index length (#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 庄天翼 <zty0826@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
- Loading branch information
14 people
authored
Apr 2, 2020
1 parent
3239e7d
commit 194a4ff