Added `thanos bucket replicate` #2113

daixiang0 · 2020-02-10T08:57:37Z

Signed-off-by: Xiang Dai 764524258@qq.com

Implement thanos-replicate to support replicate from one object storage to another.

TODO:

I added CHANGELOG entry for this change.
Add document for replcate.
Add mixin about replcate.

yeya24 · 2020-02-10T14:26:48Z

Hmm, do we plan to add thanos-replicate here?

daixiang0 · 2020-02-11T02:51:34Z

Yep, here is a discuss in the slack.

yeya24 · 2020-02-11T02:57:13Z

Yep, here is a discuss in the slack.

Thanks for clarifying 😄

bwplotka

Awesome, it looks almost ready!

👍

cc @brancz

CHANGELOG.md

cmd/thanos/bucket.go

docs/components/bucket.md

go.mod

pkg/replicater/scheme.go

cmd/thanos/bucket.go

pkg/replicater/scheme.go

cmd/thanos/bucket.go

pkg/replicator/scheme.go

bwplotka

Awesome! Few suggestions and LGTM ❤️

Good work!

Also cc @brancz for quick look, as he initially wrote this. 💪

bwplotka · 2020-02-18T08:00:11Z

cmd/thanos/bucket.go

+	cmd := root.Command("replicate", fmt.Sprintf("Replicate data from one object storage to another. NOTE: Currently it works only with Thanos blocks (%v has to have Thanos metadata).", block.MetaFilename))
+	httpMetricsBindAddr, _ := regHTTPFlags(cmd)
+	toObjStoreConfig := regCommonObjStoreFlags(cmd, "-to", false, "The object storage which replicate data to.")
+	resolution := cmd.Flag("resolution", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()


hm, I think we might need actually Int64s for it as we might want to replicate more than one resolution. Let's add TODO for now.

Also resolution-level ?

Suggested change

resolution := cmd.Flag("resolution", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()

// TODO(bwplotka): Allow to replicate many compaction levels.

resolution := cmd.Flag("resolution-level", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()

Not resolved

In fact it does.

How? I proposed resolution-level (: I might be wrong, but you should comment that I am wrong, because of X not ignore it. But I am pretty sure you just did not notice (:

I thought that you mean add more level support for both resolution and compaction, so i added two TODOs here.

Kinpin supports Int but not Int64, so still need to use String here.

cmd/thanos/bucket.go

pkg/replicator/scheme.go

pkg/replicator/replicater.go

Signed-off-by: Xiang Dai <764524258@qq.com>

bwplotka

Some comments are still not addressed but was marked as addressed.

bwplotka · 2020-02-18T18:11:14Z

cmd/thanos/bucket.go

+	cmd := root.Command("replicate", fmt.Sprintf("Replicate data from one object storage to another. NOTE: Currently it works only with Thanos blocks (%v has to have Thanos metadata).", block.MetaFilename))
+	httpMetricsBindAddr, _ := regHTTPFlags(cmd)
+	toObjStoreConfig := regCommonObjStoreFlags(cmd, "-to", false, "The object storage which replicate data to.")
+	resolution := cmd.Flag("resolution", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()


Not resolved

cmd/thanos/bucket.go

mixin/thanos/dashboards/replicate.libsonnet

daixiang0 · 2020-02-19T07:48:34Z

@bwplotka hi, i do not understand why those codes are old when you review, they were resolved.
Now, i have confirmed them again. For mixin part, i am still working on it.

bwplotka · 2020-02-19T08:42:18Z

Thank You!

…

On Wed, 19 Feb 2020 at 07:48, Xiang Dai ***@***.***> wrote: @bwplotka <https://github.com/bwplotka> hi, i do not understand why those codes are old when you review, they were resolved. Anyway, i have confirmed them again. For mixin part, i am still working on it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2113?email_source=notifications&email_token=ABVA3O5Q5ANWQVBTPWOYCHLRDTP5JA5CNFSM4KSKAQWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMGWP3I#issuecomment-588081133>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O5CVVU4MJUKM2U6YOTRDTP5JANCNFSM4KSKAQWA> .

Signed-off-by: Xiang Dai <764524258@qq.com>

brancz · 2020-02-20T09:32:56Z

Can you mark this PR as work in progress until it’s reviewable in its entirety? Thanks! :)

kakkoyun

One thing that bothers me, is the namespacing of the sub-subcommand. I think all the references to this have to be under bucket replicate from docs, dashboards and alerts. Otherwise, it could create confusion.

CHANGELOG.md

Signed-off-by: Xiang Dai <764524258@qq.com>

bwplotka

LGTM !

Signed-off-by: Xiang Dai <764524258@qq.com>

kakkoyun

lgtm

bwplotka

🎉 Amazing! LGTM ❤️

bwplotka · 2020-02-26T18:49:18Z

CHANGELOG.md

@@ -29,6 +29,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel
 - [#2049](https://github.com/thanos-io/thanos/pull/2049) Tracing: Support sampling on Elastic APM with new sample_rate setting.
 - [#2008](https://github.com/thanos-io/thanos/pull/2008) Querier, Receiver, Sidecar, Store: Add gRPC [health check](https://github.com/grpc/grpc/blob/master/doc/health-checking.md) endpoints.
 - [#2145](https://github.com/thanos-io/thanos/pull/2145) Tracing: track query sent to prometheus via remote read api.
+- [#2113](https://github.com/thanos-io/thanos/pull/2113) Bucket: Added `thanos bucket replicate`.


It got into wrong release section, so we need to be careful when merging release-0.11 to master @metalmatze

* bucket: Implement replcate Signed-off-by: Xiang Dai <764524258@qq.com> * Document replicate Signed-off-by: Xiang Dai <764524258@qq.com> * feedback Signed-off-by: Xiang Dai <764524258@qq.com> * feedback Signed-off-by: Xiang Dai <764524258@qq.com> * remove version check Signed-off-by: Xiang Dai <764524258@qq.com> * update CHANGLOG Signed-off-by: Xiang Dai <764524258@qq.com> * add chan for SIGHUP Signed-off-by: Xiang Dai <764524258@qq.com> * add existence check Signed-off-by: Xiang Dai <764524258@qq.com> * Add mixin Signed-off-by: Xiang Dai <764524258@qq.com> * rename as replicate Signed-off-by: Xiang Dai <764524258@qq.com> * feedback Signed-off-by: Xiang Dai <764524258@qq.com> * update mixin Signed-off-by: Xiang Dai <764524258@qq.com> * update CHANGLOG Signed-off-by: Xiang Dai <764524258@qq.com> * add bucket prefix Signed-off-by: Xiang Dai <764524258@qq.com>

daixiang0 changed the title ~~[WIP] bucket: Implement replcate~~ [WIP] bucket: Implement replicate Feb 10, 2020

daixiang0 changed the title ~~[WIP] bucket: Implement replicate~~ [WIP] Bucket: Implement replicate Feb 10, 2020

daixiang0 force-pushed the repl branch from e51a0a7 to 70c46b0 Compare February 10, 2020 10:00

daixiang0 force-pushed the repl branch 3 times, most recently from 07e6c5e to 1927577 Compare February 11, 2020 04:27

bwplotka reviewed Feb 11, 2020

View reviewed changes

bwplotka requested review from squat and brancz February 11, 2020 09:50

daixiang0 force-pushed the repl branch 5 times, most recently from 611a218 to 3a4a697 Compare February 12, 2020 03:45

yeya24 reviewed Feb 12, 2020

View reviewed changes

cmd/thanos/bucket.go Outdated Show resolved Hide resolved

daixiang0 force-pushed the repl branch from 3a4a697 to e0d13d2 Compare February 12, 2020 10:01

daixiang0 requested a review from bwplotka February 12, 2020 11:54

daixiang0 changed the title ~~[WIP] Bucket: Implement replicate~~ Bucket: Implement replicate Feb 12, 2020

daixiang0 changed the title ~~Bucket: Implement replicate~~ Bucket: Added thanos bucket replicate Feb 12, 2020

daixiang0 changed the title ~~Bucket: Added thanos bucket replicate~~ Added thanos bucket replicate Feb 12, 2020

GiedriusS reviewed Feb 12, 2020

View reviewed changes

cmd/thanos/bucket.go Outdated Show resolved Hide resolved

GiedriusS requested changes Feb 12, 2020

View reviewed changes

pkg/replicater/scheme.go Outdated Show resolved Hide resolved

daixiang0 force-pushed the repl branch 4 times, most recently from 1c7ad57 to 2ca1cdb Compare February 14, 2020 01:33

GiedriusS reviewed Feb 16, 2020

View reviewed changes

cmd/thanos/bucket.go Outdated Show resolved Hide resolved

cmd/thanos/bucket.go Outdated Show resolved Hide resolved

pkg/replicator/scheme.go Outdated Show resolved Hide resolved

bwplotka approved these changes Feb 18, 2020

View reviewed changes

daixiang0 added 2 commits February 18, 2020 16:11

rename as replicate

772a358

Signed-off-by: Xiang Dai <764524258@qq.com>

feedback

cd8a067

Signed-off-by: Xiang Dai <764524258@qq.com>

bwplotka reviewed Feb 18, 2020

View reviewed changes

daixiang0 force-pushed the repl branch from 2265953 to b57c426 Compare February 20, 2020 08:28

update mixin

4bda22a

Signed-off-by: Xiang Dai <764524258@qq.com>

daixiang0 force-pushed the repl branch from b57c426 to 4bda22a Compare February 20, 2020 08:41

kakkoyun reviewed Feb 20, 2020

View reviewed changes

brancz reviewed Feb 20, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

daixiang0 changed the title ~~Added thanos bucket replicate~~ [WIP] Added thanos bucket replicate Feb 21, 2020

update CHANGLOG

ffece8a

Signed-off-by: Xiang Dai <764524258@qq.com>

daixiang0 force-pushed the repl branch from 80ab201 to ffece8a Compare February 21, 2020 04:07

daixiang0 requested review from bwplotka and brancz and removed request for squat February 24, 2020 10:13

bwplotka approved these changes Feb 24, 2020

View reviewed changes

add bucket prefix

dcf97d1

Signed-off-by: Xiang Dai <764524258@qq.com>

daixiang0 force-pushed the repl branch from 2eb2690 to dcf97d1 Compare February 25, 2020 03:38

kakkoyun approved these changes Feb 25, 2020

View reviewed changes

daixiang0 changed the title ~~[WIP] Added thanos bucket replicate~~ Added thanos bucket replicate Feb 25, 2020

daixiang0 requested a review from bwplotka February 26, 2020 05:23

bwplotka mentioned this pull request Feb 26, 2020

Consider merging this with Thanos shipper package. observatorium/thanos-replicate#7

Open

bwplotka approved these changes Feb 26, 2020

View reviewed changes

bwplotka merged commit 0f6df8b into thanos-io:master Feb 26, 2020

bwplotka reviewed Feb 26, 2020

View reviewed changes

daixiang0 deleted the repl branch February 27, 2020 01:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added `thanos bucket replicate` #2113

Added `thanos bucket replicate` #2113

daixiang0 commented Feb 10, 2020 •

edited

Loading

yeya24 commented Feb 10, 2020

daixiang0 commented Feb 11, 2020

yeya24 commented Feb 11, 2020

bwplotka left a comment

bwplotka left a comment

bwplotka Feb 18, 2020

bwplotka Feb 18, 2020

daixiang0 Feb 19, 2020

bwplotka Feb 24, 2020

daixiang0 Feb 25, 2020

daixiang0 Feb 25, 2020

bwplotka left a comment

bwplotka Feb 18, 2020

daixiang0 commented Feb 19, 2020 •

edited

Loading

bwplotka commented Feb 19, 2020 via email

brancz commented Feb 20, 2020

kakkoyun left a comment •

edited

Loading

bwplotka left a comment

kakkoyun left a comment

bwplotka left a comment

bwplotka Feb 26, 2020

	resolution := cmd.Flag("resolution", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()
	// TODO(bwplotka): Allow to replicate many compaction levels.
	resolution := cmd.Flag("resolution-level", "Only blocks with this resolution will be replicated.").Default(strconv.FormatInt(downsample.ResLevel0, 10)).HintAction(listResLevel).Int64()

Added thanos bucket replicate #2113

Added thanos bucket replicate #2113

Conversation

daixiang0 commented Feb 10, 2020 • edited Loading

yeya24 commented Feb 10, 2020

daixiang0 commented Feb 11, 2020

yeya24 commented Feb 11, 2020

bwplotka left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daixiang0 commented Feb 19, 2020 • edited Loading

bwplotka commented Feb 19, 2020 via email

brancz commented Feb 20, 2020

kakkoyun left a comment • edited Loading

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

kakkoyun left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Added `thanos bucket replicate` #2113

Added `thanos bucket replicate` #2113

daixiang0 commented Feb 10, 2020 •

edited

Loading

daixiang0 commented Feb 19, 2020 •

edited

Loading

kakkoyun left a comment •

edited

Loading