REPLICATOR: Add TimePartitionMetaFilter to thanos replicate. #2979

OGKevin · 2020-08-04T12:39:18Z

This will allow time based replication.

Its blocked by #2906

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Added TimePartitionMetaFilter for thanos replication, this means that replicate can now be time based.

Verification

pkg/block/metadata/deletionmark.go

cmd/thanos/tools_bucket.go

bwplotka

Nice, just comments around deletion not being part of replicator

CHANGELOG.md

cmd/thanos/tools_bucket.go

pkg/block/block.go

pkg/replicate/replicator.go

OGKevin · 2020-08-10T12:33:55Z

Im waiting on the depended MR to get merged to continue on this one. As the dependent MR will most likely introduce more merging conflicts.

stale · 2020-10-17T09:00:00Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

OGKevin · 2020-10-17T19:06:53Z

@bwplotka can the stale label be removed? As this is blocked on #2906

kakkoyun · 2020-10-19T08:26:13Z

@yeya24 FYI this is also blocked by #2906

kakkoyun · 2020-10-19T08:27:14Z

@OGKevin In the meanwhile, could you maybe rebase and also sign DCO?

This will allow time based replication. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

OGKevin · 2020-10-31T12:12:35Z

@bwplotka its ready for 👀!

bwplotka

Looks good generally, some suggestions.

Thanks 👍

CHANGELOG.md

bwplotka · 2020-11-02T14:46:20Z

pkg/replicate/replicator.go

+		fromBkt,
+		"",
+		reg,
+		[]thanosblock.MetadataFilter{thanosblock.NewTimePartitionMetaFilter(*minTime, *maxTime)},


Can we add this only if minTime and maxTime are passed? (:

Wouldn't the behaviour be the same? As the max and min are almost infinite. What is the reason for only passing it if its set to non default value?

Just no risk ;p

pkg/replicate/scheme.go

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

CHANGELOG.md

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

bwplotka

Amazing! Looks good to me 👍

Thanks!

…io#2979) * Add TimePartitionMetaFilter to thanos replicate. This will allow time based replication. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Update docs and changelog. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Fix lint Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Add posibility to delete blocks older than max time Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Fix metric naming. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Fix order of arguments for max and min time. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Remove deletion of blocks from replicator. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Fix import formatting Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Revert unneeded changes due to replicator no longer deleting blocs Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> * Fix typo Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com> Signed-off-by: Oghenebrume50 <raphlbrume@gmail.com>

OGKevin force-pushed the add-time-filter-to-replicate branch 2 times, most recently from 9f52fb1 to cc16483 Compare August 4, 2020 12:54

OGKevin commented Aug 5, 2020

View reviewed changes

pkg/block/metadata/deletionmark.go Outdated Show resolved Hide resolved

OGKevin force-pushed the add-time-filter-to-replicate branch from bdc6443 to 87d81a3 Compare August 5, 2020 11:49

OGKevin marked this pull request as ready for review August 5, 2020 11:55

brancz reviewed Aug 6, 2020

View reviewed changes

cmd/thanos/tools_bucket.go Outdated Show resolved Hide resolved

OGKevin force-pushed the add-time-filter-to-replicate branch from d57faab to e3c36f0 Compare August 6, 2020 12:22

bwplotka requested changes Aug 6, 2020

View reviewed changes

stale bot added the stale label Oct 17, 2020

stale bot removed the stale label Oct 17, 2020

yeya24 mentioned this pull request Oct 20, 2020

tools: Be able to copy blocks of data for a give time-frame from one bucket to another. #3340

Closed

OGKevin mentioned this pull request Oct 30, 2020

Refactor replicate execution method to not iterate the origin bucket #2906

Merged

2 tasks

Add TimePartitionMetaFilter to thanos replicate.

eeb4150

This will allow time based replication. Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

OGKevin force-pushed the add-time-filter-to-replicate branch from e3c36f0 to a4d95b8 Compare October 31, 2020 11:08

OGKevin added 3 commits October 31, 2020 12:13

Update docs and changelog.

1066cc8

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

Fix lint

c8f6b6e

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

Add posibility to delete blocks older than max time

abbf93a

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

OGKevin force-pushed the add-time-filter-to-replicate branch 2 times, most recently from 2d98401 to e5297d7 Compare October 31, 2020 11:20

OGKevin added 2 commits October 31, 2020 12:21

Fix metric naming.

9318d90

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

Fix order of arguments for max and min time.

c18d069

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

OGKevin force-pushed the add-time-filter-to-replicate branch from e5297d7 to c18d069 Compare October 31, 2020 11:22

OGKevin and others added 2 commits October 31, 2020 12:52

Remove deletion of blocks from replicator.

a66b40c

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

Fix import formatting

7c7e09a

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

bwplotka reviewed Nov 2, 2020

View reviewed changes

Revert unneeded changes due to replicator no longer deleting blocs

431021a

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

kakkoyun reviewed Nov 3, 2020

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Fix typo

47073a6

Signed-off-by: Kevin Hellemun <17928966+OGKevin@users.noreply.github.com>

bwplotka approved these changes Nov 3, 2020

View reviewed changes

bwplotka merged commit 3ce844b into thanos-io:master Nov 3, 2020

OGKevin deleted the add-time-filter-to-replicate branch November 3, 2020 22:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REPLICATOR: Add TimePartitionMetaFilter to thanos replicate. #2979

REPLICATOR: Add TimePartitionMetaFilter to thanos replicate. #2979

OGKevin commented Aug 4, 2020 •

edited

Loading

bwplotka left a comment

OGKevin commented Aug 10, 2020

stale bot commented Oct 17, 2020

OGKevin commented Oct 17, 2020

kakkoyun commented Oct 19, 2020

kakkoyun commented Oct 19, 2020

OGKevin commented Oct 31, 2020 •

edited

Loading

bwplotka left a comment •

edited

Loading

bwplotka Nov 2, 2020

OGKevin Nov 2, 2020

bwplotka Nov 3, 2020

bwplotka left a comment

REPLICATOR: Add TimePartitionMetaFilter to thanos replicate. #2979

REPLICATOR: Add TimePartitionMetaFilter to thanos replicate. #2979

Conversation

OGKevin commented Aug 4, 2020 • edited Loading

Changes

Verification

bwplotka left a comment

Choose a reason for hiding this comment

OGKevin commented Aug 10, 2020

stale bot commented Oct 17, 2020

OGKevin commented Oct 17, 2020

kakkoyun commented Oct 19, 2020

kakkoyun commented Oct 19, 2020

OGKevin commented Oct 31, 2020 • edited Loading

bwplotka left a comment • edited Loading

Choose a reason for hiding this comment

bwplotka Nov 2, 2020

Choose a reason for hiding this comment

OGKevin Nov 2, 2020

Choose a reason for hiding this comment

bwplotka Nov 3, 2020

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

OGKevin commented Aug 4, 2020 •

edited

Loading

OGKevin commented Oct 31, 2020 •

edited

Loading

bwplotka left a comment •

edited

Loading