receiver: Split - Upstream the changes from the PR #3580 #3845

yashrsharma44 · 2021-02-26T18:03:08Z

Rebased #3580 with the main branch.

Notes:

There were few issues with the dynamic type checking for error, I am getting the following error -
errs, ok := unwrappedErr.(errutil.MultiError)
unwrappedErr (variable of type error) cannot have dynamic type errutil.MultiError (missing method Error)

Fixed it by creating another method for multierror - Error(). Let me know if the above makes sense.
Would love some reviews on this.
cc @brancz

pkg/receive/multitsdb.go

test/e2e/e2ethanos/services.go

test/e2e/receive_test.go

test/e2e/e2ethanos/services.go

yashrsharma44 · 2021-02-26T19:37:40Z

I have updated the changes, take a look whenever free 🤗 @yeya24

test/e2e/e2ethanos/services.go

cmd/thanos/receive.go

cmd/thanos/receive_route.go

yashrsharma44 · 2021-03-01T12:19:19Z

I wonder if this suggestion is helpful in make lint -

/home/yash/golib/src/github.com/thanos-io/thanos/pkg/route/handler_test.go:234:31: declaration "NewCounterVec" from package "github.com/prometheus/client_golang/prometheus" shouldn't be used, suggested: "github.com/prometheus/client_golang/prometheus/promauto.{NewCounter,NewCounterVec,NewCounterVec,NewGauge,NewGaugeVec,NewGaugeFunc,NewHistorgram,NewHistogramVec,NewSummary,NewSummaryVec}"
/home/yash/golib/src/github.com/thanos-io/thanos/pkg/route/handler_test.go:240:28: declaration "NewCounterVec" from package "github.com/prometheus/client_golang/prometheus" shouldn't be used, suggested: "github.com/prometheus/client_golang/prometheus/promauto.{NewCounter,NewCounterVec,NewCounterVec,NewGauge,NewGaugeVec,NewGaugeFunc,NewHistorgram,NewHistogramVec,NewSummary,NewSummaryVec}"
/home/yash/golib/src/github.com/thanos-io/thanos/pkg/route/handler_test.go:246:33: declaration "NewGauge" from package "github.com/prometheus/client_golang/prometheus" shouldn't be used, suggested: "github.com/prometheus/client_golang/prometheus/promauto.{NewCounter,NewCounterVec,NewCounterVec,NewGauge,NewGaugeVec,NewGaugeFunc,NewHistorgram,NewHistogramVec,NewSummary,NewSummaryVec}"

because a similar issue was reported here after applying the suggestion - prometheus/client_golang#716

yashrsharma44 · 2021-03-02T06:01:13Z

Any idea to get the tests fixed? 🤔 cc @brancz @kakkoyun @yeya24

brancz · 2021-03-02T14:00:07Z

I don't have time for this right now, but what I'm a bit confused about is that

There seem to be commits missing compared to the other PR
The e2e tests were passing on the last state of the other PR

yeya24 · 2021-03-02T15:38:11Z

I will do some investigation today.

cmd/thanos/receive_route.go

yeya24 · 2021-03-03T15:32:09Z

Can you please fix the Circle CI test and linting tests?

kakkoyun

Either we're missing some context on this or we miss certain parts of the code 😅

cmd/thanos/receive_route.go

cmd/thanos/main.go

pkg/errutil/multierror.go

pkg/route/config_test.go

kakkoyun · 2021-03-04T14:28:12Z

pkg/route/error.go

+// determineWriteErrorCause extracts a sentinel error that has occurred more than the given threshold from a given fan-out error.
+// It will inspect the error's cause if the error is a MultiError,
+// It will return cause of each contained error but will not traverse any deeper.
+func determineWriteErrorCause(err error, threshold int) error {


Let's make sure we have parity

thanos/pkg/receive/handler.go

Line 689 in befb025

func determineWriteErrorCause(err error, threshold int) error {

Can you elaborate on this?

pkg/route/hashring.go

cmd/thanos/receive.go

cmd/thanos/main.go

yashrsharma44 · 2021-03-11T08:12:44Z

Fixed the failing tests as we were using MultiError type assertion instead of NonNilMultiError assertion. Now the tests are passing.
cc @kakkoyun @squat can you review if the current changes make sense in accordance to the main branch?

Signed-off-by: Chans321 <tsschand@gmail.com> fixes in e2e test for new receieve architecture - more fixes pending fixes in receive and e2e test of receive Signed-off-by: Chans321 <tsschand@gmail.com> fixed connection reset by peer error in receive e2e tests. all e2e tests pass Signed-off-by: Chans321 <tsschand@gmail.com> pkg/route: Fix unit tests route: Make hashring only work on labels route: Incorporate thanos-io#2899 fixed import issues and changed the query test signature Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> made some corrections while rebasing receiver changes Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> Added a method Error() to implement golang error interface Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> linting Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> use the new promauto package Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed unused params Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> rename method and add comments Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed unused flags in receiver service in e2e tests Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> replaced promauto with prometheus, as earlier was causing unnecessary panic Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> changes for debug Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed entry for receive test for e2e Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> added a readiness probe Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> fix E2E tests for receive router Signed-off-by: yeya24 <yb532204897@gmail.com>

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

Signed-off-by: Chans321 <tsschand@gmail.com> fixes in e2e test for new receieve architecture - more fixes pending fixes in receive and e2e test of receive Signed-off-by: Chans321 <tsschand@gmail.com> fixed connection reset by peer error in receive e2e tests. all e2e tests pass Signed-off-by: Chans321 <tsschand@gmail.com> pkg/route: Fix unit tests route: Make hashring only work on labels route: Incorporate thanos-io#2899 fixed import issues and changed the query test signature Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> made some corrections while rebasing receiver changes Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> Added a method Error() to implement golang error interface Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> linting Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> use the new promauto package Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed unused params Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> rename method and add comments Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed unused flags in receiver service in e2e tests Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> replaced promauto with prometheus, as earlier was causing unnecessary panic Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> changes for debug Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed entry for receive test for e2e Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> added a readiness probe Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> fix E2E tests for receive router Signed-off-by: yeya24 <yb532204897@gmail.com> revert changes for cmd/main Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> nitpicks Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed redundant health check Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> Add receive router registration Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> not to be commited Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> modified pkg/receive/handler.go Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> modified pkg/receive/multitsdb.go Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> modified pkg/route Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> modified cmd/thanos/receive_route.go Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> replaced MultiError with NonNilMultiError Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> changed to promauto Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> removed unused variables Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> changed labelpb.Zlabel to timeseries Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> whitespace removal and revert the labels Signed-off-by: Yash Sharma <yashrsharma44@gmail.com> use the main branch's implementation of MultiError Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

… update-receive

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

kakkoyun

I have just skimmed again and pointed out the major things. I'll review it in depth when I have time.

We should be super considered about merging this. @yashrsharma44 Please check all the subsequent PRs after this branched is created. And also we should have all the necessary facilities for the new subcommand. Moreover I don't see any changes on alerts or dashboard don't we have any changes regarding those parts?

I understand this is just a rebase effort but this could break receiver on master. If there are things missing in the previous PR, these should be documented by an issue or a document so that we can fix all those before the next major release. cc @Chans321

pkg/receive_route/hashring.go

kakkoyun · 2021-03-24T09:04:00Z

docs/components/receive.md

@@ -108,29 +108,6 @@ Flags:
                                 TLS CA to verify clients against. If no client
                                 CA is specified, there is no client
                                 verification on server side. (tls.NoClientCert)
-      --remote-write.address="0.0.0.0:19291"


Yes, we remove these however, where are we adding those? We need to generate documentation for router as well.

yashrsharma44 · 2021-03-30T05:51:37Z

TODO:

Traverse PRs and add anything missing to this PR. - Need to be taken care in this PR
Make sure documentation for the new command exists. - Enable autogeneration of documentation for receive-router
Make sure dashboards and alerts work with the new command. - We can create an issue to track the shortcomings related to dashboards and alerts

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

bwplotka · 2021-04-09T10:33:32Z

As promised, the modification of the proposal is ready: #4036 🤗 PTAL before moving with this PR.

kakkoyun · 2021-05-19T13:13:15Z

Deprecated by #4231

yeya24 reviewed Feb 26, 2021

View reviewed changes

test/e2e/e2ethanos/services.go Outdated Show resolved Hide resolved

cmd/thanos/receive.go Outdated Show resolved Hide resolved

cmd/thanos/receive_route.go Show resolved Hide resolved

kakkoyun changed the title ~~Upstream the changes from the PR #3580~~ receiver: Split - Upstream the changes from the PR #3580 Mar 1, 2021

yashrsharma44 force-pushed the update-receive branch 3 times, most recently from 09022da to 4361243 Compare March 2, 2021 05:21

yashrsharma44 commented Mar 3, 2021

View reviewed changes

cmd/thanos/receive_route.go Show resolved Hide resolved

yashrsharma44 force-pushed the update-receive branch from b35dace to c35d09c Compare March 3, 2021 07:21

kakkoyun reviewed Mar 4, 2021

View reviewed changes

kakkoyun marked this pull request as draft March 4, 2021 14:54

kakkoyun changed the title ~~receiver: Split - Upstream the changes from the PR #3580~~ WIP: receiver: Split - Upstream the changes from the PR #3580 Mar 4, 2021

squat marked this pull request as ready for review March 10, 2021 10:19

yashrsharma44 force-pushed the update-receive branch from c35d09c to c34eac8 Compare March 10, 2021 10:50

yeya24 reviewed Mar 10, 2021

View reviewed changes

cmd/thanos/main.go Outdated Show resolved Hide resolved

yashrsharma44 force-pushed the update-receive branch 3 times, most recently from 47799f5 to d30458e Compare March 11, 2021 05:52

Chans321 and others added 6 commits March 14, 2021 22:08

revert changes for cmd/main

edb6bd7

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

nitpicks

a74f1c4

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

removed redundant health check

7565f96

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

Add receive router registration

83e198c

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

not to be commited

97ae1ee

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

yashrsharma44 added 10 commits March 14, 2021 22:08

modified pkg/receive/handler.go

e79ff16

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

modified pkg/receive/multitsdb.go

760dc0f

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

modified pkg/route

c987ad0

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

modified cmd/thanos/receive_route.go

694b869

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

replaced MultiError with NonNilMultiError

bd90f93

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

changed to promauto

45a9c2c

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

removed unused variables

6a3ac83

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

changed labelpb.Zlabel to timeseries

5d332c5

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

whitespace removal and revert the labels

9893292

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

use the main branch's implementation of MultiError

6e61af6

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

yashrsharma44 force-pushed the update-receive branch from b05c9fd to 6e61af6 Compare March 14, 2021 16:38

yashrsharma44 changed the title ~~WIP: receiver: Split - Upstream the changes from the PR #3580~~ receiver: Split - Upstream the changes from the PR #3580 Mar 14, 2021

metalmatze mentioned this pull request Mar 23, 2021

Probe Statefulset Pods until healthy upon scale up observatorium/thanos-receive-controller#71

Merged

Chans321 and others added 4 commits March 23, 2021 21:49

Merge branch 'update-receive' of github.com:yashrsharma44/thanos into…

9258dbe

… update-receive

Merge branch 'main' into update-receive

8fb03a2

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

renamed route to receive_route

794de1e

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

kakkoyun requested changes Mar 24, 2021

View reviewed changes

yashrsharma44 added 2 commits March 30, 2021 12:16

Merge branch 'main' into update-receive

251893f

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

fixed some regression issues which crept from previous changes

dae3830

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

metalmatze mentioned this pull request Mar 30, 2021

Create benchmarks/receive for load and auto scale testing of receive thanos-io/thanosbench#34

Closed

temp fix

1944f29

Signed-off-by: Yash Sharma <yashrsharma44@gmail.com>

bwplotka mentioned this pull request Apr 9, 2021

Proposal: Modification of Receive Split proposal. #4036

Merged

kakkoyun closed this May 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

receiver: Split - Upstream the changes from the PR #3580 #3845

receiver: Split - Upstream the changes from the PR #3580 #3845

yashrsharma44 commented Feb 26, 2021 •

edited

Loading

yashrsharma44 commented Feb 26, 2021

yashrsharma44 commented Mar 1, 2021

yashrsharma44 commented Mar 2, 2021 •

edited

Loading

brancz commented Mar 2, 2021

yeya24 commented Mar 2, 2021

yeya24 commented Mar 3, 2021

kakkoyun left a comment

kakkoyun Mar 4, 2021

yashrsharma44 Mar 11, 2021

yashrsharma44 commented Mar 11, 2021

kakkoyun left a comment

kakkoyun Mar 24, 2021

yashrsharma44 commented Mar 30, 2021

bwplotka commented Apr 9, 2021 •

edited

Loading

kakkoyun commented May 19, 2021

receiver: Split - Upstream the changes from the PR #3580 #3845

receiver: Split - Upstream the changes from the PR #3580 #3845

Conversation

yashrsharma44 commented Feb 26, 2021 • edited Loading

yashrsharma44 commented Feb 26, 2021

yashrsharma44 commented Mar 1, 2021

yashrsharma44 commented Mar 2, 2021 • edited Loading

brancz commented Mar 2, 2021

yeya24 commented Mar 2, 2021

yeya24 commented Mar 3, 2021

kakkoyun left a comment

Choose a reason for hiding this comment

kakkoyun Mar 4, 2021

Choose a reason for hiding this comment

yashrsharma44 Mar 11, 2021

Choose a reason for hiding this comment

yashrsharma44 commented Mar 11, 2021

kakkoyun left a comment

Choose a reason for hiding this comment

kakkoyun Mar 24, 2021

Choose a reason for hiding this comment

yashrsharma44 commented Mar 30, 2021

bwplotka commented Apr 9, 2021 • edited Loading

kakkoyun commented May 19, 2021

yashrsharma44 commented Feb 26, 2021 •

edited

Loading

yashrsharma44 commented Mar 2, 2021 •

edited

Loading

bwplotka commented Apr 9, 2021 •

edited

Loading