Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat sicedar: added readiness prober #1395

Merged
merged 7 commits into from
Aug 28, 2019
Merged

feat sicedar: added readiness prober #1395

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3b2d411
feat sicedar: added readiness prober
FUSAKLA Aug 10, 2019
9193da3
cr: inline the component usage
FUSAKLA Aug 14, 2019
cdc30f4
cr: refactor sidecar prober logic
FUSAKLA Aug 14, 2019
153cd3f
CR: revert to original PR logic wothout touching sidecar liveness
FUSAKLA Aug 19, 2019
d351d03
CR: removed readiness handling from periodical prometheus health checks
FUSAKLA Aug 20, 2019
60897a8
fix prober: fixed false error logging
FUSAKLA Aug 27, 2019
210075a
CR: fix prober comment to work with godoc
FUSAKLA Aug 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel

- [#1358](https://github.com/thanos-io/thanos/pull/1358) Added `part_size` configuration option for HTTP multipart requests minimum part size for S3 storage type
- [#1363](https://github.com/thanos-io/thanos/pull/1363) Thanos Receive now exposes `thanos_receive_hashring_nodes` and `thanos_receive_hashring_tenants` metrics to monitor status of hash-rings
- [#1395](https://github.com/thanos-io/thanos/pull/1395) Added `/-/ready` and `/-/healthy` endpoints to Thanos sidecar.

### Changed

Expand Down
2 changes: 1 addition & 1 deletion cmd/thanos/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func main() {
tracingConfig := regCommonTracingFlags(app)

cmds := map[string]setupFunc{}
registerSidecar(cmds, app, "sidecar")
registerSidecar(cmds, app)
registerStore(cmds, app, "store")
registerQuery(cmds, app, "query")
registerRule(cmds, app, "rule")
Expand Down
29 changes: 21 additions & 8 deletions cmd/thanos/sidecar.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ import (
"sync"
"time"

"github.com/thanos-io/thanos/pkg/prober"

FUSAKLA marked this conversation as resolved.
Show resolved Hide resolved
"github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/oklog/run"
opentracing "github.com/opentracing/opentracing-go"
"github.com/opentracing/opentracing-go"
"github.com/pkg/errors"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/common/model"
Expand All @@ -26,13 +28,14 @@ import (
"github.com/thanos-io/thanos/pkg/store"
"github.com/thanos-io/thanos/pkg/store/storepb"
"google.golang.org/grpc"
kingpin "gopkg.in/alecthomas/kingpin.v2"
"gopkg.in/alecthomas/kingpin.v2"
)

const waitForExternalLabelsTimeout = 10 * time.Minute

func registerSidecar(m map[string]setupFunc, app *kingpin.Application, name string) {
cmd := app.Command(name, "sidecar for Prometheus server")
func registerSidecar(m map[string]setupFunc, app *kingpin.Application) {
comp := component.Sidecar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would maybe inline those, would be more verbose.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I inlined it in the registerSidecar function, not sure if you want to inline it everywhere even in the runSidecar ?

cmd := app.Command(comp.String(), "sidecar for Prometheus server")

grpcBindAddr, httpBindAddr, cert, key, clientCA := regCommonServerFlags(cmd)

Expand All @@ -54,7 +57,7 @@ func registerSidecar(m map[string]setupFunc, app *kingpin.Application, name stri

uploadCompacted := cmd.Flag("shipper.upload-compacted", "[Experimental] If true sidecar will try to upload compacted blocks as well. Useful for migration purposes. Works only if compaction is disabled on Prometheus.").Default("false").Hidden().Bool()

m[name] = func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ bool) error {
m[comp.String()] = func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ bool) error {
rl := reloader.New(
log.With(logger, "component", "reloader"),
reloader.ReloadURLFromBase(*promURL),
Expand All @@ -77,6 +80,7 @@ func registerSidecar(m map[string]setupFunc, app *kingpin.Application, name stri
objStoreConfig,
rl,
*uploadCompacted,
comp,
)
}
}
Expand All @@ -96,6 +100,7 @@ func runSidecar(
objStoreConfig *pathOrContent,
reloader *reloader.Reloader,
uploadCompacted bool,
comp component.Component,
) error {
var m = &promMetadata{
promURL: promURL,
Expand All @@ -117,6 +122,12 @@ func runSidecar(
uploads = false
}

readinessProber := prober.NewProber(comp, logger, prometheus.WrapRegistererWithPrefix("thanos_", reg))
FUSAKLA marked this conversation as resolved.
Show resolved Hide resolved
// Initiate default HTTP listener providing metrics endpoint and readiness/liveness probes.
if err := defaultHTTPListener(g, logger, reg, httpBindAddr, readinessProber); err != nil {
return errors.Wrap(err, "create readiness prober")
}

// Setup all the concurrent groups.
{
promUp := prometheus.NewGauge(prometheus.GaugeOpts{
Expand Down Expand Up @@ -148,6 +159,7 @@ func runSidecar(
"err", err,
)
promUp.Set(0)
readinessProber.SetNotReady(err)
return err
}

Expand All @@ -156,6 +168,7 @@ func runSidecar(
"external_labels", m.Labels().String(),
)
promUp.Set(1)
readinessProber.SetReady()
lastHeartbeat.Set(float64(time.Now().UnixNano()) / 1e9)
return nil
})
Expand All @@ -176,8 +189,10 @@ func runSidecar(
if err := m.UpdateLabels(iterCtx, logger); err != nil {
level.Warn(logger).Log("msg", "heartbeat failed", "err", err)
promUp.Set(0)
readinessProber.SetNotReady(err)
FUSAKLA marked this conversation as resolved.
Show resolved Hide resolved
} else {
promUp.Set(1)
readinessProber.SetReady()
lastHeartbeat.Set(float64(time.Now().UnixNano()) / 1e9)
}

Expand All @@ -195,9 +210,7 @@ func runSidecar(
cancel()
})
}
if err := metricHTTPListenGroup(g, logger, reg, httpBindAddr); err != nil {
return err
}

{
l, err := net.Listen("tcp", grpcBindAddr)
if err != nil {
Expand Down
8 changes: 8 additions & 0 deletions tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar-lts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ spec:
containerPort: 10902
- name: grpc
containerPort: 10901
livenessProbe:
httpGet:
port: 10902
path: /-/healthy
readinessProbe:
httpGet:
port: 10902
path: /-/ready
volumeMounts:
- name: prometheus
mountPath: /var/prometheus
Expand Down
8 changes: 8 additions & 0 deletions tutorials/kubernetes-demo/manifests/prometheus-ha-sidecar.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,14 @@ spec:
containerPort: 10902
- name: grpc
containerPort: 10901
livenessProbe:
httpGet:
port: 10902
path: /-/healthy
readinessProbe:
httpGet:
port: 10902
path: /-/ready
volumeMounts:
- name: prometheus
mountPath: /var/prometheus
Expand Down