Multi-prometheus proxy (mpp) exists to forward incoming query requests to one of a set of multiple prometheus instances deployed as HA duplicates of each other.
As the recommended pattern for running Prometheus in HA mode is to run duplicate instances (same configuration, scraping the same targets independently), a method is needed to route queries appropriately between those instances to provide a seemless experience for clients when individual instance failures occur.
MPP acts as a proxy sitting in front of multiple prometheus instances, choosing one or more instances to receive requests based on a configurable selector strategy. Candidate instances are found using a configured locator mechanism, of which Marathon, Kubernetes and endpoints file are supported.
Request are buffered and retried (up to 2 times), and a re-selection is performed in process after the first failure for a given request--which should result in seemless failover, assuming a viable backend candidate is available.
Prometheus endpoints can be discovered via the Marathon API, the Kubernetes API, or by providing an endpoints file which is scanned on a regular interval.
Marathon discovery is configured using:
--marathon-url: The marathon API endpoint to contact--marathon-apps: A comma-separated list of apps to query for endpoints--marathon-principal-secret: (optional) A DCOS principal-secret object used to authenticate to Marathon--marathon-auth-endpoint: (optional) Overrides the auth-endpoint contained within the principal secret object
Kubernetes discovery is configured using:
--kubeconfig: (optional) The path to the kubeconfig file used to locate the cluster and authenticate; if not specified, in-cluster config will be used--kube-namespace: The namespace in which the pods/endpoints exist--kube-service-name: The name of a service whose endpoints will be used to locate prometheus--kube-pod-label-selector: A pod-selector string used to locate the pods containing the endpoints--kube-port: The port for pods/endpoints on which prometheus is lisetening; if not specified, the first TCP port found will be used
Endpoints file discovery is configured using:
--endpoints-file-- the path to a file containing one endpoint per line.
Traffic is routed based on the chosen --routing-strategy:
-
single-most-data: This strategy always routes traffic to a single prometheus endpoint, determined by whichever endpoint contains the most data, measured by total ingested samples count. -
minimum-history:{min-history-duration}: This strategy routes traffic to a randomly selected prometheus endpoint having at least M of sample history. -
random: This strategy routes traffic to a randomly selected prometheus endpoint.
Stickiness is configured via the --affinity-options flag, which accepts a comma-separated
list of one or more of the options:
cookies: (endabled by default) Sets a cookie on the first request and uses that in subsequent requests to route to the same backend.sourceip: Examines theX-Forwarded-Forheader, falling back to the actual source IP; the chosen route is stored in an LRU-based cache for that value.
The proxy responds with OK to requests on the /mpp/health path.
Prometheus-formatted metrics are available at the /mpp/metrics path.
NAME:
mpp - Launch a dynamically configured proxy over multiple prometheus endpoints
which selects endpoints based on configurable criteria.
USAGE:
mpp [global options] command [command options] [arguments...]
VERSION:
v0.2.0-a2
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--kubeconfig value The path to a kubeconfig file used to communicate with the kubernetes api server
to locate prometheus instances [$MPP_KUBECONFIG]
--kube-service-name value The service name used to locate prometheus endpoints; take precedence over 'kube-pod-label-selector' [$MPP_SERVICE_NAME]
--kube-pod-label-selector value The label selector used to find prometheus pods [$MPP_KUBE_POD_LABEL_SELECTOR]
--kube-namespace value The namespace in which prometheus pods/endpoints exist [$MPP_KUBE_NAMESPACE]
--kube-port value The port (name or number) where prometheus is listening on individual pods/endpoints [$MPP_KUBE_PORT]
--marathon-url value The URL for the marathon API endpoint used to locate prometheus instances [$MPP_MARATHON_URL]
--marathon-apps value A comma-separated list of marathon app IDs whose tasks will be queried for
prometheus endpoints [$MPP_MARATHON_APPS]
--insecure value, -k value Whether connections to https endpoints with unverifiable certs are allowed [$MPP_INSECURE_CERTS]
--marathon-principal-secret value The principal secret used to handle authentication with marathon [$MPP_MARATHON_PRINCIPAL_SECRET]
--marathon-auth-endpoint value The authentication endpoint to use with the 'marathon-principal-secret', overriding the value
contained within the secret [$MPP_MARATHON_AUTH_ENDPOINT]
--endpoints-file value A file path containing a list of endpoints to use, one per line. This file is re-read at every
selection interval [$MPP_ENDPOINTS_FILE]
--routing-strategy value The strategy to use for choosing viable prometheus enpoint(s) from those located;
valid choices include: 'single-most-data', 'random', 'minimum-history' (default: "single-most-data") [$MPP_ROUTING_STRATEGY]
--selection-interval value The interval at which selections are performed; note that selection is
automatically performed upon backend failures (default: "10s") [$MPP_SELECTION_INTERVAL]
--affinity-options value A comma-separated list of sticky-session modes to enable, of which 'cookies', and 'sourceip'
are valid options (default: "cookies") [$MPP_AFFINITY_OPTIONS]
--port value The port on which the proxy will listen (default: 9090) [$MPP_PORT]
--verbose, -V Log debugging information [$MPP_VERBOSE]
--help, -h show help
--version, -v print the version
The proxy displays a status summary (see below) on the /mpp/status path.
