* add consul_exporter integration

* update changelog

* add consul to integrations test env

* s/statsd/consul

* Add support to skip TLS verification for Tempo exporter

This will bump the opentelemetry-collector to v.0.16.0 which now
supports insecure_skip_verify for TLS. This can be configured under the
push_config to be passed to the otel collector.

* Run go mod tidy to fix go.mod/go.sum

* Vendor dependencies after update go.mod

* Document Opentemetry-collector bump in CHANGELOG

* docs: Add insecure_skip_verify to tempo_config section

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* override Loki User-Agent header

Fixes grafana#188

* update CHANGELOG

* revert hacky workaround, no longer needed now that fix in loki is merged

* use prometheus version for setting user-agent header

This avoids the same problem that Loki originally had and allows
downstream importers that use prometheus/common/version to import
the loki package without it overwriting their build info.

* Added config environment expansion

* Implemented optional env-var config substitution using CLI flag

* Update pkg/config/config.go

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

Closes grafana#279

* grafanacloud-specific install script

* remove empty comment

* Update cmd/agentctl/main.go

Co-authored-by: Evan <10603766+hoenn@users.noreply.github.com>

* appease the linter

* Update production/grafanacloud-install.sh

Co-authored-by: Jack Baldry <jack.baldry@grafana.com>

* Update production/grafanacloud-install.sh

Co-authored-by: Jack Baldry <jack.baldry@grafana.com>

* Update production/grafanacloud-install.sh

Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>

* review feedback

* agentctl: change to consistent command function naming

* print success message after installing/starting agent service

Co-authored-by: Evan <10603766+hoenn@users.noreply.github.com>
Co-authored-by: Jack Baldry <jack.baldry@grafana.com>
Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>

* Update ADOPTERS.md

Signed-off-by: yeya24 <yb532204897@gmail.com>

* prepare for 0.9.1 release

* add v0.9.1 entry to changelog

* add dynamic integration registration system

This commit adds a dynamic integration registration system similar to
(and mostly copied from) Prometheus' SD registration mechanism.

* fix lint error

* fix runtime errors

* refactor integrations

1. Integration interface moved to integrations
2. Integration Config interface now responsible for providing
   name/config

* marshal manager config with embedded integrations

* Fixing issue 206

Added mapping for listen address so that it propagates correctly and is honored by the http server.

* Added changelog doc

* Added additional comments on changelog

* Clean up formatting

* Changed host to pointer and updated tests to support that

* Cleaned up spacing

* update process exporter to v0.7.5

* update depcheck entry and CHANGELOG

* don't mount /var/log as readonly

* update phrasing in changelog to explain what the issue was

Remove WAL directories that are no longer associated with a `ManagedInstance`
that this agent is responsible for. The storage directories used by active
instances are compared to all subdirectories under the `wal_directory` root.
Any that are not associated with an instance and that haven't been modified
in over a configured amount of time are removed.

How often the check is run and the cutoff for how recently a WAL must have
been modified are controlled by the `wal_cleanup_period` and
`wal_cleanup_age` settings respectively, under the top-level Prometheus
configuration. When omitted, they default to a period of 30 minutes and
recently modified threshold of 12 hours.

Fixes grafana#132

* Use a newer libsystemd in Docker containers

Corresponds to grafana/loki#2957

* changelog

* merge apt-get commands to create less layers

Having the warning prefix every version is getting silly. I've
created grafana#317 to track the issue and added a consolidated warning
at the top of the CHANGELOG.

…afana#322)

* add sigv4 as a custom authentication method for remote_write data

* address review feedback i gave myself

* require region

* docs, changelog

* make region requirements more clear

* Explicitly specify API server Service name so DNS resolves to IP matching cert.

* Update changelog

* add server_name to jsonnet

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* support ARMv6 linux builds

* Only zip files starting with `agent`

Also fixes bug where the deployment scrape config was included with host
filtering enabled, where the inverse should've been true.

* allow specifying installation namespace in script

* specify ARN and add to ServiceAccount annotation

* fix NAMESPACE typo

* add namespace to serviceaccount

* add region

* update changelog, docs

* Configuration file validation subcommand for agentctl

Perform basic YAML and type checking for an Agent configuration file
and exit with `1` if there are any errors loading, parsing, or validating
the file.

Fixes grafana#329

* CR feedback

* prepare for v0.11.0 release

* fix `make agentctl` and `make agentctl-image`

* Register integrations in agentctl

* Update changelog

* Changelog typo

* Update CHANGELOG.md

* build agentctl with CGO_ENABLED=1

importing integrations requires CGO_ENABLED=1 on macOS. Making this
change also requires installing libsystemd-dev in the agentctl Docker
containers, as CGO_ENABLED=1 will cause promtail to include the systemd
tailer.

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Make CollectorIntegration more flexible

Refactor the CollectorIntegartion to support multiple collectors and an
optional runner function (not used in this commit).

This changes will be used in the oncoming ElasticSearch integration.

* ElasticSearch exporter integration

Vendored github.com/justwatchcom/elasticsearch_exporter

This will be used in conjunction with grafana/jsonnet-libs#427

* Remove `es.` prefixes from ElasicSearch configs

And make consistent usage of underscores in the yaml keys.

* Fix the docs, add elasticsearch_exporter reference

* Remove unneeded `_ =` in elasticsearch_exporter

* Credits to @colega for elasticsearch_exporter

* Consistent ES exporter config with other exporters

Move the init(), use pointer receivers.

* Add elasticsearch_expoter example

Also renamed elasticsearch `uri` config key to `address`.

* Add elasticsearch_exporter to depcheck.yml

* only reference elasticsearch_exporter once in depcheck file

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* support other architectures for debs/rpms

* fix tests

* check for RELEASE_TAG when making and testing packages

* include discovered labels in /agent/api/v1/targets

* fix indentation, leaked print line debug

Added clarifying rules from grafana#316 (comment) for the default script config to scrape pods.

prerequisite for grafana#353

* Lint scripts with ShellCheck.

* Lint all scripts.

* Document shell type and correct all errors.

* Correct build.sh

Upstream PR has been merged, we can use it directly now

…rafana#353)

* pkg/loki: support multiple instances of promtail

* fix build, lint errors

* docs, fix configs

* hard break

* changelog

* create positions directory if it doesn't exist

* fix bugs from testing

* remove test extra loki config

* update remaining docs to use new loki config

* add more detail to the migration guide, config reference

* fix loki port

…fana#365)

* support multiple tempo instances

* add warning about port overrides

* fix tabs in yaml config

* remove test configs

* allow failed agents in ring

also registers ring metrics when scraping service mode is enabled

* changelog

includes grafana/loki#3284 and grafana/loki#3246

* try to fix CI flakiness

* setup qemu

* undo temporary changes

…nts (grafana#382)

* scraping service: improve sharding mechanism

Sharding will now be performed based on the name of the config file
instead of the contents of the config file. This guarantees that in a
static ring, the same config name will always hash to the same Agent.
This improves guarantees of the hashing, but also ensures that changing
a config will not cause it to move to another Agent and cause gaps in
metrics as the scrape responsibility moves.

* fix lint error

…ipt (grafana#369)

* update go for builds to 1.15.7

* override seego image

* changelog

* update CI to build seego override

* transparently build seego in Makefile targets

* fix dist-packages job

* remove buildx options from custom seego image

* grafana/agent/seego doesn't work in buildx

* update to go 1.15.8

* go mod vendor

…rafana#396)

* allow relying on the default value of the -addr flag in config-sync

Closes grafana#395

* changelog

* fix: Remove unnecessary tls_config and bearer_token_file from kubernetes-pods scrape job

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* fix: Remove unnecessary tls_config and bearer_token_file from kube-state-metrics scrape config

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* fix: Remove unnecessary tls_config and bearer_token_file from node_exporter scrape config

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

* fix: Remove unnecessary tls and bearer tokens

Signed-off-by: Jack Baldry <jack.baldry@grafana.com>

…afana#401)

* reduce likelihood of data loss when remote_write has an outage

* take out old config block

* s/some/some new

* allow specifying wal truncate frequencies per integration

* rebase and fix default time in docs

* s/master/main in docs

* add branch name change message to CHANGELOG

* update CI

* infer sigv4 region from environment

* document test

* aws.StringValue

* update prometheus to 2.25.0

* changelog

* update docs to include new headers field in remote_write

…grafana#417)

* stop scraping service instances on shutdown

This is needed for the config reloading, but is already handled today:
when the overall Agent stop, it stops all intances, including ones
retrieved via the scraping service. However, if _just_ the scraping
service was stopped, then its instances would continue running in the
background. This fixes that problem.

* log error when failing to delete config on sharding instance manager shutdown

* s/error/err for consistency

* Implement ApplyConfig for Loki

* add integration test for ApplyConfig

* fix lint, race errors, map deletion

* make windows test happy

* fixes a conflict in go mod tidy by explicitly excluding the package we don't want

* fix indentation

* speed up test-packages job

Only builds amd64 packages for test-packages, speeding up the job.

* combine build and test jobs

* sync up build flags for testing and building

* implement ApplyConfig for tempo

* add integration tests, fix bugs in ApplyConfig from testing

* i before e except after c

* fix test failure

…ana#425)

…ations (grafana#418)

* Added the ability to specify certificates so that integrations can be server and read from HTTPS.

* Add servername as a required field.

* Add configuration comments and move the configuration around.

* Update changelog

* Add comment

* Fix Tests

* Clean up imports with goimports

* Update configuration-reference.md

* Move checks to separate function for cleanup.

* Move config into two separate locations due to conflicts and update doc.

* Clean up config doc and fix tests.

* Remove unneeded err and cleanup

* Remove unneeded comments

* Feedback from PR and fixing of Sever

* Update pkg/integrations/manager.go

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* Moved check inside the check for integrations enabled.

* Update linter to most recent to avoid mlock issue.

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* prepare for 0.13.0 release

* Update CHANGELOG.md

Co-authored-by: mattdurham <mattdurham@ppog.org>

* Expose OTLP exporter compression setting

* Update changelog

* Change default compression to gzip and 'none' to disable

* Updated to use UnmarshalYAML for defaults and validation

* Update tests, add new test for no compression

* tweak organization of applying defaults a little

* move CHANGELOG entry to unreleased after rebase

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* pkg/prom/instance: create a ModalManager for ApplyConfig

ApplyConfig for the pkg/prom will need to be able to mutate the instance
grouping mode and other settings. This PR enables that by doing the
following:

1. Create a ModalManager that wraps around a GroupManager or a (new)
   DistinctManager. The ModalManager can change modes at the cost of
   needing to re-apply all instances.

2. Remove CountingManager and migrate its logic to ModalManager

3. Separate out storage of the ModalManager and BasicManager to be able
   to control both independently

* remove DistinctManager, use underlying manager directly

* simplify GroupManager.Stop

* removed unused logger from GroupManager

* document active/wrapped managers a bit better in ModalManager

* Fix panic running GCA on an M1 Mac

* changelog

* go mod tidy / go mod vendor

* add support for M1

* Changes to allow config to follow hierarchy

* Follow config hierCarchy

* Align yaml

* Remove unnecessary check for nil

* Fix for arguments in tests.

* Clean up based on feedback and fixed lint issue.

* Go fmt

* Documentation updated and changelog

* fmt of Yaml

* Make defaultRemoteWrite name consistent

* Make defaultRemoteWrite name consistent

A remote_write timestamp of 0 preventing truncations from occurring.
Ignoring 0s was an unnecessary safety measure, and only caused problems
when considering the min/max WAL sample lifetime.

Closes grafana#409.

* remove sharding intance manager

* add test to ensure sharding behaves properly

* handle watch events properly

* add test for unowned config in watchKV

* add comments based on feedback

)

* store text in codec instead of instance.Config

* it's ok to store default values that never change across process lifetime

* don't need to force the name from the client anymore

* remove superflous comment

…#442)

this commit reduces the number of places defaults are applied to
instance configs:

1. When the Agent YAML is unmarshaled
2. When the instance config is loaded from the KV store

* initial commit of a configstore

* review feedback

* add HTTP api for configstore

matches functionalities of HTTP API used for Scraping Service

* fix errors from rebase

* Checkin for windows service support

* Update documentation

* Add documentation

* Fix linting errors

* Fix error return issues

* Fix go.mod and some naming cleanup

* Remove windows documentation

* Fix modules.txt

* Run go mod vendor

* Fix modules.txt v2

* Update cmd/agent/main.go

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* Update cmd/agent/service_windows.go

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* Checking in changes from PR review

* Fix formatting

* Resolving PR comments:
- Renaming AgentServer to Entrypoint
- Using channels to convey errors and teardown the server
- Other minor cleanups

* Switch up channels so they are handled by the caller.

* Fix args

* Fix naming issues from PR

Co-authored-by: Robert Fratto <robertfratto@gmail.com>

* wire in configstore into HA code

* register API metrics

Fix the syntax in the docs used for environment variable substitution.

* rename Grafana Cloud Agent to Grafana Agent

* update changelog to mention project rename

* add use cases of the grafana agent to readme

* "telemetry collector", link to Prometheus

* initial commit of pkg/prom/cluster package

* add tests

* fix test / race condition problems

* address review feedback

gRPC service can't be registered after calling Serve on the gRPC server.
Since Serve was being called in a separate goroutine, it was a race
condition for whether it would fail.

* initial commit

* add tests

* address review feedback