Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logcli remote storage. #1814

Merged
merged 4 commits into from
Mar 30, 2020
Merged

Conversation

cyriltovena
Copy link
Contributor

This allows to query a remote storage (eg. bigtable/gcs) using the logcli. It requires to provide a loki configuration files.

example of usage:

logcli query --org-id=3927 --local-config=/Users/ctovena/local-gcs.yaml {container_name="ingress-nginx"}

Example of configuration:

chunk_store_config:
limits_config:
  enforce_metric_name: false
  ingestion_burst_size_mb: 20
  ingestion_rate_mb: 10
  ingestion_rate_strategy: global
  max_global_streams_per_user: 10000
  max_query_length: 12000h
  max_query_parallelism: 32
  max_streams_per_user: 0
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
    - from: "2018-04-15"
      index:
        period: 168h
        prefix: loki_dev_index_
      object_store: gcs
      schema: v9
      store: bigtable
    - from: "2019-10-30"
      index:
        period: 168h
        prefix: loki_dev_index_
      object_store: gcs
      schema: v11
      store: bigtable
storage_config:
  bigtable:
    instance: dev-us-central1
    project: dev
  gcs:
    bucket_name:  dev
  max_chunk_batch_size: 50

This is very handy to debug store queries.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Copy link
Contributor

@sandeepsukhani sandeepsukhani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small suggestion otherwise it LGTM.

@@ -169,6 +169,7 @@ func newQuery(instant bool, cmd *kingpin.CmdClause) *query.Query {
cmd.Flag("exclude-label", "Exclude labels given the provided key during output.").StringsVar(&query.IgnoreLabelsKey)
cmd.Flag("include-label", "Include labels given the provided key during output.").StringsVar(&query.ShowLabelsKey)
cmd.Flag("labels-length", "Set a fixed padding to labels").Default("0").IntVar(&query.FixedLabelsLen)
cmd.Flag("local-config", "Execute the current query using a configured storage from a given Loki configuration file.").Default("").StringVar(&query.LocalConfig)
Copy link
Contributor

@sandeepsukhani sandeepsukhani Mar 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

local-config sounds a little confusing. Should we call it storage-config or store-config instead?
What do you think?

@cyriltovena
Copy link
Contributor Author

Good idea !

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena cyriltovena merged commit ba51aad into grafana:master Mar 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants