Document multi-tenant queries.

[jeschkies]

What this PR does / why we need it:
This change adds documentation for multi-tenant queries. It describes the API usage, limitations and possible filters in queries.

Which issue(s) this PR fixes:
Fixes #5993

Special notes for your reviewer:

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[jeschkies]

I wonder whether we should document how to setup a datasource in Grafana for multi-tenant queries. It is not trivial IMHO.

[xom4ek]

That didnt work.
When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that:

[jeschkies]

Inserting. The piping worked for me. Let me investigate. Thanks for trying it out.

[jeschkies]

Ok, I've tried it with the Docker compose setup in tools/dev/loki-boltdb-storage-s3. How did you try it out?

[xom4ek]

analytics:
  reporting_enabled: false
auth_enabled: true
common:
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    s3:
      access_key_id: access
      bucketnames: bucketname
      endpoint: endpoint
      http_config:
        idle_conn_timeout: 90s
        insecure_skip_verify: false
        response_header_timeout: 0s
      region: region
      secret_access_key: secret
compactor:
  shared_store: aws
  working_directory: /var/loki/compactor
frontend:
  log_queries_longer_than: 20s
memberlist:
  join_members:
  - loki-memberlist
querier:
  multi_tenant_queries_enabled: true
schema_config:
  configs:
  - from: "2021-09-07"
    index:
      period: 24h
      prefix: loki_index_
    object_store: aws
    schema: v11
    store: boltdb-shipper
storage_config:
  aws:
    access_key_id: access
    bucketnames: bucketname
    endpoint: endpoint
    http_config:
      idle_conn_timeout: 90s
      insecure_skip_verify: false
      response_header_timeout: 0s
    region: region
    secret_access_key: secret
  boltdb_shipper:
    active_index_directory: /var/loki/index
    cache_location: /var/loki/index_cache
    shared_store: s3
tracing:
  enabled: true

So i found that when use simple-scale helm chart.

If use X-Scope-OrgID='default|system'

If use X-Scope-OrgID='default

[xom4ek]

Seems like on my instance multi-tenant queries didnt work.
I need send __tenant_id__ from log collector? Or loki create that labels selves?

[jeschkies]

What are the screenshots showing? What are tags here? X-Scope-OrgID='default|system' image should be correct on the datasource. However, that also means Promtail is sending logs either as default or system tenant.

[xom4ek]

Let me explain.

When i use Loki datasource and set X-Scope-OrgID=default or X-Scope-OrgID=system - we get that graph:

But when i set X-Scope-OrgID=default|system i get "no data".

So my first screenshots just trace from loki where read service get 0 Series when tenants_ids=[default|system] and 3k when tenants_ids=[default]

About promtail - already have tenants in loki and X-Scope-OrgID without multitenant_query work fine - just in code and your example i see __tenant_id label and didnt see that in my loki installation

[xom4ek]

That didnt work. When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that:

That didnt work correct - loki just take first of that Headers

[jeschkies]

Thanks for your patience. I'm not really sure where you see tenants_ids=[default|system].

Anyhow, which version of Loki are you using? It must be a recent main build after at least commit c7cbeab.

[xom4ek]

Yes - i found problem.

We didnt update master branch after release.
So right now everything fine.
Thanks for help!

[KMiller-Grafana]

Docs LGTM.