Quickwit performance over large index

[mzupan]

I have the following index

Number of published documents:
119,144,410,710
Size of published documents (uncompressed):
313,456,623.89 MB
Number of published splits:
9999
Size of published splits:
33,896,928.36 MB

The index mapping is

{
  "field_mappings": [
    {
      "name": "timestamp",
      "type": "datetime",
      "fast": true,
      "fast_precision": "seconds",
      "indexed": true,
      "input_formats": [
        "iso8601"
      ],
      "output_format": "unix_timestamp_secs",
      "stored": true
    },
    {
      "name": "message",
      "type": "text",
      "fast": false,
      "fieldnorms": false,
      "indexed": true,
      "record": "position",
      "stored": true,
      "tokenizer": "default"
    }
  ],
  "tag_fields": [],
  "store_source": false,
  "index_field_presence": false,
  "timestamp_field": "timestamp",
  "mode": "dynamic",
  "dynamic_mapping": {
    "indexed": true,
    "tokenizer": "raw",
    "record": "basic",
    "stored": true,
    "expand_dots": true,
    "fast": {
      "normalizer": "raw"
    }
  },
  "max_num_partitions": 200,
  "tokenizers": []
}

Right now its all kubernetes logs so looking to do a lot of needle/haystack type searches in it like as follows

cluster:nameHere message:user@gmail.com

My grafana query looks like

{
  "request": {
    "url": "api/ds/query?ds_type=quickwit-quickwit-datasource&requestId=explore_66g",
    "method": "POST",
    "data": {
      "queries": [
        {
          "refId": "A",
          "datasource": {
            "type": "quickwit-quickwit-datasource",
            "uid": "a278d508-3294-473a-934d-65f398cb9b3e"
          },
          "query": "message:\"user@gmail.com\"",
          "alias": "",
          "metrics": [
            {
              "type": "logs",
              "id": "3",
              "settings": {
                "limit": "100",
                "sortDirection": "desc"
              }
            }
          ],
          "bucketAggs": [
            {
              "type": "date_histogram",
              "id": "2",
              "settings": {
                "interval": "auto"
              },
              "field": ""
            }
          ],
          "timeField": "",
          "datasourceId": 32,
          "intervalMs": 900000,
          "maxDataPoints": 1906
        }
      ],
      "from": "1721631600000",
      "to": "1723273199000"
    },
    "hideFromInspector": false
  },
  "response": {
    "message": "An error occurred within the plugin",
    "messageId": "plugin.downstreamError",
    "statusCode": 500,
    "traceID": ""
  }
}

I'd be fine with like 2 week search if possible but right now its OOMing a search node each time I do it

searcher:
  aggregation_memory_limit: 2G
  fast_field_cache_capacity: 4G
  split_footer_cache_capacity: 1500M
  partial_request_cache_capacity: 256M
  split_cache:
    max_num_bytes: 2G
    max_num_splits: 20000
    num_concurrent_downloads: 2

Each searcher node right now is 10Gi of Ram

[fulmicoton]

We will work on limiting the memory usage. In the meanwhile, as a workaround, you can reduce your memory usage by reducing
max_num_concurrent_split_searches in the searcher config.
https://quickwit.io/docs/configuration/node-config#searcher-configuration

[mzupan]

@fulmicoton i'm not too worried about memory usage its mostly a needle/haystack search timeout in like 8minutes and wondering if there's anything I can do to speed it up or searching over 200tbs is just not doable or tuning the index batter to handle it.

I'll toss 40gig of memory at the pods if that fixes the issue

[mzupan]

Update on this.. it completes outside of grafana in 45s which I feel is fine for that amount of data over a 2w search

└─> curl -w "@curl-format.txt" -X POST http://127.0.0.1:7280/api/v1/logs/search \
  -d '{"query": "foobar@gmail.com AND 17709", "start_timestamp": 1722554566, "end_timestamp": 1723764166}' \
  -H "Content-Type: application/json"
{
  "num_hits": 0,
  "hits": [],
  "elapsed_time_micros": 44831462,
  "errors": []
}     

time_namelookup:  0.000146s
        time_connect:  0.000405s
     time_appconnect:  0.000000s
    time_pretransfer:  0.000449s
       time_redirect:  0.000000s
  time_starttransfer:  45.096754s
                     ----------
          time_total:  45.096998s

Not sure why it timesout in grafana

[fulmicoton]

The query is very simple but this is still 313TB of uncompressed data. 45s is not so bad if you only have a few searchers. How many searcher node/vCPUs per node do you have?

[fmassot]

Note: Grafana runs (at least) two queries in parallel: one for log volume and one for logs.

[mzupan]

@fulmicoton

24 searchers...

    Limits:
      memory:  24Gi
    Requests:
      cpu:      2
      memory:   24Gi

i'm honestly fine with 45s over that type query

@fmassot in grafana there doesn't seem to be a way to turn it off.. and it doesn't seem to be honoring the timeout. I have it set for 300s but its timing out in 60s

These two queries always timeout in a minute

{"queries":[{"refId":"A","datasource":{"type":"quickwit-quickwit-datasource","uid":"a278d508-3294-473a-934d-65f398cb9b3e"},"query":"message:\"import\" AND message:\"4493429\"","alias":"","metrics":[{"type":"logs","id":"3","settings":{"limit":"100","sortDirection":"desc"}}],"bucketAggs":[{"type":"date_histogram","id":"2","settings":{"interval":"auto"},"field":""}],"timeField":"","datasourceId":32,"intervalMs":300000,"maxDataPoints":1914}],"from":"1723209983498","to":"1723814783498"}

{"queries":[{"refId":"log-volume-A","query":"message:\"import\" AND message:\"4493429\"","metrics":[{"type":"count","id":"1"}],"bucketAggs":[{"id":"3","type":"date_histogram","settings":{"interval":"auto","min_doc_count":"0","trimEdges":"0"},"field":""}],"datasource":{"type":"quickwit-quickwit-datasource","uid":"a278d508-3294-473a-934d-65f398cb9b3e"},"datasourceId":32,"intervalMs":3600000,"maxDataPoints":1914}],"from":"1723209983498","to":"1723814783498"}

My grafana datasource is this

[mzupan]

looking over this more how its handled in loki is loki also seems to run 1 query for the data and another for the graph

I could be wrong but it seems the grafana issue seems to be quickwit running both queries at the same time. Loki seems to get the data first and once the data is populated it runs another query to generate the graph