Indexing traces into Quickwit with tracegen on i5-13600K

[fmassot]

I indexed traces in Quickwit with tracegen with 4 quickwit nodes started with docker-compose on 1 instance i5-13600K.

It's pretty easy to play with tracegen + Quickwit and docker-compose so I just wanted to share the results and config.

Indexing throughput

I progressively increased the number of spans sent to Quickwit, thus the increasing ramp until reaching the plateau.

Memory consumption

It's the total RSS of the 4 quickwit nodes.

Quickwit config

4 nodes with Docker compose

version: "3"

networks:
  default:
    name: quickwit-network
    ipam:
      config:
      - subnet: 172.16.7.0/24
        gateway: 172.16.7.1

services:
  quickwit-01:
    image: quickwit/quickwit:edge
    networks:
      default:
        aliases:
          - quickwit
    volumes:
      - ./qwdata/indexes:/quickwit/qwdata/indexes
      - ./../config/quickwit.yaml:/quickwit/config/quickwit.yaml
    ports:
      - 7280:7280
    command: ["run"]
    environment:
      - QW_NODE_ID=quickwit-01
      - QW_DISABLE_TELEMETRY=1

  quickwit-02:
    image: quickwit/quickwit:edge
    networks:
      default:
        aliases:
          - quickwit
    volumes:
      - ./qwdata/indexes:/quickwit/qwdata/indexes
      - ./../config/quickwit.yaml:/quickwit/config/quickwit.yaml
    environment:
      - QW_NODE_ID=quickwit-02
      - QW_DISABLE_TELEMETRY=1
      - QW_PEER_SEEDS=quickwit-01
    command: ["run", "--service", "indexer", "--service", "searcher"]

  quickwit-03:
    image: quickwit/quickwit:edge
    networks:
      default:
        aliases:
          - quickwit
    volumes:
      - ./qwdata/indexes:/quickwit/qwdata/indexes
      - ./../config/quickwit.yaml:/quickwit/config/quickwit.yaml
    environment:
      - QW_NODE_ID=quickwit-03
      - QW_DISABLE_TELEMETRY=1
      - QW_PEER_SEEDS=quickwit-01,quickwit-02
    command: ["run", "--service", "indexer", "--service", "searcher"]

  quickwit-04:
    image: quickwit/quickwit:edge
    networks:
      default:
        aliases:
          - quickwit
    volumes:
      - ./qwdata/indexes:/quickwit/qwdata/indexes
      - ./../config/quickwit.yaml:/quickwit/config/quickwit.yaml
    environment:
      - QW_NODE_ID=quickwit-04
      - QW_DISABLE_TELEMETRY=1
      - QW_PEER_SEEDS=quickwit-01,quickwit-02,quickwit-03
    command: ["run", "--service", "indexer", "--service", "searcher"]

Trace gen config

For each node in [quickwit-01, quickwit-02, quickwit-03, quickwit-04], execute the command:

docker run -e OTEL_EXPORTER_OTLP_ENDPOINT=http://${NODE_ID}:7281 --net quickwit-network jaegertracing/jaeger-tracegen -trace-exporter otlp-grpc -service quickwit-test -traces 100000000 -workers 20 -spans 100

Index config

version: 0.6

index_id: otel-traces-v0_6

doc_mapping:
  mode: strict
  field_mappings:
    - name: trace_id
      type: bytes
      fast: true
    - name: trace_state
      type: text
      indexed: false
    - name: service_name
      type: text
      tokenizer: raw
    - name: resource_attributes
      type: json
      tokenizer: raw
    - name: resource_dropped_attributes_count
      type: u64
      indexed: false
    - name: scope_name
      type: text
      indexed: false
    - name: scope_version
      type: text
      indexed: false
    - name: scope_attributes
      type: json
      indexed: false
    - name: scope_dropped_attributes_count
      type: u64
      indexed: false
    - name: span_id
      type: bytes
    - name: span_kind
      type: u64
    - name: span_name
      type: text
      tokenizer: raw
    - name: span_fingerprint
      type: text
      tokenizer: raw
    - name: span_start_timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
      indexed: false
      fast: true
      fast_precision: milliseconds
    - name: span_end_timestamp_nanos
      type: datetime
      input_formats: [unix_timestamp]
      output_format: unix_timestamp_nanos
      indexed: false
      fast: false
    - name: span_duration_millis
      type: u64
      indexed: false
      fast: true
      stored: false
    - name: span_attributes
      type: json
      tokenizer: raw
    - name: span_dropped_attributes_count
      type: u64
      indexed: false
    - name: span_dropped_events_count
      type: u64
      indexed: false
    - name: span_dropped_links_count
      type: u64
      indexed: false
    - name: span_status
      type: json
      indexed: true
    - name: parent_span_id
      type: bytes
      indexed: false
    - name: events
      type: array<json>
      tokenizer: raw
    - name: event_names
      type: array<text>
      tokenizer: default
      record: position
      stored: false
    - name: links
      type: array<json>
      tokenizer: raw

  timestamp_field: span_start_timestamp_nanos

  # partition_key: hash_mod(service_name, 100)
  # tag_fields: [service_name]

indexing_settings:
  commit_timeout_secs: 30
  split_num_docs_target: 3000000
  merge_policy:
    type: no_merge

search_settings:
  default_search_fields: []

Node config

version: 0.6

ingest_api:
  max_queue_memory_usage: 4G
  max_queue_disk_usage: 8G