How to control request size within a batch?

[cringelord000222]

Hi all,

I've tried --max-batch-size command however it doesn't work as I've expected, I thought it was supposed to limit tgi_batch_current_size. I'd like to control the queue size & how many requests per inference batch. Can someone please clarify on this?

These are my launch commands and version:

# from docker compose file

image: ghcr.io/huggingface/text-generation-inference:2.0.4
command: --model-id /models/meta-llama/Meta-Llama-3-70B-Instruct --quantize bitsandbytes-nf4 --num-shard 2 --max-input-tokens 5000 --max-total-tokens 6000 --max-batch-size 10

My situation:

1. Concurrently sent 40 requests to TGI server, --max-batch-size 10
tgi_batch_current_size goes over 10 after a short while

# from /metrics

tgi_request_count 40
tgi_request_success 15
tgi_batch_inference_count{method="prefill"} 15
tgi_batch_inference_count{method="decode"} 559
tgi_batch_concat{reason="backpressure"} 7
tgi_batch_concat{reason="wait_exceeded"} 7
tgi_batch_inference_success{method="prefill"} 15
tgi_batch_inference_success{method="decode"} 558
tgi_batch_current_max_tokens 65664
tgi_queue_size 13
tgi_batch_current_size 12

Expected situation:

1. Concurrently sent 40 requests to TGI server, --max-batch-size 10
tgi_batch_current_size stays at 10, and the other 30 request will stay in tgi_queue_size until its done.

TLDR: tgi_batch_current_size doesn't align with --max-batch-size.

Thanks in advance.