[Neuron] Adding support for context-lenght, token-gen buckets.#7885

Merged

simon-mo merged 1 commit intovllm-project:mainfrom hbikki:neuron-bucketing

Aug 29, 2024

+33-11

Commits on Aug 28, 2024

[Neuron] Adding support for context-lenght, token-gen buckets for latency optimization with neuron device.
Harsha Bikki
committed