Latency metrics - Time to first token, Tokens per second

[nikcaryo-super]

(Time to first token) can be derived from completion_start_time - start_time -> open for PRs on this or feel free to add it to the ideas section on GH discussions: https://github.com/langfuse/langfuse/discussions
https://discord.com/channels/1111061815649124414/1111063505563562064/1200205414365679819

really useful as latency metrics for Voice AI applications

[clemra]

Hi @nikcaryo-super -- just wanted to let you know that we have now started to work on this and will ship soon. Will ping you once it's live.

[nikcaryo-super]

thank you!

[maxdeichmann]

Hi @nikcaryo-super, both of the metrics are available by now on the generations table :)

[idotr7]

i want to measure time to first token metric, i am using ollama, how do i do it?

[marcklingen]

time to first token is based on start_time and completion_start_time. How do you integrate with Langfuse?

[idotr7]

i use ollama with the low level sdk and want to also measure the ttft.
i use local docker, very convenient, i like to manage my self.

[marcklingen]

In that case, you need to set completion_start_time when creating generations via the low-level sdk. Thereby, you'll see ttft in the UI.

[idotr7]

it still not clear how to use this.
i tried like this:
def stream_response(chat_messages: List[Dict[str, str]], trace):

span = trace.generation(name="response generation", input={"messages": chat_messages})
start_time=time.time()
response_stream = ollama_client.chat(model="qwen2:0.5b", stream=True, messages=chat_messages, keep_alive=-1, options=ollama.Options(num_ctx=32768, f16_kv=True))
response = ""
for response_part in response_stream:
    part_content = response_part['message']['content']
    yield part_content
    response += part_content

span.end(output=response, completion_start_time= time.time()- start_time)
trace.update(output=response)

the result: -1721567355.07s