SBS Markdown mode for dataset runs

[pfurovYnP]

Describe the feature or potential improvement

Sometimes, the best scoring tool is human comparison so being able to inspect what your model generated Side by Side with expected output or output from another run would be helpful. As for now, you can only see previews side-by-side without any markdown support

I think, this evolves into being able to score each generation of each run in the same view as well

Scenario: you have an llm based system that creates telegram posts, by transforming the big input into a summarized post. You are using Langfuse to trace the system but you also want to evaluate it and see how the post would change if you use another version of your prompt.

Additional information

No response

[marliessophie]

Hey @pfurovYnP thanks for your feedback! Very helpful. I'll address your points one-by-one:

• See previews side-by-side WITH markdown support: I'd like to refer you to this issue that tracks our progress on adding markdown support for input/output (e.g. for tracing/datasets). So that should be ready in the near feature
• Scoring generations in dataset run view: you're scenario makes a tone of sense. Am I understanding correctly you'd love to see say a table action for annotation e.g. in the dataset item view, where you can compare an item across runs, and score each corresponding item/linked trace/linked observation directly? Am I correct to assume that in your scenario the resulting key info of interest are the score analytics per dataset run? And you are using human annotation mainly, correct? We're currently building out our annotation tooling so your feedback very much appreciated!

[pfurovYnP]

HI! You stated everything correctly, I have nothing to add. An annotation tooling integrated with langfuse dataset runs sound nice!