Use the log10 CLI to list and download the completions, feedback, and feedback task data at log10.io.
Here's a demo video.
Install the log10-io python package (version >= 0.6.7) and setup Log10
$ pip install 'log10-io[cli]'You can list all your completions using log10 completions list. In addition, you can filter the completions by tag names (--tags) and created date --from and --to. For instance, here is the command to filter the completions with two tag names foo and bar and created between 2024-02-01 and 2024-02-29
$ log10 completions list --tags foo,bar --from 2024-2-1 --to 2024-2-29Filter with tags: foo,bar
Filter with created date: 2024-02-01 to 2024-02-29
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ ID ┃ Status ┃ Created At ┃ Prompt ┃ Completion ┃ Tags ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ 497974e8-c1ed-4de7-90ab-8f104eb870te │ success │ 18 days ago │ You are a ping pong machine. │ │ bar, foo │
│ │ │ │ Ping? │ Pong! │ │
│ │ │ │ │ │ │
│ e8e4eaf5-871b-4e1d-8688-1234071aca1c │ success │ 18 days ago │ Where is the Eiffel Tower? │ The Eiffel Tower is located in Paris, ... │ bar, foo │
│ │ │ │ │ │ │
│ eb5ee140-6d93-4908-837f-9872345f1677 │ success │ 19 days ago │ You are a ping pong machine. │ Pong! │ bar, foo │
│ │ │ │ Ping? │ │ │
│ │ │ │ │ │ │
│ 98250940-bd7c-4b1d-9834-12b8a2fdec73 │ success │ 19 days ago │ Where is the Eiffel Tower? │ The Eiffel Tower is located in Paris, ... │ bar, foo │
│ │ │ │ │ │ │
│ b922d686-4aae-42ef-815e-60987sdfge16 │ success │ 19 days ago │ Where is the Eiffel Tower? │ The Eiffel Tower is located in Paris, ... │ bar, foo │
│ │ │ │ │ │ │
└──────────────────────────────────────┴─────────┴─────────────┴──────────────────────────────┴───────────────────────────────────────────┴──────────┘
total_completions=5You can download these completions into a JSONL file by using log10 completions download with the same options and specify the output file --file:
$ log10 completions download --tags foo,bar --from 2024-2-1 --to 2024-2-29 --file foo_bar.jsonl
Filter with tags: foo,bar
Filter with created date: 2024-02-01 to 2024-02-29
Download total completions: 5/5
Do you want to continue? [y/N]: y
100%|███████████████████████████████████████████████████████████████████████████| 5/5 [00:03<00:00, 1.31it/s]To retrieve details for a specific completion, use log10 completions get.
For instance,
$ log10 completions get --id 497974e8-c1ed-4de7-90ab-8f104eb870teoutput (only showing part of the full raw output):
{
"id": "497974e8-c1ed-4de7-90ab-8f104eb870te",
"created_at": "2024-02-23T19:37:17.709536+00:00",
"status": "finished",
"duration": 384,
"kind": "completion",
"request": {
"prompt": [
"You are a ping pong machine.\nPing?\n"
],
"model": "gpt-3.5-turbo-instruct",
"temperature": 0.5,
},
"response": {
"choices": [
{
"finish_reason": "stop",
"index": 0,
"text": "\nPong!"
}
],
}
}
You can load completions' prompt messages and compare with other LLM models by using log10 completions benchmark_models.
For instance,
log10 completions benchmark_models --ids 25572f3c-c2f1-45b0-9de8-d96be4c4e544 --models=gpt-3.5-turbo,mistral-small-latest,claude-3-haiku-20240307output
Running gpt-3.5-turbo
Running mistral-small-latest
Running claude-3-haiku-20240307
completion_id: 25572f3c-c2f1-45b0-9de8-d96be4c4e544
original_request:
{
"model": "gpt-4-0125-preview",
"messages": [
{
"role": "system",
"content": "Summarize the article in 30 words."
},
{
"role": "user",
"content": "\"Story of Your Life\" is a science fiction novella by American writer Ted Chiang, first published in Starlight 2 in 1998, and in 2002 in Chiang's collection of short stories, Stories of Your Life and Others. Its major themes are language and determinism. \"Story of Your Life\" won the 2000 Nebula Award for Best Novella, as well as the 1999 Theodore Sturgeon Award. It was nominated for the 1999 Hugo Award for Best Novella. The novella has been translated into Italian, Japanese, French and German.[1] A film adaptation of the story, Arrival, was conceived and adapted by Eric Heisserer. Titled and directed by Denis Villeneuve, it was released in 2016. It stars Amy Adams, Jeremy Renner, and Forest Whitaker and was nominated for eight Academy Awards, including Best Picture and Best Adapted Screenplay; it won the award for Best Sound Editing.[2][3][4] The film also won the 2017 Ray Bradbury Award for Outstanding Dramatic Presentation and the Hugo Award for Best Dramatic Presentation."
}
],
"temperature": 0.2
}
╭─────────────────────────┬───────────────────────────────────────────────────────┬──────────────────────────────────┬───────────────╮
│ Model │ Content │ Total Token Usage (Input/Output) │ Duration (ms) │
├─────────────────────────┼───────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────┤
│ gpt-4-0125-preview │ "Story of Your Life" by Ted Chiang explores language │ 323 (255/68) │ 2527 │
│ │ and determinism, winning the 2000 Nebula and 1999 │ │ │
│ │ Theodore Sturgeon Awards. Adapted into the film │ │ │
│ │ "Arrival" by Denis Villeneuve in 2016, it received │ │ │
│ │ critical acclaim and multiple awards, including an │ │ │
│ │ Academy Award for Best Sound Editing. │ │ │
├─────────────────────────┼───────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────┤
│ gpt-3.5-turbo │ "Story of Your Life" is a science fiction novella by │ 295 (255/40) │ 2345 │
│ │ Ted Chiang, exploring themes of language and │ │ │
│ │ determinism. It won awards and was adapted into the │ │ │
│ │ film Arrival in 2016. │ │ │
├─────────────────────────┼───────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────┤
│ mistral-small-latest │ "Story of Your Life" is a Ted Chiang novella │ 342 (282/60) │ 2087 │
│ │ exploring language and determinism, winning Nebula │ │ │
│ │ and Sturgeon Awards. It was adapted into the 2016 │ │ │
│ │ film "Arrival," which received multiple Academy Award │ │ │
│ │ nominations and won for Best Sound Editing. │ │ │
├─────────────────────────┼───────────────────────────────────────────────────────┼──────────────────────────────────┼───────────────┤
│ claude-3-haiku-20240307 │ "Story of Your Life" is a science fiction novella by │ 320 (274/46) │ 1944 │
│ │ Ted Chiang, exploring themes of language and │ │ │
│ │ determinism, winning multiple awards and inspiring a │ │ │
│ │ film adaptation, Arrival, which was critically │ │ │
│ │ acclaimed. │ │ │
╰─────────────────────────┴───────────────────────────────────────────────────────┴──────────────────────────────────┴───────────────╯
You can also filter the completions by tags and save the results using --file or -f.
Specify the output file using .md for a markdown report, .csv for comma-separated values, or .jsonl for JSON Lines format.
And run our prompt analyzer (auto-prompt) using --analyze_prompt.
To start adding feedback, first you need to define a feedback task with log10 feedback-task create. Then you can add feedback to a logged completions with log10 feedback create. For more details, you can read more in log10's user documentation.
To list all feedback tasks, use log10 feedback-task list
$ log10 feedback-task list
Feedback Tasks
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ID ┃ Created At ┃ Name ┃ Required ┃ Instruction ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 04405cbc-3420-4901-97b6 │ 5 days ago │ Demo TLDR Dataset Summary grading w/ note │ coherence, accuracy, coverage, overall │ │
│ 15a5e099-a56a-49d0-b488 │ 29 days ago │ emoji_feedback_task │ feedback │ Provide feedback using emojis │
└─────────────────────────┴─────────────┴────────────────────────────────────────────┴────────────────────────────────────────────┴───────────────────────────────────────────────────┘and retrieve details about a specific task with log10 feedback-task get --id
To list and download your current feedback, use log10 feedback list and log10 feedback download.
For instance you can list all feedback filtered by a feedback task --task_id:
$ log10 feedback list --task_id 04405cbc-3420-4901-97b6
Feedback
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ID ┃ Task Name ┃ Feedback ┃ Completion ID ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 4f5fefd1-06a4-415e-b56f │ Demo TLDR Dataset Summary grading w/ note │ {"note": "sounds like the girl is really into OP and OP is wary about committing too easily │ 0b5bc07c-a976-49cc-bd160 │
│ │ │ -which is absolutely not the right interpretation.", "overall": 2, "accuracy": 3, "coverage": 2, │ │
│ │ │ "coherence": 7} │ │
│ 5525bf49-a870-465a-b6b7 │ Demo TLDR Dataset Summary grading w/ note │ {"note": "The summary conveys the main idea of the post.", "overall": 7, "accuracy": 7, │ 431ee175-3a37-436a-bd627 │
│ │ │ "coverage": 7, "coherence": 7} │ │
│ 9874ca7b-1c01-4785-8331 │ Demo TLDR Dataset Summary grading w/ note │ {"note": "•explicit purpose statement will make summary perfect. ", "overall": 6, "accuracy": 7, │ 447c3b69-6aea-4f4d-95846 │
│ │ │ "coverage": 6, "coherence": 7} │ │
│ ad7dd317-3a19-4633-bf80 │ Demo TLDR Dataset Summary grading w/ note │ {"note": "Missing details of why.", "overall": 4, "accuracy": 7, "coverage": 4, "coherence": 7} │ 1d2396bf-df44-4eaf-a9f03 │
│ 4b5edcb1-ca79-42a8-a0f5 │ Demo TLDR Dataset Summary grading w/ note │ {"note": "Should mention they're male.", "overall": 6, "accuracy": 7, "coverage": 6, │ 264a3ca1-7bcc-4679-b80eb │
│ │ │ "coherence": 7} │ │
└─────────────────────────┴───────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────┘And to download to a JSONL file, use log10 feedback download --task_id 04405cbc-3420-4901-97b6 --file feedback.jsonl
To leverage your feedback and use AI to generate more feedback automatically, use log10 feedback predict. Please refer to this doc for more details.
To get auto generated feedback for a completion, use log10 feedback autofeedback get
$ log10 --help
Usage: log10 [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
auto-prompt Analyze prompts and messages to get suggestions
completions Manage logs from completions i.e.
feedback Manage feedback for completions i.e.
feedback-task Manage tasks for feedback i.e.$ log10 auto-prompt --help
Usage: log10 auto-prompt [OPTIONS] COMMAND [ARGS]...
Analyze prompts and messages to get suggestions
Options:
--help Show this message and exit.
Commands:
analyze Analyze a prompt or messages and provide suggestions on how to improve it. $ log10 auto-prompt analyze --help
Usage: log10 auto-prompt analyze [OPTIONS]
Analyze a prompt or messages and provide suggestions on how to improve it.
Options:
-p, --prompt TEXT The prompt to analyze. Provide a string or a file
containing the prompt. We allow three formats: 1) string
prompt, e.g. "Summarize this article in 3 sentences." 2)
messages, e.g. [{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi"}] 3) log10
completion, e.g. {..., "request": {..., "messages":
[{"role": "user", "content": "Hello"}, {"role":
"assistant", "content": "Hi"}], ...}, "response": {...}}
The prompt input could be a string or a file path. If
it's a file path, read the file.$ log10 completions --help
Usage: log10 completions [OPTIONS] COMMAND [ARGS]...
Manage logs from completions i.e. logs from users
Options:
--help Show this message and exit.
Commands:
benchmark_models Compare completions using different models and...
download Download completions to a jsonl file
get Get a completion by id
list List completionslog10 completions benchmark_models --help
Usage: log10 completions benchmark_models [OPTIONS]
Compare completions using different models and generate report
Options:
--ids TEXT Log10 completion IDs. Provide a comma-separated list
of completion IDs or a path to a JSON file containing
the list of IDs.
--tags TEXT Filter completions by specific tags. Separate multiple
tags with commas.
--limit TEXT Specify the maximum number of completions to retrieve
filtered by tags.
--offset TEXT Set the starting point (offset) from where to begin
fetching completions filtered by tags.
--models TEXT Comma separated list of models to compare
--temperature FLOAT Temperature
--max_tokens INTEGER Max tokens
--top_p FLOAT Top p
--analyze_prompt Run prompt analyzer on the messages.
-f, --file FILE Specify the filename to save the results. Specify the
output file using `.md` for a markdown report, `.csv`
for comma-separated values, or `.jsonl` for JSON Lines
format. Only .md, .csv, and .jsonl extensions are
supported.$ log10 completions download --help
Usage: log10 completions download [OPTIONS]
Download completions to a jsonl file
Options:
--limit TEXT Specify the maximum number of completions to
retrieve.
--offset TEXT Set the starting point (offset) from where
to begin fetching completions.
--timeout INTEGER Set the maximum time (in seconds) allowed
for the HTTP request to complete.
--tags TEXT Filter completions by specific tags.
Separate multiple tags with commas.
--from [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
Define the start date for fetching
completions (inclusive). Use the format:
YYYY-MM-DD.
--to [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
Set the end date for fetching completions
(inclusive). Use the format: YYYY-MM-DD.
-f, --file TEXT Specify the filename and path for the output
file.$ log10 completions get --help
Usage: log10 completions get [OPTIONS]
Get a completion by id
Options:
--id TEXT Completion ID$ log10 completions list --help
Usage: log10 completions list [OPTIONS]
List completions
Options:
--limit INTEGER Specify the maximum number of completions to
retrieve.
--offset INTEGER Set the starting point (offset) from where
to begin fetching completions.
--timeout INTEGER Set the maximum time (in seconds) allowed
for the HTTP request to complete.
--tags TEXT Filter completions by specific tags.
Separate multiple tags with commas.
--from [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
Define the start date for fetching
completions (inclusive). Use the format:
YYYY-MM-DD.
--to [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S]
Set the end date for fetching completions
(inclusive). Use the format: YYYY-MM-DD.$ log10 feedback --help
Usage: log10 feedback [OPTIONS] COMMAND [ARGS]...
Manage feedback for completions i.e. capturing feedback from users
Commands:
create Add feedback to a group of completions associated with a task
download Download feedback based on the provided criteria.
get Get feedback based on provided ID.
list List feedback based on the provided criteria.
predict$ log10 feedback create --help
Usage: log10 feedback create [OPTIONS]
Add feedback to a group of completions associated with a task
Options:
--task_id TEXT Task ID
--values TEXT Feedback in JSON format
--completion_tags_selector TEXT
Completion tags selector
--comment TEXT Comment$ log10 feedback download --help
Usage: log10 feedback download [OPTIONS]
Download feedback based on the provided criteria. This command allows
fetching feedback for a specific task or across all tasks, with control over
the starting point and the number of items to retrieve.
Options:
--offset INTEGER The starting index from which to begin the feedback fetch.
Defaults to 0.
--limit INTEGER The maximum number of feedback items to retrieve. Defaults
to 25.
--task_id TEXT The specific Task ID to filter feedback. If not provided,
feedback for all tasks will be fetched.
--filter TEXT The filter applied to the feedback. If not provided,
feedback will not be filtered. e.g. `log10 feedback list
--filter 'Coverage <= 5'`.
-f, --file FILE Path to the file where the feedback will be saved. The
feedback data is saved in JSON Lines (jsonl) format. If
not specified, feedback will be printed to stdout.$ log10 feedback get --help
Usage: log10 feedback get [OPTIONS]
Get feedback based on provided ID.
Options:
--id TEXT Get feedback by ID$ log10 feedback list --help
Usage: log10 feedback list [OPTIONS]
List feedback based on the provided criteria. This command allows fetching
feedback for a specific task or across all tasks, with control over the
starting point and the number of items to retrieve.
Options:
--offset INTEGER The starting index from which to begin the feedback fetch.
Defaults to 0.
--limit INTEGER The maximum number of feedback items to retrieve. Defaults
to 25.
--task_id TEXT The specific Task ID to filter feedback. If not provided,
feedback for all tasks will be fetched.
--filter TEXT The filter applied to the feedback. If not provided,
feedback will not be filtered. e.g. `log10 feedback list
--filter 'Coverage <= 5'`.$ log10 feedback predict --help
Usage: log10 feedback predict [OPTIONS]
Generate feedback with existing human feedback based on in context learning
Options:
--task_id TEXT Feedback task ID
--content TEXT Completion content
-f, --file TEXT File containing completion content
--completion_id TEXT Completion ID
--num_samples INTEGER Number of samples to use for few-shot learning$ log10 feedback autofeedback get --help
Usage: log10 feedback autofeedback get [OPTIONS]
Get an auto feedback by completion id
Options:
--completion-id TEXT Completion ID [required]$ log10 feedback-task --help
Usage: log10 feedback-task [OPTIONS] COMMAND [ARGS]...
Manage tasks for feedback i.e. instructions and schema for feedback
Commands:
create
get
list$ log10 feedback-task create --help
Usage: log10 feedback-task create [OPTIONS]
Options:
--name TEXT Name of the task
--task_schema TEXT Task schema
--instruction TEXT Task instruction
--completion_tags_selector TEXT Completion tags selector$ log10 feedback-task get --help
Usage: log10 feedback-task get [OPTIONS]
Options:
--id TEXT Get feedback task by ID
--help Show this message and exit.$ log10 feedback-task list --help
Usage: log10 feedback-task list [OPTIONS]
Options:
--limit INTEGER Number of feedback tasks to fetch
--offset INTEGER Offset for the feedback tasks