Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While looking through @sydney-runkle's work on execution limits, I became pretty strongly convinced that this type should be called/thought of as
Usage
rather thanCost
, and I wanted to rename it.Some more specific reasons:
I think it makes more sense to describe the method that returns this struct as "Return the usage of the request" than "Return the cost of the request" when we are just getting token counts and not a monetary value. The word "cost" just feels like it should be referring to money.
Right now, the docstring of the Cost class says:
I think the fact that we need to use quotes in
"token count" costs
is representative that it's not obvious/expected that cost isn't money, and if we call this classUsage
, we can avoid that phrase"token count" costs
entirely. We can just say "you'll need to look up the documentation of the model you're using to convert usage into monetary costs".Usage
provides a clear place to track request count (which we want to add limits to), and it doesn't feel at all out of place weird to add that field to a type calledUsage
in the way it kind of does feel weird to add it to a type calledCost
. (Though even if we keep the name asCost
, I want to add request count to that structure rather than store usage state on something else..)Anthropic uses the term "usage" for everything related to token counts — see here https://github.com/anthropics/anthropic-sdk-python#token-counting. (Sydney pointed this out to me after I suggested it; I wanted to use that term independently/before I realized Anthropic did too.) I see this as confirmation that this name is a reasonable choice, and using it also has the benefit of matching existing LLM SDK conventions (i.e., anthropic's).
I'm going to open some subsequent PRs implementing the ExecutionLimit stuff based on @sydney-runkle's work on top of this, I just wanted to make this its own PR to keep it easier to review diffs.