Script to figure out model sequence length limits #1579
Conversation
|
Can we make this a scenario that we run officially (kind of like a unit test) rather than a one-off script? |
|
Sure, I could look into that. Fox now, since @yifanmai needs this (I think) to fix the canary runs. Should we keep it as a script and then I could make it into a scenario in a different PR? |
|
This can't be a scenario because scenarios cannot access the tokenizer. |
yifanmai
changed the title
scripts/compute_request_limits.py
Outdated
|
|
||
| # model_name, tokenizer_name, prefix and suffix are passed as arguments | ||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--model_name", type=str, default="writer/palmyra-base") |
There was a problem hiding this comment.
Optional suggestion: helm-run and helm-summarize use hyphens for flags; you could considering doing that here for consistency. Note that argparse will autoconvert field names to use underscore e.g. args.model_name.
There was a problem hiding this comment.
Not sure I understand this, I will just skip it for now
| return lower_bound + max_prompt_length | ||
|
|
||
|
|
||
| def check_limits( |
There was a problem hiding this comment.
Is this just for double-checking i.e. if the measurement was performed correctly, should this method always return that the limits are correct??
There was a problem hiding this comment.
Normally it should, it's was just for me to check that my implementation was correct and that I did not have a +1 problem with my binary search
This script is used to find the limits required by the window service