You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Depending on how slow tokenization is, we could tokenize one document at a time and check whether throwing away that document saves enough tokens.
The call to count_openai_tokens(text=doc.content, tokenizer=self._tokenizer) would need to go into the same loop as n_skipped_tokens += doc_token_len. What do you think? It's not the focus of this PR so maybe this improvement could become a separate issue?
Yeah, I think this is a good idea. This would avoid a user accidentally sending 100+ docs to the PromptNode or AnswerGenerator and wonder what is taking so long. But this PR is already fairly large so I think opening a separate issue for this would be a good idea.
We're tokenizing all documents here:
haystack/haystack/nodes/answer_generator/openai.py
Lines 316 to 317 in c3a38a5
and here:
haystack/haystack/nodes/answer_generator/openai.py
Line 307 in c3a38a5
Depending on how slow tokenization is, we could tokenize one document at a time and check whether throwing away that document saves enough tokens.
The call to
count_openai_tokens(text=doc.content, tokenizer=self._tokenizer)
would need to go into the same loop asn_skipped_tokens += doc_token_len
. What do you think? It's not the focus of this PR so maybe this improvement could become a separate issue?Originally posted by @julian-risch in #4179 (comment)
Yeah, I think this is a good idea. This would avoid a user accidentally sending 100+ docs to the PromptNode or AnswerGenerator and wonder what is taking so long. But this PR is already fairly large so I think opening a separate issue for this would be a good idea.
Originally posted by @sjrl in #4179 (comment)
The text was updated successfully, but these errors were encountered: