Generator streamlining, docs #682
Conversation
…nerate() return would/would not be filtered for None that was predicated on whether a model supported multiple outputs
leondz
added
documentation
| #. Otherwise, we need to assemble the outputs over multiple calls. There are two options here. | ||
| #. Is garak running with ``parallel_attempts > 1`` configured? In that case, start a multiprocessing pool with as many workers as the value of ``parallel_attempts``, and have each one of these work on building the required number of generations, in any order. | ||
| #. Otherwise, call ``_call_model()`` repeatedly to collect the requested number of generations. | ||
| #. Strip ``None`` responses from the outputs, and return the resulting list of prompt responses. |
There was a problem hiding this comment.
This is the current practice, however I am interested in the reasoning for this. Is there value in knowing that call returned None, in theory we can determine this happened from inspection of configuration, however I wonder if a None response might be valuable in some probes patterns. Consider if a model is setup to mitigate certain responses by simply closing the connection. In practice this is not a good pattern as a 200 response with a rejection is the common response, yet I can akin it to the idea that if you call someone to ask a question they might simply hang up the phone.
There was a problem hiding this comment.
This was a great time to ask this question. None is intended to indicate that generation failed, so would definitely be good for the situation you mention. Perhaps then generate() should not filter out Nones at all. But also - perhaps when a model fails when given a blank prompt, it should return None, instead of exception handling and a string. I think that we're in a better place if any str-typed output from a generator is an indication that the model successfully output a string, and that this output is the string component of the output. Wdyt?
cf.: #689 (comment)
Added generator docs
Made streamlining changes to
generators.base.GeneratorlogicResolves #662