Long context and optimizers

[betimd]

Hey there,

We have signature like below:
class InputToScreenSignature(dspy.Signature): """Your task is to get project (name and description as input) and requirements (screen description and fields) and return a json document based on context document provided in the input on markdown format. """ context = dspy.InputField(default=None) project = dspy.InputField() requirements = dspy.InputField() screen_json = dspy.OutputField(desc="answer as json") # key-value pairs

Context is a document around 20 pages long and contains all design construction (json product), structure, components relation and what component could be parent, what could be child, and on, and on.

We are optimizing it using BootstrapFewShotWithRandomSearch optimizer with a considerable dataset (~500 rows) gold (human written) rows.

When used compiled module, and generate screen_json, we usually get half document. Our initial thoughts are that we're hitting max_tokens (4096) and we don’t get a full generation for a screen. If this is a problem (happened to others as well), how do we pass this issue, is there any approach to ask model to continue "part-2" of generation? Or finetuning with whole document doesn't require us to send to forward function full document always (that would minimize max_tokens spent :( ). Or any other idea at all...

I'd really love to hear your feedback and opinions on how to approach this issue!

Thanks in advance!