Update global state from content

[diptanu]

Certain structured extraction tasks are related to maintaining a global state as content is ingested, such as summary of chat messages, emails threads, or update code snippets which are used by an agent or another extractor for structured extraction and planning. For example, Evaporate uses a two step process for structured extraction, the second step prompts the LLM to synthesize code by prompting it documents and important attributes of the doc.

Make Indexify manage global state derived from ingested content, with sampling functions to filter content. So instead of writing extracted information as rows or columns in the underlying storage, it should also have the concept of updating an artifact.

Extractor(Content, State) -> State^