Initial plugin design

[simonw]

The goal of this plugin is to provide a UI for extracting structured data from unstructured text, using the trick described in https://til.simonwillison.net/gpt3/openai-python-functions-data-extraction

Datasette is all about tables, so a plugin which makes it as easy as possible to turn unstructured data into table data makes a ton of sense.

[simonw]

Assorted ideas:

• You can select an existing table to write data to, or you can define a new table
• GPT-assisted schema creation could be available too, including giving it the example data and having it suggest a schema that could make sense
• Sources of data could include:
  • Paste text into a textarea (most common option)
  • Provide a URL and have the plugin fetch the content, strip (most of) the tags and proceed from there
  • Select one or more image, run Tesseract OCR on it (as seen on https://til.simonwillison.net/tools/annotated-presentations) and process the results - no need to upload the images to the server for this
  • PDF files! These do get uploaded to an S3 bucket and then processed by https://github.com/simonw/s3-ocr or equivalent
• Extracted data can be previewed? Though now we have https://datasette.io/plugins/datasette-write-ui that might not be so important any more

[simonw]

Most basic version: you select an existing table (hence avoiding the need to implement a schema editing tool) and paste text into a textarea. I'll build that first.

[simonw]

It's going to need a description for each column - it can guess in some cases, but the option to give it clues will help a lot.

[simonw]

I got this working, but it was really slow... because the OpenAI APIs take a while to stream back all of that JSON.

I had a note about that https://til.simonwillison.net/gpt3/openai-python-functions-data-extraction where I mentioned that maybe ijson could help with that.

So I spent some time and figured out the ijson recipe for it, described in a new TIL: https://til.simonwillison.net/json/ijson-stream

Short version:

events = ijson.sendable_list()
coro = ijson.items_coro(events, "items.item")

seen_events = set()

for chunk in chunks:
    coro.send(chunk.encode("utf-8"))
    if events:
        # Any we have not seen yet?
        unseen_events = [e for e in events if json.dumps(e) not in seen_events]
        if unseen_events:
            for event in unseen_events:
                seen_events.add(json.dumps(event))
                print(json.dumps(event))
                time.sleep(1)