Add example extractors

[yenicelik]

Implemented and tested these extractors:

• Invoice Extractor (using Donut)
• Language Extractor (provided full-text, chunks the text and predicts the language for each chunk)
• Identity Extractor (hashes the input-bytes into a vector using sha256, useful for quick duplicate-checks)

How to reproduce:

(1) Package the extractor into a docker image cargo run extractor package --dev -v --config-path extractors/simple_invoice_parser.yaml
(2) Run the extractor, mounting any files required

docker run \
--mount type=bind,source="$(pwd)/data",target=/indexify/data \ 
-it \
--rm \
-e RUST_LOG=debug \ 
yenicelik/simple-invoice-parser extractor extract \ 
--file data/...

cargo run can be replaced by ./target/debug/indexify or indexify if the binaries were added to the PATH.

For maintainers & contributors:

These steps are usually not needed when working only extractor! Currently, the extractor-base image is pulled from DockerHub, due to how BuildKit works (it does not use a local registry). Unfortunately, it seems tedious to resolve this (see moby/buildkit#2343). If you need to modify the rust code to run the extractors, please run

• create an account for dockerhub
• modify the Makefile credentials at the top of the Makefile
• run make build-base-extractor-push, this will build the image, and push it to dockerhub
• modify dockerfiles/Dockerfiles.extractor to use your own FROM ..., (i.e. FROM yenicelik/indexify-extractor-base instead FROM diptanu/indexify-extractor-base).
• Then you can go to the "How to reproduce" section above

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
indexify	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 22, 2023 1:59pm

[diptanu]

Looks great overall! Thanks David <3
The main comment is to break down the PR into many commits, and move some code around.

[yenicelik]

Please make sure to merge #146 before this

[yenicelik]

Will move the extractors here https://github.com/tensorlakeai/indexify-extractors and remove them from diptanu/indexify