Tools for running OCR against files stored in S3
Background on this project: s3-ocr: Extract text from PDF files stored in an S3 bucket
Install this tool using pip:
pip install s3-ocr
You can see the results of running this tool against three PDFs from the Internet Archive (one, two, three) in this example table hosted using Datasette.
The start command takes a list of keys and submits them to Textract for OCR processing.
You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using s3-credentials.
You can start the process running like this:
s3-ocr start name-of-your-bucket my-pdf-file.pdf
The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:
s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf
OCR can take some time. The results of the OCR will be stored in textract-output in your bucket.
To process every file in the bucket with a .pdf extension use --all:
s3-ocr start name-of-bucket --all
To process every file with a .pdf extension within a specific folder, use --prefix:
s3-ocr start name-of-bucket --prefix path/to/folder
Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...
Start OCR tasks for PDF files in an S3 bucket
s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf
To process every file with a .pdf extension:
s3-ocr start name-of-bucket --all
To process every .pdf in the PUBLIC/ folder:
s3-ocr start name-of-bucket --prefix PUBLIC/
Options:
--all Process all PDF files in the bucket
--prefix TEXT Process all PDF files within this prefix
--dry-run Show what this would do, but don't actually do it
--no-retry Don't retry failed requests
--access-key TEXT AWS access key ID
--secret-key TEXT AWS secret access key
--session-token TEXT AWS session token
--endpoint-url TEXT Custom endpoint URL
-a, --auth FILENAME Path to JSON/INI file containing credentials
--help Show this message and exit.
The s3-ocr status <bucket-name> command shows a rough indication of progress through the tasks:
% s3-ocr status sfms-history
153 complete out of 532 jobs
It compares the jobs that have been submitted, based on .s3-ocr.json files, to the jobs that have their results written to the textract-output/ folder.
Usage: s3-ocr status [OPTIONS] BUCKET
Show status of OCR jobs for a bucket
Options:
--access-key ...
The s3-ocr inspect-job <job_id> command can be used to check the status of a specific job ID:
% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
{
"DocumentMetadata": {
"Pages": 583
},
"JobStatus": "SUCCEEDED",
"DetectDocumentTextModelVersion": "1.0"
}
Usage: s3-ocr inspect-job [OPTIONS] JOB_ID
Show the current status of an OCR job
s3-ocr inspect-job <job_id>
Options:
--access-key ...
Once an OCR job has completed you can download the resulting JSON using the fetch command:
s3-ocr fetch name-of-bucket path/to/file.pdf
This will save files in the current directory with names like this:
4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json
The number of files will vary depending on the length of the document.
If you don't want separate files you can combine them together using the -c/--combine option:
s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json
The output.json file will then contain data that looks something like this:
{
"Blocks": [
{
"BlockType": "PAGE",
"Geometry": {...}
"Page": 1,
...
},
{
"BlockType": "LINE",
"Page": 1,
...
"Text": "Barry",
},
Usage: s3-ocr fetch [OPTIONS] BUCKET KEY
Fetch the OCR results for a specified file
s3-ocr fetch name-of-bucket path/to/key.pdf
This will save files in the current directory called things like
a806e67e504fc15f...48314e-1.json a806e67e504fc15f...48314e-2.json
To combine these together into a single JSON file with a specified name, use:
s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json
Use "--output -" to print the combined JSON to standard output instead.
Options:
-c, --combine FILENAME Write combined JSON to file
--access-key ...
If you don't want to deal with the JSON directly, you can use the text command to retrieve just the text extracted from a PDF:
s3-ocr text name-of-bucket path/to/file.pdf
This will output plain text to standard output.
To save that to a file, use this:
s3-ocr text name-of-bucket path/to/file.pdf > text.txt
Separate pages will be separated by three newlines. To separate them using a ---- horizontal divider instead add --divider:
s3-ocr text name-of-bucket path/to/file.pdf --divider
Usage: s3-ocr text [OPTIONS] BUCKET KEY
Retrieve the text from an OCRd PDF file
s3-ocr text name-of-bucket path/to/key.pdf
Options:
--divider Add ---- between pages
--access-key ...
If you move files around within your S3 bucket s3-ocr can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run s3-ocr start against those new files.
The s3-ocr dedupe command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the ETag for each file, which represents the MD5 hash of the file contents.
The command will write out new .s3ocr.json files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run s3-ocr start.
s3-ocr dedupe name-of-bucket
Add --dry-run for a preview of the changes that will be made to your bucket.
Usage: s3-ocr dedupe [OPTIONS] BUCKET
Scan every file in the bucket checking for duplicates - files that have not
yet been OCRd but that have the same contents (based on ETag) as a file that
HAS been OCRd.
s3-ocr dedupe name-of-bucket
Options:
--dry-run Show output without writing anything to S3
--access-key ...
To keep track of which files have been submitted for processing, s3-ocr will create a JSON file for every file that it adds to the OCR queue.
This file will be called:
path-to-file/name-of-file.pdf.s3-ocr.json
Each of these JSON files contains data that looks like this:
{
"job_id": "a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe",
"etag": "\"b0c77472e15500347ebf46032a454e8e\""
}The recorded job_id can be used later to associate the file with the results of the OCR task in textract-output/.
The etag is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.
This design for the tool, with the .s3-ocr.json files tracking jobs that have been submitted, means that it is safe to run s3-ocr start against the same bucket multiple times without the risk of starting duplicate OCR jobs.
The s3-ocr index <bucket> <database_file> command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:
% s3-ocr index sfms-history index.db
Fetching job details [####################################] 100%
Populating pages table [####################----------------] 55% 00:03:18
The schema of the resulting database looks like this (excluding the FTS tables):
CREATE TABLE [pages] (
[path] TEXT,
[page] INTEGER,
[folder] TEXT,
[text] TEXT,
PRIMARY KEY ([path], [page])
);
CREATE TABLE [ocr_jobs] (
[key] TEXT PRIMARY KEY,
[job_id] TEXT,
[etag] TEXT,
[s3_ocr_etag] TEXT
);
CREATE TABLE [fetched_jobs] (
[job_id] TEXT PRIMARY KEY
);The database is designed to be used with Datasette.
Usage: s3-ocr index [OPTIONS] BUCKET DATABASE
Create a SQLite database with OCR results for files in a bucket
Options:
--access-key ...
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd s3-ocr
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
To regenerate the README file with the latest --help:
cog -r README.md