Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a cut off for size of text to add to ES #224

Open
karindalziel opened this issue Mar 22, 2024 · 1 comment · Fixed by #226
Open

Add a cut off for size of text to add to ES #224

karindalziel opened this issue Mar 22, 2024 · 1 comment · Fixed by #226

Comments

@karindalziel
Copy link
Member

karindalziel commented Mar 22, 2024

we have found that more than a million characters in the text field makes ES choke. Probably the best way to go about this would be to have a max size in the config, and then call that in the script (just in case we discover performance issues at different ingest sizes.)

Also add a note to the log/output that the file was truncated so we can split it up, which should be best practice anyway.

@techgique
Copy link
Member

This issue links to a couple related PRs for code to reference: https://github.com/whitmanarchive/whitman-issues/issues/665

@wkdewey wkdewey linked a pull request Aug 8, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants