Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Textractor pipeline into multiple pipelines #828

Closed
davidmezzetti opened this issue Dec 3, 2024 · 0 comments
Closed

Refactor Textractor pipeline into multiple pipelines #828

davidmezzetti opened this issue Dec 3, 2024 · 0 comments
Assignees
Milestone

Comments

@davidmezzetti
Copy link
Member

Currently, the Textractor pipeline consists of three different components.

  • Logic to download/retrieve content
  • A file to html converter currently backed by Apache Tika
  • A html to markdown converter

This logic should be split up into multiple pipelines.

  • FileToHTML pipeline that converts a file to html using a text extraction backend (Tika currently)
  • HTMLToMarkdown pipeline that converts html to markdown
  • Existing logic to call those pipelines and download data remains in the Textractor pipeline

This will make it easier to add new file to html conversion engines.

@davidmezzetti davidmezzetti added this to the v8.1.0 milestone Dec 3, 2024
@davidmezzetti davidmezzetti self-assigned this Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant