feat: Create a backend to transform PubMed XML files to DoclingDocument

[lucas-morin]

feat: Create a backend to transform PubMed XML files to DoclingDocument

• Introduce PubMedDocumentBackend to (1) parse elements from PubMed XML files and to (2) convert them to a DoclingDocument. Convert to DoclingDocument the authors, the abstract, the main-body text, the tables, the tables captions, the figures captions and the references. The hierarchy of the main-body text is preserved.
• Add end-to-end tests for PubMed XML conversion (./tests/test_backend_xml.py).

Known limitations:

• Compared to the original PDF documents, some sections are missing: page footers, page headers, figures, sections outside the main-body (for instance: Authors’ Contribution, Funding, Conflict of Interest, Acknowledgment, or Keywords).
• Column headers for tables with multi-column headers are duplicated.
• Some PubMed XML tables (stored as HTML) are occasionally not properly converted to Docling tables and then ignored.

Issue resolved by this Pull Request:
Resolves #446 (Partially)

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

Wonderful, this rule succeeded.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

[ceberam]

Some additional comments:

• Rename the input format objects from PUBMED to XML_PUBMED for verity clarity (first the type, then the collection)
• rename test files from nxml to xml (since the .xml is the only extension for PubMed, if InputFormat object is not provided)
• remove opencv-python = "^4.10.0.84" from pyproject.toml. It's already a dependency of rapidocr-onnxruntime.
• rebase to latest main
• style suggestions (but it's up to you):
  • no need to import Dict, List, .... from typing library, you can use the built-in types dict, list, ... respectively
  • you can shorten expressions like if len(reference_node.xpath("article-title")) == 0 by using falsy and truthy values: if not reference_node.xpath("article-title"), and instead of if reference["author_names"] != "" you can use if reference["author_names"]

[dolfim-ibm]

I would like to understand how this is affecting the backend/type selection:

1. What happens if we input is a .xml file not matching the PubMed schema?
2. What happens if the input is .nxml instead of .xml?

[dolfim-ibm]

I think we should switch all function typing to Pydantic BaseModel.

[lucas-morin]

1. If we input a .xml file which do not match the JATS schema, docling raises an error:

docling.exceptions.ConversionError: File format not allowed: /home/lum/Documents/public/docling/tests/data/pubmed/test.xml

2. If we rename one of the test documents from .xml to .nxml, the conversion remains the same.

[dolfim-ibm]

1. If we input a .xml file which do not match the JATS schema, docling raises an error:

docling.exceptions.ConversionError: File format not allowed: /home/lum/Documents/public/docling/tests/data/pubmed/test.xml

2. If we rename one of the test documents from `.xml` to `.nxml`, the conversion remains the same.

This is good. My last question on this topic, do we have already a way for a different XML format?

[ceberam]

This is good. My last question on this topic, do we have already a way for a different XML format?

@dolfim-ibm This is addressed in #606 , which should be rebased (and therefore slightly refactored) once this PR #557 is merged.

@lucas-morin as discussed, let's give a more readable and type-safe schema to the parsed content instead of generic dictionaries (dict[str, Any]) by leveraging pydantic models. Also, we should rebase to pick-up the commits from last week.

[lucas-morin]

Now, I improved the typing by removing Any types in dictionaries. I used TypedDict instead of Pydantic BaseModel, is it also fine?

[PeterStaar-IBM]

let's group all xml-related backends together in a subfolder,

from docling.backend.pubmed_backend import PubMedDocumentBackend

should become,

from docling.backend.xml.pubmed_backend import PubMedDocumentBackend

[ceberam]

🏆

The code was updated accordingly.