Implement more pdf importers #7947

btut · 2021-07-30T12:07:25Z

This PR aims to implement more pdf importers.
Currently, pdfs can be imported using the PdfContentImporter that is tailored to some IEEE and Springer formats. We want to add:

PdfGrobidImporter.java: Query grobid.jabref.org Grobid (Commits cherry-picked from Implement an interface to import PDF metadata from multiple sources (XMP, Grobid, ...) #7929)
PdfEmbeddedBibFileImporter.java: Support for pdfs that have a BibTeX file embedded into them (e.g. generated using the authorarchive package)
PdfVerbatimBibTextImporter.java: Support for pdfs that have their own BibTeX entry on the first page (e.g. generated using the coverpage package)
PdfMergeMetadataImporter.java: An Importer that:
- Calls a list of other pdf importers
- Merges the result (importers with higher priority have the highest credibility, fields from lower-priority importers are only used if the higher-priority importers did not deduct that field)
- In case a DOI or ISBN was found by any of the used importers, call the DOI / ISBN fetcher to improve the entry even more

The PdfMergeMetadataImporter will be used when users import PDFs into JabRef.

- Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
- Tests created for changes (if applicable)
- Manually tested changed features in running JabRef (always required)
- Screenshots added in PR description (for UI changes)
- Checked documentation: Is the information available and up to date? If not created an issue at https://github.com/JabRef/user-documentation/issues or, even better, submitted a pull request to the documentation repository.

Implemented an Importer that querries Grobid for metadata of a pdf. The necessary Grobid functionality (retrieving BibTeX for a pdf) is not yet available in Grobid, but we opened a PR that implements it (kermitt2/grobid#800).

It's no longer necessary to set the POST data by bytes as we use JSoup for that.

CHANGELOG.md

src/test/java/org/jabref/logic/importer/util/GrobidServiceTest.java

Grobid cannot predict a citationkey

Co-authored-by: Christoph <siedlerkiller@gmail.com>

Users can perform a PDF import on already imported pdf's to improve the quality of the entry

Fixes JabRef#7984

When importing, try importers that can tell if they are suitable for a certain file format or not. Some importers only check if a file is present, not if it in the correct format (isRecognizedFormat is always true if an existing file is given). They are used last. The List of importers now reflects that prioritization. It is not sorted by importer names anymore. The getter-methods getImportFormats and getImportFormatList still sort the List by name for the View.

src/main/resources/l10n/JabRef_en.properties

Siedlerchr · 2021-08-16T08:26:04Z

you seem to have an error:

/home/runner/work/jabref/jabref/src/test/java/org/jabref/logic/importer/WebFetchersTest.java:65: error: method getEntryBasedFetchers in class WebFetchers cannot be applied to given types;
Set idFetchers = WebFetchers.getEntryBasedFetchers(importFormatPreferences);
^
required: ImportFormatPreferences,FilePreferences,BibDatabaseContext,Charset
found: ImportFormatPreferences
reason: actual and formal argument lists differ in length

…PdfImporters

btut · 2021-08-18T14:50:30Z

One last thing before we merge: The verbatim importer currently checks for bibtex on the first page. As discussed with @calixtus and @koppor, there are cases where a verbatim bibtex code is present, but on a later page. Shall we increase that limit? What would be a good number to use?

calixtus · 2021-08-18T15:03:42Z

Wouldn't make this PR more complex now than it is. We can discuss this probably on JabCon. Let's just merge it and see after that.
Just restarted the fetcher tests, let's see, if Grobid turns green. :-)

btut · 2021-08-18T16:05:53Z

Wouldn't make this PR more complex now than it is.

It's just changing a number here. Nothing complex about it. We just need to decide if we want that behaviour or not.

Just restarted the fetcher tests, let's see, if Grobid turns green. :-)

Adapted them to the new Grobid output. Should be fine now.

…PdfImporters

* upstream/main: (110 commits) Extract PushTo names into model (#8005) Refactor processCitation in GrobidService to match processPdf (#8003) Improved progress indication for fulltext-index operations (#7981) Reordered Pdf-Importer priorities (#8001) Implement more pdf importers (#7947) Adding icon picker for group dialog issue#6142 (#7776) Fix possible NPE in exporter with empty charset (#7979) Fix icon color (#7994) Bump slf4j-api from 2.0.0-alpha2 to 2.0.0-alpha4 (#7991) Bump classgraph from 4.8.112 to 4.8.114 (#7990) Bump mariadb-java-client from 2.7.3 to 2.7.4 (#7992) Bump jsoup from 1.14.1 to 1.14.2 (#7993) New yaml issue template (#7983) [Bot] Update CSL styles (#7985) Reordered items in main table right-click menu (#7952) Fulltext Index: Only index local pdf files (#7980) Bump WyriHaximus/github-action-wait-for-status from 1.3 to 1.4 (#7973) Bump byte-buddy-parent from 1.11.9 to 1.11.12 (#7974) Bump classgraph from 4.8.110 to 4.8.112 (#7975) Bump checkstyle from 8.45 to 8.45.1 (#7978) ... # Conflicts: # src/main/java/module-info.java

btut added 7 commits July 30, 2021 13:46

GrobidPdfMetadataImporter implemented

8a7b80f

Implemented an Importer that querries Grobid for metadata of a pdf. The necessary Grobid functionality (retrieving BibTeX for a pdf) is not yet available in Grobid, but we opened a PR that implements it (kermitt2/grobid#800).

Fixed class when accessing resources

6fe2a23

Use FileHelper method to get extension

f99bc52

Use jsoup to issue POST request

1d64d80

Removed unnecessary field

f591bfc

Reverted URLDownload

b2bd365

It's no longer necessary to set the POST data by bytes as we use JSoup for that.

Changelog entry

e458c77

Siedlerchr reviewed Jul 30, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Siedlerchr reviewed Jul 30, 2021

View reviewed changes

src/test/java/org/jabref/logic/importer/util/GrobidServiceTest.java Show resolved Hide resolved

btut and others added 21 commits July 30, 2021 16:46

Add pdf link to imported entry

5478585

Remove citationkey from Grobid

d0cc663

Grobid cannot predict a citationkey

FirstPageImporter

2cd78fc

Fixed grammar mistake in CHANGELOG.md

eb22157

Co-authored-by: Christoph <siedlerkiller@gmail.com>

Fixed Grobid tests

3ac0094

Fixed Grobid URL

c87ed4e

Checkstyle

3d8c4da

Fixed doc

168b866

Checkstyle

42adea9

Use JSoup for plaintext citations as well

73dc505

Renamed FirstPageImporter to PdfVerbatimBibTextImporter

7ce7105

Fixed getName (no importer)

53d8e9a

Renamed Grobid importer to match convention

9080f14

PdfEmbeddedBibTeXImporter

2757be6

Renamed PdfEmbeddedBibTeXImporter to PdfEmbeddedBibFileImporter

8a05c3e

Checkstyle

0c488ec

Remove debug output

02057f0

Checkstyle

3d66855

PdfMergeMetadataImporter

fd8918b

Add DOI and ISBN fetching in PdfMergeMetadataImporter

56868f5

Fixed concurrent list access

479a0bc

btut added 4 commits August 14, 2021 20:50

Use MergeImporter as WebFetcher

5cbf919

Users can perform a PDF import on already imported pdf's to improve the quality of the entry

Only force BibTeX import if everything else fails

1cb4dfc

Fixes JabRef#7984

Checkstyle

7ba8b40

btut self-assigned this Aug 16, 2021

Siedlerchr reviewed Aug 16, 2021

View reviewed changes

src/main/resources/l10n/JabRef_en.properties Outdated Show resolved Hide resolved

btut mentioned this pull request Aug 16, 2021

Implement an interface to import PDF metadata from multiple sources (XMP, Grobid, ...) #7929

Merged

5 tasks

Siedlerchr approved these changes Aug 16, 2021

View reviewed changes

btut added 6 commits August 16, 2021 11:12

Fixed WebFetchersTest

18dbb67

Grobid does not need localization

3d46df4

Followup on removed Grobid localization

40b2759

Fixed tests

6324cf2

Merge branch 'main' of github.com:JabRef/jabref into improvement/more…

5cf2af7

…PdfImporters

Checkstyle

b555ada

Siedlerchr approved these changes Aug 16, 2021

View reviewed changes

Merge branch 'main' of github.com:JabRef/jabref into improvement/more…

a7604b6

…PdfImporters

btut added 2 commits August 18, 2021 17:47

Grobid Fetcher and Tests adapted to updated Grobid

4cff87c

Adapted GrobidServiceTest to updated Grobid

4ac2002

Merge branch 'main' of github.com:JabRef/jabref into improvement/more…

43e22b7

…PdfImporters

Siedlerchr approved these changes Aug 18, 2021

View reviewed changes

Siedlerchr merged commit 0b02dd4 into JabRef:main Aug 18, 2021

calixtus removed status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers status: depends-on-external A bug or issue that depends on an update of an external library labels Aug 18, 2021

btut deleted the improvement/morePdfImporters branch August 19, 2021 07:05

koppor mentioned this pull request Sep 22, 2021

Think about adding GROBIT for "PDF to BibTeX" functionality JabRef/jabref-koppor#327

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement more pdf importers #7947

Implement more pdf importers #7947

btut commented Jul 30, 2021 •

edited by calixtus

Loading

Siedlerchr commented Aug 16, 2021

btut commented Aug 18, 2021

calixtus commented Aug 18, 2021

btut commented Aug 18, 2021

Implement more pdf importers #7947

Implement more pdf importers #7947

Conversation

btut commented Jul 30, 2021 • edited by calixtus Loading

Siedlerchr commented Aug 16, 2021

btut commented Aug 18, 2021

calixtus commented Aug 18, 2021

btut commented Aug 18, 2021

btut commented Jul 30, 2021 •

edited by calixtus

Loading