appjsonify
1 is a handy PDF-to-JSON conversion tool for academic papers implemented in Python.
appjsonify
allows you to obtain a structured JSON file that can be easily used for various downstream tasks such as paper recommendation, information extraction, and information retrieval from papers.
- Linux or macOS (Not tested on Windows)
- Python 3.10 or later
- pdfplumber
- registrable
- tqdm
- pillow
- pdf2image
- torch
- detectron2
Please manually install it based on the instructions.
If your environment does not have poppler
, please install it.
This is necessary to obtain PDF images using pdf2image
.
For more details, refer to Prerequisites.
pip install appjsonify
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
git clone https://github.com/hitachi-nlp/appjsonify.git
python -m pip install --editable .
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
appjsonify
offers two options to structure your paper PDF file into a JSON file.
- Use the existing templates
Suitable if a paper adopts theAAAI
,ACL
,ICML
,ICLR
,NeurIPS
,IEEE
,ACM
, orSpringer
styles. See Templates for more details. - Configure pipelines and parameters by yourself
If a paper does not adopts the above formats, you need to specify the processing pipeline and its parameters. Please refer to Build your own pipeline for further information.
appjsonify
provides two types of the templates for each of the following paper types: AAAI
, ACL
, ICML
, ICLR
, NeurIPS
, IEEE
, ACM
, and Springer
.
One is more accurate but slower due to the use of machine learning based models, the other is less accurate but faster due to its rule based approach.
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI
If your environment has a GPU(s), it is better to also specify
--detectron_device_mode cuda
to speed up the process.
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type AAAI2
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACL2
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICML2
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ICLR2
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type NeurIPS2
Currently only tested with IEEE BigData papers.
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type IEEE2
Currently only tested with TALLIP papers.
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type ACM2
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer
appjsonify /path/to/pdf/dir/or/path /path/to/output/dir --paper_type Springer2
--verbose
: If you want to check the intermediate processing results, please set this flag. The log files will be saved underoutput_dir
. Optionally, you can use the following four flags to add the corresponding information.--show_pos
: Bounding box information.--show_font
: Font name and size information.--show_style
: Style information (e.g.,section
,body
,abstract
, etc.)--show_meta
: Supplementary information (e.g., information on objects and footnotes.)--insert_page_break
: Insert breaks between pages.
--save_image
: If you are using a more accurate but slower version of templates orload_objects_with_ml
,appjsonify
can save detected table and figure images if this flag is set. In addition to this, please also specify the output directory path as--output_image_dir
.
appjsonify
also allows users to build their own academic paper PDF-to-JSON processing pipeline.
For more details, please refer to Available Modules and Document Handling in appjsonify
.
Users can add their own modules to appjsonify
for more flexible document processing.
To add modules, appjsonify
must be installed in editable mode.
See Customize appjsonify
for more details and feel free to make a PR if you wish to add your module to this repository and package!
Contributions are more than welcome! Feel free to raise an issue and/or make a PR. Possible future work is as follows:
- Better documentation
- More paper templates
- More robust references extraction
- Powerful mathematical equation support
- Robust algorithm description detection
- Multilingual support
- Add more test scripts
If you use appjsonify
in your work, please cite the following.
@article{yamaguchi2023appjsonify,
title={appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit},
author={Atsuki Yamaguchi and Terufumi Morishita},
year={2023},
eprint={2310.01206},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
© 2023 Atsuki Yamaguchi and Terufumi Morishita (Hitachi, Ltd.)
This work is licensed under the MIT license unless specified.
appjsonify
uses the follwoing publicly available works.
- pdfplumber by Jeremy Singer-Vine (MIT License).
- registrable by epwalsh (Apache License 2.0).
- tqdm (MIT License, Mozilla Public License 2.0 (MPL 2.0)).
- pillow by Jeffrey A. Clark (Historical Permission Notice and Disclaimer License).
- pdf2image by Edouard Belval (MIT License).
- torch (BSD-style license).
- Detectron2 by Facebook AI Research (Apache License 2.0) in detectron2_demo.
- DocBank pretrained model by Microsoft Research Asia (Apache License 2.0) in docbank.py.
- TableBank pretrained model by Microsoft Research Asia (Apache License 2.0) in tablebank.py.
- PubLayNet pretrained model by hpanwar08 (Apache License 2.0) in publaynet.py.
Footnotes
-
Academic Paper PDF jsonify ↩