WPextract is a tool to create datasets from WordPress sites.
- Archives posts, pages, tags, categories, media (including files), comments, and users
- Uses the WordPress API to guarantee 100% accurate and complete content
- Resolves internal links and media to IDs
- Automatically parses multilingual sites to create parallel datasets
See the complete documentation for more detailed usage.
- Install with
pipx$ pipx install wpextract - Download site data
$ wpextract download "https://example.org" out_dl - Process into a dataset
$ wpextract extract out_dl out_data
WPextract was built by Freddy Heppell of the GATE Project at the School of Computer Science, University of Sheffield, originally created to scrape mis/disinformation websites for research.
Available under the Apache 2.0 license. See LICENSE for more information.
Note
This software was developed for our EMNLP 2023 paper Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study. The code has been updated since the paper was written; for archival purposes, the precise version used for the study is available on Zenodo.
We'd love to hear about your use of our tool, you can email us to let us know! Feel free to create issues and/or pull requests for new features or bugs.
If you use this tool in published work, please cite our EMNLP paper:
Freddy Heppell, Kalina Bontcheva, and Carolina Scarton. 2023. Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5729–5741, Singapore. Association for Computational Linguistics.
Permanent references to each release of this software are available from Zenodo.