Skip to content

v0.4.1

Latest
Compare
Choose a tag to compare
@thammegowda thammegowda released this 26 Apr 05:08
· 2 commits to master since this release
  • Better parallelization: parallel and mono data are scheduled at once (previously it was one after the other)
  • mtdata cache added. Improves concurrency by supporting multiple recipes
  • Added WMT general test 2022 and 2023
  • Added news commentary 18.1. news crawl 2023
  • mtdata-bcp47 : -p/--pipe to map codes from stdin -> stdout
  • mtdata-bcp47 : --script {suppress-default,suppress-all,express}
  • Uses pigz to read and write gzip files by default when pigz is in PATH. export USE_PIGZ=0 to disable