Version 3.0.0

• Orchestration schemes added under themis analyze prefix
  • themis analyze fallback
  • themis analyze voting-router
  • themis analyze nlc-as-router
• themis analyze purview evaluate command-line tool to deal with annotator disagreement
• Standardized confidence methods used by Oracle and the orchestration schemes
• Extracted calculations from plot.py to metrics.py
• Question sampling corrects the frequency to match the size of the test set

Version 2.4.0

• Additional analyses
• Help text for individual commands
• Bug fixes

Version 2.3.1

This is identical to 2.3.0 except that the version number in 2.3.0 was incorrectly still listed as 2.2.0.

• Collate answers across multiple systems
• By default don't remove newlines from answer text to match Annotation Assist
• Results analysis
• Oracle combination of systems
• Extract corpus from TREC files

Version 2.2.0

• Rewrite corpus downloading logic to fix incorrect restarts
• Validation utilities to compare truth and usage logs to the corpus (xmgr validate-truth, xmgr validate-answers)
• Print human-readable representation of truth (examine-truth)
• Utilities to download PAU ids lists and PAUs from XMGR (xmgr pau-ids, xmgr-pau)
• Strip newlines from NLC training files because NLC cannot handle newlines

Version 2.0.0

Overhauled the command line interface.

Annotation Assist input generation may incorporate previous annotations.

Version 1.0.0

This toolkit enables you to download corpus and truth from XMGR, harvest questions from XMGR usage logs, annotate them using the Annotation Assist tool and compare system performance between WEA, Solr, and NLC.

This version was used to do the original analysis of Deakin and Met Health.

Version 2.1.0

Fix package installation issues.
Improve error messages from bad XMGR URLs.
Handle non-existent mappings in truth.