Skip to content

CrowdTruth metrics 2.0

Compare
Choose a tag to compare
@ancadumitrache ancadumitrache released this 03 Jul 17:00
· 160 commits to master since this release

First stable release of the CrowdTruth metrics 2.0, attempting continuous way of integrating the 3 components of the crowdsourcing disagreement triangle (workers, annotations and media units). The quality scores of the workers and annotations are used to calculate the quality scores of the media units, and vice versa, in a continuous loop, until the scores are stabilized.

A full description of the metrics is available here.