Evaluation of ChoCo

Please, note that this Wiki page is work-in-progress!

JAMS test-driven evaluation

The goal of this first validation step is to assess the quality and the expressivity of the JAMS files resulting from the JAMification process of ChoCo's partitions. Please, refer to choco/jams_test.py to create test data and run the final evaluation.

Testing scripts for the JAMification of ChoCo partitions.

positional arguments:
  {create,test}         Either `create` for generating the test samples or `test` for running the JAMS-based tests.
  partition_dir         Path to the directory of the ChoCo partition in which the "meta.csv" file can be found.
  {audio,score}         Type of music content in the collection.

optional arguments:
  -h, --help            show this help message and exit
  --n_samples N_SAMPLES
                        Number of test samples to draw from the partition, create the test metadata, and skim the JAMS.
  --keep_n KEEP_N       Number of chord/key observations per annotation that will be retained for the evaluation.
  --seed SEED           A random seed for reproducible sampling.

Create a test set for the partition

The first thing to do is to create a test set of silver JAMS files that will be manually edited, extended, revised, and eventually enriched by human annotators. For a single partition, say isophonics, the following script generates a test set of 4 samples where only the first 5 observations are retained per chord/key annotation.

cd choco
jams_tests.py create ../partitions/isophonics/choco/ audio --n_samples 4 --keep_n 5 --seed 1234

Conversion rule evaluation

Quality measures of the ChoCo KG