Skip to content

Commit

Permalink
Finalize Submission system (#2)
Browse files Browse the repository at this point in the history
* first draft of template

* starting some test ideas

* add pytest tests for submissions

* add travis dependencies

* only test tests

* add requirements

* make requirements explicit

* add detailed description of how to do a pull request

* add further notes

* keep it

* Update README.md

* add a warning to use the same ffmpeg version across evaluation and processing

* Merge branch 'submission_system' of github.com:sigsep/sigsep-mus-2018 into submission_system

* check existence of description.md
  • Loading branch information
faroit authored Feb 18, 2018
1 parent df5cba0 commit 6e08688
Show file tree
Hide file tree
Showing 13 changed files with 438 additions and 7 deletions.
108 changes: 108 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
#### joe made this: http://goel.io/joe

#####=== Python ===#####

.pytest_cache/

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

#####=== Matlab ===#####
##---------------------------------------------------
## Remove autosaves generated by the Matlab editor
## We have git for backups!
##---------------------------------------------------

# Windows default autosave extension
*.asv

# OSX / *nix default autosave extension
*.m~

# Compiled MEX binaries (all platforms)
*.mex*

# Simulink Code Generation
slprj/

#####=== OSX ===#####
.DS_Store
.AppleDouble
.LSOverride

# Icon must end with two \r
Icon

# Thumbnails
._*

# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns

# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
Expand Down
23 changes: 23 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
sudo: false

cache:
directories:
- $HOME/env

language: python

python:
- "3.6"

before_install:
- bash .travis_dependencies.sh
- export PATH="$HOME/env/miniconda$TRAVIS_PYTHON_VERSION/bin:$PATH";
- hash -r
- source activate test-environment

install:
- pip install jsonschema pytest

script:
- python --version
- py.test tests -v
40 changes: 40 additions & 0 deletions .travis_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/sh

ENV_NAME="test-environment"
set -e

conda_create ()
{

hash -r
conda config --set always_yes yes --set changeps1 no
conda update -q conda
conda config --add channels pypi
conda info -a
deps='pip pytest'

conda create -q -n $ENV_NAME "python=$TRAVIS_PYTHON_VERSION" $deps
conda update --all
}

src="$HOME/env/miniconda$TRAVIS_PYTHON_VERSION"
if [ ! -d "$src" ]; then
mkdir -p $HOME/env
pushd $HOME/env

# Download miniconda packages
wget http://repo.continuum.io/miniconda/Miniconda-3.16.0-Linux-x86_64.sh -O miniconda.sh;

# Install both environments
bash miniconda.sh -b -p $src

export PATH="$src/bin:$PATH"
conda_create

source activate $ENV_NAME

source deactivate
popd
else
echo "Using cached dependencies"
fi
65 changes: 58 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# SISEC MUS 2018 Submission

For the MUS18 submission we are handling the submission of the results through GitHub.
This is the submission system for the Professionally-produced music task of the [2018 SiSEC](https://sisec.inria.fr/2018-professionally-produced-music-recordings/).

>:warning: This means it will be necessary for any participant to have a github account. If you don't have experience with git, don't worry, as everything can be done from within the browser. Also we will add a full tutorial here.
For the this year, we are handling the submission of the results through GitHub.
This means it will be necessary for any participant to have a github account. If you don't have experience with git, don't worry, as everything can be done from within the browser. Also we will add a full tutorial here.

## Dates

- The submission system will be ready by __March 1st, 2018__ But you can start working on your separation method already!
- __March 1st, 2018__: The submission system is open.
- __April 1st 2018__: Deadline for submission results.

## Submission

Expand All @@ -18,17 +20,66 @@ Download the dataset via the link on the [dataset website](https://sigsep.github

### Step 2: Compute Estimates

Compute estimates using the python based [musdb tools](https://github.com/sigsep/sigsep-mus-db).
Compute estimates using the python based [musdb tools](https://github.com/sigsep/sigsep-mus-db) or through your own custom method.

In any case, make sure, your estimates folder does follow a certain schema. Lets assume your method is called `ABCD`:

```
ABC1/ <---------------------------------- Your estimate short name (max 4 characters)
└── test/ <------------------------------ The test subfolder (do not submit the results for train)
├── Arise - Run Run Run/ <----------- 50 subfolders, one for each track
│ ├── vocals.wav <----------------- One wav file for every target you want to submit (up to 5)
│ ├── accompaniment.wav
│ ├── drums.wav
│ ├── bass.wav
│ └── other.wav
└── Zeno - Signs/
```

### Step 3: Compute Evaluation Scores

t.b.a.
Evaluate your estimate folder using the BSSEval v4 metric using the [museval package](https://github.com/sigsep/sigsep-mus-eval). We provide a docker image for the evaluation tools so that you do not need to set up a python environment.

```
ABC1/ <---------------------------------- Your estimate short name (max 4 characters)
└── test/ <------------------------------ The test subfolder (do not submit the results for train)
├── Arise - Run Run Run.json <-------- 50 json files that include your bsseval v4 scores
└── Zeno - Signs.json
```

:warning: If you apply your separation method on the default dataset encoded in the STEM format (not converted to `wav` first), please make sure you use the same ffmpeg version for separation and evaluation. We tested several different machines and ffmpeg version and did not run into any problems, but we cannot guarantee that the decoded outputs of two different ffmpeg versions are identical and would not affect the bsseval scores. E.g. when silence > 512 samples would be added in the beginning of a target source.

### Step 4: Submission to Github

t.b.a
To submit your Evaluation scores, we kindly ask you to create a Pull Request on github for your submission.

The first step is to prepare your evaluation score folder so that it matches our [/submission-template/ABC1](/submission-template) directory in this repository (`ABC1` in the example above). Make sure you add the [description.md](/submission-template/ABC1/description.md) file to briefly describe your proposed method. Then put your `submission` directory into the [/submissions](/submissions) subfolder.

In the following we describe the basic steps to create this pull request with your submission through the github user interface (if you already know git, you can safely ignore many steps):

1. Create a fork of this repository by clicking on the "Fork" button at the top right of this page.
2. In your new repository, navigate to the submissions directory (after navigating, the URL should be something like https://github.com/your-github-username/sigsep-mus-2018/tree/master/submissions).
3. Click the "Upload files" button.
4. Upload your submission directory, this should upload a 51 files in total (50 json files and the `description.md`). Hit "Commit changes" at the bottom of the page.
5. Click the "New Pull Request" button.
6. On the next page, click the "Create Pull Request" button.
7. Make the pull request title the same as your submission folder, and click the "Create Pull Request" button.
8. This will take you the pull request page for your paper, with a URL like https://github.com/sigsep/sigsep-mus-2018/pull/2. Save this URL for future reference. At this point, you have finished your submission; don't close the pull request.
9. Your submission will be tested by an [automated test system](/tests). We may instruct you to update your submission with a few changes. To do so, simply follow steps 2-4 again with your updated submission directory (i.e. navigate to the submissions directory in your fork and upload your new updated folder). The pull request will be updated automatically. Once the tests pass, we will merge your submission.
10. This the merging should take place before the deadline of __April 1st 2018__.

#### Further Notes

- You can submit multiple submission directories in one pull request.
- :warning: please do not delete your estimates as we will aggregate the wav files later through a separate upload system.

## Support

If you need help with your submission, you can use our
If you need help with your submission, you can use our chat or contact @faroit.
[![Join the chat at https://gitter.im/sigsep-mus-2018/Lobby](https://badges.gitter.im/sigsep-mus-2018/Lobby.svg)](https://gitter.im/sigsep-mus-2018/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

### Acknowledgements

We thank [Colin Raffel](http://colinraffel.com) for the inspiration to use git for handling the SiSEC submissions.
Empty file added submission-template/.gitkeep
Empty file.
Empty file.
28 changes: 28 additions & 0 deletions submission-template/ABC1/description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# ABC1 <!-- Your submission short name in <=4 characters -->
Firstname1 Lastname1, Firstname2 Lastname2 <!-- Authors -->
Affiliations <!-- Affiliations -->
contact@author.com <!-- one corresponding mail address -->

## Additional Info

* __is_blind:__ no <!-- if you used supervised learning, answer no -->
* __additional_training_data:__ yes <!-- if you used more data than musdb (not including data augmentation)-->

## Supplemental Material

* __Code:__ http://github.com/my/repo
* __Demos:__ http://www.demo.com


## Method

Here we describe the method in detail. You can use math formula $a=b$ and include images.

![](https://docs.google.com/drawings/d/e/2PACX-1vRXc_l4uNUTSOoAif8r4O-AKfAVBBMUSPVG_VMu79LjcZLb4xKLgFTVoSqVodvGetEvdeakfb4Nul-3/pub?w=394&h=384)

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

## References

- A. List, Of. Authors, Proceedings of LVA/ICA, 2018
- A. List, Of. Authors, Proceedings of Lorem Ipsum, 2017
Empty file.
Empty file added submissions/.gitkeep
Empty file.
52 changes: 52 additions & 0 deletions tests/data/mus18_test_tracks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
[
"Signe Jakobsen - What Have You Done To Me",
"Enda Reilly - Cur An Long Ag Seol",
"The Doppler Shift - Atrophy",
"Sambasevam Shanmugam - Kaathaadi",
"Secretariat - Borderline",
"Hollow Ground - Ill Fate",
"AM Contra - Heart Peripheral",
"Tom McKenzie - Directions",
"Speak Softly - Broken Man",
"Ben Carrigan - We'll Talk About It All Tonight",
"Carlos Gonzalez - A Place For Us",
"BKS - Bulldozer",
"Secretariat - Over The Top",
"Georgia Wonder - Siren",
"Speak Softly - Like Horses",
"Raft Monk - Tiring",
"Lyndsey Ollard - Catching Up",
"The Mountaineering Club - Mallory",
"BKS - Too Much",
"The Long Wait - Dark Horses",
"Skelpolu - Resurrection",
"Detsky Sad - Walkie Talkie",
"PR - Oh No",
"Al James - Schoolboy Facination",
"The Easton Ellises - Falcon 69",
"PR - Happy Daze",
"Triviul feat. The Fiend - Widow",
"Juliet's Rescue - Heartbeats",
"Side Effects Project - Sing With Me",
"M.E.R.C. Music - Knockout",
"Moosmusic - Big Dummy Shake",
"Nerve 9 - Pray For The Rain",
"Mu - Too Bright",
"Bobby Nobody - Stitch Up",
"Motor Tapes - Shore",
"Timboz - Pony",
"James Elder & Mark M Thompson - The English Actor",
"Girls Under Glass - We Feel Alright",
"Arise - Run Run Run",
"Angels In Amplifiers - I'm Alright",
"Buitraker - Revo X",
"The Easton Ellises (Baumi) - SDRNR",
"Forkupines - Semantics",
"The Sunshine Garcia Band - For I Am The Moon",
"Punkdisco - Oral Hygiene",
"Zeno - Signs",
"We Fell From The Sky - Not You",
"Cristina Vane - So Easy",
"Little Chicago's Finest - My Own",
"Louis Cressy Band - Good Time"
]
47 changes: 47 additions & 0 deletions tests/data/museval.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{
"type": "object",
"properties": {
"targets": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"frames": {
"type": "array",
"items": {
"type": "object",
"properties": {
"time": {
"type": "number"
},
"duration": {
"type": "number"
},
"metrics": {
"type": "object",
"properties": {
"SDR": {
"type": "number"
},
"SIR": {
"type": "number"
},
"SAR": {
"type": "number"
},
"ISR": {
"type": "number"
}
}
}
}
}
}
}
}
}
}
}
2 changes: 2 additions & 0 deletions tests/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
jsonschema
pytest
Loading

0 comments on commit 6e08688

Please sign in to comment.