A java musical chord recognition utility using pre-trained neural network Tensorflow model and DSP tools extended from TarsosDSP.
WARNING: This library is still in development that has unorganized source code and some features are not implemented on top of another
The main processing tools in package dir you might want to use are:
Chord Processor
: A sub-TarsosDSP class extended fromAudioProcessor
that wraps Tensorflow model and other main processing componentsChord Predictor
: A singleton class that predict the chroma given in the params.CRP Vector Factory
: A factory for creating chroma object, specifically CRP chroma, from samples given.
This toolkit has currently the following main component classes:
Sound File
: A naive Interface for retrieving full samples from WAV or MP3 file.Signal
: A naive float array of raw sample retrieved from Sound File or from other srcLog Frequency Vector
: A full descriptive class that has pitch information from a sampleChroma Vector
: Interface class that has currently 3 different types of chroma classes. For the details of these different chroma, please refer to the Method and Approach sectionSpectrum
: A full description abstract class that has spectral information with the given sub-classed typeTensorflow Model
: A specific tensorflow model loader and runner with its Java API. The loaded model is pretrained and stored in main resources package
Avalaible chord sets that can be predicted by the pre-trained neural model is located in the Chord.java class. They are in all keys with only major and minor scale. The chords list given the model output are respectively:
{
"A#maj", "A#min", "Amaj", "Amin", "Bmaj", "Bmin", "C#maj", "C#min",
"Cmaj", "Cmin", "D#maj", "D#min", "Dmaj", "Dmin", "Emaj", "Emin",
"F#maj", "F#min", "Fmaj", "Fmin", "G#maj", "G#min", "Gmaj", "Gmin"
}
The model is set to have ~0.7 testing accuracy with the testing subset consisting of ~400 samples of chroma (specifically CRP) vectors
For easy quick use:
Import locally the full-bundled fat jar library, jchoreco-x.x.x-all.jar
, without implementing any dependencies in your code
(But this might cause big-sized package in your built app)
Otherwise, for lightweight use:
Use locally the smaller-packaged, jchoreco-x.x.x.jar
, and the following dependencies must be manually included in your project.
// gradle example (kotlin)
implementation("com.github.axet:TarsosDSP:2.4") // TarsosDSP libraries
implementation("com.github.wendykierp:JTransforms:3.1") // Pure java FFT libraries
implementation("org.tensorflow:tensorflow-core-platform:0.4.0") // Tensorflow API
implementation("javazoom:jlayer:1.0.1") // MP3 Decoder
implementation("org.apache.commons:commons-math3:3.6.1") // For CommonProcessor tools
implementation("com.github.yannrichet:JMathPlot:1.0.1") // For plotting using swing ui
For all examples, it is assumed the chord model is loaded in the package.
Refer to Usage Example for details.
From file, predict chord cautiously of all samples in the file with the following code
String pathFile = "/path/to/file.wav";
Signal signal = new WAVFile(new File(pathFile)).getSamples(0);
try (ChordPredictor predictor = ChordPredictor.getInstance()) {
String chord = predictor.predict(CRPVectorFactory.from_signal(signal));
System.out.println("Predicted chord: "+chord);
}
Use either SoundFile
or TarsosDSP's AudioDispatcher
to retrieve, respectively, full or buffer
samples from an audio. For SoundFile
:
File file = new File(pathString); // java.io.File
// Caution: This retrieves all sample from the file
SoundFile sound = new MP3File(file);
// or
SoundFile sound = new WAVFile(file);
Signal signal = sound.getSamples(0); // get signal from channel 1
// or
float[] samples = signal.getData(); // if you need float samples, retrieve its data of the audio
float sampleRate = signal.getSampleRate();
For TarsosDSP's` approach, refer to its example applications
Simply use CRPVectorFactory
to exctract the chroma vector from samples sources
CRP chromaCRP = CRPVectorFactory.from_signal(signal);
// or
float[] chromaCRP = CRPVectorFactory.from_floatSamples(samples, sampleRate);
CAUTION : Using this library chord predictor/processor is not thread-safe. This is because the current tensorflow model java api (v0.4.0) declare that some of its resources must be manually released by the caller.
To avoid memory leak, use pre-cautiously with try-with-resources block or call
close()
method from the predictor/processor class to release the resources.
Alert: Make sure the pre-trained tensorflow model is provided in the main resource package
Surround with try-with-resources block:
// use try-with-resources block
try (ChordPredictor predictor = ChordPredictor.getInstance()) {
String chord = predictor.predict(chromaCRP); // predict the chord
System.out.println(chord); // printing the chord
// or predict list of processed crp
List<float[]> chromaList = ...;
List<String> chords = predictor.predict(chromaList);
}
or manually call the close()
:
// use with manual close()
ChordPredictor predictor = ChordPredictor.getInstance();
System.out.println(predictor.predict(chromaCRP));
predictor.close(); // call the close() whenever finished using it
TarsosDSP's AudioDispatcher
runs on an audio input stream and provide the sample buffer
every once a time defined by the constructor. Adding the ChordProcessor
to the dispatcher
is an idea to predict chord in buffer.
ALERT:
ChordProcessor
is always closed after the dispatcher finished. But it is always a best practice to surround it with try-with-resources block or call the close() again.
To retrieve the chord bytes of lined string ( the processor output a form of(chord + '\n')
) ,
create a class of java OutputStream
first:
int BUFFER_SIZE = 1024 * 16; // 16384 bytes or more is good for chord audio buffering
AudioDispatcher dispatcher = AudioDispatcherFactory.fromFile(file, BUFFER_SIZE, BUFFER_SIZE/2);
OutputStream output = System.out; // Set the output stream as the console
try (ChordProcessor chordProcessor =
new ChordProcessor(SAMPLE_RATE, BUFFER_SIZE, output)) {
dispatcher.addAudioProcessor(chordProcessor);
dispatcher.run();
}
There are 2 provided examples in the main package:
- CLIChordRecognizer: a chord processing from microphone printing to console. Warning: Interrupt the program by entering chars, not other keyboard interrupt.
- SwingChordRecognizer: a chord processing from microphone with simple swing GUI.
The methodology for this project to achieve chord recognition is based on the following works:
- Abadi, Martín, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, et al. 2015. “TensorFlow: Large-scale machine learning on heterogeneous systems,” Software available from tensorflow.org. https://www.tensorflow.org/.
- Chollet, François. 2015. “keras.” 2.7.0. GitHub. https://github.com/fchollet/keras.
- Chordify B.V. n.d. “About Chordify.” Chordify. Accessed 2022. https://chordify.net/pages/about/.
- Fujishima, Takuya. 1999. “Realtime Chord Recognition of Musical Sound: a System Using Common Lisp Music.” 1999. https://quod.lib.umich.edu/i/icmc/bbp2372.1999.446/1.
- Garg, Vijay K. 2007. “Pulse Code Modulation.” In Wireless Communications & Networking. N.p.: Elsevier Science.
- Harris, Sarah L., and David Harris. 2021. “3.4.2 State Encodings.” In Digital Design and Computer Architecture, RISC-V Edition. N.p.: Elsevier Science.
- Jiang, Nanzhu, Peter Grosche, Verena Konz, and Meinard Müller. 2011. “Analyzing Chroma Feature Types for Automated Chord Recognition.” https://www.researchgate.net/publication/228445808_Analyzing_Chroma_Feature_Types_for_Automated_Chord_Recognition/comments.
- Jones, Russell. 2007. Understanding Basic Music Theory. Edited by Catherine Schmidt-Jones. http://cnx.org/content/col10363/1.3/.
- Mauch, Matthias, and Chris Cannam. n.d. “Chordino and NNLS Chroma.” isophonics. Accessed 2022. http://www.isophonics.net/nnls-chroma.
- Mauch, Matthias, and Simon Dixon. 2010. “Approximate Note Transcription for the Improved Identification of Difficult Chords.” Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR 2010). https://www.researchgate.net/publication/220723830_Approximate_Note_Transcription_for_the_Improved_Identification_of_Difficult_Chords.
- Microsoft. n.d. “FileDialog Class.” Microsoft Documentation. Accessed January, 2022. https://docs.microsoft.com/en-us/dotnet/api/system.windows.forms.filedialog.
- Müller, Meinard. 2007. Information Retrieval for Music and Motion. N.p.: Springer.
- Müller, Meinard, and Sebastian Ewert. 2010. “Towards Timbre-Invariant Audio Features for Harmony-Based Music.” IEEE Transactions on Audio Speech and Language Processing 18 (3): 649 - 662. 10.1109/TASL.2010.2041394.
- O'Donnell, Michael J. 2004. “Digital Sound Modeling lecture notes for Com Sci 295.” Perceptual Foundations of Sound. http://people.cs.uchicago.edu/~odonnell/Scholar/Work_in_progress/Digital_Sound_Modelling/lectnotes/node4.html.
- Oracle Corporation. 2021. “JavaFX Documentation Project.” GitHub Pages. https://fxdocs.github.io/docs/html5/.
- Osmalskyj, Julien, Jean J. Embrechts, Marc V. Droogenbroeck, and Sébastien Piérard. 2012. “Neural networks for musical chords recognition.” (January). https://www.researchgate.net/publication/252067543_Neural_networks_for_musical_chords_recognition.
- Peeters, Geoffroy. 2006. “Chroma-based estimation of musical key from audio-signal analysis.” (October). https://www.researchgate.net/publication/220723813_Chroma-based_estimation_of_musical_key_from_audio-signal_analysis.
- “Pulse-Code Modulation Codec-Filters.” 2018. In The Communications Handbook, edited by Jerry D. Gibson. N.p.: CRC Press.
- Reenskaug, Trygve. 1979. “Models-Views-Controllers,” A whitepaper report of MVC. https://folk.universitetetioslo.no/trygver/1979/mvc-2/1979-12-MVC.pdf.
- Shah, Ayush K., Manasi Kattel, Araju Nepal, and D. Shrestha. 2019. “Chroma Feature Extraction.” (January). https://www.researchgate.net/publication/330796993_Chroma_Feature_Extraction.
- Sheh, Alexander, and Daniel P. Ellis. 2003. “Chord Segmentation and Recognition using EM-Trained Hidden Markov Models.” https://www.researchgate.net/publication/36709060_Chord_Segmentation_and_Recognition_using_EM-Trained_Hidden_Markov_Models.
- Six, Joren, Olmo Cornelis, and Marc Leman. 2014. “TarsosDSP, a Real-Time Audio Processing Framework in Java.” Proceedings of the 53rd AES Conference (AES 53rd). http://0110.be/files/attachments/411/aes53_tarsos_dsp.pdf.
- Wendykier, Piotr. 2015. “JTransforms.” 3.1. https://github.com/wendykierp/JTransforms.
- Witte, Jorine. 2021. “AI-technology behind the chords of Chordify - our algorithm explained - Blog.” Chordify. https://chordify.net/pages/technology-algorithm-explained/.
- Zhou, Xinquan, and Alexander Lerch. 2015. “Chord Detection Using Deep Learning.” International Conference on Music Information Retrieval (ISMIR), (January). https://www.researchgate.net/publication/282859516_Chord_Detection_Using_Deep_Learning.