A virtual tutor to teach you how to play the piano. Built entirely using openCV. The virtual piano is an interface that allows users to simulate a musical instrument by printing a template on a sheet of paper. The user then ’plays’ the virtual instrument as if it were a real one. The user records this with a video camera, feeds the video to the application, and appropriate music is generated by the application. This is achieved by calibration of the template, detecting the position of fingers, identifying the hit position and playing notes corresponding to that position.
Suteparuk[3] leverages the change in activity maps caused by the depression of actual piano keys to detect keypresses. The problem with this approach is that in the case of a virtual piano, we have no key depression to leverage to detect keypresses. This scenario is much harder to detect as touches to the paper could be for a few microseconds and hence our system needs to work in real time. Rastogi and Joshi explored the idea of virtual musical instruments for multiple instruments with intriguing results[1]. They use a marker and detect the position of the marker tip to achieve a keypress. The problem with that is that a. We are restricted to the number of markers one person can hold to play b. The learning of playing this virtual instrument does not contribute to the case where one plays an actual piano. One idea that is explored in this paper is playing the instrument using fingers instead of using a specialized marker. Our work focuses on improving the constraints of this system by allowing users to play virtual piano without any form of specialized markers, with their bare hands.
[1] “Virtual Musical Instruments” http://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/Rastogi_Joshi.pdf
[2] “Digital Waveguide Architectures for Virtual Musical Instruments” - Julius O. Smith III https://ccrma.stanford.edu/~jos/asahb04/asahb04.pdf
[3] “Detection of Piano Keys Pressed in Video” - Potcharapol Suteparuk
[4] “Augmented Piano Reality” - Ihab Zaqout1 , Samar Elhissi2 , Aya Jarour3 and Heba Elowini4 http://www.sersc.org/journals/IJHIT/vol8_no10_2015/13.pdf
[5] “Detection and tracking of pianist hands and fingers” - Dmitry O. Gorodnichy1 and Arjun Yogeswaran2 http://www.videorecognition.com/doc/publications/2001-2007/06-crv-see-pianist-last-0.pdf
[6] “NYU Hand pose Dataset” http://cims.nyu.edu/~tompson/NYU_Hand_Pose_Dataset.htm
[7] “UCI Skin Segementation” https://archive.ics.uci.edu/ml/datasets/Skin+Segmentation#
[8] Barehanded Music: Real-time Hand Interaction for Virtual Piano - Hui Liang, Yong-Jin Li, Jun Luo, Ying He
[9] Depth Map Prediction from a Single Image using a Multi-Scale Deep Network - David Eigen, Christian Puhrsch, Rob Fergus https://papers.nips.cc/paper/5539-depth-map-prediction-from-a-single-image-using-a-multi-scale-deep-network.pdf
[10] A method for Stochastic optimization - https://arxiv.org/pdf/1412.6980.pdf
[11] “Estimation-theoretic approach to dynamic range enhancement using multiple exposures,” M. Robertson, S. Borman, and R. Stevenson,
[12] “FAST APPROXIMATE NEAREST NEIGHBORS WITH AUTOMATIC ALGORITHM CONFIGURATION” - Marius Muja, David G. Lowe