Using a Leap Motion Controller, this program will search for specific hand signals and interpret it using a text-to-speech service.
Sign language is a useful tool for the hearing impaired and those who are vocally challenged. However, it is still not widely known by the masses. Only 0.002% of the US population are able to use and understand sign language, which requires interpreters to communicate these people’s messages.
Technology is rapidly evolving, with real-time verbal translation on its way. Using a Leap Motion Controller, we are able to translate sign language for the non-verbal to better communicate with those without relying on the use of an interpreter or the other person knowing sign language themselves.
Interpretr. was produced using the Java programming language and the Leap Motion API. These two technologies allowed our team to effectively use the Leap Motion Controller to create our prototype sign language interpreter. Additionally, Interpretr. uses the FreeTTS API to give aural feedback to the user when they successfully sign a specific phrase or word.
- Install a Java IDE such as Eclipse or NetBeans
- Download this project and import it into your workspace within the IDE
- Execute the project from the Main.java class
- Plug in a Leap Motion Controller via USB (make sure the drivers are correctly installed)
- Execute the project and run the GUI
- As of this time, only three specific sign languages are recognized:
- "Hello" (all fingers up)
- "I love you" (thumb, index finger and pinky finger up)
- "I understand" (only index finger up)
Stefano Gregor Unlayao, Rafael Flora, Jeffrey Jaca & Michael Marc