speakAR is a mobile application powered by Natural Language Processing and 3D Generative AI. This application transforms human speech into 3D models that you can view in your surroundings through augmented reality. With speakAR, you can bring your words to life and immerse yourself in a world of interactive visualizations.
This is a state-of-the-art visualisation tool with infinite use-cases. speakAR will be leveraged by other developers for artistic expression, experimental ventures, and other practical scenarios. For example, its scope will be expanded to education as a visual supplement to verbal lectures or as a learning aid for neurodiverse learners. It will also be used in healthcare to visualize detailed anatomy of the human body. In short, 1 tech, endless possibilities.
- Natural Language Processing (NLP): Converts complex speech (natural language) to text.
- 3D Generative AI: Uses MirageML, DreamFusion (diffusion model), PointE, and ShapeE to generate 3D models of the inputted speech and ensures conversion from text to 3D is accurate, smooth, and free of distortions.
- Augmented Reality (AR) Integration: Overlays the generated 3D visuals onto the real world for an immersive experience using ARKit/ARCore.
- Mobile Application: Cross-platform interactive application (Android and iOS) created using Flutter, Dart, XCode, and Android Studio.
- Launch the App: Tap on the
speakARicon on your device. - Speak: Describe the image you want to visualize in natural language.
- Instant Visualization: Watch as your description is instantly converted into a 3D model and overlayed onto your surroundings.
- Interact: Use touch or gesture controls to interact with the image, such as rotating or animating it.
