Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World

Air-writing is a challenging task that combines the fields of computer vision and natural language processing, offering an intuitive and natural approach for human-computer interaction. However, current air-writing solutions face two primary challenges: (1) their dependency on complex sensors ($e.g.$, Radar, EEGs and others) for capturing precise handwritten trajectories, and (2) the absence of a video-based air-writing dataset that covers a comprehensive vocabulary range. These limitations impede their practicality in various real-world scenarios, including the use on devices like iPhones and laptops. To tackle these challenges, we present the groundbreaking air-writing Chinese character video dataset (AWCV-100K), serving as a pioneering benchmark for video-based air-writing. This dataset captures handwritten trajectories in various real-world scenarios using commonly accessible RGB cameras, eliminating the need for complex sensors. AWCV-100K includes 8.8 million video frames, encompassing the complete set of 3,755 characters from the GB2312-80 level-1 set (GB1). Furthermore, we introduce our baseline approach, the video-based character recognizer (VCRec). VCRec adeptly extracts fingertip features from sparse visual cues and employs a spatio-temporal sequence module for analysis. Experimental results showcase the superior performance of VCRec compared to existing models in recognizing air-written characters, both quantitatively and qualitatively. This breakthrough paves the way for enhanced human-computer interaction in real-world contexts. Moreover, our approach leverages affordable RGB cameras, enabling its applicability in a diverse range of scenarios.

AWCV-100K-UCAS2024

AWCV-100K-UCAS2024 is a high-quality and large-scale benchmark to create a challenging real-world experimental environment for Air-Writing.

Demo

AWCV-100K.mp4

Download

Baiduyun Disk AWCV-100K-UCAS2024

Latest News

• [2024.4.2] Accepted by TCSVT
• [2024.3.25] TCSVT Minor Revision
• [2024.2.12] TCSVT Major Revision
• [2023.12.23] The paper “Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World” was submitted in TCSVT.

Publications

"Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World" Accepted by TCSVT.

More Details

To be continued...

Website License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.