The assistive-rehab project

[pattacini]

About

assistive-rehab is a project carried out in collaboration with Fondazione Don Carlo Gnocchi to develop the capabilities of R1 for rehabilitation purposes and elderly care.
Scenarios included are:

1. hospital scenario, with the robot at the hospital having to:

   • assist and monitor patients during daily rehabilitation;
   • help patients to execute clinical tests and share the results with clinicians;
   • guide patients from their room to the gym;

2. domestic scenario, with the robot at home having to:

   • assist the patient (detect falls, manage his/her physical activity, execute fetch and carry tasks...);
   • tele-rehabilitate the patient (the therapist remotely controls the movement of the robot, which in turn assesses in real-time the performance of the activity).

During the first year of the project, we have focused on the hospital scenario, in particular on assisting and monitoring patients in the rehabilitation of upper limbs. We have developed an application where R1 shows with its arms to the user a physical exercise of the upper limbs while verifying in real-time the correct execution and providing verbal feedback.

Software

The framework developed includes the following features:

• 3D skeleton reconstruction, which merges 2D skeletons from yarpOpenPose and depth map from a RGB-D camera, including real-time optimization to retrieve keypoints not directly observable in 2D:

• action recognition, which relies on Long Short Term Memory network (LSTM) using skeleton's 2D keypoints as input. We get the following confusion matrix using as actions 4 exercises plus random motion and no motion (static):

• motion analysis, which relies on Dynamic Time Warping for temporal alignment and Fourier Transform for speed analysis;
• verbal feedback, provided to the user;
• offline interactive report, generated after the exercise session, which includes patient progress:

The software is open-source, available at https://github.com/robotology/assistive-rehab. The website points at the documentation and includes some tutorials.

Next

We plan to focus on the remaining points of the hospital scenario. Clinical tests are very useful for clinicians to get quantitative and objective data.

The complete application will include the robot going to the patient's room and guiding him to the area where the test will be performed. The robot will help the patient in executing the tests while extracting real-time metrics and sending the complete data to the clinicians.

Credits go to @vvasco ✨