NAMES OF COLLABORATORS HERE
For lab this week, we focus on creating interactive systems that can detect and respond to events or stimuli in the environment of the Pi, like the Boat Detector we mentioned in lecture. Your observant device could, for example, count items, find objects, recognize an event or continuously monitor a room.
This lab will help you think through the design of observant systems, particularly corner cases that the algorithms need to be aware of.
- Install VNC on your laptop if you have not yet done so. This lab will actually require you to run script on your Pi through VNC so that you can see the video stream. Please refer to the prep for Lab 2.
- Install the dependencies as described in the prep document.
- Read about OpenCV,Pytorch, MediaPipe, and TeachableMachines.
- Read Belloti, et al.'s Making Sense of Sensing Systems: Five Questions for Designers and Researchers.
- Pull the new Github Repo
- Raspberry Pi
- Webcam
- Show pictures, videos of the "sense-making" algorithms you tried.
- Show a video of how you embed one of these algorithms into your observant system.
- Test, characterize your interactive device. Show faults in the detection and how the system handled it.
Building upon the paper-airplane metaphor (we're understanding the material of machine learning for design), here are the four sections of the lab activity:
A) Play
B) Fold
C) Flight test
D) Reflect
For this first demo, you will be using PyTorch and running a MobileNet v2 classification model in real time (30 fps+) on the CPU. We will be following steps adapted from this tutorial.
To get started, install dependencies into a virtual environment for this exercise as described in prep.md.
Make sure your webcam is connected.
You can check the installation by running:
python -c "import torch; print(torch.__version__)"
If everything is ok, you should be able to start doing object recognition. For this default example, we use MobileNet_v2. This model is able to perform object recognition for 1000 object classes (check classes.json to see which ones.
Start detection by running
python infer.py
The first 2 inferences will be slower. Now, you can try placing several objects in front of the camera.
Read the infer.py script, and get familiar with the code. You can change the video resolution and frames per second (fps). You can also easily use the weights of other pre-trained models. You can see examples of other models here.
The following sections describe tools (MediaPipe and Teachable Machines).
A recent open source and efficient method of extracting information from video streams comes out of Google's MediaPipe, which offers state of the art face, face mesh, hand pose, and body pose detection.
To get started, install dependencies into a virtual environment for this exercise as described in prep.md:
Each of the installs will take a while, please be patient. After successfully installing mediapipe, connect your webcam to your Pi and use VNC to access to your Pi, open the terminal, and go to Lab 5 folder and run the hand pose detection script we provide: (it will not work if you use ssh from your laptop)
(venv-ml) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 5
(venv-ml) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python hand_pose.py
Try the two main features of this script: 1) pinching for percentage control, and 2) "Quiet Coyote" for instant percentage setting. Notice how this example uses hardcoded positions and relates those positions with a desired set of events, in hand_pose.py.
Consider how you might use this position based approach to create an interaction, and write how you might use it on either face, hand or body pose tracking.
(You might also consider how this notion of percentage control with hand tracking might be used in some of the physical UI you may have experimented with in the last lab, for instance in controlling a servo or rotary encoder.)
Google's TeachableMachines is very useful for prototyping with the capabilities of machine learning. We are using a python package with tensorflow lite to simplify the deployment process.
To get started, install dependencies into a virtual environment for this exercise as described in prep.md:
After installation, connect your webcam to your Pi and use VNC to access to your Pi, open the terminal, and go to Lab 5 folder and run the example script: (it will not work if you use ssh from your laptop)
(venv-tml) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python tml_example.py
Next train your own model. Visit TeachableMachines, select Image Project and Standard model. The raspberry pi 4 is capable to run not just the low resource models. Second, use the webcam on your computer to train a model. Note: It might be advisable to use the pi webcam in a similar setting you want to deploy it to improve performance. For each class try to have over 150 samples, and consider adding a background or default class where you have nothing in view so the model is trained to know that this is the background. Then create classes based on what you want the model to classify. Lastly, preview and iterate. Finally export your model as a 'Tensorflow lite' model. You will find an '.tflite' file and a 'labels.txt' file. Upload these to your pi (through one of the many ways such as scp, sftp, vnc, or a connected visual studio code remote explorer).
Include screenshots of your use of Teachable Machines, and write how you might use this to create your own classifier. Include what different affordances this method brings, compared to the OpenCV or MediaPipe options.
In an earlier version of this class students experimented with observing through audio cues. Find the material here: Audio_optional/audio.md. Teachable machines provides an audio classifier too. If you want to use audio classification this is our suggested method.
In an earlier version of this class students experimented with foundational computer vision techniques such as face and flow detection. Techniques like these can be sufficient, more performant, and allow non discrete classification. Find the material here: CV_optional/cv.md.
- Pick one of the models you have tried, and experiment with prototyping an interaction.
- This can be as simple as the boat detector showen in a previous lecture from Nikolas Matelaro.
- Try out different interaction outputs and inputs.
***Describe and detail the interaction, as well as your experimentation here.***
Now flight test your interactive prototype and note down your observations: For example:
- When does it what it is supposed to do?
- When does it fail?
- When it fails, why does it fail?
- Based on the behavior you have seen, what other scenarios could cause problems?
***Think about someone using the system. Describe how you think this will work.***
- Are they aware of the uncertainties in the system?
- How bad would they be impacted by a miss classification?
- How could change your interactive system to address this?
- Are there optimizations you can try to do on your sense-making algorithm.
Now that you have experimented with one or more of these sense-making systems characterize their behavior. During the lecture, we mentioned questions to help characterize a material:
- What can you use X for?
- What is a good environment for X?
- What is a bad environment for X?
- When will X break?
- When it breaks how will X break?
- What are other properties/behaviors of X?
- How does X feel?
***Include a short video demonstrating the answers to these questions.***
Following exploration and reflection from Part 1, finish building your interactive system, and demonstrate it in use with a video.
***Include a short video demonstrating the finished result.***