Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Berry: add support for Tensorflow Lite for microcontrollers (including speech input) #18119

Merged
merged 1 commit into from
Mar 5, 2023

Conversation

Staars
Copy link
Contributor

@Staars Staars commented Mar 5, 2023

Description:

Add module TFL to Berry in order to run generic TensorFlow Lite models.

Uses optimizations for ESP32 platform (kernels, ESP-DSP, RTOS) where possible.

Avoids boiler plate code as much as possible with a very short API (might be extended in the future if needed).

Can run specialized sessions to fulfill realtime requirements - first example is speech recognition, that can run on a simple ESP32 (no PSRAM mandatory, although it would be utilized). Will play nicely with Edgeimpulse, that provides an easy way to train a model (with compatible feature extraction!) from scratch.
IMPORTANT NOTE: Speech recognition on an ESP32 will never be as robust as on much more powerful smartphone, smart speakers and similar equipment!! But it can do some things surprisingly well.

Initial documentation: https://staars.github.io/docs/TFL/

Checklist:

  • The pull request is done against the latest development branch
  • Only relevant files were touched
  • Only one feature/fix was added per PR and the code change compiles without warnings
  • The code change is tested and works with Tasmota core ESP8266 V.2.7.4.9
  • The code change is tested and works with Tasmota core ESP32 V.2.0.7
  • I accept the CLA.

NOTE: The code change must pass CI tests. Your PR cannot be merged unless tests pass

@arendst arendst merged commit 83f039c into arendst:development Mar 5, 2023
@Jason2866
Copy link
Collaborator

Wow! Amazing work

@s-hadinger
Copy link
Collaborator

Awesome!

@TheChatty
Copy link
Contributor

Amazing... can TFL also be used in to recognize water meter digits like in AI-on-the-edge?

@Staars
Copy link
Contributor Author

Staars commented Mar 6, 2023

Not yet, but this is on the to-do-list.
In fact my initial plan was to start with image recognition and especially digits recognition, but after playing with the speech recognition training pipeline of EdgeImpulse I was too curious to see, if this can be done for Tasmota.
For the next time I would like to consolidate the speech recognition part and provide simpler Berry templates.
After that I will try to add the webcam/image part, but I can not provide an exact time line.

@Staars Staars deleted the tflm branch March 11, 2023 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants