Handwritten Digit Recognition

The goal of the project is to implement CNN based digit-recognition system in a edge and resource constrained device. More specifically, I used a esp32-cam module, which have 2mp camera, 512kb RAM and 4Mb flash memory. My object is to capture image in a loop, crop that image, send to the cnn model’s input tensor, run the model ( invoke ) and output the result based on maximum probability returned

Install the ESP IDF

• Guide : https://rupakbiswas-2304.github.io/esp32/installation/

File Structure:

• main.cc : main file that runs
• model_data.h : contains the model’s input and output tensor
• model_data_archive.h : contains the old model’s weights and bias ( not used )
• CMakeLists.txt : cmake file to build the project
• Makefile : make file to build the project
• README.md : this file
• image_provider.h : contains the image provider function definition
• image_provider.cc : contains the image provider function implementation

Dependencies

• IDF
• ESP32-CAM
• ESP32-CAM Arduino Library
• Tensorflow Lite for Microcontrollers
• Tensorflow
• Matplotlib

Building the Model

• Goto the python folder
• run the main.py file
• It will download the mnist dataset and train the model
• It will save the model in the models folder
• Next for converting the model, run the convert_lite.py file
• It will convert the model to tflite format
• For testing the file, run python test.py

Load and run the example

To flash (replace /dev/ttyUSB0 with the device serial port):

idf.py --port /dev/ttyUSB0 flash

Monitor the serial output:

idf.py --port /dev/ttyUSB0 monitor

Use Ctrl+] to exit.

The previous two commands can be combined:

idf.py --port /dev/ttyUSB0 flash monitor

Some Output Dump

Getting image... Image Captured ... Image Size: 320 x 240 Printing Image -89 -88 -90 -88 -90 -85 -83 -90 -88 -86 -84 -85 -88 -85 -87 -84 -84 -86 -88 -85 -83 -86 -86 -85 -83 -85 -84 -89 -92 -92 -90 -92 -93 -91 -88 -87 -99 -92 -93 -93 -93 -93 -95 -93 -91 -91 -90 -94 -90 -89 -94 -95 -93 -94 -95 -96 -88 -89 -87 -93 -91 -91 -98 -91 -92 -91 -92 -94 -93 -93 -91 -91 -92 -92 -93 -96 -92 -93 -94 -85 -89 -92 -93 -92 -91 -89 -89 -96 -92 -89 -96 -82 -90 -91 -94 -92 -92 -93 -90 -89 -93 -87 -93 -92 -87 -96 -94 -90 -93 -94 -92 -91 -93 -85 -79 -94 -93 -94 -92 -86 -85 -92 -92 -90 -90 -94 -92 -90 -92 -95 -94 -92 -96 -85 -91 -94 -94 -92 -94 -89 -94 -90 -86 -91 -93 -92 -93 -93 -83 -90 -89 -91 -93 -93 -92 -90 -91 -95 -94 -95 -96 -83 -92 -91 -93 -92 -93 -91 -94 -91 -89 -90 -93 -89 -92 -93 -89 -94 -92 -89 -92 -91 -91 -89 -87 -95 -93 -92 -95 -93 -94 -92 -91 -95 -95 -93 -92 -93 -92 -92 -91 -90 -90 -92 -92 -94 -92 -87 -91 -91 -89 -94 -95 -93 -93 -92 -95 -92 -92 -90 -91 -94 -96 -91 -91 -88 -90 -91 -92 -91 -94 -94 -94 -95 -89 -84 -91 -94 -97 -94 -95 -82 -89 -93 -92 -93 -94 -92 -95 -95 -94 -90 -96 -88 -88 -90 -91 -92 -91 -91 -92 -93 -88 -90 -89 -93 -82 -82 -89 -88 -88 -92 -92 -94 -93 -92 -94 -94 -93 -90 -97 -93 -92 -92 -90 -90 -93 -92 -92 -98 -92 -92 -92 -95 -95 -90 -92 -91 -92 -90 -91 -94 -93 -93 -92 -94 -94 -98 -93 -89 -91 -92 -91 -90 -92 -93 -93 -94 -94 -93 -94 -93 -94 -94 -93 -94 -93 -92 -91 -93 -94 -92 -92 -94 -93 -97 -92 -91 -93 -89 -91 -91 -92 -93 -92 -93 -90 -91 -94 -93 -94 -94 -94 -93 -93 -94 -92 -93 -93 -92 -92 -92 -91 -92 -92 -91 -93 -94 -91 -92 -93 -95 -92 -93 -90 -89 -92 -92 -92 -93 -94 -91 -91 -93 -94 -93 -89 -88 -93 -89 -89 -88 -90 -90 -91 -92 -92 -90 -91 -91 -93 -93 -93 -93 -95 -92 -93 -92 -91 -92 -93 -91 -92 -93 -90 -92 -88 -90 -94 -93 -95 -91 -90 -91 -91 -90 -91 -93 -93 -93 -91 -92 -91 -90 -94 -92 -88 -86 -92 -87 -91 -93 -90 -89 -90 -92 -92 -87 -95 -93 -93 -93 -92 -93 -93 -94 -93 -94 -91 -91 -86 -84 -93 -90 -88 -87 -91 -92 -93 -93 -91 -92 -92 -93 -91 -86 -88 -88 -90 -94 -92 -89 -90 -91 -91 -94 -90 -89 -89 -93 -93 -90 -90 -89 -91 -93 -93 -84 -90 -91 -91 -93 -91 -87 -85 -90 -89 -95 -92 -91 -89 -90 -91 -93 -89 -88 -88 -88 -90 -90 -88 -88 -91 -82 -89 -88 -88 -86 -88 -91 -90 -96 -89 -76 -90 -89 -91 -90 -93 -93 -92 -91 -90 -90 -88 -88 -87 -92 -91 -90 -92 -89 -92 -91 -91 -89 -88 -88 -89 -95 -88 -90 -91 -90 -90 -91 -89 -85 -92 -84 -86 -89 -92 -86 -95 -96 -92 -92 -92 -92 -91 -89 -89 -89 -96 -89 -87 -93 -96 -94 -89 -88 -91 -91 -94 -88 -85 -83 -89 -90 -91 -89 -90 -88 -91 -91 -90 -89 -90 -92 -91 -92 -92 -91 -88 -91 -92 -91 -92 -88 -92 -93 -87 -88 -90 -92 -89 -88 -89 -91 -91 -90 -90 -87 -91 -88 -91 -92 -90 -89 -91 -90 -92 -90 -91 -92 -93 -93 -92 -91 -90 -88 -84 -92 -89 -82 -91 -89 -91 -89 -88 -84 -90 -88 -90 -90 -89 -87 -90 -92 -92 -89 -95 -82 -93 -90 -88 -85 -89 -87 -86 -96 -88 -87 -87 -90 -90 -87 -81 -92 -87 -90 -92 -91 -89 -89 -91 -94 -88 -90 -93 -90 -90 -87 -87 -87 -90 -88 -89 -93 -88 -86 -80 -87 -85 -88 -91 -91 -91 -90 -87 -86 -88 -85 -90 -88 -86 -90 -91 -91 -91 -88 -81 -86 -91 -91 -82 -90 -88 -87 -85 -87 -86 -92 -88 -89 -90 -89 -88 -93 -88 -87 -88 -87 -87 -88 -91 -92 -91 -88 -86 -85 -87 -86 -84 -92 -86 -86 -82 -88 -86 -86 -85 -88 -89 -88 -89 -90 -90 -90 -88 -83 -87 -93

Quantized ! Number of Elements = 10 prediction = 7 Getting image... Image Captured ... Image Size: 320 x 240 Printing Image -83 -84 -82 -81 -83 -78 -76 -80 -81 -82 -81 -79 -82 -80 -79 -81 -80 -83 -79 -80 -80 -81 -81 -80 -77 -76 -77 -92 -94 -94 -93 -92 -90 -83 -91 -90 -89 -90 -87 -87 -88 -88 -86 -92 -89 -91 -91 -88 -86 -88 -90 -85 -86 -91 -88 -92 -93 -91 -91 -93 -88 -91 -90 -91 -90 -90 -89 -87 -88 -85 -83 -89 -89 -89 -91 -91 -92 -90 -89 -90 -89 -90 -90 -93 -94 -91 -90 -90 -91 -91 -87 -92 -90 -91 -90 -90 -88 -85 -86 -88 -90 -90 -91 -89 -89 -91 -89 -90 -91 -93 -91 -93 -92 -93 -91 -90 -91 -90 -89 -92 -92 -90 -89 -88 -87 -91 -89 -89 -91 -89 -89 -88 -88 -89 -91 -92 -92 -92 -92 -90 -90 -90 -91 -90 -89 -88 -91 -90 -89 -89 -89 -88 -89 -89 -90 -90 -91 -85 -89 -88 -88 -89 -90 -91 -92 -90 -90 -90 -90 -90 -91 -93 -92 -90 -91 -90 -89 -91 -90 -91 -90 -87 -89 -90 -90 -89 -89 -91 -91 -90 -88 -88 -91 -90 -90 -92 -90 -90 -92 -92 -91 -88 -89 -91 -91 -90 -90 -91 -89 -88 -88 -89 -90 -91 -89 -89 -88 -89 -89 -89 -89 -90 -89 -90 -89 -90 -92 -91 -91 -89 -88 -91 -90 -89 -89 -87 -88 -88 -88 -90 -89 -89 -89 -90 -89 -89 -90 -90 -90 -89 -89 -90 -91 -90 -90 -90 -90 -89 -90 -91 -91 -89 -89 -89 -89 -90 -87 -86 -89 -90 -90 -89 -90 -90 -89 -91 -90 -90 -91 -89 -91 -92 -91 -91 -89 -89 -89 -91 -89 -90 -91 -90 -91 -90 -87 -86 -89 -89 -89 -88 -89 -90 -91 -90 -90 -90 -91 -90 -89 -88 -90 -90 -90 -86 -89 -90 -89 -90 -88 -89 -91 -90 -89 -91 -91 -89 -89 -87 -88 -91 -90 -90 -90 -91 -89 -91 -89 -89 -92 -91 -90 -90 -89 -89 -89 -90 -88 -89 -91 -89 -90 -90 -87 -90 -88 -86 -88 -89 -90 -88 -90 -91 -89 -91 -89 -90 -92 -90 -85 -88 -89 -89 -89 -90 -89 -89 -91 -88 -88 -89 -88 -88 -86 -89 -89 -89 -88 -86 -87 -90 -88 -89 -89 -89 -90 -90 -88 -85 -89 -90 -90 -87 -88 -90 -91 -90 -89 -88 -88 -89 -85 -89 -88 -88 -88 -86 -87 -88 -88 -89 -89 -89 -88 -89 -88 -88 -90 -89 -88 -88 -89 -90 -91 -92 -89 -88 -87 -87 -89 -88 -88 -89 -86 -85 -89 -88 -88 -90 -90 -87 -88 -89 -89 -86 -90 -89 -88 -89 -89 -88 -88 -90 -90 -87 -87 -88 -89 -88 -87 -88 -86 -82 -83 -87 -88 -85 -87 -87 -91 -91 -89 -88 -90 -90 -90 -89 -89 -88 -88 -87 -85 -86 -88 -87 -87 -87 -88 -88 -87 -86 -88 -88 -88 -80 -85 -88 -91 -90 -89 -88 -89 -89 -89 -89 -86 -84 -86 -81 -82 -87 -87 -87 -87 -82 -83 -89 -86 -86 -88 -86 -86 -89 -90 -87 -86 -90 -89 -87 -89 -87 -88 -88 -87 -86 -83 -75 -90 -87 -86 -86 -86 -84 -85 -89 -87 -87 -86 -85 -86 -88 -90 -86 -86 -89 -90 -89 -88 -86 -87 -86 -87 -87 -87 -85 -88 -85 -86 -84 -85 -86 -87 -85 -87 -85 -86 -86 -85 -90 -89 -88 -88 -89 -88 -88 -86 -84 -85 -88 -86 -86 -86 -85 -84 -83 -84 -83 -86 -84 -86 -86 -85 -84 -83 -84 -85 -87 -81 -94 -83 -88 -86 -86 -85 -85 -84 -85 -85 -85 -86 -83 -82 -82 -86 -86 -85 -84 -85 -88 -87 -86 -86 -85 -84 -86 -85 -86 -87 -86 -84 -84 -86 -86 -84 -86 -85 -84 -85 -84 -83 -85 -84 -85 -85 -84 -83 -84 -81 -84 -85 -84 -86 -86 -84 -86 -84 -85 -85 -85 -88 -85 -86 -87 -84 -85 -84 -84 -85 -84 -85 -85 -85 -84 -83 -83 -84 -85 -83 -84 -84 -82 -85 -86 -85 -85 -85 -85 -85 -83 -86 -81 -83 -80 -83 -82 -83 -84 -84 -84 -83 -84 -84 -84 -86 -83 -84 -85 -83 -86 -86 -85 -84 -84 -83 -85 -85 -83 -84 -82 -84 -83 -85 -83 -84 -83 -83 -83 -82 -78 -80 -84 -84 -83 -86 -84 -84 -84 -85 -82 -82 -84 -83 -83 -84 -81 -82 -82 -82 -80 -84 -85 -85 -83 -82 -83 -81 -80 -80 -82 -81 -81 -83 -82 -81 -83

Quantized ! Number of Elements = 10 prediction = 8 Getting image... Image Captured ... Image Size: 320 x 240 Printing Image 97 101 106 111 114 118 123 126 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 10 15 20 29 34 38 43 48 52 55 59 61 63 65 67 68 70 71 73 74 75 75 77 77 79 80 80 82 17 21 26 34 39 44 49 52 55 58 61 63 65 68 70 70 71 73 76 77 79 80 81 81 82 83 84 85 23 27 31 38 44 49 52 55 59 61 63 65 67 69 71 72 75 77 78 79 82 83 83 85 87 87 88 88 26 30 36 41 47 51 54 56 60 62 65 67 69 71 73 75 77 78 80 83 84 85 85 87 89 89 89 91 31 35 39 44 49 55 56 57 61 64 68 70 72 75 76 77 80 81 83 85 86 86 88 89 90 92 93 93 36 40 45 47 51 56 57 59 64 66 69 71 74 76 78 80 82 83 85 86 88 90 91 92 93 94 95 96 41 44 47 50 53 58 61 62 66 69 71 74 76 78 82 83 84 86 89 90 91 93 93 94 95 96 97 98 43 46 50 51 54 59 63 65 68 72 74 75 79 81 82 84 86 87 90 92 94 95 96 97 98 98 99 100 45 47 52 55 56 60 64 67 71 73 75 77 82 84 85 87 90 91 93 94 96 98 99 102 102 102 102 104 47 49 53 56 58 62 67 68 71 75 77 79 83 85 87 90 92 93 95 97 99 100 102 102 103 105 106 106 47 51 56 58 61 63 69 70 73 77 81 82 85 87 90 91 94 96 98 99 101 103 104 105 106 108 108 110 50 52 58 61 63 66 71 73 75 79 83 85 88 90 91 94 98 99 102 102 104 106 106 108 110 110 111 112 53 55 59 63 65 68 72 74 77 81 84 88 91 92 95 97 100 102 105 105 107 108 110 111 112 113 115 116 54 57 62 66 68 71 74 76 79 83 87 91 93 95 97 100 102 104 106 107 110 111 113 114 115 116 119 119 57 59 63 68 69 72 77 79 82 85 89 92 94 97 100 102 104 106 108 109 113 113 116 117 119 119 120 121 61 62 65 70 73 74 80 82 85 88 91 94 97 100 103 105 109 109 112 113 115 117 119 120 122 123 125 124 62 64 68 72 75 77 82 84 88 90 94 97 100 103 107 108 111 112 114 116 118 121 123 124 125 126 127 127 65 67 70 74 78 80 84 88 91 92 97 101 106 106 110 111 114 115 118 120 123 124 126 127 127 127 127 127 66 68 72 76 80 82 86 90 92 95 99 103 106 108 111 113 116 119 121 123 125 126 127 127 127 127 127 127 68 70 72 78 82 84 88 91 95 96 100 105 108 110 112 114 118 120 123 125 127 127 127 127 127 127 127 127 70 74 76 81 84 86 91 96 98 99 103 108 111 114 116 118 121 123 127 127 127 127 127 127 127 127 127 127 72 76 79 83 87 90 92 97 101 102 105 109 114 117 120 122 125 126 127 127 127 127 127 127 127 127 127 127 74 78 81 85 91 93 95 99 104 106 109 113 117 120 124 126 127 127 127 127 127 127 127 127 127 127 127 127 76 80 84 86 92 95 97 102 105 108 111 116 119 121 125 127 127 127 127 127 127 127 127 127 127 127 127 127 80 83 87 89 95 98 101 104 108 113 115 118 122 125 127 127 127 127 127 127 127 127 127 127 127 127 127 127 82 86 90 93 98 101 105 108 113 116 118 121 125 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 85 88 92 96 100 105 108 111 116 119 122 124 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 127 86

Quantized ! Number of Elements = 10 prediction = 5