Skip to content

Commit

Permalink
- Fix: image convertes
Browse files Browse the repository at this point in the history
- Add: Camera version driver
- Performance improvements
- Minor imprvements
  • Loading branch information
cnadler86 committed Dec 2, 2024
1 parent eae7247 commit 176b0a3
Show file tree
Hide file tree
Showing 10 changed files with 384 additions and 229 deletions.
10 changes: 7 additions & 3 deletions .github/workflows/ESP32.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ jobs:
cd esp-idf
./install.sh all
cd components
git clone https://github.com/cnadler86/esp32-camera
latest_cam_driver=$(curl -s https://api.github.com/repos/espressif/esp32-camera/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
git clone --depth 1 --branch $latest_cam_driver https://github.com/espressif/esp32-camera.git
# git clone https://github.com/cnadler86/esp32-camera.git
cd ~/esp-idf/
source ./export.sh
Expand Down Expand Up @@ -145,6 +147,8 @@ jobs:
# Build MicroPython for each board
- name: Build MicroPython
run: |
cd ~/esp-idf/components/esp32-camera
CAM_DRIVER=$(git describe --tags --always --dirty)
cd ~/micropython/ports/esp32
source ~/esp-idf/export.sh
Expand All @@ -153,9 +157,9 @@ jobs:
IFS='-' read -r BOARD_NAME BOARD_VARIANT <<< "${BUILD_TARGET}"
if [ -n "${BOARD_VARIANT}" ]; then
IDF_CMD="idf.py -D MICROPY_BOARD=$BOARD_NAME -D USER_C_MODULES=${{ github.workspace }}/src/micropython.cmake -D MICROPY_BOARD_VARIANT=$BOARD_VARIANT -B build-$BUILD_TARGET"
IDF_CMD="idf.py -D MICROPY_BOARD=$BOARD_NAME -D USER_C_MODULES=${{ github.workspace }}/src/micropython.cmake -D MICROPY_BOARD_VARIANT=$BOARD_VARIANT -B build-$BUILD_TARGET -D MP_CAMERA_DRIVER_VERSION=$CAM_DRIVER"
else
IDF_CMD="idf.py -D MICROPY_BOARD=$BOARD_NAME -D USER_C_MODULES=${{ github.workspace }}/src/micropython.cmake -B build-$BUILD_TARGET"
IDF_CMD="idf.py -D MICROPY_BOARD=$BOARD_NAME -D USER_C_MODULES=${{ github.workspace }}/src/micropython.cmake -B build-$BUILD_TARGET -D MP_CAMERA_DRIVER_VERSION=$CAM_DRIVER"
fi
if [ -n "${CAMERA_MODEL}" ]; then
echo "FW_NAME=${CAMERA_MODEL}" >> $GITHUB_ENV
Expand Down
74 changes: 48 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

[![ESP32 Port](https://github.com/cnadler86/micropython-camera-API/actions/workflows/ESP32.yml/badge.svg)](https://github.com/cnadler86/micropython-camera-API/actions/workflows/ESP32.yml)

This project aims to support various cameras on different MicroPython ports, starting with the ESP32 port and Omnivision (OV2640 & OV5640) cameras. The project implements a general API for cameras in micropython (such as circuitpython have done).
This project aims to support various cameras (e.g. OV2640, OV5640) on different MicroPython ports, starting with the ESP32 port. The project implements a general API, has precompiled FW images and supports a lot of cameras out of the box.
At the moment, this is a micropython user module, but it might get in the micropython repo in the future.
The API is stable, but it might change without previous announce.

Expand Down Expand Up @@ -134,6 +134,17 @@ See autocompletions in Thonny in order to see the list of methods.
If you want more insides in the methods and what they actually do, you can find a very good documentation [here](https://docs.circuitpython.org/en/latest/shared-bindings/espcamera/index.html).
Note that each method requires a "get_" or "set_" prefix, depending on the desired action.

To get the version of the camera driver used:

```python
import camera
vers = camera.Version()
```

### Additional information

The FW images support the following cameras out of the box, but is therefore big: OV7670, OV7725, OV2640, OV3660, OV5640, NT99141, GC2145, GC032A, GC0308, BF3005, BF20A6, SC030IOT

## Build your custom FW

### Setting up the build environment (DIY method)
Expand All @@ -142,20 +153,20 @@ To build the project, follow these instructions:

- [ESP-IDF](https://docs.espressif.com/projects/esp-idf/en/v5.2.3/esp32/get-started/index.html): I used version 5.2.3, but it might work with other versions (see notes).
- Clone the micropython repo and this repo in a folder, e.g. "MyESPCam". MicroPython version 1.24 or higher is required (at least commit 92484d8).
- You will have to add the ESP32-Camera driver (I used v2.0.13). To do this, add the following to the respective idf_component.yml file (e.g. in micropython/ports/esp32/main_esp32s3/idf_component.yml):
- You will have to add the ESP32-Camera driver (I used v2.0.15). To do this, add the following to the respective idf_component.yml file (e.g. in micropython/ports/esp32/main_esp32s3/idf_component.yml):

```yml
espressif/esp32-camera:
git: https://github.com/cnadler86/esp32-camera #At the moment I maintain a fork because of some unsolved bugs and conveniance.
git: https://github.com/espressif/esp32-camera.git
```
Alternatively, you can clone the <https://github.com/cnadler86/esp32-camera> repository inside the esp-idf/components folder instead of altering the idf_component.yml file.
Alternatively, you can clone the <https://github.com/espressif/esp32-camera> repository inside the esp-idf/components folder instead of altering the idf_component.yml file.
### Add camera configurations to your board (optional, but recommended)
#### Supported Camera Models
This project supports various camera models out of the box. You typically only need to add a single line to your board config file ("mpconfigboard.h).
This project supports various boards with camera interface out of the box. You typically only need to add a single line to your board config file ("mpconfigboard.h).
Example (don't forget to add the empty line at the bottom):
```c
Expand Down Expand Up @@ -188,7 +199,6 @@ Below is a list of supported `MICROPY_CAMERA_MODEL_xxx` definitions:
#### For unsupported camera models

If your board is not yet supported, add the following lines to your board config-file "mpconfigboard.h" with the respective pins and camera parameters. Otherwise, you will need to pass all parameters during construction.
Don't forget the empty line at the bottom.
Example for Xiao sense:

```c
Expand All @@ -214,6 +224,9 @@ Example for Xiao sense:
#define MICROPY_CAMERA_GRAB_MODE (1) // 0=WHEN_EMPTY (might have old data, but less resources), 1=LATEST (best, but more resources)

```
#### Customize additional camera settings

If you want to customize additional camera setting or reduce the FW size by removing support for unused camera sensors, then take a look at the kconfig file of the esp32-camera driver and specify these on the sdkconfig file of your board.

### Build the API

Expand All @@ -239,28 +252,37 @@ If you experience problems, visit [MicroPython external C modules](https://docs.

## FPS benchmark

I didn't use a calibrated osziloscope, but here is a benchmark with my ESP32S3 (GrabMode=LATEST).
Using fb_count=2 doubles the FPS for JPEG. This might also aplly for other PixelFormats.

| Frame Size | GRAYSCALE | RGB565 | YUV422 | JPEG | JPEG (fb = 2) |
|------------|-----------|--------|--------|--------|---------------|
| R96X96 | 12.5 | 12.5 | 12.5 | No img | No img |
| QQVGA | 12.5 | 12.5 | 12.5 | 25 | 50 |
| QCIF | 11 | 11 | 11.5 | 25 | 50 |
| HQVGA | 12.5 | 12.5 | 12.5 | 25 | 50 |
| R240X240 | 12 | 12.5 | 11.5 | 25 | 50 |
| QVGA | 12 | 11 | 12 | 25 | 50 |
| CIF | 12.5 | No img | No img | 6 | 12.5 |
| HVGA | 2.5 | 3 | 2.5 | 12.5 | 25 |
| VGA | 3 | 3 | 3 | 12.5 | 25 |
| SVGA | 3 | 3 | 3 | 12.5 | 25 |
| XGA | No img | No img | No img | 6 | 12.5 |
| HD | No img | No img | No img | 6 | 12.5 |
| SXGA | 2 | 2 | 2 | 6 | 12.5 |
| UXGA | No img | No img | No img | 6 | 12.5 |
I didn't use a calibrated osziloscope, but here is a benchmark with my ESP32S3 (GrabMode=LATEST, fb_count = 1, jpeg_quality=85%).
Using fb_count=2 theoretically can double the FPS (see JPEG with fb_count=2). This might also aplly for other PixelFormats.

| Frame Size | GRAYSCALE | RGB565 | YUV422 | JPEG | JPEG -> RGB565 | JPEG -> RGB888 | JPEG (fb=2) |
|------------|-----------|--------|--------|--------|----------------|----------------|-------------|
| R96X96 | 12.5 | 12.5 | 12.5 | No img | No img | No img | No img |
| QQVGA | 12.5 | 12.5 | 12.5 | 25 | 25 | 25 | 50 |
| QCIF | 11 | 11 | 11.5 | 25 | 25 | 25 | 50 |
| HQVGA | 12.5 | 12.5 | 12.5 | 25 | 16.7 | 16.7 | 50 |
| R240X240 | 12.5 | 12.5 | 11.5 | 25 | 16.7 | 12.5 | 50 |
| QVGA | 12 | 11 | 12 | 25 | 12.5 | 12.5 | 50 |
| CIF | 12.5 | No img | No img | 6.3 | 1.6 | 1.6 | 12.5 |
| HVGA | 3 | 3 | 2.5 | 12.5 | 6.3 | 6.3 | 25 |
| VGA | 3 | 3 | 3 | 12.5 | 3.6 | 3.6 | 25 |
| SVGA | 3 | 3 | 3 | 12.5 | 2.8 | 2.5 | 25 |
| XGA | No img | No img | No img | 6.3 | 1.6 | 1.6 | 12.5 |
| HD | No img | No img | No img | 6.3 | 1.4 | 1.3 | 12.5 |
| SXGA | 2 | 2 | 2 | 6.3 | 1 | 1 | 12.5 |
| UXGA | No img | No img | No img | 6.3 | 0.7 | 0.7 | 12.5 |


Looking at the results: image conversion make only sense for frame sized below QVGA or if capturing the image in the intended pixelformat and frame size combination fails.

## Troubleshoot

You can find information on the following sites:
- [ESP-FAQ](https://docs.espressif.com/projects/esp-faq/en/latest/application-solution/camera-application.html)
- [ChatGPT](https://chatgpt.com/)
- [Issues in here](https://github.com/cnadler86/micropython-camera-API/issues?q=is%3Aissue)

## Future Plans

- Edge case: enable usage of pins such as i2c for other applications
- Provide examples in binary image
- Include camera driver version in API
3 changes: 2 additions & 1 deletion examples/CameraSettings.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ async def stream_camera(writer):
cam.init()
if not cam.get_bmp_out() and cam.get_pixel_format() != PixelFormat.JPEG:
cam.set_bmp_out(True)

await asyncio.sleep(1)

writer.write(b'HTTP/1.1 200 OK\r\nContent-Type: multipart/x-mixed-replace; boundary=frame\r\n\r\n')
await writer.drain()

Expand Down
21 changes: 11 additions & 10 deletions examples/benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,21 @@
gc.enable()

def measure_fps(duration=2):
start_time = time.time()
while time.time() - start_time < 0.5:
start_time = time.ticks_ms()
while time.ticks_ms() - start_time < 500:
cam.capture()

start_time = time.time()
start_time = time.ticks_ms()
frame_count = 0

while time.time() - start_time < duration:
cam.capture()
frame_count += 1
while time.ticks_ms() - start_time < duration*1000:
img = cam.capture()
if img:
frame_count += 1

end_time = time.time()
fps = frame_count / (end_time - start_time)
return fps
end_time = time.ticks_ms()
fps = frame_count / (end_time - start_time) * 1000
return round(fps,1)

def print_summary_table(results, cam):
print(f"\nBenchmark {os.uname().machine} with {cam.get_sensor_name()}, GrabMode: {cam.get_grab_mode()}:")
Expand Down Expand Up @@ -78,7 +79,7 @@ def print_summary_table(results, cam):
print('Set', p, f,f'fb={fb}',':')

try:
cam.reconfigure(frame_size=f_value)
cam.reconfigure(frame_size=f_value) #set_frame_size fails for YUV422
time.sleep_ms(10)
img = cam.capture()

Expand Down
107 changes: 107 additions & 0 deletions examples/benchmark_img_conv.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
from camera import Camera, FrameSize, PixelFormat
import time
import gc
import os
gc.enable()

def measure_fps(cam,out_fmt,duration=2):
start_time = time.ticks_ms()
while time.ticks_ms() - start_time < 500:
cam.capture(out_fmt)

start_time = time.ticks_ms()
frame_count = 0

while time.ticks_ms() - start_time < duration*1000:
img = cam.capture(out_fmt)
if img:
frame_count += 1

end_time = time.ticks_ms()
fps = frame_count / (end_time - start_time) * 1000
return round(fps,1)

def print_summary_table(results, cam):
print(f"\nBenchmark {os.uname().machine} with {cam.get_sensor_name()}, GrabMode: {cam.get_grab_mode()}:")

fb_counts = sorted(results.keys())
frame_size_names = {getattr(FrameSize, f): f for f in dir(FrameSize) if not f.startswith('_')}

header_row = f"{'Frame Size':<15}"
sub_header_row = " " * 15

for fb in fb_counts:
for p in results[fb].keys():
header_row += f"{'fb_count ' + str(fb):<15}"
sub_header_row += f"{p:<15}"

print(header_row)
print(sub_header_row)

frame_sizes = list(next(iter(next(iter(results.values())).values())).keys())

for f in frame_sizes:
frame_size_name = frame_size_names.get(f, str(f))
print(f"{frame_size_name:<15}", end="")

for fb in fb_counts:
for p in results[fb].keys():
fps = results[fb][p].get(f, "N/A")
print(f"{fps:<15}", end="")
print()

if __name__ == "__main__":
cam = Camera(pixel_format=PixelFormat.JPEG)
results = {}

try:
for fb in [1, 2]:
cam.reconfigure(fb_count=fb, frame_size=FrameSize.QQVGA)
results[fb] = {}
for p in dir(PixelFormat):
if not p.startswith('_'):
p_value = getattr(PixelFormat, p)
try:
if p_value == PixelFormat.JPEG:
continue
cam.capture(p_value)
results[fb][p] = {}
gc.collect()
except:
continue
for f in dir(FrameSize):
if not f.startswith('_'):
f_value = getattr(FrameSize, f)
if f_value > cam.get_max_frame_size():
continue
gc.collect()
print('Set', p, f,f'fb={fb}',':')

try:
cam.set_frame_size(f_value)
time.sleep_ms(10)
img = cam.capture(p_value)
if img:
print('---> Image size:', len(img))
fps = measure_fps(cam,p_value,2)
print(f"---> FPS: {fps}")
results[fb][p][f_value] = fps
else:
print('No image captured')
results[fb][p][f_value] = 'No img'

print(f"---> Free Memory: {gc.mem_free()}")
except Exception as e:
print('ERR:', e)
results[fb][p][f_value] = 'ERR'
finally:
time.sleep_ms(250)
gc.collect()
print('')

except KeyboardInterrupt:
print("\nScript interrupted by user.")

finally:
print_summary_table(results, cam)
cam.deinit()
25 changes: 17 additions & 8 deletions src/micropython.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,28 @@ target_sources(usermod_mp_camera INTERFACE
${CMAKE_CURRENT_LIST_DIR}/modcamera_api.c
)

target_include_directories(usermod_mp_camera INTERFACE
${CMAKE_CURRENT_LIST_DIR}
${IDF_PATH}/components/esp32-camera/driver/include
${IDF_PATH}/components/esp32-camera/driver/private_include
${IDF_PATH}/components/esp32-camera/conversions/include
${IDF_PATH}/components/esp32-camera/conversions/private_include
${IDF_PATH}/components/esp32-camera/sensors/private_include
)
if(EXISTS "${IDF_PATH}/components/esp32-camera")
target_include_directories(usermod_mp_camera INTERFACE
${CMAKE_CURRENT_LIST_DIR}
${IDF_PATH}/components/esp32-camera/driver/include
${IDF_PATH}/components/esp32-camera/driver/private_include
${IDF_PATH}/components/esp32-camera/conversions/include
${IDF_PATH}/components/esp32-camera/conversions/private_include
${IDF_PATH}/components/esp32-camera/sensors/private_include
)
else()
target_include_directories(usermod_mp_camera INTERFACE
${CMAKE_CURRENT_LIST_DIR})
endif()

if (MICROPY_CAMERA_MODEL)
target_compile_definitions(usermod_mp_camera INTERFACE MICROPY_CAMERA_MODEL_${MICROPY_CAMERA_MODEL}=1)
endif()

if (MP_CAMERA_DRIVER_VERSION)
target_compile_definitions(usermod_mp_camera INTERFACE MP_CAMERA_DRIVER_VERSION=\"${MP_CAMERA_DRIVER_VERSION}\")
endif()

target_link_libraries(usermod INTERFACE usermod_mp_camera)

micropy_gather_target_properties(usermod_mp_camera)
Loading

0 comments on commit 176b0a3

Please sign in to comment.