Reading flash from ESP32-S3 with read_flash command is very slow (ESPTOOL-774)

[ciniml]

Operating System

Ubuntu 22.04

Esptool Version

v4.7.dev3-5-gda4a486

Python Version

Python 3.10.12

Chip Description

ESP32-S3

Device Description

ESP32-S3-WROOM-1-N8R8

Hardware Configuration

Connect "USB" port (which is connected to internal USB Serial/JTAG) to host PC.
Force ESP32-S3 to enter download mode by pushing BOOT and RESET button on the board.

How is Esptool Run

No IDE

Full Esptool Command Line that Was Run

esptool.py --port /dev/ttyACM1 --baud 1500000 read_flash 0 16777216 dump.bin

Esptool Output

$ esptool.py --port /dev/ttyACM1 --baud 1500000 read_flash 0 16777216 dump.bin
/home/kenta/.espressif/python_env/idf5.1_py3.10_env/bin/esptool.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').run_script('esptool==4.7.dev0', 'esptool.py')
esptool.py v4.7-dev
Serial port /dev/ttyACM1
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded PSRAM 8MB (AP_3v3)
Crystal is 40MHz
MAC: 7c:df:a1:e0:98:b8
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 1500000
Changed.
16777216 (100 %)
16777216 (100 %)
Read 16777216 bytes at 0x00000000 in 2074.0 seconds (64.7 kbit/s)...
Hard resetting via RTS pin...

More Information

The command takes around 15 minutes to complete. (around 8000 bytes/s to reading flash)

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

• I confirm I have read the troubleshooting guide.

[ciniml]

For some reason, the communication speed does not seem to change on the ESP32-S3 regardless of the --baud option setting.

With the same settings (only changing the read size to 4 [MiB]), I tried it on the ESP32-C3 and confirmed that the read speed changes with the --baud option setting.

[ciniml]

Through investigation, I confirmed that implementing the following measures improves the read speed to a normal level.

• Similar to flush timing in the USB-OTG driver, change it so that it only flushes when 64 bytes have accumulated or a 0xc0 transfer.
  • https://github.com/ciniml/esptool/blob/improve_esp32s3_read_flash_speed/flasher_stub/stub_io.c#L178
• When using the built-in USB Serial/JTAG, set the divider in uart_div_modify to 0.
  • https://github.com/ciniml/esptool/blob/improve_esp32s3_read_flash_speed/flasher_stub/stub_io.c#L309

[radimkarnis]

Hello @ciniml,
thank you for the report, thorough investigation, and a patch!

I can confirm this is a real issue and can be fully reproduced. We will fix this ASAP.

[ciniml]

Hello @radimkarnis .

I am testing the stub on Windows, but it seems that the transfer stops midway. It's likely that further adjustment of the flush timing is needed. I will continue to investigate what is happening.

[ciniml]

Hello.

Further investigation has revealed that in the problematic Windows environment, reducing the accumulating amount from 64 bytes to 63 bytes prevents the system from hanging.

The exact cause is not well understood, but when observing packet captures, it appears that packets of an unnaturally long length occur regularly with 64 bytes.
In the case of 63 bytes, such phenomena were not observed.

https://github.com/ciniml/esptool/tree/improve_esp32s3_read_flash_speed

[peterdragun]

Hi @ciniml, thank you for the investigation and providing the fix it is much appreciated. I have just one question.

When using the built-in USB Serial/JTAG, set the divider in uart_div_modify to 0.
https://github.com/ciniml/esptool/blob/improve_esp32s3_read_flash_speed/flasher_stub/stub_io.c#L309

What is the issue with setting the divider? I have tried the fix with and without this part and I cannot see a difference. Can you please explain why you think this change is needed?