Can't connect ESP32

[roaldarbol]

I'm testing out an ESP32 (yesterday I worked on a RPi Pico), and I am getting errors connecting it with Belay. It seemingly fails in Pyboard, however if I connect thorugh Pyboard directly, it works.

pyb = pyboard.Pyboard('/dev/cu.SLAB_USBtoUART', 115200)
# pyb = belay.Device('/dev/cu.SLAB_USBtoUART', 115200)

pyb.close()
# Works without errors

# pyb = pyboard.Pyboard('/dev/cu.SLAB_USBtoUART', 115200)
pyb = belay.Device('/dev/cu.SLAB_USBtoUART', 115200)

pyb.close()
# Produces an error >>>

---------------------------------------------------------------------------
PyboardError                              Traceback (most recent call last)
Cell In [24], line 2
      1 # pyb = pyboard.Pyboard('/dev/cu.SLAB_USBtoUART', 115200)
----> 2 pyb = belay.Device('/dev/cu.SLAB_USBtoUART', 115200)
      4 pyb.close()
      6 # poll = select.poll()
      7 # poll.register(pyb.serial, select.POLLIN)

File /usr/local/anaconda3/envs/buxr/lib/python3.10/site-packages/belay/device.py:464, in Device.__init__(self, startup, attempts, *args, **kwargs)
    461 self.attempts = attempts
    462 self._cmd_history = []
--> 464 self._connect_to_board(**self._board_kwargs)
    466 self.task = _TaskExecuter(self)
    467 self.thread = _ThreadExecuter(self)

File /usr/local/anaconda3/envs/buxr/lib/python3.10/site-packages/belay/device.py:523, in Device._connect_to_board(self, **kwargs)
    521 else:
    522     soft_reset = True
--> 523 self._board.enter_raw_repl(soft_reset=soft_reset)

File /usr/local/anaconda3/envs/buxr/lib/python3.10/site-packages/belay/pyboard.py:382, in Pyboard.enter_raw_repl(self, soft_reset)
    380 if not data.endswith(b"raw REPL; CTRL-B to exit\r\n>"):
    381     print(data)
--> 382     raise PyboardError("could not enter raw repl")
...
    386 # Waiting for "soft reboot" independently to "raw REPL" (done below)
    387 # allows boot.py to print, which will show up after "soft reboot"
    388 # and before "raw REPL".

PyboardError: could not enter raw repl

Before the error it seems to print:

b'MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32\r\nType "help()" for more information.\r\n>>> '

[BrianPugh]

I am unable to reproduce; the examples work fine for me on an esp32.

As a debugging step, what does the following do for you:

python -m serial.tools.miniterm /dev/tty.usbserial-0001 115200
# After it's booted and you're in a prompt, then press CTRL+A 

It should look like:

MacBook-Air:~/projects/belay$ python -m serial.tools.miniterm /dev/tty.usbserial-0001 115200
--- Miniterm on /dev/tty.usbserial-0001  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4540
ho 0 tail 12 room 4
load:0x40078000,len:12344
ho 0 tail 12 room 4
load:0x40080400,len:4124
entry 0x40080680
MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32
Type "help()" for more information.
>>>
raw REPL; CTRL-B to exit
>

[roaldarbol]

That's strange... I get:

(buxr) ➜  ~ python -m serial.tools.miniterm /dev/tty.usbserial-0001 115200
--- Miniterm on /dev/tty.usbserial-0001  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32
Type "help()" for more information.
>>> 
raw REPL; CTRL-B to exit
>

I'm still unable to connect though Belay only. I'm in a conda environment, but I don't think that should make any difference.

[roaldarbol]

I see the ports fine too:

from serial.tools import list_ports

working_serial = []
for port in list(list_ports.comports()):
    working_serial.append(port.device)
print(working_serial)
# ['/dev/cu.BLTH', '/dev/cu.Bluetooth-Incoming-Port', '/dev/cu.usbserial-0001', '/dev/cu.SLAB_USBtoUART']

I am connecting it to a external USB hub.

[BrianPugh]

I wonder if there is some subtle race condition going on; do you have an unusually fast computer 😉 ?

Try putting a time.sleep(0.01) after the self.serial.write here and here.

If that does anything, then we know we're on the right trail.

[roaldarbol]

Figured it out! It is a speed problem - adding time.sleep(1) after the while loop solves the issue:

# flush input (without relying on serial.flushInput())
        n = self.serial.inWaiting()
        while n > 0:
            self.serial.read(n)
            n = self.serial.inWaiting()
        time.sleep(1)

0.5 seconds is not enough - 1 second does the trick. Apparently I wasn't the only one who's had the issue, see the same issue with ampy: scientifichackers/ampy#19 (comment).
(Took me 4 hours, but worth it I hope)😅

[BrianPugh]

glad you found a solution! Right now I'm trying to think of a generic solution, and I think there are 2 paths:

Option 1: Figure out why Pyboard.read_until is not working as expected. Clearly the reported data

b'MicroPython v1.19.1 on 2022-06-18; ESP32 module with ESP32\r\nType "help()" for more information.\r\n>>> '

does not end with b"raw REPL; CTRL-B to exit\r\n>". So why does Pyboard.read_until return? You're not reaching the 10s timeout, are you? This is a bit hard on my side since I cannot reporduce this. Once we fix this, then hopefully we don't need an inelegant sleep.

Option 2 (backup): If we cannot get option 1 working, what are your thoughts if I add an additional kwarg raw_repl_delay=0 to the Device constructor and pipe it through to enter_raw_repl? so, in your case, it would look like:

pyb = belay.Device('/dev/cu.SLAB_USBtoUART', 115200, raw_repl_delay=1)

I'd be hesitent to hard program in a value because:

1. I'd say anything above 1 second is too long for cases that don't need it. I'd realistically draw the line at 0.25 seconds.
2. Some people said 1 second wasn't enough.

[BrianPugh]

I think I understand the problem a little better now; does your current setup have a main.py on board?

[roaldarbol]

I actually do reach the 10s timeout, so it isn’t reading anything. What happens is beyond me. I think the second solution is a good one, especially, as you say, you’re unable to reproduce the error and I also wouldn’t want to hardcode such a long sleep in. And since it’s still an issue both in pyboard.py and ampy I wouldn’t spend more time than necessary on it.

[roaldarbol]

Maybe just call the argument delay for simplicity and cover it in the documentation.🤷🏻‍♂️

[roaldarbol]

Nope, there’s no main.py on board. Just the autogenerated boot.py with everything commented out.

[BrianPugh]

sounds good, here are a few more thoughts:

1. I'm going to modify read_until to raise an exception if the timeout is reached. All the locations its called is always followed by a check to see if the end_marker is actually in the returned data. And, in all cases, it seems to be an irrecoverable error if not. So raising an exception will make a lot of code calling code a little terser and the issue more obvious. No action required on your side.
2. Shot in the dark, if you were to put self.read_until(1, b">>>") instead of your time.sleep(1), does that solve the issue either?

EDIT: fixed command in (2)

[roaldarbol]

I think I did try with >>> earlier, and it didn’t help (but let me give it another try in the morning, as I’m not 100% where I placed it).

[BrianPugh]

took care of (1) in #39, will wait on your response for implementing the fix for this delay issue.

[roaldarbol]

Alright, back at it. self.read_until(1, b">>>") works like a charm! I timed it, and for me it takes roughly 650ms - which corresponds quite well with the minimum time.sleep() needed to make the connection. 😃 (And just for feedback, #39 works without issues for me too).

[BrianPugh]

excellent!! so last test, does everything still work if you comment out the while loop above it? So your code should look like:

    def enter_raw_repl(self, soft_reset=True):
        self.serial.write(b"\r\x03\x03")  # ctrl-C twice: interrupt any running program
        # flush input (without relying on serial.flushInput())
        n = self.serial.inWaiting()
        # while n > 0:
        #     self.serial.read(n)
        #     n = self.serial.inWaiting()
        self.read_until(1, b">>>")
        self.serial.write(b"\r\x01")  # ctrl-A: enter raw REPL
        # and the rest of the normal code here

If that works, thats the solution we'll go with, and the action items will be:

1. Make a PR here with the change that worked (whether or not that includes deleting the while loop).
2. Make a PR into the micropython/micropython repository with the upgrade.

If you're comfortable, I'd suggest you make the PR for (1) here and (2). However, if you don't want to, I'd be more than happy to.

EDIT: unfortunately, waiting for ">>>" doesn't reliably work in my setup; I'll try and come up with a PR that works for me and I think will work for you, and you can test it.

[BrianPugh]

alright @roaldarbol , I think I solved it; give #41 a try. If it works for you, I'll merge it and make a PR to upstream micropython.

[roaldarbol]

Really neat @BrianPugh! I'll test first thing in the morning (I'm on London time).

[roaldarbol]

Fixed with #41.