Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better support for using esptool as Python module (ESPTOOL-157) #208

Open
projectgus opened this issue Jun 14, 2017 · 20 comments
Open

Better support for using esptool as Python module (ESPTOOL-157) #208

projectgus opened this issue Jun 14, 2017 · 20 comments

Comments

@projectgus
Copy link
Contributor

Following on from #157. It would be good to provide more generic-code-access friendly methods, for people building UIs and the like.

Non-exhaustive list:

  • Move module-level generic functions into classes where possible.
  • Test cases of keeping a single ESPLoader object around and calling multiple functions on it.
  • Allow redirection of output to status callbacks rather than only print().
@marcelstoer
Copy link
Contributor

This is most welcome. It's actually gotten worse since 1.3. Below is a comparison of the code I had to write for marcelstoer/nodemcu-pyflasher to get the same results (incl. console output) as what esptool.py produces.

integration code comparison between 1.3  and 2.0

It even got worse (i.e. more code that I had to write) between the latest beta and the final 2.0.

@projectgus
Copy link
Contributor Author

Thanks for showing that. I guess I'm not super surprised, given the number of additional features in 2.0. But in both cases you essentially have to "mock up" a command line, and that shouldn't be necessary. I'll use your code as an example when it comes to designing a more useful Python API.

@murilopolese
Copy link

This would be indeed much appreciated! I have been trying to avoid "mocking up" a command line but I wasn't able to find my way through the code on how to do it (I'm just starting with Python). Thanks @marcelstoer for the snippet! This is great!

Before finding this solution I was fiddling around with a local copy of esptool.py where the main() function accepts a non required Namespace as argument. In case I called main() with this argument I would use it instead of the result of parser.parse_args(). It worked pretty well as I could perform all the actions I needed with a fairly small amount of code.

My instincts tell me that is not a very good idea but I can't reason exactly why. Could you help me understanding why the main() function shouldn't accept an optional Namespace argument? Or maybe it could but what problems that wouldn't solve and which new problems that would create?

@Pimmetje
Copy link

It would also be nice to be able to catch the print output (callback). But i guess that would require a lot of refactoring on the esptool

@marcelstoer
Copy link
Contributor

@Pimmetje for my PyFlasher I had to redirect stdout for that: https://github.com/marcelstoer/nodemcu-pyflasher/blob/master/Main.py#L165

@Pimmetje
Copy link

Pimmetje commented Nov 3, 2018

Thx @marcelstoer your code gave me a clue on how to make it work. I can now get my compiled projects from my CI server and upload them to a ESP with a few clicks :). I wished arduino had a python version of AVRdude. Remark i had to hack the code a bit because the Serial port was not closed after a upload.

@marcelstoer
Copy link
Contributor

I recently got an issue about exactly that: marcelstoer/nodemcu-pyflasher#37

@Pimmetje
Copy link

Pimmetje commented Nov 3, 2018

Thx, i already found that one. I also updated the esptool but no joy on that end

@murilopolese
Copy link

@Pimmetje for my PyFlasher I had to redirect stdout for that: https://github.com/marcelstoer/nodemcu-pyflasher/blob/master/Main.py#L165

Just to update it as it's now pointing to an unrelated line of code: https://github.com/marcelstoer/nodemcu-pyflasher/blob/861f75415c3874e7e2f1000e8ea419da70f6fd19/Main.py#L188

@projectgus
Copy link
Contributor Author

projectgus commented Feb 11, 2019

One more requirement (noted in #407) is for the ESPLoader object to be able to track something about the state of the chip (ie has bootloader connection been established). But still allow for case where serial port is connected to bootloader before the ESPLoader function is even called (this can probably be done because the bootloader will respond to SYNC packets at any time, I think.)

@doayee
Copy link

doayee commented Mar 27, 2019

+1 on this particularly with regards to error handling. I just published my ESP32 flasher gui but am conscious it could handle errors much better. Particularly with the case where the supplied binary sizes and start addresses are detected to result in overlap, for some reason I couldnt catch the ArgumentError that this is supposed to throw at all, though I appreciate I am new to the game?

@jziolkowski
Copy link

Any progress on that one? I want to build a custom GUI app using esptool under the hood and I'd really hate to have to use subprocess or hacking the code to achieve that ;)

@radimkarnis radimkarnis changed the title Better support for using esptool as Python module Better support for using esptool as Python module (ESPTOOL-157) Dec 11, 2020
@someburner
Copy link

I broke out a very old version of esptool into just the class components required for flashing esp8266, and made it so I could interact with the esp object without needing to invoke commands from a shell, and allowed for custom return values. But for the ESP32 this is rather daunting since there are so many hacks and changes added to __init__.py.

Being able to interact with an esp object and perform multiple read/write/other operations without having to parse stdout and without having to leave stub mode would be so much cleaner.

Could probably also just write a special command that does everything I want it to do but then I have to use a custom esptool.py version. And because there are all kinds of dependency checks with esp-idf, it might not play nice when trying to use other tools like the nvs_gen.py utility. The issue is that for anything other than a super basic manufacturing process you really need to have a variety of custom io with the chip. All of it is already there in esptool, just not easily accessible to developers, since each command must be run individually.

For example, in our production process we need to: read flash_id, check if XMC, apply XMC fix, then set custom SPI parameters to check SPI connection to a device connected to VSPI, then read the MAC so we have it in our database for customers and/or print on label, then generate NVS partition data, then write the binaries.

If all of this could be done in python with a standard esp object/class, it would make life so much easier.

@radimkarnis
Copy link
Collaborator

Thank you @someburner for the valuable insight, it is appreciated.

I agree with your opinion and can see how this is getting worse with every feature/hack being added to esptool.

There wasn't any progress or driving force to do this since 2019, I will try to change that and put this higher on our priority list!

@someburner
Copy link

I might even suggest not trying to change esptool.py now, but rather just make a library that is like esp-serial-flasher but for python, and keeping extensibility for custom manufacturing scripts in mind. Seems like that way you eliminate all the argparse stuff and boil it down to what the classes actually need. Then later they can be reconciled.

We have manufactured in the 10s of thousands of device for esp8266 (ESP-12S) and moving to ESP32 soon, I'm happy to provide more detailed input / testing.

@marcelstoer
Copy link
Contributor

not trying to change esptool.py now, but rather just make a library

It should be possible to refactor all essential functionality out of the esptool.py into something like esptool-lib.py. I expect the only thing left in the former will be the argparse stuff and the calls to the latter.

@M-Bab
Copy link

M-Bab commented Aug 24, 2022

Just another example how useful a smart API could be: I am using esptool-ftdi which resets the ESP via RTS/CTS instead of RTS/DTS. This script uses the very little documented functionality to replace the serial communication object. I could get it working with esptool~=3.0 but not with esptool>=4 because the external_esp object is significantly more complex to overwrite than the serial object.

This is obsolete now. It works just as fine with esptool 4 now.

@someburner
Copy link

someburner commented Mar 23, 2023

I've modified esptool to my liking, but just sharing in case this ever happens:

When using something like pyinstaller to generate a bundled application, it has a hard time locating references to the stub json files, since loader.py uses a relative path that I suppose is not inferred properly.

Instead of hassling with a bunch of custom pyinstaller/scripting, I just made a script to wrap all the json files into their own python folders and import them as a string instead.

Pretty sure this is how esptool used to do it back in the day anyways. I'm sure there's ways to configure static resources but those always seem to break or the api changes.

from .pystubs import *

def get_stub_json_str(chip_name):
    chip_name = re.sub(r"[-()]", "", chip_name.lower())
    chip_name = chip_name.replace("esp", "")
    py_mod = "stub_flasher_" + chip_name + "._STUBSTR"
    return eval(py_mod)

class PyStubFlasher:
    """Loading static json files doesnt play well with pyinstaller. Just load
    json strings from a python module instead.
    """
    def __init__(self, jsStr):
        stub = json.loads(jsStr)
        self.text = base64.b64decode(stub["text"])
        self.text_start = stub["text_start"]
        self.entry = stub["entry"]

        try:
            self.data = base64.b64decode(stub["data"])
            self.data_start = stub["data_start"]
        except KeyError:
            self.data = None
            self.data_start = None

...

stub = PyStubFlasher(get_stub_json_str(self.CHIP_NAME))

And a script to just generate them into esptool/pystubs. E.g. esptool/pystubs/stub_flasher_32.py looks like:

#
# stub_flasher_32.py
# generated python wrapper for stub_flasher_32.json
#
_STUBSTR = """
{
    "entry": 1074521516,
    "text": "CAD0CR7kCBn0AuFWoJQlkQtQreSSAA",
    "text_start": 1074520064,
    "data": "CMD8Pw==",
    "data_start": 1073605544
}
"""

@someburner
Copy link

someburner commented Apr 17, 2023

@radimkarnis

Another related feature that would be helpful:

Currently read_flash requires a filename to write data to. In a custom esptool application it may be preferred to avoid writing data to disk and simply return the data read as a byte array, could be an option or a different command.

Similarly it may be useful to provide input data streams for write_flash, although it seems like that might already be possible with current arg structure.

@mickeyl
Copy link

mickeyl commented Feb 26, 2024

Very good ideas here, sadly very little progress. I really wonder what people use in their production lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests