"Integrated handling for filesystem and gzipped Binaries" (Restart with fresh Fork)

[gneiss15]

ATENTION: This is tested on Linux only! See below for open questions/prerequisites.

Currently there are some manual steps necessary to prepare the files necessary for OTA updates (especially for ESP8266HTTPUpdateServer)

The steps are:

Compile the sketch
Export the generated sketch-binary
Generate the filesystem
Export the fs-binary
Generate gzipped versions of binaries
Generate signed versions of gzipped binaries
Which of the above steps are necessary depends on the sketch (does it use a fs?) and the kind of OTA chosen (signed binaries necessary?) or preferred usage (use of gzipped binaries for faster upload?).

There are some problems/inconveniences:
2. The signed sketch-binary is not exported when using "Export compiled Binary".
(This point will be solved with my Pull-Request #8255)
3. & 4. AFAIK, the only “integrated” way to generate the fs is using one of the tools:
“arduino-esp8266littlefs-plugin” or “arduino-esp8266fs-plugin”.
Both tools are designed to create & upload the fs, but not export them.
So if preparing the files for an OTA when no board is connected, you have to:

ignore errors shown because upload is not possible
find & navigate to the “build.path” and copy the fs-binary manually
& 6. AFAIK, there is no “integrated” way to perform these steps.
This Pull-Request will integrate all of the above steps into the “compile and/or upload” action of Arduino.

This is done by adding 3 new entries (“Upload”, "Filesystem" and "Export") to the tools-menu.
These entries are presented just like any other options for esp8266 (“Upload speed” ... “Non-32-Bit Access”) and placed right after the last.

Below is the text that is already integrated into "doc/filesystem.rst"
--- schnipp ---

Menu-Entry “Upload” lets You choose between:

• Sketch
• Filesystem
• Both
  This Menu controls which parts are uploaded when "Sketch->Upload" (Ctrl-U)
  is selected from the Arduino Menu.

Menu-Entry “Filesystem” lets You choose between:

• Off
• LitteFs
• SPIFFS
  This Menu controls which Fs will be created when "Sketch->Compile" (Ctrl-R)
  is selected from the Arduino Menu. The Fs will always be created inside the
  "export" dir (see below).

Menu-Entry “Export” lets You choose between:

• Off
• .bin & .bin.signed)
• Create & Export gzipped Binaries too
  This Menu controls export & generation of (extra) Sketch-Binaries.

The "export" dir
................
After a (successful) compile, the files will be exported to a subdir
“bin/{variant}” of Your sketch directory
(“{variant}” replaced with the name of the board as shown behind
“Tools -> Board:”).
Signed variants of all Binaries are generated similar to the
“automatic signing” done for normal sketch binary.

As valid for the "old" tools mentioned below, it is nessesary to place
files you want to be inside the generated file system into a directory
named data inside Your sketch directory.
--- schnapp ---

Open questions/prerequisites:

1. As stated above I had tested this on Linux only.
   Hopefully some cant test it under windows..

2. tkinter should be available.
   if it is not, and creating of fs is requested AND the data dir is empty, I didn't know how to ask the user: “Are you sure you want to create an empty ... image?”.
   So in this case a message is given and the fs will not be created.
   (So its not a real problem if tkinter isn't installed)

An additional question for developers:
As mentioned inside function main of “postbuild.py”, there is a strange behavior of the arg "{build.path}" supplied via “platform.txt”. If someone can explain me why this “string” behave as described in the comments, I would be happy ;-)

[gneiss15]

As agreed in my old (now closed) PR #8266, I had remade it by starting with a fresh fork of esp8266/Arduino.
After creating the new fork, I reintegrate MY changes from PR #8266 and do some tests here (all work).
So I hope this is now ready for integrating.

@d-a-v Can You please review this ?
@earlephilhower Can You please review this ?

[d-a-v]

Sorry for taking so long to check on your work, I'm just not used to using filesystems flashing (I prefer to use network protocols and download files on first run). I have been trying the "alpha" release on a dummy linux account that I use for testing.
It appears that mklittlefs is not found after installing.

In platform.txt python3 for example is accessed through {runtime.tools.python3.path}/python3 so mklittlefs should be accessible with {runtime.tools.mklittlefs.path}/mklittlefs. Maybe this variable could be passed to upload.py?

Edit

This is also true with xtensa-gcc path.
FWIW Here's part of the content of the json installer. mklittlefs/spiffs are next to python3 and gcc

               "toolsDependencies": [
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "xtensa-lx106-elf-gcc"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "mkspiffs"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.0.4-gcc10.3-1757bed",
                     "name": "mklittlefs"
                  },
                  {
                     "packager": "esp8266",
                     "version": "3.7.2-post1",
                     "name": "python3"
                  }
               ],

Their path shouldn't be guessed (it's actually ~/.arduino15/packages/esp8266/tools/mklittlefs/mklittlefs and the core is at ~/.arduino15/packages/esp8266/hardware/esp8266/0.0.2/{cores,library,tools,...})

[gneiss15]

@d-a-v:

1. mklittlefs will be "pulled in" on a install "Using git version" (see https://github.com/esp8266/Arduino) of the esp8266 toolchain only after You "Initialize the submodules" as described here:
   https://arduino-esp8266.readthedocs.io/en/latest/installing.html#using-git-version

2. I resolve the path to the FS-build-tool (mklittlefs or mkspiffs) with the following code (inside postbuild.py):
   mkFsPath = Which( mkFsName, os.path.join( ToolsDir, mkFsName ) )
   if mkFsPath is None:
   mkFsPath = Which( "%s.exe" % mkFsName, os.path.join( ToolsDir, mkFsName ) )

This assumes that these tools are located below "{runtime.platform.path}/tools".
With this assumption the path to both fs-build-tools (on linux and window) should be correctly resolved.
It will result in mkFsPath being None if the tool wasn't installed (as it must be the case on Your test).

The message You see is a bit confusing, because the second parameter that should indicate the path were the tool was searched was chosen wrong.
That's why You see a "None" after "not exist:"
And Yes, these check and resulting error messages need a correction. I will commit these ASAP.

But as long as You had done the "Initialize the submodules", the code will resolve the path to the FS-tool correctly...

3. The approach shown above may fail if You had installed these tools at a different place (but platform.txt has correctly been adopted).
   So Your approach of using the path from platform.txt is definitely better.

This will require:

• add two extra parameter (tools.mkspiffs.path and tools.mklittlefs.path) to "postbuild.py" (that, not upload.py is the part that handles FS creation)
• I then still need to try if we are running on linux or windows....
• so the above code (using Which) is still preffered
• OK I will try to implement all of the above..

OK, I had implemented these changes and will commit them hereafter
With this new commit You will get the correct message if You haven't installed the selected FS-tools

[gneiss15]

Sorry, Commit text is wrong it should have been "Improved..." not "Aproved..."

[gneiss15]

@d-a-v
@earlephilhower
Almost 30 Days without any reaction ?

[d-a-v]

#6690 is mine, is more than 2 years old, and hasn't received any public comment since 9 months.

Please be patient, we don't have more free time than you do.

[gneiss15]

@d-a-v That sound like You haven't enough manpower. May I help anywhere ?