Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flash-size agnostic builds #6690

Merged
merged 75 commits into from
Feb 10, 2022
Merged

flash-size agnostic builds #6690

merged 75 commits into from
Feb 10, 2022

Conversation

d-a-v
Copy link
Collaborator

@d-a-v d-a-v commented Oct 29, 2019

(edit 2021/12/01)

This PR adds a new option in the flash size configuration options list called
Mapping defined by Hardware and Sketch.

This option is useful because:

  • Users need not to care about flash configuration
    (If the option is selected. It is not currently the default)

  • A binary can indifferently be flashed over 512k or 16MB flash chips, or esp8285 if dout is selected

    • FS and EEPROM location will automatically match legacy addresses as if flash size was correctly selected

    • Three configurations are available, they reflects standard / current menus flash mapping

      • FLASH_MAP_OTA_FS (default if FLASH_MAP_SETUP_CONFIG() is not used)
      • FLASH_MAP_MAX_FS
      • FLASH_MAP_NO_FS

      Sketch can use FLASH_MAP_SETUP_CONFIG(config) to select a configuration, for example: FLASH_MAP_SETUP_CONFIG(FLASH_MAP_MAX_FS)

    • Sketch can also define another configuration by providing a custom mapping structure.

  • No more FS overwrite / reformatting when bad configuration is selected and flashed

  • Binary releases for projects can see their number of builds reduced

When not enabled, this PR makes no change. and flash size and configuration needs to be correctly selected as before (with the same symbols).

Documentation update:

When the option is enabled, the magic word at physical address 0x0 in flash which contains "software" flash size is ignored; its reading is disabled and is replaced by reading the "chip vendor id" (ESP.getFlashChipRealSize()).

Fixes: #6679
Fixes: #2321
Closes: #6806 via #6813

@d-a-v
Copy link
Collaborator Author

d-a-v commented Oct 29, 2019

@TD-er for a start I ran the same binary DOUT lib-eeprom read/write sketch on esp8285(1M) and esp8266(4M) with good results.

#include <EEPROM.h>

#if AUTOFLASHSIZE
void flashinit()
{
  FLASHMAPCONFIG(FLASH_MAP_MAX_FS); // maximum for FS
  // or FLASH_MAP_OTA_FS (maximum space for OTA, rest of free space for FS)
  // or FLASH_MAP_NO_FS (no FS needed)
}
#endif

void setup() {
  Serial.begin(115200);

  Serial.printf("\n\nsize=%d\nsize=%d\naddr=0x%08x\n\n", ESP.getFlashChipSize(), ESP.getFlashChipRealSize(), EEPROM_start);

  EEPROM.begin(512);
  Serial.printf("%02x %02x %02x %02x\nwait 10s\n",
                EEPROM.read(0),
                EEPROM.read(1),
                EEPROM.read(2),
                EEPROM.read(3));
  delay(10000);
  Serial.printf("write\n");
  EEPROM.write(0, 0xde);
  EEPROM.write(1, 0xad);
  EEPROM.write(2, 0xbe);
  EEPROM.write(3, 0xef);
  EEPROM.commit();
}

void loop() {
}

esp8285:

size=1048576
size=1048576
addr=0x402fb000

esp8266:

size=4194304
size=4194304
addr=0x405fb000

@TD-er
Copy link
Contributor

TD-er commented Oct 29, 2019

So, you've already started with it :)
The earliest test I can do with it is tomorrow evening. (I also have 2M modules)
Have got to finish something else now and will be away most of tomorrow.

@TD-er
Copy link
Contributor

TD-er commented Nov 4, 2019

I have not yet tested it, but looked through the code.
The flashInit function is called from the core main function and does look at the flash size to pick a layout.
Can I later set it again to another layout at runtime?
Or is it possible to set the magic byte to define some alternate setting and then reboot to load the alternate setting?

The reason I'm asking is that I cannot support a single binary for 4M flash with either 1 MB SPIFFS (default for all nodes out there running my software) or 2 MB SPIFFS if it is only looking at the flash size.
So either I have to have my own detection algorithm to see where the SPIFFS is located and change at runtime, or only try once to find any existing 1M SPIFFS and if it isn't there switch it to the 4M2M option, write that to the magic byte and reboot.
If it still cannot find a valid SPIFFS, then format it as a 2M SPIFFS.,

N.B. the reason I have not tested it yet, is mainly because of the amount of work needed to get a PR ready within PlatformIO. I also just made a feature request for it: platformio/platformio-core#3239
In the retrieved packages directory of PIO, the git config is not complete or correct.
I do see the pull requests on origin, but when I want to checkout based on a PR, I get an error like "missing delta bases"

@d-a-v
Copy link
Collaborator Author

d-a-v commented Nov 4, 2019

The flashInit function is called from the core main function and does look at the flash size to pick a layout.

That is on purpose.
The parameter (what you call layout) is in fact an array of layouts indexed by the flash chip size.
There are three hardcoded layout sets, but you can use a self-defined one.

Can I later set it again to another layout at runtime?

FSes and EEPROM should be closed before doing that, but yes I don't see any reason against that.

@d-a-v
Copy link
Collaborator Author

d-a-v commented Nov 4, 2019

As @earlephilhower said elsewhere, we cannot autodetect SPIFFS because there is no signature. I don't know about LittleFS.
In either case the information about location and size can anyway be written somewhere and allow to build the right descriptor.

@TD-er
Copy link
Contributor

TD-er commented Nov 4, 2019

For my own purpose, I could also use the EPROM part to store something with a very specific signature to help detect the used offset.
It does probably have 2 possible locations at most for any given flash size.

@d-a-v
Copy link
Collaborator Author

d-a-v commented Nov 4, 2019

this file contains all the addresses computed by boards.txt.py.
I think the easier path is to use the example provided in OP then change FS addresses:

... (later)
close_all_FSes(); // or not
FS_start = myFS_start;
FS_stop = myFS_end;
SPIFFS.begin()...

@d-a-v d-a-v added the alpha included in alpha release label Jul 16, 2020
Copy link
Collaborator

@mcspr mcspr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in the chats, still wip but it might be much nicer to edit it in the tree instead of amending the PR ad-infinitum

Only two sore points

  • we still have flash layout as symbols & defines, I wonder if we should move closer to something 'iterable' so we see all of things at once as [begin, end] ranges. at least of the sake of the reader, not necessary to be closer to partition schemes of the updated sdks
  • boards.txt.py python could do better :>

cores/esp8266/Arduino.h Outdated Show resolved Hide resolved
@@ -24,8 +24,7 @@ extern "C" {
#include "user_interface.h"
}

extern "C" uint32_t _FS_start;
extern "C" uint32_t _FS_end;
#include <flash_hal.h>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#include <flash_hal.h>
#include "flash_hal.h"

Copy link
Collaborator Author

@d-a-v d-a-v Feb 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one has been reverted because

  • mock version of flash_hal.h needs to be loaded first, to redefine FS_start and FS_end

This is only needed in sources files

  • located in cores/esp8266/
  • which make use of FS_start or FS_end

cores/esp8266/spiffs_api.h Outdated Show resolved Hide resolved
doc/filesystem.rst Outdated Show resolved Hide resolved
tests/host/common/littlefs_mock.h Outdated Show resolved Hide resolved
tests/host/common/spiffs_mock.h Outdated Show resolved Hide resolved
libraries/EEPROM/EEPROM.cpp Outdated Show resolved Hide resolved
d-a-v and others added 8 commits February 7, 2022 22:42
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
Co-authored-by: Max Prokhorov <prokhorov.max@outlook.com>
@d-a-v d-a-v merged commit f60defc into esp8266:master Feb 10, 2022
@d-a-v d-a-v deleted the nosizeconf branch February 10, 2022 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
alpha included in alpha release
Projects
None yet
3 participants