Statically allocated DRAM is limited to 160KB (IDFGH-1187)

[dsyleixa]

code compiles and works for M4 plus TFT HX8357,
but fails for ESP32;
Arduino IDE 1.8.8
Board: Adafruit Feather ESP32
core: ESP32 espressive 1.0.2

compile errors (when linking) for my code

• what am I missing?
  Perhaps too much code and too little RAM at ESP32?

Arduino: 1.8.8 (Windows 7), Board: "Adafruit ESP32 Feather, 80MHz, 921600, None"

D:\arduino\arduino-builder -dump-prefs -logger=machine -hardware D:\arduino\hardware -hardware D:\arduino\portable\packages -tools D:\arduino\tools-builder -tools D:\arduino\hardware\tools\avr -tools D:\arduino\portable\packages -built-in-libraries D:\arduino\libraries -libraries D:\arduino\portable\sketchbook\libraries -fqbn=esp32:esp32:featheresp32:FlashFreq=80,UploadSpeed=921600,DebugLevel=none -ide-version=10808 -build-path C:\Users\hw\AppData\Local\Temp\arduino_build_226627 -warnings=none -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.esptool_py.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.esptool_py-2.6.1.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.mkspiffs.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.mkspiffs-0.2.3.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.xtensa-esp32-elf-gcc.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -prefs=runtime.tools.xtensa-esp32-elf-gcc-1.22.0-80-g6c4433a-5.2.0.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -verbose D:\Akten\Programmierung\ArduinoProgs\ConwaysGameOfLife\ConwayGoL_M4_Logic_NOR\ConwayGoL_M4_Logic_NOR.ino
D:\arduino\arduino-builder -compile -logger=machine -hardware D:\arduino\hardware -hardware D:\arduino\portable\packages -tools D:\arduino\tools-builder -tools D:\arduino\hardware\tools\avr -tools D:\arduino\portable\packages -built-in-libraries D:\arduino\libraries -libraries D:\arduino\portable\sketchbook\libraries -fqbn=esp32:esp32:featheresp32:FlashFreq=80,UploadSpeed=921600,DebugLevel=none -ide-version=10808 -build-path C:\Users\hw\AppData\Local\Temp\arduino_build_226627 -warnings=none -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.esptool_py.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.esptool_py-2.6.1.path=D:\arduino\portable\packages\esp32\tools\esptool_py\2.6.1 -prefs=runtime.tools.mkspiffs.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.mkspiffs-0.2.3.path=D:\arduino\portable\packages\esp32\tools\mkspiffs\0.2.3 -prefs=runtime.tools.xtensa-esp32-elf-gcc.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -prefs=runtime.tools.xtensa-esp32-elf-gcc-1.22.0-80-g6c4433a-5.2.0.path=D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0 -verbose D:\Akten\Programmierung\ArduinoProgs\ConwaysGameOfLife\ConwayGoL_M4_Logic_NOR\ConwayGoL_M4_Logic_NOR.ino
Using board 'featheresp32' from platform in folder: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2
Using core 'esp32' from platform in folder: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2

...

Linking everything together...
"D:\arduino\portable\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-gcc" -nostdlib "-LD:\arduino\portable\packages\esp32\hardware\esp32\1.0.2/tools/sdk/lib" "-LD:\arduino\portable\packages\esp32\hardware\esp32\1.0.2/tools/sdk/ld" -T esp32_out.ld -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.rom.spiram_incompatible_fns.ld -u ld_include_panic_highint_hdl -u call_user_start_cpu0 -Wl,--gc-sections -Wl,-static -Wl,--undefined=uxTopUsedPriority -u __cxa_guard_dummy -u __cxx_fatal_exception -Wl,--start-group "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\sketch\ConwayGoL_M4_Logic_NOR.ino.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Wire\Wire.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\SPI\SPI.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\Adafruit_GFX.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\Adafruit_SPITFT.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit-GFX-Library-master\glcdfont.c.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit_HX8357_Library-master\Adafruit_HX8357.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\libraries\Adafruit_STMPE610-master\Adafruit_STMPE610.cpp.o" "C:\Users\hw\AppData\Local\Temp\arduino_build_226627\core\core.a" -lgcc -lopenssl -lbtdm_app -lfatfs -lwps -lcoexist -lwear_levelling -lesp_http_client -lprotobuf-c -lhal -lnewlib -ldriver -lbootloader_support -lpp -lfreemodbus -lmesh -lsmartconfig -ljsmn -lwpa -lethernet -lphy -lfrmn -lapp_trace -lfr_coefficients -lconsole -lulp -lwpa_supplicant -lfreertos -lbt -lmicro-ecc -lesp32-camera -lcxx -lxtensa-debug-module -ltcp_transport -lmdns -lvfs -lmtmn -lesp_ringbuf -lsoc -lcore -lfb_gfx -lsdmmc -llibsodium -lcoap -ltcpip_adapter -lprotocomm -lesp_event -limage_util -lc_nano -lesp-tls -lasio -lrtc -lspi_flash -lwpa2 -lwifi_provisioning -lesp32 -lface_recognition -lapp_update -lnghttp -lspiffs -lface_detection -lespnow -lnvs_flash -lesp_adc_cal -llog -ldl_lib -lsmartconfig_ack -lexpat -lfd_coefficients -lm -lmqtt -lc -lheap -lmbedtls -llwip -lnet80211 -lesp_http_server -lpthread -ljson -lesp_https_ota -lstdc++ -Wl,--end-group -Wl,-EL -o "C:\Users\hw\AppData\Local\Temp\arduino_build_226627/ConwayGoL_M4_Logic_NOR.ino.elf"
d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: C:\Users\hw\AppData\Local\Temp\arduino_build_226627/ConwayGoL_M4_Logic_NOR.ino.elf section .dram0.bss' will not fit in region dram0_0_seg'

d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: DRAM segment data does not fit.

d:/arduino/portable/packages/esp32/tools/xtensa-esp32-elf-gcc/1.22.0-80-g6c4433a-5.2.0/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `dram0_0_seg' overflowed by 74776 bytes

collect2.exe: error: ld returned 1 exit status

Bibliothek Wire in Version 1.0.1 im Ordner: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2\libraries\Wire wird verwendet
Bibliothek SPI in Version 1.0 im Ordner: D:\arduino\portable\packages\esp32\hardware\esp32\1.0.2\libraries\SPI wird verwendet
Bibliothek Adafruit-GFX-Library-master in Version 1.4.10 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit-GFX-Library-master wird verwendet
Bibliothek Adafruit_HX8357_Library-master in Version 1.1.4 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit_HX8357_Library-master wird verwendet
Bibliothek Adafruit_STMPE610-master in Version 1.0.1 im Ordner: D:\arduino\portable\sketchbook\libraries\Adafruit_STMPE610-master wird verwendet
exit status 1
Fehler beim Kompilieren für das Board Adafruit ESP32 Feather.

complete error msg in attachement
compileError.txt

source code also attached
ConwayGoL_M4_Logic_NOR.zip

[negativekelvin]

Yes too much static ram usage. This is the wrong place to post

Read this
espressif/arduino-esp32#1163
Ask for help here
https://esp32.com/viewforum.php?f=19

[dsyleixa]

hmmm - that's weird:

ESP32: 4 MByte flash, 520 KB SRAM
https://learn.adafruit.com/adafruit-huzzah32-esp32-feather/overview

M4: 512 KB flash, 192 KB RAM
https://learn.adafruit.com/adafruit-huzzah32-esp32-feather/overview

if ESP32 RAM is so restricted, then that is big drawback and that issue should be fixed ASAP!

[dsptech]

from https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/system/mem_alloc.html

"Note:
Due to a technical limitation, the maximum statically allocated DRAM usage is 160KB. The remaining 160KB (for a total of 320KB of DRAM) can only be allocated at runtime as heap."

[dsyleixa]

OMG, that should have been made absolutely clear BEFORE I had purchased that thing!
OTOH, Adafruit advertises the ESP by
520 KB SRAM

Can that "technical limitation" be reworked and fixed?

[projectgus]

Hi @dsyleixa,

The total SRAM is 520KB but this is DRAM+IRAM, and only DRAM (320KB) is normally used for data, IRAM is normally used for executable instructions. It is possible to store data in IRAM but there are restrictions.

Regarding how Adafruit advertise the ESP32, you'll have to take this up with them. But they are technically correct.

Working around the static memory limitation should be possible without too much work. If you have some larger buffer like this:

static uint8_t framebuffer[131072];

Then change it to a pointer, and allocate it during setup with malloc() or calloc(). This makes it dynamic memory not static memory:

static uint8_t *framebuffer;

void setup() {
   framebuffer = calloc(1, 131072);
   assert(framebuffer != NULL);

   // do other setup stuff
}

Also, as @negativekelvin mentions, for Arduino issues please post on the Arduino issue tracker or the Arduino forum of esp32.com.

[dsyleixa]

sorry, but I can't handle pointer and allocates, that is far beyond my skills :(

I don't post to Arduino.cc forum, because there are too many outrageous and insulting users.

[projectgus]

There is an Arduino subforum on esp32.com, that's the one I was referring to:
https://esp32.com/viewforum.php?f=19

[dsyleixa]

ok, perhaps in future.
As to now, to me it's an ESP32 hardware issue which should be reworked and fixed, not discussed ;)

[projectgus]

There's no hardware issue. This is a software issue in ESP-IDF that the amount of statically assigned DRAM in an app is limited to 160KB. We don't have a GitHub issue tracking this, so we can start tracking it here.

We don't have an ETA for fixing this issue as it's complex to fix (for architectural reasons) and it's generally easy to work around by assigning large buffers dynamically (see above post with suggestion). But we do plan to fix it at some point.

(There is also the related but different issue #3211 )

[galah92]

I'm also looking for a solution for that, dynamic allocation makes my code much more error-prone.

[SoucheSouche]

The DRAM limitations are now clearly stated in the documentation.

[zikalino]

closing the issue

[dsyleixa]

my point is to change 2x160kB to 1x320kB to be able to address it as a unit in its entirety.
My point is not about documentation or discussion.

[dsyleixa]

please reopen this issue because it is not fixed yet.

[SoucheSouche]

Hi @dsyleixa,

I am sorry we closed the issue after more than 2 years of inactivity without giving out more information about why we closed it.

The reason why ~2x160KB of DRAM is available instead of the ~1x320KB on esp32 is because some functions used during startup are stored starting from the address 0x3ffexxxx which happens to be in the middle of the DRAM section starting at 0x3ffbxxxx making it not continuous and preventing you from creating static arrays bigger than approx. 160KB.

Fixing the problem would require moving said functions at the end of the DRAM memory but as @projectgus mentioned more than 2 years ago this is far from being a trivial fix and it would require a lot of architectural changes which explains why as of now it has not been addressed.

As I can't guaranty that this issue will ever be fixed, I would encourage you to consider (or reconsider) the following workaround:

• Try your application on one of our other boards that have a DRAM section big enough for your application case since this technical limitation is only present on esp32 board.
• Reconsider the usage of dynamic allocation.
• Try using 2 arrays instead of 1. I am not sure if this alternative would meet your application needs but if you can't statically allocate an array of 160000 bytes, it is however possible to allocate 2 arrays of 80000 bytes instead.

We have an internal ticket reporting this issue so it won't be forgotten and we don't want you to rely on this issue being fixed
to continue with your project development. For those reason I would recommend against reopening it but feel free to let me know if you disagree with this decision now that I provided more information about it.

I am sorry again that this issue was left unaddressed for so long.