Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic when driving larger matrixes (one long strip) on 32MB 8MB ESP32-S3-DevKitC1 #580

Open
GameTec-live opened this issue Dec 28, 2023 · 70 comments

Comments

@GameTec-live
Copy link

Bug report

32MB FLASH 8MB PRAM ESP32-S3-DevKitC1(ESP32-S3-DevKitC-1-N32R8V)

Problem

Steps

  1. Modify env:demo to compile properly for the chip (SPIFFS Failing on a 32MB 8MB ESP32-S3-DevKitC1 #579)
[env:demo]
extends         = dev_esp32-s3
build_flags     = -DDEMO=1
                  ${dev_esp32-s3.build_flags}
                  ${psram_flags.build_flags}
board_build.partitions = config/partitions_custom_8M.csv
board_upload.flash_size = 32MB
board_build.flash_mode = qio
board_build.arduino.memory_type = opi_opi
  1. In global.h define a matrix width and height larger than 36 (so 37+ results in the panic)
  2. See core 1 panic and the system reboot

Example

Notes
In this case a "Matrix" is just a bunch of daisychained LED strips going back and forth.

My globals.h
globals.h.txt

Same exact config file works fine on a generic, less powerful, esp32.
And reportedly running 1500 leds on one controller isnt optimal, but it works fine and i get a decent frame rate on my underpowered esp32. It shouldnt crash anyways ;)

Monitor Log:
https://hastebin.skyra.pw/wikevijele.yaml

@rbergen
Copy link
Collaborator

rbergen commented Jan 4, 2024

@GameTec-live The hastebin link at the end of your description leads to an empty page with a blinking cursor in my browser.

@GameTec-live
Copy link
Author

GameTec-live commented Jan 4, 2024

@GameTec-live The hastebin link at the end of your description leads to an empty page with a blinking cursor in my browser.

oh, weird... ill reupload later... (when I'm home)

@GameTec-live
Copy link
Author

crashlog.log
Sorry for the late reply, but here you go...

@robertlipe
Copy link
Contributor

robertlipe commented Jan 4, 2024 via email

@GameTec-live
Copy link
Author

GameTec-live commented Jan 4, 2024

changed build_type under base from release to debug and set monitor_filters = esp32_exception_decoder under [env:demo]

heres the new log:
debug-log.log

@robertlipe
Copy link
Contributor

robertlipe commented Jan 4, 2024 via email

@GameTec-live
Copy link
Author

probably the stupidiest thing youve heard in a while, but.... i do have a pico debug probe (SWD) and cant figure out where im supposed to hook it up...

@robertlipe
Copy link
Contributor

robertlipe commented Jan 5, 2024 via email

@GameTec-live
Copy link
Author

Okay, thanks for the help, i managed to get a debugger running (it was a driver issue...)... I don't know what the value of mCur is supposed to be, etc... I'll poke around a bit more and report back...

@GameTec-live
Copy link
Author

cant really see anything unusal? (i mean; i also dont know what those values are supposed to be lol)
mPixelData seems to be large or even "infinite" as i can request a lot more than 1500 array entries with the debugger... mCur also goes past 1500...

@rbergen
Copy link
Collaborator

rbergen commented Jan 7, 2024

@GameTec-live Glad to see you got the debugger working, and I appreciate the earlier upload of the debug log. As @robertlipe already indicated, it does show that the actual problem (which is a form of illegal memory access) takes place several levels below "our" code. In fact, the backtrace doesn't even include references to any code that's part of NightDriverStrip.

Concerning your last comment, we're obviously not looking over your shoulder, so we can't see what you are looking at. Also, even if we could then figuring out what the cause of the invalid behavior is would effectively require us to debug the dependency libraries involved. Which isn't entirely impossible, but very difficult if we can't ourselves debug trace through the code just before the problem occurs.

Without having the hardware and software setup that triggers these crashes available, I therefore think we won't be able to solve this. You could (still) consider raising a bug report in the dependent libraries (Espressif ESP-IDF and/or FastLED) and see if they are able to provide pointers to what's actually behind this.

I'll leave this issue open in case someone else runs into the same problem, and may be able to provide additional information that can help get to the root of this.

@GameTec-live
Copy link
Author

Yeah, kinda hard to replicate a issue without hardware... Ill open a issue over at FastLED... Thanks for the help though...

@prschguy1
Copy link

hey GameTec-live
While i am not one of the bigger brains on this repository, perhaps I can provide some help to the problems you are experiencing.
here. My understanding is that demo is intended for strip effects only. I was a bit surprised that you tried putting spectrum into a demo build. have never tried that, but assumed it wouldn't work as newer spectrum builds use dma. .Instead of using demo as your build, might suggest using spectrum-elecrow, I have that working properly on the chip you specify. it looks good with pdm mic, remote, display, and all the effects. when I run it with strip effects, I run out of memory, but runs well on spectrum. might give it a try. Memory usage looks pretty good here. just my 2 cents.
Capture

20240108_191857.mp4

@GameTec-live
Copy link
Author

hey GameTec-live
While i am not one of the bigger brains on this repository, perhaps I can provide some help to the problems you are experiencing.
here. My understanding is that demo is intended for strip effects only. I was a bit surprised that you tried putting spectrum into a demo build. have never tried that, but assumed it wouldn't work as newer spectrum builds use dma. .Instead of using demo as your build, might suggest using spectrum-elecrow, I have that working properly on the chip you specify. it looks good with pdm mic, remote, display, and all the effects. when I run it with strip effects, I run out of memory, but runs well on spectrum. might give it a try. Memory usage looks pretty good here. just my 2 cents.
Capture

20240108_191857.mp4

thx for the info, ig ill try that... My matrix is just one long strip though...

@GameTec-live
Copy link
Author

altough using the spectrum-elecrow project seems to work, it doesnt help me much as spectrum drives hub whatever its called matrixes and i just have one long strip snaking back and forth...

@prschguy1
Copy link

All of the spectrum builds use a series of strips that zig-zag back and forth as you describe. They can be identified as there is only 1 pin used for output led_pino. The hub75 is currently only used for the Mesmerizer builds that require around 14 output pins depending on what you are doing. Dave does a great job of describing all of this here:

https://www.youtube.com/watch?v=COJnlehBcKw&t=224s

The video I posted is using a standard zig-zag strip as you describe at 16 pixels high by 48 pixels wide on spectrum build, using the chip you specified.

@GameTec-live
Copy link
Author

ah, ok... so ig I will fumble a bit more with spectrum elecrow and try to use that... thanks...

@GameTec-live
Copy link
Author

GameTec-live commented Jan 12, 2024

nvm, setting the matrix to the required size crashes elecrow too...

@prschguy1
Copy link

Perhaps if you could articulate exactly what you are trying to accomplish, we might be able to help. As far as I am seeing, this chip is working without any crashes for the spectrum build. While there aren't a lot of spectrum effects here, intend to test it more thoroughly.

@GameTec-live
Copy link
Author

Perhaps if you could articulate exactly what you are trying to accomplish, we might be able to help. As far as I am seeing, this chip is working without any crashes for the spectrum build. While there aren't a lot of spectrum effects here, intend to test it more thoroughly.

drive my 50x30 matrix (being one long strip) with the new, more powerful ESP32-S3... It works fine with my current, not as powerful ESP32 (afaik its even singlecore?), had to disable nice to haves like the webserver though for it to run a stream from the computer at a decent framerate which im hoping to fix with this a lot more powerful variant...

@GameTec-live
Copy link
Author

Stupid question, but can some of the devs or someone more competent than me try and compile demo with a 50x30 matrix?
I tried to compile elecrow, 50x30, similar error. Tried to compile demo for a seeed studio esp32 c3 (ik, not officially supported) and it's still the same error (from what it looks like)
Havnt tried a nodemcu (clone) yet as thats currently driving the matrix and id rather not break it until my replacement mcu works... :/

@robertlipe
Copy link
Contributor

robertlipe commented Feb 12, 2024 via email

@GameTec-live
Copy link
Author

GameTec-live commented Feb 12, 2024

Ok, that makes sense then, was just the one I had laying around... Id still be interested if someone else can replicate this issue or if its just me or maybe even a defect MCU... And having a reproducable thing / minimum reproducable example might help speed things up here (or over at fastLED)

@prschguy1
Copy link

Hello [GameTec-live], have looked at your situation a good bit over the last couple of weeks, and do get similar results. While I can approach your matrix size, cannot quite get there. Have a similar problem when trying to use this chip with 4 channel strip effects. Have tried different board define files as well as different memory tables, but still have not solved this problem. While an alternate led program makes both of our build problems work, I would like to get it working here. What I have learned is programs can make use of 16 mb, and the 32mb that these chips support is only useful for storage. Have tried both of our builds with unexpected maker s3 pro, wemos s3, and generic and official builds of esp32-s3 in various memories. Have a wemos d32 pro, and m5stack I'll try our build on next. M5 stack used to work on my build, but suspect that no longer works. Will let you know what I find.

@GameTec-live
Copy link
Author

Ah, so it isn't just me XD
Thanks for trying to help though.

@GameTec-live
Copy link
Author

@robertlipe while ordering other stuff, i threw in a N16R8, so the 16MB version you apperently use... I still get the same panic...

@robertlipe
Copy link
Contributor

robertlipe commented May 15, 2024 via email

@davepl
Copy link
Contributor

davepl commented May 15, 2024 via email

@GameTec-live
Copy link
Author

@rbergen you seem to be misremembering, as atleast for me on my 2 MCUs, with PSRAM enabled, driving a smaller matrix (eg 10x10) boots up perfectly, etc...
For completeness have the logs and git diffs anyways:
N16R8-smallWorking-diff.patch
N32R8V-smallWorking-diff.patch
N32R8V-smallWorking-log.txt
N16R8-smallWorking-log.txt

@rbergen
Copy link
Collaborator

rbergen commented May 20, 2024

No, I'm not - or at least not in that way. The point is that there are now 3 problems with (certain) S3 boards we know of. Your problem - that being the one this issue concerns primarily - is the second problem in @robertlipe's summary. There are two others, though. One is the first Robert mentions, the other the one I mentioned in my previous comment.

@robertlipe
Copy link
Contributor

robertlipe commented May 20, 2024 via email

@rbergen
Copy link
Collaborator

rbergen commented May 20, 2024

That means we have only Gametec's still live and in play, right?

Based on what you say in the rest of your comment, that's quite likely true.

If we track problems we used to have, I'm sure we'll all lose our minds even more quickly.

That may be true for some, many or most, but not me. I need to keep some record of problems we used to have to retain my sanity. If only because some "solved" problems have the unpleasant habit of rearing their heads again sometime later.

But, I can be quiet about it if that's generally preferred. :)

@davepl
Copy link
Contributor

davepl commented May 20, 2024 via email

@robertlipe
Copy link
Contributor

robertlipe commented May 20, 2024 via email

@GameTec-live
Copy link
Author

Dont want to be annoying (greatly apprecite the work your doing here), but has there been any progress on this issue?

@davepl
Copy link
Contributor

davepl commented Jun 30, 2024 via email

@robertlipe
Copy link
Contributor

robertlipe commented Jun 30, 2024 via email

@GameTec-live
Copy link
Author

So progress but also not? fun

Thanks for the update though

@GameTec-live
Copy link
Author

As the FastLED issue has been closed unceremoniously last week, I decided to try it again. Cloned the repo again, made the required changes, built, uploaded aaaand... Panic... Yaaay...

@rbergen
Copy link
Collaborator

rbergen commented Sep 5, 2024

That's obviously disappointing. "Made the required changes" includes making sure PlatformIO pulls in the right version of FastLED? I don't have the code in front of me, but I half-remember the FastLED version was pinned.

@GameTec-live
Copy link
Author

In the PIO logs it said that it pulled version 3.7.5 (or whatever the latest version is)

@rbergen
Copy link
Collaborator

rbergen commented Sep 5, 2024

Thanks for checking. Well, I guess that settles that then. In the sense that it doesn't.

@GameTec-live
Copy link
Author

😂 yep, sadly lol

@robertlipe
Copy link
Contributor

robertlipe commented Sep 5, 2024 via email

@GameTec-live
Copy link
Author

Ig, but idk how id reproduce this in plain FastLED?

@zackees
Copy link

zackees commented Sep 6, 2024

You can reopen the issue. Zack's trying to burn down the list of ancient bugreports that have been open for up to 8 years and apparently just cut too deeply. Be prepared to build up a test case that repros this in "raw" FastLED, though. It's increasingly time to replace FastLED with with Makuna NeoPixel or maybe I2SClocklessLedDriver. Maybe. I'm leaning toward the former. But given the requirement of building and running every combination, I cannot undertake that here. I'll probably eventually do it in a fork and get "advanced" features like RGBWW, changing pin numbers and strip types without recompiling, etc.

On Thu, Sep 5, 2024 at 5:49 AM GameTec-live @.> wrote: 😂 yep, sadly lol — Reply to this email directly, view it on GitHub <#580 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD3ZQNEEVRXOA6LPEMBDZVAZKJAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZRGIYDQMJWGY . You are receiving this because you were mentioned.Message ID: @.>

I already tried to import the I2SClocklessDriver.

Here's the bad news: Doesn't work in C3,S3,C6, only works on Esp32dev.

The developer said he has a working version for s3 but I didn't see it, maybe I didn't look good enough.

Additionally the readme states that the library works on Arduino/PlatformIO as a drop in for FastLED.

Problem is: The developer forgot to put in a header include for the library.json file. I fixed the issue and issued a pull request. But it's been three days and the developer hasn't merged it in.

I looked at the neopixel bus to see what they did and - yup, same problem, only works on ESP32DEV and all other platforms are compiled out.

Major bummer. Though I like the thought of running 24 channels of I2S WS2812 data, the driver is about only 33% done, realistically for someone that is new and wants to come in and fix it. For the original developer, this is probably 90% done. I've reached out to the developers of this to help them get it in for the library but haven't heard back.

Bummer, because this is a fantastic feature. But the ESP32DEV user base is just too small to justify me wrapping my head on all the crazy espressif headers to get this done. Our user base wants WS2812 RGBW and that's my top priority for the next release.

@robertlipe
Copy link
Contributor

robertlipe commented Sep 6, 2024 via email

@GameTec-live
Copy link
Author

Ill try and poke a bit more at nightdriver and fastled... for reference ill also drop the links to the exact 2 boards i have (incase someone is crazy enough and wants to try on the exact boards):
https://www.berrybase.at/espressif-esp32-s3-devkitc-1-n32r8v-esp32-s3-wroom-2-8mb-psram-32mb-flash
https://de.aliexpress.com/item/1005006418608267.html

@zackees
Copy link

zackees commented Sep 6, 2024

While you have that can open, Zack, please add another W to that request. Another project just went to the effort to add RGBW, then their top request became RGBWW when the additional work from there is mostly mechanical. It's a bummer that atruct Pixel no longer fits in a word which can be a drag on low memory configurations. This problem, though, is likely something that's just not pinned in IRAM. Someone with a debugger just needs to repro it. I've tried to repro it (when filed) and can't. Perhaps with a better repro case, Zach and/or I can repro it in NightdriverLED context. I do agree that there are a ton of packages out there that just haven't kept up in the later silicon, but as existence proof, I have WLED running on S3, son Makuna code must work there, though I have only run it to a few hundred pixels. Oh, one of the things that NightdriverLED had to work around is the antiquated use of 'register'..I haven't looked lately but has that been fixed in FastLED? It's a problem in the post c++2017 world. P. S. Are you Zack or Zach? Over the lunch table I never noticed. On Fri, Sep 6, 2024, 11:04 AM Zachary Vorhies @.> wrote:

You can reopen the issue. Zack's trying to burn down the list of ancient bugreports that have been open for up to 8 years and apparently just cut too deeply. Be prepared to build up a test case that repros this in "raw" FastLED, though. It's increasingly time to replace FastLED with with Makuna NeoPixel or maybe I2SClocklessLedDriver. Maybe. I'm leaning toward the former. But given the requirement of building and running every combination, I cannot undertake that here. I'll probably eventually do it in a fork and get "advanced" features like RGBWW, changing pin numbers and strip types without recompiling, etc. … <#m_4762911349048884551_> On Thu, Sep 5, 2024 at 5:49 AM GameTec-live @.
> wrote: 😂 yep, sadly lol — Reply to this email directly, view it on GitHub <#580 (comment) <#580 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD3ZQNEEVRXOA6LPEMBDZVAZKJAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZRGIYDQMJWGY https://github.com/notifications/unsubscribe-auth/ACCSD3ZQNEEVRXOA6LPEMBDZVAZKJAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZRGIYDQMJWGY . You are receiving this because you were mentioned.Message ID: @.
> I already tried to import the I2SClocklessDriver. Here's the bad news: Doesn't work in C3,S3,C6, only works on Esp32dev. The developer said he has a working version for s3 but I didn't see it, maybe I didn't look good enough. Additionally the readme states that the library works on Arduino/PlatformIO as a drop in for FastLED. Problem is: The developer forgot to put in a header include for the library.json file. I fixed the issue and issued a pull request. But it's been three days and the developer hasn't merged it in. I looked at the neopixel bus to see what they did and - yup, same problem, only works on ESP32DEV and all other platforms are compiled out. Major bummer. Though I like the thought of running 24 channels of I2S WS2812 data, the driver is about only 33% done, realistically for someone that is new and wants to come in and fix it. For the original developer, this is probably 90% done. I've reached out to the developers of this to help them get it in for the library but haven't heard back. Bummer, because this is a fantastic feature. But the ESP32DEV user base is just too small to justify me wrapping my head on all the crazy espressif headers to get this done. Our user base wants WS2812 RGBW and that's my top priority for the next release. — Reply to this email directly, view it on GitHub <#580 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD37UG2REUDFRXBQY2KDZVHHBPAVCNFSM6AAAAABBFMSILWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZUGM3TMMZWGE . You are receiving this because you were mentioned.Message ID: @.
**>

I really don't understand why everyone wants to make the RGBW implementation so freaking difficult.

This is my opinion, but Adafruit did it the completely wrong way. They said: there are 4 pixels being outputted so let's give the user 4 pixels to manipulate.

Everyone that I've talked to about the RGBW issue seems to have the same opinion.

I come from the games industry. We use RGB8 everywhere until HD became a thing, and that was only what, RGB10?

Going beyond RGB8 on the client side seems like absolute madness. Are we going to partition our entire blending / fx code too? Are we going to do it again with RGBWW? Shoot me.

RGB8 It's good enough for games, it's good enough for our users.

So what's my approach?

Convert RGB -> RGBW at the fastled driver level. Boom. Done, easy peezy. All the sketches that work today for the RGB pixel type now magically work on the RGBW output pixel type. Blending code stays the same. If RGBWW becomes the rage, then that's supported too very easily (well for non AVR chipsets unless someone wants to change all that asm code).

RGB8 is fine, the problem is that the RGB8 on the pixel chipsets don't gamma correct. But future ones do or will give you 16 bit per channel brightness or maybe they'll get smart and realize we only need 5+ bits of global brightness per pixel. I've already done this for APA102HD mode. No gamma correction necessary by the user, it's automatic. Client still uses RGB8, it just looks fantastic.

RGB8 is good enough for artists IF the pixels themselves are gamma corrected. This is what games do already and it works great. Will some insist on a W component? Maybe, but they are in the minority. Most users just want to plug in a strip and have it work well.

What makes this even easier right now is that it appears that the WS2812 makers have so far standardized W as the last component. So 3-component ordering may stay the same. Though I suspect that the entire industry may just assume standardized ordering going forward since it's a major headache for them to support anything but the standard GRB ordering, which becomes the default for WS2812 on the next release, as god intended it.

What's great about the RGBW types is that it seems to reduce power usage. Full blast white for RGB for 30 pixels is 3.8 watts. Full blast white for RGBW (using the white color stealing algorithm) reduces that to 2.6 watts. So it cuts the power down by a 1/3rd.

Experimental support for RGBW will be released on Monday. Fun times.

(btw it's Zach but you can use either spelling - i don't care)

@zackees
Copy link

zackees commented Sep 6, 2024

Ill try and poke a bit more at nightdriver and fastled... for reference ill also drop the links to the exact 2 boards i have (incase someone is crazy enough and wants to try on the exact boards): https://www.berrybase.at/espressif-esp32-s3-devkitc-1-n32r8v-esp32-s3-wroom-2-8mb-psram-32mb-flash https://de.aliexpress.com/item/1005006418608267.html

Use our clone-and=compile mode for best results. Just clone our repo, open it up in VSCode and hit compile. It's that easy.

Your changes go in dev/dev.ino. It's already preset to the esp32-s3 using the open source platformio codebase and uses the new arduion 3+ framework which includes idf 5.1.

Once you've repro'd the case, push it to your fork and send me a link and I can see if I can repro it with my XIAO esp32 s3.

@GameTec-live
Copy link
Author

@zackees heres the fun part, I cant rly reproduce it with a simple fastled sketch alone, tried for ~1h now, so ig its the combination of smthng Nightdriver is doing and Fastled thats the problem?

@zackees
Copy link

zackees commented Sep 6, 2024

Can you package it all up into one repo?

@GameTec-live
Copy link
Author

That's like this repo or what do you mean? All you need to do is change the config as ive shown already.I can generate a new patchfile for the current repo versions as the older ones dont apply anymore and youd have to modify the files by hand... I'll send them in a bit...

@GameTec-live
Copy link
Author

N32R8V-diff.patch

@robertlipe
Copy link
Contributor

robertlipe commented Sep 7, 2024 via email

@robertlipe
Copy link
Contributor

robertlipe commented Sep 7, 2024 via email

@GameTec-live
Copy link
Author

@robertlipe appreciate the effort, but then smthng must be different? Ive tried 2 different boards, from different sellers, so I doubt that both just happened to randomly be broken in the same way?

@zackees
Copy link

zackees commented Sep 7, 2024

That's just not true. I have strips in my collection that use different ordering for the white channels. At least one of them swaps a white with Green. One of them swaps a white with Blue. I can't remember any putting a/either white FIRST, but at least one of the whites always seems to be last.

Okay. This is exactly the real world information that i was looking for but couldn't find. Thank you. Looks like I have to define a total ordering for RGBW then. Luckily there is enough room in the EOrdering enum for one additional component.

Cheers as always.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants