Odd output corruption with multiple GPIO outputs using large LED count

[blazoncek]

Describe the bug
We at WLED are experiencing an odd issue when we define multiple outputs (different GPIO ports) with large LED count (>2000).
Currently a well documented issue is 75x40 LED matrix using 5 GPIO outputs (5x600=3000 pixels) on ESP32.
If the number of GPIOs is reduced (keeping the same total LED count) the output corruption is reduced to occasional flicker.
The output corruption can also be measured with logic analyser on actual GPIO pins.

To Reproduce
Steps to reproduce the behavior:

1. Create 75x40 WS2812B matrix using 5 x 600 LED strips
2. Use WLED 0.13-b4 and assign GPIOs 16, 3, 23, 26, and 27 with LED count of 600 each
3. Start any effect available in WLED
4. See corrupted output or occasional flicker

Expected behavior
No output corruption

Development environment (please complete the following information):

• Build Environment [VSC+PIO; platform=espressif32@2.0]
• Board target [ESP32 WROOM]
• Library version [v2.6.7]

Minimal Sketch that reproduced the problem:
Unfortunately N/A.

Additional context
Everything seems fine with low GPIO ports used and lower LED count.
Free heap is still plenty at 150kB.
Video of corruption available upon request.
Contact persons involved are also @jdavis7765 and @pbolduc with WLED issue #2268

The issue seems to disappear when downgrading to WLED 0.12 which used NeoPixelBus 2.6.0.

[pbolduc]

Lets leave this issue open for now, but I think we need to be sure to isolate the issue. WLED 0.12 -> 0.13-b4 has other changes too that we cannot rule out. I was using the WS281x output which appears to map to NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32RmtNWs2812xMethod> method. I had configured 600 LEDS per pin and 5 total pins. ESP32 D1 Mini. Here was my logic analyzer output for reference. From top to bottom, pins 16, 3, 1, 4, 15.

[Makuna]

@pbolduc Do you know what channels of the RMT were used for that output?

Providing a minimum sketch that demonstrates the problems would be helpful with no WLED present. It would confirm if its truly with NeoPixelBus or some other artifact of WLED.

JFYI: You will notice an increased "start" lag after about five channels. This is a bug in the IDF, there are "issues" tracking it in the ESP32 Arduino and the Espressif IDF code bases; but seem unlikely to be fixed since they are near two years old.

[pbolduc]

I do not know off hand. I will have to review the code and verify. However, it is the start of my work day and wont be able investigate further until this evening.

[blazoncek]

    #ifdef ARDUINO_ARCH_ESP32
      case I_32_RN_NEO_3: busPtr = new B_32_RN_NEO_3(len, pins[0], (NeoBusChannel)channel); break;
      case I_32_I0_NEO_3: busPtr = new B_32_I0_NEO_3(len, pins[0]); break;
      #ifndef CONFIG_IDF_TARGET_ESP32S2
      case I_32_I1_NEO_3: busPtr = new B_32_I1_NEO_3(len, pins[0]); break;
      #endif
      case I_32_RN_NEO_4: busPtr = new B_32_RN_NEO_4(len, pins[0], (NeoBusChannel)channel); break;
      case I_32_I0_NEO_4: busPtr = new B_32_I0_NEO_4(len, pins[0]); break;
      #ifndef CONFIG_IDF_TARGET_ESP32S2
      case I_32_I1_NEO_4: busPtr = new B_32_I1_NEO_4(len, pins[0]); break;
      #endif
      case I_32_RN_400_3: busPtr = new B_32_RN_400_3(len, pins[0], (NeoBusChannel)channel); break;
      case I_32_I0_400_3: busPtr = new B_32_I0_400_3(len, pins[0]); break;
      #ifndef CONFIG_IDF_TARGET_ESP32S2
      case I_32_I1_400_3: busPtr = new B_32_I1_400_3(len, pins[0]); break;
      #endif
      case I_32_RN_TM1_4: busPtr = new B_32_RN_TM1_4(len, pins[0], (NeoBusChannel)channel); break;
      case I_32_I0_TM1_4: busPtr = new B_32_I0_TM1_4(len, pins[0]); break;
      #ifndef CONFIG_IDF_TARGET_ESP32S2
      case I_32_I1_TM1_4: busPtr = new B_32_I1_TM1_4(len, pins[0]); break;
      #endif
    #endif

Initialization code. And the channel corresponds to the output used (0 based)
The B_32_...are #defines for NPB classes.

/*** ESP32 Neopixel methods ***/
#ifdef ARDUINO_ARCH_ESP32
//RGB
#define B_32_RN_NEO_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32RmtNWs2812xMethod>
#define B_32_I0_NEO_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32I2s0800KbpsMethod>
#ifndef CONFIG_IDF_TARGET_ESP32S2
#define B_32_I1_NEO_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32I2s1800KbpsMethod>
#endif
//RGBW
#define B_32_RN_NEO_4 NeoPixelBrightnessBus<NeoGrbwFeature, NeoEsp32RmtNWs2812xMethod>
#define B_32_I0_NEO_4 NeoPixelBrightnessBus<NeoGrbwFeature, NeoEsp32I2s0800KbpsMethod>
#ifndef CONFIG_IDF_TARGET_ESP32S2
#define B_32_I1_NEO_4 NeoPixelBrightnessBus<NeoGrbwFeature, NeoEsp32I2s1800KbpsMethod>
#endif
//400Kbps
#define B_32_RN_400_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32RmtN400KbpsMethod>
#define B_32_I0_400_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32I2s0400KbpsMethod>
#ifndef CONFIG_IDF_TARGET_ESP32S2
#define B_32_I1_400_3 NeoPixelBrightnessBus<NeoGrbFeature, NeoEsp32I2s1400KbpsMethod>
#endif
//TM1814 (RGBW)
#define B_32_RN_TM1_4 NeoPixelBrightnessBus<NeoWrgbTm1814Feature, NeoEsp32RmtNTm1814Method>
#define B_32_I0_TM1_4 NeoPixelBrightnessBus<NeoWrgbTm1814Feature, NeoEsp32I2s0Tm1814Method>
#ifndef CONFIG_IDF_TARGET_ESP32S2
#define B_32_I1_TM1_4 NeoPixelBrightnessBus<NeoWrgbTm1814Feature, NeoEsp32I2s1Tm1814Method>
#endif

[pbolduc]

Based on that in my example, it was channels 0, 1, 2, 3 and 4 from top to bottom.

[blazoncek]

JFYI: You will notice an increased "start" lag after about five channels. This is a bug in the IDF, there are "issues" tracking it in the ESP32 Arduino and the Espressif IDF code bases; but seem unlikely to be fixed since they are near two years old.

Measuring actual time taken to update all strips show that 5x600 takes ~40ms, but 4x600 takes only ~15ms (this is including all math for effects)

[Makuna]

@blazoncek If you add another channel, you would notice it would jump even more. The start of the sixth channel usually coincides with first channel finishing sending all data.

Here is the "issues" which include a similar image showing stagger start, but not the corrupted channel.

espressif/arduino-esp32#2885

espressif/esp-idf#3645

[blazoncek]

Indeed. Increasing pixel count on existing 4 channels (to 720/channel) only increased time taken to ~22ms. Dealing with almost same amount of total pixels).
So the timing issue is indeed related to RMT.
We are still facing occasional flicker though. As reported by @jdavis7765.
Original corruption was apparently resolved by using espressif32@3.3.2 platform.

Could this flickering be also due to RMT buffer timings? We did see those a lot after we switched to AsyncTCP @ 1.2.0 which was later solved by @pbolduc by modifying AsyncTCP code.

[Makuna]

Underneath the IDF API it is using an interrupt to request data translation (through the API) to what is needed for real RMT hardware buffers as it fills them to send (the RMT buffers are tiny). But this very short piece of code for very small amounts of data (100-300us of RMT transmission per callback I believe). So if you have other ISRs that hog the cpu (take longer than the 100-300us), then yes, this could starve the RMT data sending pump causing incorrect timing.

Often Web interfaces/libraries do WAY TO MUCH in their ISRs rather than a better model of flag request then push to a thread/task to do the work outside the ISR. Also, it can be doing too much on one core versus using one core for ISR and the other core for those "tasks".

[pbolduc]

The modifications I did in AsyncTCP were primarily worked on by others, I just created a consolidated fork WLED can use. The main cause of flickering in the updated AsyncTCP library wasn't related to interrupts, but more related to the main upstream fork adding a call to enable and then disable Task WDT on each TCP data packet. For WLED, I added a config value to skip those Task WDT registration/deregistration call on each packet. So not really related. I didn't fundamentally change any of the lower level lwIP calls.

[embedded-creations]

I'm wondering if the I2S parallel output driver integrated into FastLED is worth porting over to NeoPixelBus. For WLED, after 5 WS2812 outputs are enabled, it could disable the RMT drivers and start using I2S. I'm not sure of the RAM cost to compare RMT and I2S.

https://www.reddit.com/r/FastLED/comments/bjq0sm/new_24way_parallel_driver_for_esp32/

[Makuna]

@embedded-creations Try to keep on topic within "Issues". See #270 for what you are suggesting.

[blazoncek]

@Makuna do you agree to close this issue since it may be related to core RMT buffer behavior?

[Makuna]

@blazoncek That makes the most sense at this point. We can always reopen if new information appears.