Screen recovery failing.

[Z0ld3n]

Hi Rudolph,

Firstly thanks for this amazing library!

I am facing issues during screen recovery from errors. I can fairly consistently put my screen in a state where it will never recover by introducing some short interferences on SPI lines. Basically after doing so EVE_busy() will never return E_OK (always returns EVE_FIFO_HALF_EMPTY and is indefinitely stuck in a loop) so I decided to introduce a retry counter and another define EVE_AFTER_RESET to indicate that reset happened and return it from EVE_busy(). I did so because I noticed another issue with recovery. After recovery extended fonts are not recovered, I have to perform another set of fonts initialization (using EVE_cmd_flashread and EVE_cmd_setfont2), and here comes my issue, every few recoveries fonts are not reinitialized even though functions are executed without error. What am I missing to properly recover from coprocessor errors when using extended fonts? I also don't know how to debug it properly, I assume the issue must be somewhere between BT816 and the display flash.

Im using STM32F303RE and Riverdi RVT43ALBFWN00 display.

[RudolphRiedel]

Hello,

and thank you for the report, looks like I need to actually make EVE cry for once to test this better.
I have to admit that I never ran into this issue and interference on the SPI lines is kind of an odd concept to me. :-)

That EVE_busy() returns EVE_FIFO_HALF_EMPTY is of course an error and I have to admit an obvious one - ouch. :-)
The version I am playing with has this fixed now and for the time beeing would return E_NOT_OK, now I only need to trigger a coprocessor fault and figure out how this could be debugged.

That the fonts are gone is a result of the bitmap handles getting reset.
Reloading should not be necessary though as RAM_G should not be cleared on reset.
But loading PNG images wipes out the top 42k of RAM_G, everything above 0xF5800 gets overwritten for the decoding.
And 0xC0000 to 0xC0003 gets overwritten by the ID.

[RudolphRiedel]

I did a couple of tests to make EVE cry.
EVE_Recover_Arduino_PIO_RiTFT43.zip

EVE_commands.c is updated to return E_NOT_OK on co-processor faults.
I triggered a few faults and EVE_busy() did recover from those.
Well, with the exception of CMD_CLEARCACHE as this takes >12ms before a fault is indicated.

The test simply draws a "Hello, world!" in black, triggers a fault, calls EVE_busy() and when recovering from the fault was successfull a new display list is written with "Hello, world!" in blue.

And then there is this:

if(E_NOT_OK == EVE_busy())
{
uint8_t p_data[256];
EVE_memRead_sram_buffer(EVE_RAM_ERR_REPORT, p_data, 128);
char buffer[200];
sprintf(buffer, "%s", p_data);
Serial.println("Coprocessor Fault!");
Serial.println(buffer);
}

This prints:
Coprocessor Fault!
ERROR: illegal font or bitmap handle in cmd_setscratch()

`

[Z0ld3n]

Thanks for response,

I didn't think of sending wrong command to trigger fault :) It was easiest for me to interrupt SPI cause this error firstly occurred during EMC tests of our custom board.

So according to your info i should not need to reinitialize fonts but it does not work if i don't reinitialize xfont/font handles/bitmap handles in RAM_G (and it is not working every time, sometimes i need to do it twice).

What I am getting as coprocessor fault is : ERROR: unsupported command

To make EVE respond again i have to omit restoring previous state, basically i need to remove:

    uint16_t copro_patch_pointer;
	copro_patch_pointer = EVE_memRead16(REG_COPRO_PATCH_DTR);

and:

    EVE_memWrite16(REG_COPRO_PATCH_DTR, copro_patch_pointer);
    DELAY_MS(5U); /* just to be safe */
    EVE_cs_set();
    spi_transmit((uint8_t) 0xB0U); /* high-byte of REG_CMDB_WRITE + MEM_WRITE */
    spi_transmit((uint8_t) 0x25U); /* middle-byte of REG_CMDB_WRITE */
    spi_transmit((uint8_t) 0x78U); /* low-byte of REG_CMDB_WRITE */
    spi_transmit_32(CMD_FLASHATTACH);
    spi_transmit_32(CMD_FLASHFAST);
    EVE_cs_clear();

after that i have to reinitialize flash using

EVE_init_flash() (simply sending CMD_FLASHATTACH and CMD_FLASHFAST is not enough)

and setup new handles/fonts using series of:

EVE_cmd_flashread()
EVE_cmd_setfont2()
EVE_cmd_setbitmap()

After all of this is eve is working fine. If i don't do this program is stuck in EVE_busy(). I must be missing something or doing something wrong but I have no clue what :/

I am putting font data in 0x186C0 address and am using 20800 bytes for xfont data. And this is only thing i store in RAM_G.

[RudolphRiedel]

I can't do any EMC tests myself and it was not necessary so far at work for any project featuring a display.
So sending EVE an unsupported command was the easiest thing to do and the first thing I did. :-)
I sent 0xFFFFFFFF but anything above 0xFFFFFF70 should result in "ERROR: unsupported command".
For BT815 it would be 0xFFFFFF5E.
Hmm, actually I am curious what will happen, also for the number of not documented commands in between. :-)

The sequence of commands is from the programming guide and I am not really happy with it.
In particular as the external flash might not even be used or present.
And in regards of details there are no checks or delays, it might fail regardless.
It should be better to leave out the flash commands and handle this in the application when needed.

I'll up the game, so far I was using a 4.3" with BT815, as I had it on the desk already.
I'll change over to the RVT50HQBNWC00-B with BT817.
And I'll check it with a custom font.

I am putting font data in 0x186C0 address and am using 20800 bytes for xfont data

Wow, that is a lot, how many glyphs does your font have? Looks like 5000 to 6000.
Is this for something like a multi language vendor machine?

[RudolphRiedel]

I reworked EVE_Busy() a bit more, removed the flash commands from it and added a new return code EVE_FAULT_RECOVERED to indicate that EVE_busy() detected a coprocessor fault and went thru the recovery sequence.

I ran a quick test with the small font I am using for the other examples and while I had to put EVE_cmd_setfont2() even in the first display list so it makes no difference if there is a coprocessor fault or not.

It works just fine though with not using EVE_init_flash() and EVE_cmd_flashread().
The flash is in FLASH_STATUS_FULL before the fault and after the fault.
Issuing an unsupported command is not detaching the flash.

Now I need to find a font with a lot more glyphs.

[RudolphRiedel]

I am trying to find a font with so many glyphs that the .xfont file has 20kiB.
The best I could find so far is Quivira, it has 11091 glyphs and after conversion the log says that 8089 got converted.
All of these have a .xfont file of "only" 13376 bytes.
The .glyph file for size 24 has over 5MiB.

And it just works, no need for extra steps.

Can you provide an example for a 32 bit word that brings EVE more pain than what I can come up? :-)

[Z0ld3n]

Wow, that is a lot, how many glyphs does your font have? Looks like 5000 to 6000. Is this for something like a multi language vendor machine?

I should have been clearer, it is not one xfont, but a few for different languages and font sizes.

Ok so I did some more tests thanks to your last comments and I believe that my issue is partially solved. If I don't do flash reinitialization as you suggested (it is not getting detached), EVE can almost always recover without issue, I just have to set up handles again. However, there are still some occasional circumstances where execution is stuck in EVE_busy() always returning EVE_FIFO_HALF_EMPTY. For now, I decided to check if there are 500 (seems to work fine for me) consecutive EVE_FIFO_HALF_EMPTY returns and perform another recovery without restoring the coprocessor state. This seems to solve it all.

Also, may I suggest adding some way to check if recovery happened from outside of the library code? Some global variables or introduce callback functions to perform custom recovery. There is no easy way to perform application-specific recovery atm.

[RudolphRiedel]

If I don't do flash reinitialization as you suggested (it is not getting detached),

I removed the flash commands with the update yesterday, this is something an application would need to do.
Edit: just call EVE_init_flash() as it checks the flash status and does nothing when in EVE_FLASH_STATUS_FULL.

And I just added defines like EVE_FLASH_STATUS_FULL.

I just have to set up handles again.

This one puzzles me, for some reason it does not work here when I setup the font handle before writing the first display list.
But using EVE_cmd_setfont2() in the recovery code before setting up the second display list works just fine.
I added delays and even an additional empty display list, does not work.
Adding the EVE_cmd_setfont2() to the empty display list works though.
Ok, I know that EVE_cmd_setfont2() generates a set of display list commands so maybe these need to be executed in a display list.
But then, it works outside of the display list generation the second time.

However, there are still some occasional circumstances where execution is stuck in EVE_busy() always returning EVE_FIFO_HALF_EMPTY.

Have you updated the library?
And if you can still trigger this, please use EVE_memRead16(REG_CMDB_SPACE); - I wonder if it gets set to a value that would indicate an error but not a coprocessor fault.
There are things missing from the programming guide, like that error for "unsupported command".
Oh, please also read REG_CMD_READ.

Also, may I suggest adding some way to check if recovery happened from outside of the library code?

EVE_busy() is the only function to attempt the recovery and it returns EVE_FAULT_RECOVERED now, so usually this should be enough.
There also is EVE_execute_cmd() though which waits untill EVE_busy() returns E_OK and does not return anything.

Hmm, maybe this helps as additional safeguard:

uint8_t EVE_get_and_reset_fault_state(void)
{
uint8_t ret = E_OK;

if (fault_recovered)
{
ret = EVE_FAULT_RECOVERED;
fault_recovered = 0;
}
return ret;
}

I just pushed another update.

[RudolphRiedel]

Regarding the CMD_SETFONT2 issue.
I started to write a message on the BRT forum regarding this but that it fails outside a display list makes sense.
The command generates display list commands and is documented to do so.

So the clean way to use it would be:
EVE_cmd_dl(CMD_DLSTART);
EVE_cmd_setfont2(12, MEM_FONT, 32); /* assign bitmap handle to a custom font */
EVE_cmd_dl(CMD_SWAP);

So the question is not really why the first call outside of a display list fails,
the question is why it works the second time when not using it in a display list.

Without the CMD_SWAP it should get overwritten by the following new display list.
And yet EVE_cmd_setfont2(12, MEM_FONT, 32); worked in my recovery code and the display stayed
black when I removed that line.

Adding all three lines to the recovery code works as well, as it should, that using only the one line works is merely by coincidence.

[Z0ld3n]

Have you updated the library?

All of my tests were on the latest available version of the library.

And if you can still trigger this, please use EVE_memRead16(REG_CMDB_SPACE); - I wonder if it gets set to a value that would indicate an error but not a coprocessor fault.

Oh, please also read REG_CMD_READ.

So to test this I increased my stuck in EVE_FIFO_HALF_EMPTY detection counter to 10000 (which is way longer than EVE should take to execute full DL). After reaching my "timeout" I assumed that it is in an OK state and was sending a "new" frame (it was the same DL all the time).
And these are the results:
REG_CMDB_SPACE: 3384, 2516, 4008, 3140
REG_CMD_READ: 2588, 2588, 2588, 2588
so it seems like REG_CMD_READ is stuck on this value

If after "timeout" I perform another recovery, everything is fine. (This happens occasionally and performing another reset is fine for me, but sadly I need to modify the library code to detect it :/)

About EVE_cmd_setfont2 and reinitialization: to reinitialize I am indeed writing an empty display list once (on init and on coprocessor fault):

	EVE_cmd_dl(CMD_DLSTART);
	EVE_cmd_dl(CLEAR_COLOR_RGB(255UL, 255UL, 255UL));
	EVE_cmd_dl(CLEAR(1UL, 1UL, 1UL));
	// bunch of:
	EVE_cmd_setfont2()
	// than a bunch of:
	EVE_cmd_dl(BITMAP_HANDLE())
	EVE_cmd_setbitmap()
	//	
	EVE_cmd_dl(DL_END);
	EVE_cmd_dl(DL_DISPLAY);
	EVE_cmd_dl(CMD_SWAP);

There also is EVE_execute_cmd() though which waits untill EVE_busy() returns E_OK and does not return anything.

That is what I meant. There was no way to know if a fault happened and to trigger setting up handles without modifying the library code. I guess EVE_get_and_reset_fault_state() is OK, I can just check it after each EVE_execute_cmd.

[RudolphRiedel]

And these are the results:
REG_CMDB_SPACE: 3384, 2516, 4008, 3140
REG_CMD_READ: 2588, 2588, 2588, 2588
so it seems like REG_CMD_READ is stuck on this value

Oh, it might be stuck there but there is no coprocessor fault detected.
0xd38, 0x9d4, 0xfa8, 0xc44
0x1ac

These values are perfectly fine indicating no issue with the coprocessor.
Are these really from the period during which EVE hangs with returning EVE_FIFO_HALF_EMPTY?

It might be very interesting to trigger this reliably in order to ask Bridgetek about this and what could be done.

[Z0ld3n]

Yeah, I'm sure they are from when I get EVE_FIFO_HALF_EMPTY as a return. I even added a 10s delay just to make sure.

I have one more question (maybe I should open a new discussion for this?). Up until a week ago, I was using your library version from 2022-05-06 and I was using DMA without any issues. To debug issues from this discussion I disabled DMA and updated the library, however now I wanted to enable DMA again but can't get it to work. Maybe you have any idea what change might be causing this?

[Z0ld3n]

Nevermind about DMA, it was my mistake it is working fine :)

[RudolphRiedel]

Ugh, Github only sent me the first message. :-)

What is your SPI clock then? With DMA you can probably go a lot lower.