Weird decoding glitch with AV1

[makedir]

I bought one of these Intel N100 box which has Intel 12th gen cores/GPU which is capable to hw decode AV1. Using latest stable Moonlight 5.01, latest Intel drivers and Windows 11 both on client and server. Host uses latest Sunshine nightly. Host has a RTX4080. Streaming uses 1440p@120 FPS with 50mbit.

Tested with Diablo 4, I noticed first no issue starting the stream. Decoding information stat on client showed 1 to 1.5ms decoding latency, stable 120FPS. Starting Diablo 4 everything worked fine. Walking around in town or doing dungeons, ev erything works fine, stable 120FPS.

Then I opened the map in Diablo 4, and suddenly the stream broke down and I got decoding latency of about 60-80ms. Closing the map and latency goes back down to 1 to 1.5ms. Opening the map again latency jumpes to 60ms.

I can reproduce this issue with 100% chance.

I tried to find out whats going on. I dont know why this is happening, if there is some "special" features of the Diablo 4 map which causes this, maybe because the map gives "too much" informations because of the nature of the image. The map looks like this:

I then notice, that when I move the cursor around and move the map around, the latency goes back down to 1.5ms. Letting cursor go and make the image "freeze" because of no motion in the screen, the latency jumpes back to 60ms.

I thought of this maybe is some weird enegry saving maybe with the AV1 coded and the Intel GPU, but I also tested on Desktop where also is a frozen image with no information change, or in the Diablo 4 menu where nothing is moving. The issue is not happening there.

I noticed when I stop the stream while on the map with ctrl alt shift q, then reconnect, so I am still on the map, the latency now is fine at 1.5ms even with no parts of the stream are moving, with the same image, where before the latency jumped to 60ms. Closing the map and reopeing now getting again high latency. Which goes again away when moving the map around.

I set the FPS down from 120FPS to 60FPS and now the issue doesnt seem to happen anymore.

Issue also doesnt happen with H264 and H265.

[mvillarg]

To me this looks like a bug on the Intel 12th gen hardware decoder when the image doesn't change a bit from frame to frame. Can you open Windows performance stats and take a look at both CPU % utilization and GPU decoder graph when this happens?

[makedir]

@mvillarg how? you mean perfmon? I dont see anything in there to frack gpu usage or even gpu video decode usage.

[makedir]

I was able to reproduce the issue with running HwInfo, being on desktop, and open a https://www.testufo.com/ to trigger a constant 120fps animation on screen and then switch between two tabs in Firefox between ufotest and no motion.

The moment I change to ufotest and have constant 120fps animation it seems to work fine, the moment I change to other tab and have no motion on screen, the problem starts to happen, but JUST if I also slowly move the mouse cursor around very slowly. When I dont move the mouse cursor around it doesnt seem to happen.

Pxl.20240531.101014545.mp4

Pxl.20240531.101014545b.mp4

This doesnt happen with h264 and h265, just AV1. Not sure if the issue is on the client side with Moonlight or on the server with Sunshine.

[mvillarg]

I mean you can open Windows's Task manager which has well-known performance stats for CPU, RAM, GPU and Network. When you detect the issue you can alt+tab and see what's happening there. The GPU stats include 3D workload performance, VRAM usage and both enconding & decoding performance charts.

[makedir]

@mvillarg please see what I posted on 2nd post. Issue also seems to happen with h265 but just very rarely and just for 1 second or so randomly and then goes back, also just when moving mouse around. What is weird though is, even with high mouse movement over entire screen, the issue wont dissolve with AV1 and is stuck at high latency.

Is there a way on host / Sunshine to force a constant 120FPS stream even with no motion? It seems the host is throttling FPS down to even 11fps with no motion, maybe that is the issue and the decoder gets bugged with that. Though it doesnt seem to bug out that much on h265 just av1.

[makedir]

Seems someone already suggested it a year ago here with the same issue, that the stream lags on no motion with no answer:

https://github.com/LizardByte/Sunshine/discussions/1610

Is there no option for Sunshine to force constant FPS?

[mvillarg]

Hi @makedir ,
Ok, this is simpler than it looks. What's happening here is not actually related to AV1 not being able to encode at a constant bitrate, but simply that the game is not sending draw commands to the GPU, hence Sunshine is not encoding/sending anything to Moonlight. This even occurs in Windows itself, where if you don't move the mouse the FPS counter almost drops to 0.

[makedir]

@mvillarg and you want now what say with that?

1. why doesnt it happen with h264 and h265
2. why is this also just happening if moving mouse cursor slowly like seen in the examples in combination with no other screen motion. what you said there is wrong. look at the examples again. it does not happen if not moving the mouse it just happens if you move mouse at the same time of no other draw. also moving mouse quickly doesnt resolve it, where it should with your theory.
3. why is there no option in sunshine forcing a constant fps / constant bitrate
4. what is the solution / workaround for this
5. so you are saying moonlight and sunshine are useless? because it is, if they behave like this with no constant fps force even with no draw on screen.

[mvillarg]

Hold on a second... Does that huge decoding latency (50-70ms) really affects the input or audio in the game? For example, when you start scrolling or zooming in/out on the map do you experience some sort of severe glitching/shuttering?

If you don't experience such a high input lag or audio shuttering in that specific situation where the game is not rendering any new frame I wouldn't worry too much. The high decoding time might be tightly related to both hardware or drivers issues on the N100, or just the way delta time is measured in thouse super low framerate conditions between Sunshine and Moonlight.

[makedir]

"Hold on a second... Does that huge decoding latency (50-70ms) really affects the input or audio in the game? For example, when you start scrolling or zooming in/out on the map do you experience some sort of severe glitching/shuttering?"

Yes?? of course. the entire stream laggs when this happens. or mouse cursor.

[makedir]

Nightly change seems to fix this issue I reported here with Intel N100 and AV1 @cgutman

#1304 (comment)

So it totally was a bug in Moonlight not the Intel drivers with AV1. Please push this nightly change as 6.0.1 stable.

[cgutman]

Fixed by c3bd7ed and 94943d2