Drop frames problem fps is very bad

[isnzenauz]

hi dear,
i have problem with fvp drop frames on android (mobile, tv) im trying to use AMediaCodec and FFmpeg but his give me same results frame drops i test it in: XIAOMI MI BOX S2 (Android), And mobile RED MAGIC 6, Poco x3 Pro, Tecno Spark 8 tecno spark 10 pro, give me same results, drop frames in LIVE M3u8, and Movies MP4, MKV

I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000 and the pakage is com.poppycorn.rax.poppycorn
I/AudioTrack(21381): AudioTrack::setVolume L:1.000000 R:1.000000
I/AudioTrack(21381): AudioTrack::setVolume L:1.0

The log note give me anything about dropframes

[wang-bin]

please show me the log following issue template

[azukaar]

Experiencing the same issue on Windows. I am using the Player low level object and showing the texture:

            Center(
              child: ValueListenableBuilder<int?>(
                valueListenable: player.textureId,
                builder: (context, id, _) => id == null
                    ? const SizedBox.shrink()
                    : Texture(textureId: id),
              ),
            ),

And even with a bare bone UI (nothing else but the player's texture) I get severe frame dropping compared to the vanilla Flutter player.

Here's the logs although, ignore the buffering my internet was a bit unstable at that moment: https://pastebin.com/VczcUvV7

I also have a sample to showcase the issue:

Aurora.Media.Center.2024-09-07.11-48-07.-.Trim.mp4

[azukaar]

In case it's not clear enough in the sample, here you have another player at the top and FVP at the bottom. You can clearly see the dropped frame after slowing down the footage

Timeline.1.mov

[wang-bin]

@azukaar

flutter's rendering performance is very poor, comparing to my Qt examples, higher cpu and gpu load, lower fps. It's texture api for native plugins is not good enough, we can't know the rendering context used by flutter gui, so in fvp I create another context to render video. but it's impossible to sync 2 contexts in fvp without knowing flutter's context, so if flutter does not handle resource synchronization correctly(I guess yes), you will see tearing. You can download libmdk and run ./glfwplay.exe -c:v MFT -d3d11 -log all $video_url to see drop count, if you can reproduce the same issue in glfwplay, then it's a bug in libmdk. glfwplay is a simple example without ui.

What is vanilla Flutter player?

[isnzenauz]

THE PROBLEM IN ANDROID TV,

[wang-bin]

THE PROBLEM IN ANDROID TV,

I need the log, otherwise I can't help

[azukaar]

flutter's rendering performance is very poor, comparing to my Qt examples, higher cpu and gpu load, lower fps. It's texture api for native plugins is not good enough

is there a way to fix this? for example adding a buffer for the texture synchronisation?

What is vanilla Flutter player?

I mean the basic VideoPlayer component in Flutter. It does not have the chopiness issue

You can download libmdk and run ./glfwplay.exe

OK so I tried it. It does work much better but I think. See the footage. First half is MDK, second half is MPV. BTW there is an issue in glfwplay.exe I think because cli options (like adding a video URL or adding -log all) don't do anything, so unfortunately I wasn't able to get any log.

Timeline.1.mov

[wang-bin]

@azukaar

is there a way to fix this? for example adding a buffer for the texture synchronisation?

Impossible to fix tearing issue. Must fix in flutter side.

I mean the basic VideoPlayer component in Flutter. It does not have the chopiness issue

which platform?

BTW there is an issue in glfwplay.exe I think because cli options (like adding a video URL or adding -log all) don't do anything, so unfortunately I wasn't able to get any log.

I run commands in msys2, then log will print to msys2 terminal. If no msys2, you can add -logfile glfwplay.log in the command line, then log will save to a file, but drop info is only visible in terminal. To play a local file, you can also run install.bat as admin, then right click a video file you can choose glfwplay to open.

[azukaar]

Impossible to fix tearing issue. Must fix in flutter side.

It's not a tearing issue, the issue is frame dropping / stuttering. And when using the Flutter Video_player or Media_kit packages instead, there's no issues at all, that's why I think it comes from the FVP package (not MDK lib itself, just the way the binding to Flutter is).
The video_player and media_kit also use a binding on Texture Component so it is capable or rendering the video without stutter for sure. I am having trouble finding the correct root cause thought, still investigating

Also see for the terminal, if I run the command, the terminal detaches:

[wang-bin]

It's not a tearing issue, the issue is frame dropping / stuttering

I see tearing in #134 (comment)

./glfwplay.exe -log all -logfile glfwplay.log -c:v MFT -d3d11 $video_url will save log to a file. Or install msys2 and run command in msys2 terminal to see log in terminal.

[azukaar]

I see tearing in #134 (comment)

No it's not tearing, I superposed two video. The top half is MPV, the bottom half is MDK

[wang-bin]

@azukaar

I see tearing in #134 (comment)

No it's not tearing, I superposed two video. The top half is MPV, the bottom half is MDK

I edit the screenshot. Do you mean there are 2 players in your recorded video, top left triangle is mpv, bottom right triangle is mdk? What I see is tearing at diagonal.

[azukaar]

yes correct that is it
i edited the video to show the difference better

[DovgopolsSerj]

is there a way to fix this? for example adding a buffer for the texture synchronisation?

Be sure to specify the texture size in the updateTexture(width, heigth) method.
I've encountered disgusting lags when playing 4K videos.
If you do not specify the size, the plugin gives the flutter a full texture with a size of 3840×2160 and the flutter scales it to the size of the window.
If you specify the texture size, the plugin will adjust the size itself and the flutter needs to perform much fewer actions.

[azukaar]

I see thanks, I am assuming that only is relevant when the viewport is not 4k

[wang-bin]

is there a way to fix this? for example adding a buffer for the texture synchronisation?

Be sure to specify the texture size in the updateTexture(width, heigth) method. I've encountered disgusting lags when playing 4K videos. If you do not specify the size, the plugin gives the flutter a full texture with a size of 3840×2160 and the flutter scales it to the size of the window. If you specify the texture size, the plugin will adjust the size itself and the flutter needs to perform much fewer actions.

try master branch, and set max texture size as screen size or smaller

registerWith(options: {
     'maxWidth': screenWidth,
     'maxHeight': screenHeight,
   });

[isnzenauz]

Screen.Record_2024-09-08-03-29-28.mp4

Look drops in the apk test

[isnzenauz]

4k live Android mobile

Screen.Record_2024-09-08-03-32-55.mp4

[isnzenauz]

Same Channel 4k working good in vlc player

Screen.Record_2024-09-08-03-35-16.mp4

[DovgopolsSerj]

Same Channel 4k working good in vlc player

Yes. Flutter is not the best choice for a video application. It copies each frame to his own scene. This is a very large volume of more than 9Mb for a 2000x1200 screen.
I spent a lot of time and now regret this choice.
I found a temporary solution for android. I took the native implementation of the mdk-sdk and brought out its SurfaceView as the root of FragmentActivity and a transparent FlutterFragment is located above it. Fragment, respectively, is my entire former application, only without the video texture.
The performance is staggering.

[DovgopolsSerj]

try master branch, and set max texture size as screen size or smaller

It had no effect.
I did an experiment. I took the mdk-sdk and brought it to Flutter as native Android views.
the example is taken from here:
https://github.com/wang-bin/mdk-android
In my app:

class NativeView implements PlatformView {
    private SurfaceView view;
    private MDKPlayer mPlayer;
    private static final String videoURL = Environment.getExternalStorageDirectory().getPath()
            + "/Download/test4k.webm";

    NativeView(@NonNull Context context, int id, @Nullable Map<String, Object> creationParams) {
        view = new SurfaceView(context);
        mPlayer = new MDKPlayer();
        mPlayer.setSurfaceView(view);
        mPlayer.setMedia(videoURL);
        mPlayer.setState(PlaybackState.PLAYING);
    }

    @Override
    public View getView() {
        return view;
    }

    @Override
    public void dispose() {
        mPlayer.surfaceDestroyed(view.getHolder());
        view.getHolder().getSurface().release();
        mPlayer.setState(PlaybackState.STOPPED); 
    }
}

In Flutter:

  Widget build(BuildContext context) {
    return Scaffold(
      body: AndroidView(
        viewType: "videoPlayer",
        layoutDirection: TextDirection.ltr,
        creationParams: creationParams,
        creationParamsCodec: const StandardMessageCodec(),
      ),
    );

In this case, there are no lags.
Although Native View uses the same texture copying technology.

Unfortunately, I can't build such an option in release mode.
mdk-sdk in build.gradle contains:
minifyEnabled false
But it is redefined by flutter and flutter itself is not going with this option.
As a result, the debug version works and the release crashes with an error.

[azukaar]

Although Native View uses the same texture copying technology.

I think this is the important point. Every players uses the same technique I think but only the MDK-flutter one seems to have the lags. I think there might just be a bug rather than a hard Flutter limitation.
Is there some additional computing that block the thread in the MDK flutter lib may be? Something to do with VSYNC? I am not really sure

[DovgopolsSerj]

Maybe it has some meaning. There is a constant message in the logs, similar to missed frames. But I do not know exactly what it means.

W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
D/ImageReaderSurfaceProducer(10596): ImageTextureEntry can't wait on the fence on Android < 33
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
I/chatty (10596): uid=10167(ru.sserjirk.flvideo) 1.raster identical 4 lines
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
I/chatty (10596): uid=10167(ru.sserjirk.flvideo) 1.raster identical 8 lines
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented
W/vulkan (10596): vkAcquireNextImageKHR: non-infinite timeouts not yet implemented