Orange pi 5 plus HDMI IN native V4L2 support really needed (OBS crashes) #830
Comments
MaximKurakin
changed the title
MaximKurakin
changed the title
|
Good find, have you confirmed the kernel patch works? |
|
I'm trying to build the media module right now and test in on my Orange PI 5 PLUS. But it might take a while as soon as i'm really new in kernel drivers building. Struggling with finding proper way to override kernel media module with my module at working ubuntu 24.04. What did i do:
I really need a good push in right direction =) |
|
If you have the kernel build dependencies installed correctly cd into your kernel tree and enter the below to re-build the kernel. You can then install the deb package. # Setup env
export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
export CC=aarch64-linux-gnu-gcc
export LANG=C
# Compile the kernel into a deb package
fakeroot debian/rules clean binary-headers binary-rockchip do_mainline_build=true |
|
I can try to test on my end in a few hours, i'm currently working on some U-Boot code for some new Radxa boards. |
|
I am quite sure that in order to get hdmi in to work in obs studio we would need panthor. This is what amazingfate claim. So, should be pinned here. |
|
Guys. Sorry for delay. I had to do a slight detour to GPIO and FAN settings, because during kernel compilation my orange pi was trying to melt ... So, the progress exist. I'v managed to reduce amount of v4l2-compliance errors from 9 to 7 as of today... OBS still crashes though... |
MaximKurakin
changed the title
Well according to the article new GPU driver will be merged with kernel 6.10 (we expect this driver to land in Linux v6.10). But we are sitting on 6.1. I think it will take years to get there... Do you think enabling HDMI IN v4l2 full support completerly impossible with current panfrost driver? Even if i'll fix hdmirx driver for rk3855? just let it here for future reference https://www.collabora.com/news-and-blog/news-and-events/release-the-panthor.html |
|
I mean, currently, hdmi in performs really bad in obs wheb used with v4l2 (at least when it worked). The only real way to use it is with a gst pipeline, and that's inside a windowd, not in obs itself. Panthor is meant for mainline, but was backpoorted by boogie to bsp 6.1 |
|
Ok this is my bes effort after 2 weeks of sufferring without sleep, food and life itself ... v4l2-compliance shows 0 errors and 0 warnings! I was making test like this: ffmpeg -f v4l2 -video_size 3920x2160 -i /dev/video0 -pix_fmt yuv420p -vf "format=yuv420p" -framerate 60 -c:v rawvideo -vsync 1 -f sdl "SDL output" I have tried to merge synopsis driver with rockchip because the source code is almost the same in general... But i'm week in DMA flags, bits and fencing... I need somebody more talanted and experienced to look at it. Any help is very much appreciated! I think if we can solve DMA sync issues we will finally fix rockchip driver for good and will get 4k60 as entitled in orange pi 5 plus specification... Nobody touched this driver since 2021... Let's fix it up all together! |
|
I would love to help, but I am not capable. Did you try to contact amazingfate? The only way I got decent hdmi in was with ximagesynk streaming pipelines and obs recording my desktop, at even then, I never got 1080p60 out of it. He claimed tha tin order to really use it inside obs, so, as a v4l2 source, we had to use panthor and even then it would need more work. You should at least co tact him and boogie at radxa discord, or armbian one. |
|
@MaximKurakin Did you happen to test HDMI audio input with the Collabora code? It was my understanding that they didn't support it yet. Also, you might want to look at the obs-gstreamer plugin. If performance is bad, you might be doing colour format conversion on the CPU rather than RGA. I've never tried it directly in OBS, just in gstreamer directly. |
|
Collabora driver doesn't support audio and HDCP. Thats why I went into madness of merging current rockchip driver with it. In debug mode drivers gives something like: |
I tried this plugin today (had a hard time with unclear dependencies during buld). yes i can capture gstreamer directly to obs now. but fps is still 30 and tonns of dropped frames -( Thank you for you advice anyway! Also I tried obs beam plugin. (had very sophisticated intercource with build process for arm64). Yes it works but device cannot keep up with 2k 60fps at any jpeg compression quality even in quality 1!!!! black screen always flashing from time to time. All equipment has 2.5 Gbps caps but this orange pi 5 plus really cant keep up with the massive frames comming from network... I don't know if anybody even tried to launch obs-beam on orange pi ... but I did and it doesn't work as expected -((( platform simply doen't have enough performance |
Contacted. He only introduced : media: rockchip: hdmirx: add VIDIOC_G_PARM hack fix for Armbian. BUt this fix doesn' work for Jushua Riek build. I tested it already. But he gave me hope with armbian embedded panthor driver support out of the box. SO, i'm going to give it a spin later today. Maybe with panthor driver overal graphical performance will be better? |
|
Panthor does not have better performance and has quite a few bugs and problems when playing games. |
|
To me, it has better performance,.overall. but even so, it has better features, and will have much better ones down the road. This is not about performance, but feautres |
|
Ok. No more ARMBIAN for me!!!! It freezed 3 times in 30 minuts. HDMIRX driver exactly the sam as in Josua build ... 30 fps max. 9 compliance errors an 6 warnings. Morover! OBS doesn't recognize new panthor graphics and doesn't start . THis is a dead end to me. |
|
Obs doesnt start bc it needs PAN_MESA_DEBUG=gl3, it's probably forced on panfork by default |
|
I understand your suffering! Hahah yeah, hdmi in is only really usable from android as today. |
|
I don't give up with it. I'm just taking a pause to collect more information about HDMI protocol and rk388 chip memory access features. But after 2.5 weeks reading tons of web-sites and manuals ... I need vacation |
Please contact me at armbian discord. Microlinux(salva) |
|
I'm definitely interested in collaborating on this, as I've been working with Rockchip SoCs for about a year and a half now for mobile livestreaming. I've generally found the HDMI input to work well enough, except when it doesn't. And it can be frustrating when it doesn't. Couple thoughts: Early on, I was seeing pretty choppy performance, dropped frames, etc, but it seemed like at least some of that wasn't the HDMI input itself's fault, but a downstream element not being fast enough. In my case, the HDMI input was using a pixel format the encoder I was using didn't support, so I need to use a The drivers seem very fragile in their current state. Slight disturbances to the kernel, etc, seem to break things. I've had audio just stop working multiple times, and at least at one point, the colours went all weird. When a briefly looked at OBS support, I was seeing issues due to multi-planar formats. I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/ |
|
Nice, I used that env var on my pipelines too and yes, obs seems to not support multiplannar formats. https://github.com/neofeo/BOX86-BOX64-WINEx86-TUTORIAL/blob/main/gst/hdmi_in.sh |
|
Hmm . This GST_VIDEO_CONVERT_USE_RGA didn't help me. Same frame drops and 30 fps max -(((( More insteresting is OBS crach logs: and same time hdmirx shows only this one line: I hope it will give somebody a hint where is the issue |
I'm on it. Will test it in a couple hours... |
Nope... nothing changed. The most surprizing part is that i'm grabbing 1080p60 and getting 30 fps. Then I set 30 hz to the HDMI 2 display out (i'm using orange pi 5 plus as a source and as an input just made a cable loop hdmi2 to hdmi in) and grab it - i'm getting 20 fps... oddd |
|
More research comming up: I stopped totruring hdmi in and decided to make a clean perf test of my SBC caps. Here is the result:
I did the test wrong way? Or it's my orange pi 5 plus actual limits ? |
That's odd. When I plug an NV24 input I get this: And FFmpeg is happy treating it as NV24. |
|
I'll need to test again with the new kernel. Maybe your recent patch fixed this issue... |
|
@michelehokuto friend. Shortly - with current
If you need only to record in 1080 then try ffmeg. It will also generates tons of CPU load but still can provide 60 FPS the trick is scaling with rkrga hardware support which you cannot set in gstreamer (i don't know how): If you still need NV24 then use I know this is ridiculous but rk3588 chip software support currenly in the very raw state. RK3588 spec says "hardware encoding and decoding support" but actually all needed software (ffmpeg, ffplay, vlc, gst, obs ... ) are not fully support our chip. THere are separeted efforts to enable our chip support from the hard working guys like Joshua(stitching all togather) and nyanmisaka (ffmpeg for rockchip HW encoding\decoding), collabora (https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md - attempt to enable ALL hardware in upstream kernel for rockchip) but all this attempts are still "work in progress". If you are good at codding - then hack into the source code and DYI -) but if you are just a normal user - just wait for panthor upstream support (6.10 kernel myabe?) and suffer. I'm still working on HDMIRX driver by myself and managed to add VIDIOC_G_PARM to it and also fix 4 errors out of 9 but my work is still far from completion. |
This sounds like the issues I was describing, where the capture device doesn't signal to gstreamer what format it's using. If your input is NV12, you need to use NV12. The capture device doesn't do any pixel format conversion, so choosing the wrong format will have it fail to negotiate.
Setting the environment variable |
Could prove it with this cmd (downscale 4k to FHD 60) ( to check RGA is actually involved I used this cmd sudo cat /sys/kernel/debug/rkrga/load): But my Laptop Nvidia with Optimus looks like try to play "smart" and if nothing is moving on the screen it drops fps down to 30 from time to time. Anyway, that should be irrelevant for your task.
But trying to do the same with |
Hmmm maybe something is really wrong with 1080 NV24 format? I connected Orange Pi 5 Plus HDMI 2 to HDMI IN and set 1080p60 NV24. Then tried to capture it with gstreamer: And I'm getting only 30 stable FPS and zero RGA usage even with
Possible reason one CPU core is 60% loaded. |
|
RGB capturing in 1080p60 is broken completely. 4 FPS only
Possible reason: One CPU core is always 100% loaded. Gstreamer issue? single core RGB encoding? hmmm I don't know... |
|
Then only color format which gives stable 60 FPS for 1080p is NV16!
CPU graph is fine and flat |
|
Finally I could achieve my goal to stream in 60 fps!!!! but only in 1080p =((( OPI 5 PLUS can handle 2k streaming but fps will be jumping between 44 to 56. Not eyes pleasing I must say... Short instructions for those who wants to get the same 1080p60 with camera and gstreamer hw h264 encoding to youtube:
|
|
Hello all , just want to know what I have to directly do to run HDMI input as I want to grab images using opencv. I am knew to orange pi 5 plus. |
just like @RkhanTUM, I also wanna know on how to get the input on opencv. In the comment I'm answering to I basically tell you what I tried so far. If anyone could give clear steps for beginners like me on how to get the data without obs that would be fantastic, thanks in advance |
|
@MaximKurakin can you point out in which ubuntu rockchip release you are? I can't reproduce your success. At 1080p, it won't do N16 on my side |
Hello friend. I'm sitting tight on Joshua's Ubuntu 24.04 LTS (GNU/Linux 6.1.0-1017-rockchip aarch64) release . Just because I started my infinite stream to youtube I cannot touch this devise anymore. From time to time i'm checking both armbian and Jushua releases and waiting for this HDMI-IN sortware maihem to complete. New panthor driver upstream support is what should make our life easier. Without proper kernell support currently there is no way to grab HDMI-IN port signal in the "normal" way. |
|
Yes, I would also agree thst the hdmi input and the rk3588 delivers insane quality and very low latency. With proper software, this kind of devices would outshine any capture card. Hopefully we can use it as any other v4l2 source in obs on mainline woth panthor,etc... |
|
I am downloading latest release, wish me luck! |
|
@MaximKurakin the problem remains the same, I cant use NV12 with hdmi input .. that being said, obs recording seems to have been fixed in noble!! that's great @Joshua-Riek my main script remains the same since I cant force it to use NV12 or NV16 note: most of those variables aren't really used |
|
Here's a 90% solution with v4l2, just have to clean out the qt specific code. Also switch from bgr24. Code is free as in puppy. I guess I should mention this works against mainline kernel. Haven't tried it on the bsp. |
|
@benhoff we need a more detailed explanation. |
|
@MaximKurakin so, Amazingfate is working on a PR to add proper support for v4l2 for the hdmi input on the bsp kernel. |
Good to konw that! I spent months trying to fix the HDMI-RX driver. And yes I had a good proggress but had to switch to another activities. At one of my buillds it was possible to natively use HDMI-IN via v4l2 but there were so many other issues with the signal which I didn't have time to solve. I'll be happy to share my work with someone who have mote rime than me. |
Agreed. BGR24 transcoding causes a lot of CPU load. I've made all the test in the messages abothe NV16 - #830 (comment) So you should find a way to output NV12 or NV16 to OPI 5 PLUS. It it's windows PC you can use CRU tools to create custom resolution with custom colorspace. |
|
Anyone figure out using ffmpeg with hdmirx? I just installed the most recent version 2.3.2 24.0.4. It looks like something isnt working right, not sure if I need to be on an older version? v4l2-compliance -d /dev/video0
v4l2-compliance 1.26.1, 64 bits, 64-bit time_t
Compliance test for rk_hdmirx device /dev/video0:
Driver Info:
Driver name : rk_hdmirx
Card type : rk_hdmirx
Bus info : fdee0000.hdmirx-controller
Driver version : 6.1.75
Capabilities : 0x84201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Required ioctls:
fail: v4l2-compliance.cpp(667): missing bus_info prefix ('fdee0000.hdmirx-controller')
test VIDIOC_QUERYCAP: FAIL
fail: v4l2-compliance.cpp(810): doioctl(node, ioc, nullptr) != ENOTTY
test invalid ioctls: FAIL
Allow for multiple opens:
test second /dev/video0 open: OK
fail: v4l2-compliance.cpp(667): missing bus_info prefix ('fdee0000.hdmirx-controller')
test VIDIOC_QUERYCAP: FAIL
test VIDIOC_G/S_PRIORITY: OK
test for unlimited opens: OK
Debug ioctls:
test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
test VIDIOC_LOG_STATUS: OK
Input ioctls:
test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
test VIDIOC_ENUMAUDIO: OK (Not Supported)
test VIDIOC_G/S/ENUMINPUT: OK
test VIDIOC_G/S_AUDIO: OK (Not Supported)
Inputs: 1 Audio Inputs: 0 Tuners: 0
Output ioctls:
test VIDIOC_G/S_MODULATOR: OK (Not Supported)
test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
test VIDIOC_ENUMAUDOUT: OK (Not Supported)
test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
test VIDIOC_G/S_AUDOUT: OK (Not Supported)
Outputs: 0 Audio Outputs: 0 Modulators: 0
Input/Output configuration ioctls:
test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
fail: v4l2-test-io-config.cpp(166): TIMINGS cap set, but could not get current timings
fail: v4l2-test-io-config.cpp(386): Timings check failed for input 0.
test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: FAIL
test VIDIOC_DV_TIMINGS_CAP: OK
fail: v4l2-test-io-config.cpp(571): ret != EINVAL
fail: v4l2-test-io-config.cpp(625): EDID check failed for input 0.
test VIDIOC_G/S_EDID: FAIL
Control ioctls (Input 0):
test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
test VIDIOC_QUERYCTRL: OK
test VIDIOC_G/S_CTRL: OK
test VIDIOC_G/S/TRY_EXT_CTRLS: OK
test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
Standard Controls: 3 Private Controls: 2
Format ioctls (Input 0):
test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
test VIDIOC_G/S_PARM: OK
test VIDIOC_G_FBUF: OK (Not Supported)
fail: v4l2-test-formats.cpp(374): ycbcr_enc >= 0xff
fail: v4l2-test-formats.cpp(491): testColorspace(!node->is_io_mc, pix_mp.pixelformat, pix_mp.colorspace, pix_mp.ycbcr_enc, pix_mp.quantization)
test VIDIOC_G_FMT: FAIL
fail: v4l2-test-formats.cpp(744): Video Capture Multiplanar: TRY_FMT(G_FMT) != G_FMT
warn: v4l2-test-formats.cpp(794): TRY_FMT cannot handle an invalid pixelformat.
warn: v4l2-test-formats.cpp(795): This may or may not be a problem. For more information see:
warn: v4l2-test-formats.cpp(796): http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html
test VIDIOC_TRY_FMT: FAIL
warn: v4l2-test-formats.cpp(1157): S_FMT cannot handle an invalid pixelformat.
warn: v4l2-test-formats.cpp(1158): This may or may not be a problem. For more information see:
warn: v4l2-test-formats.cpp(1159): http://www.mail-archive.com/linux-media@vger.kernel.org/msg56550.html
fail: v4l2-test-formats.cpp(1185): Video Capture Multiplanar: S_FMT(G_FMT) != G_FMT
test VIDIOC_S_FMT: FAIL
test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
test Cropping: OK (Not Supported)
test Composing: OK (Not Supported)
test Scaling: OK (Not Supported)
Codec ioctls (Input 0):
test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
test VIDIOC_G_ENC_INDEX: OK (Not Supported)
test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)
Buffer ioctls (Input 0):
fail: v4l2-test-buffers.cpp(517): node->g_dv_timings(timings)
fail: v4l2-test-buffers.cpp(642): testCanSetSameTimings(node)
test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
test CREATE_BUFS maximum buffers: OK
test VIDIOC_EXPBUF: OK
test Requests: OK (Not Supported)
Total for rk_hdmirx device /dev/video0: 46, Succeeded: 37, Failed: 9, Warnings: 6I just need ffpmeg to output a 1080p 60fps stream and thats it. |
Yes. Hdmi input in ffmpeg works out of the box now, but you need to get the latest revision of ffmpeg from git and build it yourself. You also need to use a relatively recent ubuntu build (from June or July IIRC). |
|
I am on the most recent ubuntu release and the most recent ffmpeg release. Now it doesnt even recognize /dev/video0 as a cap device: ffmpeg -version
ffmpeg version 7.0.2 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)
configuration:
libavutil 59. 8.100 / 59. 8.100
libavcodec 61. 3.100 / 61. 3.100
libavformat 61. 1.100 / 61. 1.100
libavdevice 61. 1.100 / 61. 1.100
libavfilter 10. 1.100 / 10. 1.100
libswscale 8. 1.100 / 8. 1.100
libswresample 5. 1.100 / 5. 1.100
If I try to start a capture I get: ffmpeg -f v4l2 -i /dev/video0 -c:v h264_rkmpp_encoder -b:v 600k -g 30 -f rtsp rtsp://localhost:8554/cam
[sudo] password for logan:
ffmpeg version 7.0.2 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 13 (Ubuntu 13.2.0-23ubuntu4)
configuration:
libavutil 59. 8.100 / 59. 8.100
libavcodec 61. 3.100 / 61. 3.100
libavformat 61. 1.100 / 61. 1.100
libavdevice 61. 1.100 / 61. 1.100
libavfilter 10. 1.100 / 10. 1.100
libswscale 8. 1.100 / 8. 1.100
libswresample 5. 1.100 / 5. 1.100
[video4linux2,v4l2 @ 0xaaab2269a9b0] Not a video capture device.
[in#0 @ 0xaaab2269a730] Error opening input: No such device
Error opening input file /dev/video0.
Error opening input files: No such device
|
Did you enable hdmirx in the device tree overlay? |
Oh, and the h264 rkmpp encoder still doesn't work :P. I was talking only about hdmi input through v4l2, which now works.
7.0.2 isn't enough. You have to get the latest from git. But still, it doesn't solve the issue that the rkmpp encoders don't work. |
Yeah I did enable it
Updated to most recent and it seems to not be detecting input. Not sure I need to continue to troubleshoot on this thread though. |
|
Hello all, just an update or not. So is it the usual thing or this is something new. (I am a noob here). |
|
1080p60 with full color, nto even YUV used. The patch is being pushed to rk linuxvideo |
Thank you for your tremendous efforts!!!
I'm trying to build a re-streaming platform out of orange pi 5 plus with 24.04 Ubuntu from you. But there is a bummber issue like OBS and other apps cannot open HDMI in via v4l2 propertly due to driver IOCTL missing.
So i found another fork where it seems to me all fixed and hdmirx driver patched for rockchip and v4l2 compliance runs with no errors.
https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/issues/4#note_152133
The core issue which crashes OBS is lack of VIDIOC_G_PARM ioctl.
Could you please merge those chages with you build?
Grabbing HDMI IN with gstreamer or ffmpeg is working but i'm getting 30 fps max at 4K resolution and cannot change it no matter what setting i'm trying to apply. I need to get 2K in 60 FPS. I think this orange pi is capable of doing it with no effors (spec says 4K 60fps but nobody can get it from HDMI IN so far). It would be great to just plug and play HDMI IN via V4L2 directly to OBS and check what fps will i get then.
Thank you!
The text was updated successfully, but these errors were encountered: