Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It would be nice to be able to add DTS support. #72

Open
Zibri opened this issue Sep 8, 2022 · 65 comments
Open

It would be nice to be able to add DTS support. #72

Zibri opened this issue Sep 8, 2022 · 65 comments

Comments

@Zibri
Copy link

Zibri commented Sep 8, 2022

LG OLED CX and later TVs do not support DTS anymore.

It would be nice to add it back.

@alvgalrus
Copy link

alvgalrus commented Sep 25, 2022

I was just thinking about this. If it were just a software issue, and B9/C9 and BX/CX are really similar TVs, maybe we could toggle or add support...

@aabytt
Copy link

aabytt commented Sep 30, 2022

CX/C1 is physically able to play DTS track of mkv file. So I guess this is a software issue.
Requires a furtner research on how to properly add dts support.
https://www.youtube.com/watch?v=UV0gSDck9Ew

@Zibri
Copy link
Author

Zibri commented Sep 30, 2022

it is indeed a software issue only.

@alvgalrus
Copy link

CX/C1 is physically able to play DTS track of mkv file. So I guess this is a software issue. Requires a furtner research on how to properly add dts support. https://www.youtube.com/watch?v=UV0gSDck9Ew

Do you mind sharing the steps to try that with a BX?

@pbatard
Copy link

pbatard commented Dec 8, 2022

EDIT: If you don't want to scroll through the whole thread, you can always access the latest release of dts_restore by clicking this button: Download Stats

INITIAL DISCLAIMER

Playing with root and system libraries or partitions is NEVER without a risk, and I make no guarantee whatsoever that, whereas what I am describing below does appear to works fine for my specific environment, you are not going to end up bricking your TV.

THE FOLLOWING IS THEREFORE PROVIDED "AS IS", WITHOUT GUARANTEE OF ANY KIND, AND THE ENTIRE RESPONSIBILITY OF USING THE FOLLOWING DATA, FOR ANY PURPOSE, IS ENTIRELY WITH YOU

In other words: If you don't understand what is being described below, you REALLY shouldn't play with it, because this is pretty much only a proof of concept that will NOT achieve what you expect. Instead you should wait until a safer solution, that has both been validated properly and is easier to install, comes along as what I am providinging below is mostly intended for developers or experienced users.

YOU HAVE BEEN WARNED!

Reenabling DTS on LG TVs

DTS can most certainly be reenabled.

As a matter of fact, I can give you limited DTS support right now, if you have root access:

dts_install.tar.gz

Just extract the content of the above archive to /home/root/ (you should end up with a /home/root/dts_install.sh and a /home/root/gst/ directory), run dts_install.sh and you should now be able to play mkv's with DTS tracks...

However, this comes with the following major caveats:

  1. DTS output is limited to 2 channels WITH NO DOWNMIX, so you can say goodbye to dialog or surround if you have anything that uses center or rear channels (I told you it was limited DTS support).
  2. No DTS passthrough.
  3. You're going to lose the ability to play mkv's and possibly other video files at 2x speed, regardless of whether they include a DTS track or not.
  4. I have only tested this on OLED CX, so YMMV.

In short, this mostly provides a demonstration of how DTS could be restored, for people who might be interested to take it further...

How it works

Basically, we replace all the GStreamer libraries that LG has deliberately nerfed with "unnerfed" versions (and if you are interested in recompiling your own version, rather than use the ones I provide, I'm describing how to do that below). Especially, we:

  • Restore DTS demuxing in the Matroska parsing library (libgstmatroska.so)
  • Restore DTS parsing in the global audio parsing library (libgstaudioparsers.so)
  • Restore DTS decoding in the ffmpeg based global software decoding library (libgstlibav.so)

Then we refresh the GStreamer registry (which is permanent on LG hardware and is not refreshed unless you specifically ask for it) and make sure avdec_dca (the libgstlibav.so DTS decoder) has a high enough priority in /etc/gst/gstcool.conf, the global GStreamer configuration file used by LG.

What about the limitations?

As far as I can tell, the first 2 limitations (2.0 audio with no downmix, and no passthrough) is due to LG having shoved their feature-complete audiosink (multichannel with passthrough) behind their proprietary audio decoder library. So, if you don't go through these proprietary decoders, you're limited to the default 2.0 audio. Even with stock unaltered firmware, you can actually observe the same issue (only the front channels of 5.1 or 7.1 audio being output) if you play something like a mkv with 5.1 or 7.1 FLAC audio, on account that this kind of configuration never reaches the more capable multichannel audio sink, most likely because this is handled by the non-proprietary FLAC decoder. As a matter of fact, LG does some weird gymnastic in hiding some of its A/V processing chain behind a proxy (decproxy) and fake decoders(fakedec). For more on this see https://github.com/justinjoy/gst-plugins-cool.

Note that I have also tried the dtsdec from gst-plugins-bad instead of libav (which comes with its own whole set of challenges... there's a good reason the GStreamer folks have placed it in the "bad" plugins repository), and the outcome about 2.0 audio without passthrough being the only option available appears to be the same.

So, this may very well mean that the best we can ever hope to achieve using the "insert our own DTS decoder here" approach is stereo downmix... unless we start hacking at the proprietary libgstlxaudiodec.so/libgstlxaudiosink.so libraries...

However, there still exists another approach, that should allow us to achieve full multichannel DTS, which is to write our own DTS → LPCM converter (basically audio/x-private1-dts/audio/x-private-ts-lpcm), since the latter is something that the LG processing chain should happily output as 5.1 or 7.1 on account that it's one of the supported audio formats for BluRay. This latter approach may require quite a bit more involvement than simply re-enabling DTS in the libraries where LG forcefully removed it however...

The other limitation, about loss of 2x speed playback when overriding the libraries, is most likely due to the Matroska parser override. But it doesn't worry me too much at this stage, since LG should be providing us with everything we need to recompile a libgstmatroska.so similar to theirs, where we can just re-enable DTS (but please bear in mind that this will only apply to DTS parsing, not DTS decoding).

Wait, what? LG is going to give us the source for some of the libraries they nerfed?

[EDIT 2023.01.02 - As promised, LG have now released their modified GStreamer sources here and if you are interested in using them to recompile your replacement binaries, you may also wants to take a look here.]

Yes, that is exactly right.

You see, what a lot of people don't appreciate (including major companies like LG 😄), is that the (L)GPL has actually a very well defined intent, from which the full legalese of the license is actually derived:

The (L)GPL is intended to ensure that users of software that contains (L)GPL-derived binaries can replace any of these (L)GPL-derived binaries with either an identical or modified version, that they compiled themselves.
This is meant to be accomplished by ensuing that the source code, that was used to produce these (L)PGL-derived binaries, is being published by the purveyor of the software.

This is very different from MIT or BSD, where you can take the code, modify it, and never publish these modifications. With LGPL, if you make any modification of LGPL derived source, be it to add or remove a single line, and publish a binary, you MUST publish that source (and that is regardless of whether the user can actually replace the binary or not, i.e. it still applies if you sign or encrypt your firmware or prevent root access altogether).

Well, it so happens that outside of the libgstlx... specific ones, the GStreamer libraries that LG uses are derived from LGPL v2.x code, including libgstmatroska.so, libgstaudioparsers.so and libgstlibav.so, and LG has, so far, "forgotten" to publish the sources for these libraries, even though they clearly altered them to remove DTS support (Both the LGPL licensing and the alteration are very easy to demonstrate by looking at the binaries and/or disassembling them with something like Ghidra).

Which basically means that, in terms of (most of) their GStreamer libraries, LG are currently in breach of the LGPL license.

And it also happens that I have contacted their Open Source inquiry department (at https://opensource.lge.com/inquiry) to signal this legal issue to them, and they have replied that plan to publish the relevant sources (at least for CX firmware 04.40.20 which is the one I specifically asked for, even though, technically, they are supposed to publish their changes for each firmware they publish) within the next 2 weeks.

Which means that we can most likely sort the 2x playback speed issue once we compile our libraries with whatever changes LG applied and with DTS support.

And that, my friends, is why you really want developers to use (L)GPL as their license, so that, as a user, you can't end up being locked out of running your own version, and "unnerf" whatever a manufacturer decides you can or can not do...

Of course, I'll keep you posted on the publishing of the LG GStreamer sources.

What if I want to compile the libraries myself (rather than trust yours)?

Sure. You'll need a recent Linux machine with the WebOSBrew SDK installed. Then:

. /opt/webos-sdk-x86_64/1.0.g/environment-setup-armv7a-neon-webos-linux-gnueabi
git clone git://anongit.freedesktop.org/git/gstreamer/gstreamer
cd gstreamer
git checkout tags/1.14.0 -b lg
git submodule update
autoreconf -i
./autogen.sh --noconfigure
wget https://www.gentoofan.org/gentoo/misc/gst/gstreamer-1.16.2-make43.patch
patch -p1 < gstreamer-1.16.2-make43.patch
./configure --disable-gtk-doc --disable-introspection --enable-shared --disable-orc --host=arm-webos-linux-gnueabi --with-sysroot="$SDKTARGETSYSROOT" --prefix="$SDKTARGETSYSROOT"
make install

cd ..
git clone git://anongit.freedesktop.org/gstreamer/gst-plugins-good
cd gst-plugins-good
git checkout tags/1.14.0 -b lg
./autogen.sh --noconfigure
wget https://www.gentoofan.org/gentoo/misc/gst/gst-plugins-good-1.16.2-make43.patch
patch -p1 < gst-plugins-good-1.16.2-make43.patch
./configure --disable-gtk-doc --enable-shared --disable-orc --host=arm-webos-linux-gnueabi --with-sysroot="$SDKTARGETSYSROOT" --prefix="$SDKTARGETSYSROOT" CPPFLAGS="$CPPFLAGS -I$SDKTARGETSYSROOT/include"
make

cd ..
git clone git://anongit.freedesktop.org/gstreamer/gst-libav
cd gst-libav/
git checkout tags/1.14.0 -b lg
./autogen.sh --noconfigure
./configure --enable-shared --with-libav-extra-configure="--disable-everything --enable-decoder=aac --enable-decoder=dca --enable-decoder=ac3 --enable-decoder=alac --enable-decoder=amrnb --enable-decoder=amrwb --enable-decoder=eac3 --enable-decoder=flac --enable-decoder=mp3 --enable-decoder=mulaw --enable-decoder=wmapro --enable-decoder=wmav1 --enable-decoder=wmav2 --enable-decoder=wmavoice --enable-decoder=h264" --host=arm-webos-linux-gnueabi --with-sysroot="$SDKTARGETSYSROOT" --prefix="$SDKTARGETSYSROOT" CPPFLAGS="$CPPFLAGS -I$SDKTARGETSYSROOT/include"
make

At the end of this, you should find the libgstmatroska.so and libgstaudioparsers.so in gst-plugins-good/ and libgstlibav.so in gst-libav/.

Additional notes

  • If you know your bash shell, the dts_install.sh script should be pretty explicit. We just mount an override folder for /usr/lib/gstreamer-1.0/ so that we can place our libraries there and we also override /etc/gst/gstcool.conf to set the priority of avdec_dca. Note that you may want to set GST_UPDATE_REGISTRY=0 in the script after you have ran it once, so that you don't update the registry partition every time you run the script, on account that registry changes are preserved between reboots.
  • Outside of the registry update (which is benign even if avdec_dca is no longer provided from libgstlibav.so) the changes are designed not to be permanent. This means that, if anything happens, a reboot should restore everything back to normal. Be mindful that if you have Quick Start+ enabled, because the TV doesn't actually power off, changes may still persist until you perform a formal reboot or unplug the TV altogether.
  • Again, I have only tested it on an OLED CX model, so, whereas the libraries and script are built in a manner that I believe should also work for non CX users, I really can't promise anything.
  • If you are interested in seeing what's going on with the GStreamer processing, and especially how parsers and codecs are invoked, you probably want to change DEBUG_MODE=disable to DEBUG_MODE=enable in /etc/gst/gstcool.conf (or /tmp/gstcool.conf since these are the same file). Then you will get some very verbose /tmp/gst.log.# files you can examine.

@Zibri
Copy link
Author

Zibri commented Dec 9, 2022

grat post! @pbatard ! Can you contact me? I think we can do a lot together.

@aabytt
Copy link

aabytt commented Dec 9, 2022

@pbatard great! I could play dts on CX by swapping libgstmatroska.so and libgstlibav.so with C9 libs but it worked only with gst-play. So glad you made it working.

@pbatard
Copy link

pbatard commented Dec 10, 2022

Here's a v2, that downmixes multichannel DTS to stereo and is therefore a lot more usable:

dts_install_v2.tar.gz

Especially, if you don't have an external decoder but just use the internal TV speakers, then the above is pretty much all you need to restore DTS playback.

Installation process is the same as before: Extract all the content to /home/root/ and run dts_install.sh. Note that you will need to manually run the script at each boot, since it's designed not to be permanent, and you probably want to change GST_UPDATE_REGISTRY=1 to GST_UPDATE_REGISTRY=0 after you ran the script at least once, as the registry is preserved between reboots.

If someone wants to write a DTS Enabler webos app, that uses this content to enable DTS automatically (and in a more friendly manner) feel free to go ahead. Obviously, that app will require root access. Ideally, since I pulled the downmix coefficients out of thin air, we'd probably want to have that potential webos app handle the downmix values and feed them to libgstlibav.so through a config file, so that users can modify them to their linking...

Notes:

  • If you still get This video does not support audio on first attempt, just close the video file and try again. Sometimes the LG player seems to be a little slow on realising that it can actually play DTS back again... 😉
  • This version does away with libgstaudioparsers.so altogether, since libgstlibav.so can include a DTS parser if you add --enable-parser=dca to the --with-libav-extra-configure= options of gst-libav (see previous entry).
  • Since I'm not LG, and I do publish my changes, here's the exact patch I applied to gst-libs/ext/libav/ (which is a git submodule that needs to be instantiated before you can patch it) to force Stereo downmix: Force-stereo-downmix-for-LG.patch.gz
  • If you have a DTS track that uses floating point rather than integer, the downmix will probably not work, since I only bothered to override fixed point. My understanding is that it should be uncommon enough so as not to be an issue.
  • Again, because we override libgstmatroska.so with a non LG version, you will lose 2x playback for all mkv files (and not just those that contain a DTS track).
  • It's still possible that we just haven't found the proper sink to use to get actual multichannel and/or passthrough and that there's some other audio sink besides the audiosink that's referenced in /etc/gst/gstcool.conf. Or possibly, there's something that need to be done (preliminary conf?) to have audiosink properly handle multichannel. If that's the case, that would avoid having to go through the whole shebang of writing our own DTS -> BluRay LPCM converter, which is likely to be a very lengthy process if it happens. So if you are interested in restoring DTS and can find your way around GStreamer, feel free to poke around.

@pbatard
Copy link

pbatard commented Dec 11, 2022

Just going to add that it appears that floating point DTS tracks appear to be more common than I thought, and these currently won't work, on account that the DTS codec will be trying to output audio as raw 32-bit floating point, and the default LG audio sink does not accept that. So you'll get a This file cannot be played error if the DTS track happens to be a floating point one.

It'll probably require quite a bit of time to add support for floating point DTS...

@pbatard
Copy link

pbatard commented Dec 12, 2022

Here's v3, with the following changes:

  • Fix playback for regular DTS files (the ones that were converted to floating point)
  • Fix rear right channels being downmixed to front left instead of front right
  • Enable the reading of the stereo downmix coefficient from /etc/gst/gstcool.conf
  • Only override the 2 libraries files we need instead of overriding the whole library directory.

dts_install_v3.tar.gz

Basically this version should play any DTS track (as long as it's embedded into a .mkv — There's no support for anything but .mkv) and what's more, if you're not happy with my stereo downmix values, you can set your own by editing the [downmix] section of /etc/gst/gstcool.conf.

From this, it should be possible (by somebody else, as I am not planning to do that myself) to create a webos app, that handles the script overrides and also allows the user to set their downmix parameters by updating /etc/gst/gstcool.conf.

As far as non multichannel DTS is concerned, this should accomplish pretty much all we need, bearing in mind that, once LG releases their libgstmatroska.so source, I am hopeful that we'll be able to fix the lost of 2x playback for all .mkv files.

I'm still not too sure about multichannel, as we have at least 3 vectors of attacks there:

  1. Figure out if there is a trick to enable multichannel and passthrough for the default LG audiosink, so that we can just feed the mulltichannel output from the decoder and have that "just work".
  2. Write our own supplement to libgstlxaudiodec.so, that mimics what the former does for ac3, but for DTS. Looking at the underlying liblxa.so, which is invoked by libgstlxaudiodec.so, I can definitely see some support for DTS there, so it's possible that duplicating what libgstlxaudiodec.so does might not require that much code, if it just feeds most of the stream to liblxa.so.
  3. Write our own DTS → BluRay LPCM converter.

As opposed to floating point DTS (which was actually very simple, since the issue was not that there exists "floating point DTS tracks", as I erroneously assumed, but that the libav decoder for regular DTS was set to output floating point PCM by default, and there was just one flag to toggle to change that), this is going to require a lot of time to figure out...

Finally, the source for the modified libgstlibav.so can be found at https://github.com/pbatard/gst-libav.

@pbatard
Copy link

pbatard commented Jan 3, 2023

And here we go, as a new year's present, here's a working permanent DTS restoration package for LG OLED CX.

As LG promised, they finally released the sources for GStreamer (as part of the webOS 5.0 JO_2.0_1.tar.gz archive available here, which, as I suspected, allowed me to recompile a version of libgstmatroska.so that supports 2x playback.

The nice surprise, which I wasn't really expecting, is that 2x playback also works for DTS (as long as you don't play 4k videos, but that's a global limitation that applies to AC3/AAC too).

With this sorted out, I can therefore enable a more permanent means of restoring DTS support, since there shouldn't be any loss of functionality for non DTS media.

Download

Download Stats

You can download the new archive from HERE and basically the new DTS restore process is as simple as:

  • Open a root shell to your TV (e.g. using ssh)*
  • Download, extract and run the installer by issuing:
    cd /home/root
    wget https://github.com/lgstreamer/dts_restore/releases/download/1.0/dts_restore_1.0.tgz
    tar -xzvf dts_restore_1.0.tgz
    ./dts_install.sh
    

That's it.

An uninstaller (dts_uninstall.sh) is also provided if needed, but all it does, really, since it's all that's needed, is delete the /var/lib/webosbrew/init.d/restore_dts that was created by the installer.

Oh, and this new version does not alter any of the original firmware files, such as the GStreamer registry (which we regenerate in /tmp/ on each boot and use an overlay for) and minimises RAM usage by not duplicating unnecessary content to /tmp/.

Sources

I have now moved all the sources that were used to create the content under https://github.com/orgs/lgstreamer/repositories. Especially, you will find complete details on how you can recompile your own libgstmatroska.so as well as libgstlibav.so from the sources that were released by LG.

I'm still not completely sure why, since we're using the same source and should also be using the same configure options, our libgstlibav.so is so much bigger than LG's, but I suspect this has to do with being able to locate the shared ffmpeg library in our sysroot (using LG's sysroot was too much of a hassle), and, outside of the extra space taken, I don't really see much of an issue arising from this. Plus, in case LG also nerfed their ffmpeg, this should keep us safe from surprises...

Limitations (besides root)

  • Only mkv playback supported (no .mp4, no .dts).
  • Only PCM stereo downmix supported (no multichannel, no passthrough).
  • You may still get an initial "This video does not support audio" message on first play of a .mkv with a DTS track, in which case you should just stop playback and try again (not too sure what can be done to get rid of this one)
  • The libraries and scripts have been designed for LG OLED CX models.
    While there is a good chance that, if your TV is not too dissimilar to OLED CXs, you might also be able to restore DTS support there, any issue that arises from not running this software on an OLED CX will be entirely yours to troubleshoot and support.

Further information

See https://github.com/lgstreamer/dts_restore.

Oh and moving it to its own GitHub repository allows to get some nice Download stats (see above).

@aabytt
Copy link

aabytt commented Jan 3, 2023

@pbatard great job!
FYI v3 confirmed working for C1 (2021), C2 (2022), UN7400 (2020), NANO796 (2020)

@davix004
Copy link

davix004 commented Jan 18, 2023

And here we go, as a new year's present, here's a working permanent DTS restoration package for LG OLED CX.

As LG promised, they finally released the sources for GStreamer (as part of the webOS 5.0 JO_2.0_1.tar.gz archive available here, which, as I suspected, allowed me to recompile a version of libgstmatroska.so that supports 2x playback.

The nice surprise, which I wasn't really expecting, is that 2x playback also works for DTS (as long as you don't play 4k videos, but that's a global limitation that applies to AC3/AAC too).

With this sorted out, I can therefore enable a more permanent means of restoring DTS support, since there shouldn't be any loss of functionality for non DTS media.

Download

Download Stats

You can download the new archive from HERE and basically the new DTS restore process is as simple as:

  • Open a root shell to your TV (e.g. using ssh)*
  • Download, extract and run the installer by issuing:
    cd /home/root
    wget https://github.com/lgstreamer/dts_restore/releases/download/1.0/dts_restore_1.0.tgz
    tar -xzvf dts_restore_1.0.tgz
    ./dts_install.sh
    

That's it.

An uninstaller (dts_uninstall.sh) is also provided if needed, but all it does, really, since it's all that's needed, is delete the /var/lib/webosbrew/init.d/restore_dts that was created by the installer.

Oh, and this new version does not alter any of the original firmware files, such as the GStreamer registry (which we regenerate in /tmp/ on each boot and use an overlay for) and minimises RAM usage by not duplicating unnecessary content to /tmp/.

Sources

I have now moved all the sources that were used to create the content under https://github.com/orgs/lgstreamer/repositories. Especially, you will find complete details on how you can recompile your own libgstmatroska.so as well as libgstlibav.so from the sources that were released by LG.

I'm still not completely sure why, since we're using the same source and should also be using the same configure options, our libgstlibav.so is so much bigger than LG's, but I suspect this has to do with being able to locate the shared ffmpeg library in our sysroot (using LG's sysroot was too much of a hassle), and, outside of the extra space taken, I don't really see much of an issue arising from this. Plus, in case LG also nerfed their ffmpeg, this should keep us safe from surprises...

Limitations (besides root)

  • Only mkv playback supported (no .mp4, no .dts).
  • Only PCM stereo downmix supported (no multichannel, no passthrough).
  • You may still get an initial "This video does not support audio" message on first play of a .mkv with a DTS track, in which case you should just stop playback and try again (not too sure what can be done to get rid of this one)
  • The libraries and scripts have been designed for LG OLED CX models.
    While there is a good chance that, if your TV is not too dissimilar to OLED CXs, you might also be able to restore DTS support there, any issue that arises from not running this software on an OLED CX will be entirely yours to troubleshoot and support.

Further information

See https://github.com/lgstreamer/dts_restore.

Oh and moving it to its own GitHub repository allows to get some nice Download stats (see above).

Thanks @pbatard!
I can confirm that it's working on my tv too, the 43NANO766QA 2022 model. I have the "This video does not support audio" message but afer stop and replay the file, it works well.
I opened a root shell using telnet, because when using ssh a "wget: error getting response: Connection reset by peer" message appears, but all commands passed successfully using telnet.

@Zibri
Copy link
Author

Zibri commented Jan 18, 2023

• Only PCM stereo downmix

why is that? isn't it possible to have 5+1?

P.S.
@pbatard great work, I also have the CX, great TV!
What firmware are you on at the moment?

@pbatard
Copy link

pbatard commented Jan 18, 2023

why is that?

Because the default gstreamer output device (audio sink) that is being used when not decoding audio using the closed source proprietary libraries (that call the hardware directly) only outputs stereo.

In short, that device says it accepts multichannel, but LG nerfed it so that only front left and front right are being output and everything else is discarded. And I have not discovered any "trick" to somehow toggle multichannel for that device. As a matter of fact, even when using the vanilla LG player and libraries, multichannel FLAC is not being handled properly and suffers the same issue of only FL + FR being output, because it seems FLAC is also being sent to the default limited gstreamer audio output sink.

I sure wish LG had bothered giving us a default audio device that handles multichannel and passthrough, as one would find on any regular Linux distro, but I suspect they designed the default audio output for apps audio output where only stereo was needed (note that movie playback from apps is still being handled by the video playback stack, so it's not limited by this), and they cut corners in terms of the development time they'd have to invest otherwise.

isn't it possible to have 5+1?

As I believe I explained above, there are 3 possible ways we may be able to get 5.1/7.1 which basically are:

  1. Spend an awful lot of time reverse engineering the existing closed source proprietary that handle audio decoding, and try to figure out a way to patch them for DTS output (since there seems to be hardware DTS decoding calls further down). Again, I have to stress out for anybody who believes that reverse engineering and patching existing binaries is easy that this is probably at least a 1 month FULL TIME job (i.e. 10 hours a day doing nothing else) without any clue as to whether there will be a massive roadblock that prevents the whole thing (such as the hardware calls having been left in in some libraries, but the hardware decoder having been removed altogether — Oh, and these hardware calls may be super specific to one model only, meaning you may only be able to satisfy CX owners and nobody else). In short, the cost/benefit ratio, if it's to help a handful owners, is just too high to be worth exploring.
  2. Spend a lot of time looking at the pulse audio and gstreamer audio output sink source, for which we should have most of the sources, to see if we could sort out a multichannel version. Now, because the audio output sink obviously needs to make hardware calls at some point, and we obviously don't have the technical specs from LG on how their proprietary audio hardware works, there too a lot of reverse engineering is expected to be needed, with potentially the same caveats as we have above, with whatever we do applying only to a specific model and nothing else.
  3. Spend some time crafting a gstreamer plugin that does DTS -> PCM conversion (e.g. BD PCM), so that the gstreamer native playback chain (which should be smart enough to do this automatically) will send the multichannel PCM output to LG's proprietary audio decoder library, and therefore to a working multichannel audio output. Obviously you won't get passthrough, whereas you may be able to get it with the other methods, but since it boils down to writing a standard gstreamer plugin, without having to reverse engineer or know anything about the underlying hardware, this is clearly the most attractive approach, since, unlike the other ones, one can estimate its difficulty and therefore the time it's likely to take, and it's also something you can develop without access to the hardware (since that plugin can apply to any Linux system that has gstreamer installed, and not just one with LG hardware). As far as I can tell, since we do have DTS decoding and PCM encoding from the gstreamer libav plugin, what's needed is to write a gstreamer manager plugin, that places the DTS decoding to multichannel PCM encoding into a black box that can then be presented as a DTS → multichannel PCM plugin. Of course, unless you're an experienced gstreamer developer (which I am not — I am still discovering the whole thing), whereas this looks relatively straightforward on paper, actually going through coding such a plugin is likely to be a lot more work and therefore require a lot more time, than people expect.

In short, considering that I'm planning to tackle approach 3, I'm hoping that, if I'm lucky, crafting such a plugin may only take me 1 week full time work (and will work, which is still not guaranteed), but as you can imagine, I can't exactly be taking 1 full week away from other activities just like this, so it'll probably be a month or two at best before I can spend enough time on looking into that. And that's hoping that this whole writing a gstreamer manager plugin doesn't turn out to be much more complex than I anaticipate... which, as with any software project, is almost always invariably the case.

Now of course, if someone wants to spend their own time looking at the pulse audio and gstreamer source to investigate approach 2, or even more time reverse engineering LG's proprietary libraries to poke 1, feel free to go ahead!

What firmware are you on at the moment?

As mentioned here (since this is what I used to root my TV) I'm on 04.40.20.

@furiseto
Copy link

I don't know if it helps somewhat but for some reason my LG CX could play DTS audio (with full 7.1 audio) through PLEX native app and via Direct Play too. But it only happens for one movie and it's Pan's Labyrinth (2006), with original Spanish language. I'm sorry if it's against the rule but I torrented it and the file name is Pans.Labyrinth.2006.SPANISH.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.7.1-SWTYBLZ.

So far I only notice that movie. All the other movies with DTS are always mute when playing dts via PLEX, except that one. Maybe because it is in Spanish and somehow LG lets it pass through? I'm just a casual viewer so maybe you can check it out to see if there's any trick you could discover.

@throwaway96
Copy link
Contributor

Pans.Labyrinth.2006.SPANISH.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.7.1-SWTYBLZ

Have you confirmed that the audio in that file is actually DTS?

@pbatard
Copy link

pbatard commented Jan 19, 2023

Have you confirmed that the audio in that file is actually DTS?

Or the the PLEX application isn't doing transcoding on the fly, which, AFAIK, is what PLEX server does (i.e. it queries the target device to find out what kind of multichannel audio it can output and, if needed, such as if a device cannot output DTS, transcodes DTS to PCM or EAC when sending it to the device).

The DTS decoding on CX models is pretty black and white, and LG's explicitly removed DTS demuxing from the Matroska (mkv) gstreamer plugin, so there's absolutely no way you can get DTS playback with the vanilla gstreamer plugins when playing a .mkv (which your file appears to be), even if you actually have plugins and libraries further downstream that can decode DTS.

All the other movies with DTS are always mute when playing dts via PLEX, except that one.

My guess is that this movie has an additional EAC3/DD track besides the DTS one, and your other movies don't. The LG player is designed to automatically fall back to an audio track it can play if the first audio track it tries is not supported, and the behaviour you observe with some movies (with only a single DTS track) being silent while others (with a DTS track and an EAC3 track) producing audio is exactly what one would expect.

@furiseto
Copy link

furiseto commented Jan 19, 2023

Pans.Labyrinth.2006.SPANISH.2160p.BluRay.x265.10bit.HDR.DTS-HD.MA.7.1-SWTYBLZ

Have you confirmed that the audio in that file is actually DTS?

Yup, I checked and confirmed it's indeed DTS-MA. I also tried to play it through MPC-HC (PC) and it's mute with the "Audio Not Supported" message. But through PLEX it can play all 7.1 channels.

My guess is that this movie has an additional EAC3/DD track besides the DTS one, and your other movies don't. The LG player is designed to automatically fall back to an audio track it can play if the first audio track it tries is not supported, and the behaviour you observe with some movies (with only a single DTS track) being silent while others (with a DTS track and an EAC3 track) producing audio is exactly what one would expect.

The others are also bluray movies and they also have multi-tracks with DTS and EAC3 just like this one. And they can only play audio when I choose EAC3, no sound from DTS.
That is why I was confused as heck and wonder why this specific movie, or rather, this specific file, is allowed to pass through DTS

Also here is a vid showing Plex's info screen I just recorded: https://youtu.be/9awka3CrlGs

P/s: In case anyone wondering, the soundbar you see in this vid is Samsung Q950A that supports all the sound codec and also eARC supported. The setting for the TV and soundbar is Pass-through and all HDMI bitstream.

@pbatard
Copy link

pbatard commented Jan 19, 2023

the soundbar you see in this vid is Samsung Q950A that supports all the sound codec

Hold on. What happens without the sound bar?

Pass through is a very different beast, so I could see how a track that is wrongly advertised as EAC3 but is actually DTS could end up being passed through by the PLEX app (though I have to say, this is one of the avenue I explored with the internal player as well, i.e. modify the gstreamer demuxer to set the audio type of the DTS data to x-audio/eac3 instead of x-audio/dts, but that didn't work for the LG gstreamer player). So it would be interesting to know if you still get sound output without the sound bar. And I'd be really interested to know what your soundbar see the audio as (if you had an amp, that info would usually be displayed on the front panel, but for a sound bar, I'm not sure if you can get it).

I guess if it turns out that a PLEX player that misreports DTS tracks as EAC can get the TV to just pass through the data, then we might as well try to build a PLEX app that always forces DTS as EAC always, as this should solve the multichannel output for people who have an external DTS decoding device.

@furiseto
Copy link

furiseto commented Jan 19, 2023

Hold on. What happens without the sound bar?

I just checked. Without the soundbar, and with Internal TV Speaker, the DTS audio still comes out just fine. The Plex Info still shows that it was streaming dca and 7.1.

And I'd be really interested to know what your soundbar see the audio as (if you had an amp, that info would usually be displayed on the front panel, but for a sound bar, I'm not sure if you can get it).

Yeah, unfortunately it doesn't show me what codec it is receiving. The soundbar's display only shows Dolby Atmos text whenever it detects one.

edit: remove link of the file

@pbatard
Copy link

pbatard commented Jan 19, 2023

Please don't post links to copyrighted content, even obfuscated. I would gather that if someone really wants to get their hands on the file you mention, they can do so without needing any other hints but the file name.

@furiseto
Copy link

Got it. Thanks for telling me.

@aabytt
Copy link

aabytt commented Jan 19, 2023

As far as I know plex server transcodes DTS but just leaves original label in app.
@furiseto if you are rooted connect with ssh/telnet and run ls-monitor -f com.webos.media
then launch a movie in plex and check the logs for sourceInfo event. It will show the tracks with the actual decoder info
Looks like this:

{"sourceInfo":{"numPrograms":1,"container":"mkv","seekable":true,"trickable":true,"programInfo":[{"duration":1295552,"numAudioTracks":4,"audioTrackInfo":[{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"en","codec":"ac3","sampleRate":48,"channels":6,"audioType":0} ],"numVideoTracks":1,"videoTrackInfo":[{"angleNumber":0,"level":3.1,"width":1280,"height":720,"codec":"H264","profile":"high","frameRate":23.976,"progressive":true} ],"numSubtitleTracks":2,"subtitleTrackInfo":[{"language":"ru","type":"text"},{"language":"en","type":"text"} ]} ],"mediaId":"_MQUBT01X5MiTj4"}}

@furiseto
Copy link

As far as I know plex server transcodes DTS but just leaves original label in app. @furiseto if you are rooted connect with ssh/telnet and run ls-monitor -f com.webos.media then launch a movie in plex and check the logs for sourceInfo event. It will show the tracks with the actual decoder info Looks like this:

{"sourceInfo":{"numPrograms":1,"container":"mkv","seekable":true,"trickable":true,"programInfo":[{"duration":1295552,"numAudioTracks":4,"audioTrackInfo":[{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"ru","codec":"ac3","sampleRate":48,"channels":2,"audioType":0},{"bitRate":0,"language":"en","codec":"ac3","sampleRate":48,"channels":6,"audioType":0} ],"numVideoTracks":1,"videoTrackInfo":[{"angleNumber":0,"level":3.1,"width":1280,"height":720,"codec":"H264","profile":"high","frameRate":23.976,"progressive":true} ],"numSubtitleTracks":2,"subtitleTrackInfo":[{"language":"ru","type":"text"},{"language":"en","type":"text"} ]} ],"mediaId":"_MQUBT01X5MiTj4"}}

I'm pretty amateur so my CX isn't rooted yet. So far I only used the Dev-manager method to install Homebrew and Youtube-adfree. Will try to look into alternate rooting method. If success then I will do what you said and post result.

But I'm actually confident that there isn't any transcoding process happening. Mainly because my plex server is my PC so I can easily check how much Plex's using resources from my i5 Gen10 cpu. And the absence of loading time also proves it is truly in Direct Play mode.

Still I'm gonna try to root my CX for further info.

@pbatard
Copy link

pbatard commented Jan 19, 2023

I'm actually confident that there isn't any transcoding process happening

Transcoding audio is peanuts. It's transcoding video that is CPU intensive, but for any modern processor, decoding and reencoding DTS to PCM will hardly make a dent in your resource usage.

my plex server is my PC

Then, as far as I know (though I am not that familiar with Plex), there is definitely automatic transcoding happening according to what the client device says it can handle. That's how Plex servers and clients are designed to work.

@aabytt
Copy link

aabytt commented Jan 19, 2023

I'm pretty amateur so my CX isn't rooted yet.

ok, no problem. I've installed plex and checked with my CX on the same video file as you did.
When playing with plex client it shows DTS label (as originially track is) but actually server decodes it into AAC on the fly.

"audioTrackInfo":[{"bitRate":0,"language":"es","codec":"aac","profile":"lc","level":4,"sampleRate":48,"channels":6,"pid":257,"ctag":-1,"audioType":0,"trackId":-1,"immersive":"none","periodStart":0}]

transcoding happening according to what the client device says it can handle

Actually seems Plex is not so smart :)
I’ve looked thru client options, there you can manually tick if your device supports DTS otherwise it will be always transcoded.

@furiseto
Copy link

After many attempts, I can finally root my CX and managed to pull the info:

"audioTrackInfo":[{"bitRate":0,"language":"es","codec":"ac3","sampleRate":48,"channels":6,"pid":-1,"ctag":-1,"audioType":0,"trackId":-1,"immersive":"none","periodStart":0}]

With Aabytt it's aac but with mine it's ac3.

I’ve looked thru client options, there you can manually tick if your device supports DTS otherwise it will be always transcoded.

yes, and I ticked it to DTS support. Plex leaves all the other movies with DTS audio alone, but somehow "secret transcode" Pan's Labyrinth into aac/ac3? Either CX misread the codec or it's just my failed hope all along...

I'm looking for a way to check whether Plex transcodes stuff or not in the Client's side, but I guess now I'm not much enthusiasm like before.

@aabytt
Copy link

aabytt commented Jan 26, 2023

With Aabytt it's aac but with mine it's ac3.

This file has 2 tracks. One is DTS and other is AC3. Plex streams only single selected track. I guess in your case you have AC3 selected and it is just streamed with no transcoding. If you switch to DTS track it will transcode it and you'll see AAC in audioTrackInfo

@alvgalrus
Copy link

alvgalrus commented Jan 29, 2023

@pbatard Thank you for the time you took to research how they removed the codecs!

Wouldn't it be easier to get the relevant files from a B9/C9 and replace them into the BX/CX to allow full functionality? (I mean splitter, decoder, surround and passthrough). After all, both TVs work almost the same.

@agisthos
Copy link

agisthos commented Feb 8, 2023

The act of playing the audio in a video, any audio, allows DTS-restore to work on the next (or subsequent) attempt. Until the TV is restarted that is.

@pbatard
Copy link

pbatard commented Feb 13, 2023

@agisthos, I suspect that you have been trying to install the DTS override from a telnet session, and are running into lgstreamer/dts_restore#2.

Please make sure that you use an ssh session and not a telnet session when trying to run the script, as telnet will not work because it fails to define the required environment variables.

@agisthos
Copy link

agisthos commented Feb 13, 2023

@pbatard sorry I was not clear, I got SSH working, installed your DTS_Restore via SSH and DTS audio now works great. But on the G2 all those 'this audio not supported errors' occur, triggered exactly as described above.

I then did my sisters G1 and no such errors. So either the 2022 models have new compatibility issues, or the initial failed install attempt in Telnet has mucked something up on my G2.

@pbatard
Copy link

pbatard commented Feb 13, 2023

or the initial failed install attempt in Telnet has mucked something up on my G2.

I pretty much designed the script so that it can't, and the person who ran into the telnet issue in lgstreamer/dts_restore#2 had no issue once they ran the install script from ssh after initially doing so through telnet.

And the one-time errors you see are expected (I mentioned them explicitly above, and they are also mentioned in the README). I get them all the time on my CX, and I am not planning to look into those before I start looking into multichannel DTS, so you will have to live with these for now (or, if you can figure out how to remove them on your own, then I'm interested!).

@agisthos
Copy link

@pbatard Is there a log one can view that these errors possibly go into?

@pbatard
Copy link

pbatard commented Feb 13, 2023

There sure is. From my very first post above:

If you are interested in seeing what's going on with the GStreamer processing, and especially how parsers and codecs are invoked, you probably want to change DEBUG_MODE=disable to DEBUG_MODE=enable in /etc/gst/gstcool.conf (or /tmp/gstcool.conf since these are the same file). Then you will get some very verbose /tmp/gst.log.# files you can examine.

@pbatard
Copy link

pbatard commented Feb 13, 2023

Just a quick note that I just released an updated version of DTS Restore here (v1.1).

This is a minor update that makes sure that we use the latest CX sources for the GStreamer plugins (though for the ones we are interested in, I don't believe the code has changed) as well as make sure that we properly detect and report issues with using telnet/not detecting $GST_REGISTRY_1_0. The script was also updated to remove the need to edit the path if not installed in /home/root/, as well as use a symbolic link in /var/lib/webosbrew/init.d/ rather than copying a file there.

@casenjo
Copy link

casenjo commented Mar 20, 2023

Today for some reason I started getting "Audio is not supported" when playing a file with a DTS track in it. I recently rooted my TV but I don't think that should have stopped the ability to pass-through the DTS track to my received. I hadn't experienced this issue before on my LG C1 though and I'm wondering if this could help?

It's an OLED55C1AUB TV with webOS 6.3.2 and GStreamer 1.16.2.

@aabytt
Copy link

aabytt commented Mar 20, 2023

@casenjo wait, you saying that your C1 was passing-throug DTS tracks before? It is hardly possible.

@casenjo
Copy link

casenjo commented Mar 20, 2023

@aabytt I'm not entirely sure, and if you're saying it's hardly possible then I would lean more towards it being a mistake on my end. I have quite a few video files with DTS tracks that I was able to watch just fine with the C1 but maybe my Kodi setup wasn't configured to do DTS passthrough before and now that it's enabled I'm running into the issue.

@aabytt
Copy link

aabytt commented Mar 20, 2023

@casenjo, have you been playing movies with some external device over hdmi? Then I guess pass through is possible. DTS decoding is not supported when playing media with webos itself.

@casenjo
Copy link

casenjo commented Mar 21, 2023

@aabytt no, I was playing movies and shows from Kodi on my home theatre PC connected to the TV which would then pass the audio over to my audio receiver. I never played movies directly from WebOS itself.

@agisthos
Copy link

agisthos commented Mar 31, 2023

On the LG G2 I have done a few firmware updates and DTS restore was working after each one.
But for firmware 03.30.66 I also did a factory reset and afterwards DTS playback was gone. So I am unsure if it was the firmware upgrade or the factory reset that removed it.

But anyway, after regaining root I ran this method again and DTS is back and working!

@agisthos
Copy link

agisthos commented Oct 7, 2023

Just worked then on rooted G2 with latest 3.33.65 firmware.

Best guide to ROOT the TV in the first place is by crashd
https://gist.github.com/throwaway96/e811b0f7cc2a705a5a476a8dfa45e09f

@crack666
Copy link

crack666 commented Nov 6, 2023

Hi, first off: Thanks @pbatard for your awesome work. I'm impressed what some people are able to achieve, hacking devices like TVs etc. I tried your installer on my already rooted LG OLED C22LB, so 2022 model with WebOS 7 version 03.30.45 (haven't updated since root). Got your warning message (LG OLED CX TVs running
webOS 5.x with Gstreamer 1.14.4. However, you are trying run it on a(n)
OLED77C22LB TV with webOS 7.3.0 and GStreamer 1.18.2.) and went for it - works like expected! Nice job!
Hope you find a solution for enabling multi-channeling, then I'd be perfect! Still really impressive how far you came for now.

@ChrSch84
Copy link

ChrSch84 commented Dec 1, 2023

Hello all.

Is there any news about CX DTS -> PCM passtrhough?

@pbatard
Copy link

pbatard commented Dec 1, 2023

Is there any news about CX DTS -> PCM passtrhough?

This is not going to happen. You'd have to spend weeks (or more likely months) of full time work to first reverse engineer the proprietary LG libraries and hardware and then rebuild/patch these libraries so that they can accept DTS passthrough.

Best you might get is multichannel DTS support (in the form of multichannel PCM sent to the decoder), but whereas I was planning to work on this at one stage, I currently don't think I'll ever find the time/motivation to do so.

@crack666
Copy link

crack666 commented Dec 1, 2023

hey @pbatard ever thought about setting up a Patreon/Fundraiser for this?

@pbatard
Copy link

pbatard commented Dec 1, 2023

The problem is not money. The problem is that, at best, this will help a couple thousand people entertain themselves with regards to technology that is quickly becoming obsolete (because fewer and fewer people care about features related to specific TV models that were released years ago) whereas there are plenty of other projects I can currently use my time on, where the effort I spend will ultimately be beneficial to a lot more folks.

As far as I'm concerned, no amount of money can justify spending time on this unless I happen to not have something that I consider more fruitful to do, which is unlikely to happen in the short to medium term...

@ChrSch84
Copy link

ChrSch84 commented Dec 2, 2023

multichannel PCM sent to the decoder

This would be great for 99.9% of people.

But I understand your position about the project. Sad for me and others. But I understand it. Thanks for the work so far.

Now I have to finde a way to add ac3 to all od my dts media.
This will be my fun project.

@Rutge-R
Copy link

Rutge-R commented Mar 25, 2024

@pbatard I used your patch on my C2 and it worked flawless. I'd like to thank you for the time and effort you put into this project.

@djkroko
Copy link

djkroko commented May 18, 2024

I used the DTS patch on my C29 with webOS 8.3 a few days ago. Since then i have a dead Pixel, bright red Pixel and a black vertical Line.

@pbatard
Copy link

pbatard commented May 19, 2024

@djkroko, dead pixel/deal lines are caused by pure hardware issues. They cannot be caused by software. The LG OS (where the DTS component apply, and where these components only apply to the audio processing chain) does not have that kind of premanent direct control of the individual pixels of the display.

A more valid explanation, if you want to point at fingers at things you can never have definite proof of, is that the recent solar flare is what caused your dead pixels, since it has been known to cause hardware failures in electronics all over the world. But the most likely explanation is that, hardware does fail, regardless of what you do with it and regardless of the external conditions, and that you experienced a failure that coincided (but was in no way caused) with your usage of the DTS patch.

At any rate, since you weren't using a CX model, you will have seen this very explicit message, which you accepted:

This installer was designed specifically for LG OLED CX TVs running"
webOS 5.x with Gstreamer 1.14.4. However, you are trying run it on a(n)"
C29 TV with webOS 8.3 and GStreamer <your version>."

While installing this software on an incompatible platform should not
cause irreversible damage, if you choose to proceed, you do acknowledge
that, because you are not using the relevant target system:
1. The software may not work as expected, if at all.
2. You may lose existing features and/or functionality.
3. The entire responsibility for trying this software on an unsupported
   platform lies entirely with you."

Do you wish to proceed? [y/N] 

@febman123
Copy link

febman123 commented Jun 13, 2024

I patched my rooted C1 recently and DTS works flawlessly with all tested MKVs.
Many thanks for your hard work.
Thanks again.

@williamhomyk
Copy link

williamhomyk commented Sep 19, 2024

@pbatard

I used your DTS_Restore patch on my LG CX TV to fix the issue I was having with the LG Plex app where not being able to direct play DTS audio caused high bit rate 4k video to become corrupted. It worked flawlessly and now I can play DTS Audio

https://forums.plex.tv/t/lg-cx-plex-app-video-errors-when-playing-4k-files-that-have-dts-truehd-audio/883648

Any chance you could do a similar fix for TrueHD audio on LG TV's?

@pbatard
Copy link

pbatard commented Sep 19, 2024

TrueHD should be supported natively. At any rate, I am not planning to spend any more time on this. If you or anybody else wants to add new audio functionality, you can find how this was accomplished for DTS in the repos at https://github.com/orgs/lgstreamer/repositories and do something similar (and yes, I'm afraid you will have to invest your own time on this rather than expect someone else to do it for you).

@williamhomyk
Copy link

williamhomyk commented Sep 19, 2024

TrueHD should be supported natively. At any rate, I am not planning to spend any more time on this. If you or anybody else wants to add new audio functionality, you can find how this was accomplished for DTS in the repos at https://github.com/orgs/lgstreamer/repositories and do something similar (and yes, I'm afraid you will have to invest your own time on this rather than expect someone else to do it for you).

LG doesn't handle TrueHD it tries to transcode it down to eac3 and causes this video error with high bitrate 4k Videos
aaa plex video error2

The same video corruption I was getting with dts audio files since it cant direct play the audio it also transcodes the video from MKV < MPEGTS. This problem was fixed for 4k files with DTS audio since now with your patch I can now direct play the audio. Someone please look into seeing if the same patch with DTS can be created with TrueHD

@williamhomyk
Copy link

Please add a fix for Truehd codecs not being recognized

@agisthos
Copy link

Just FYI I have a G2, rooted and with DTS Restore working but have blocked firmware updates for over a year because of the recent root exploit patching. Did not want to risk it.

But I just then updated, first to firmware 4.40.93, then TV updated WebOS to 23, then another firmware update to 13.30.85

Long story short, DTS Restore still works fine. Its now actually better, as previously the first DTS video that was played upon a TV reboot would fail (only the first one) and have to be restarted. Now this does not occur.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

16 participants