JPEG XS Codec Support

[jonathan-catt]

Starting to discuss the options to support JPEG XS as a codec option for Ultragrid. I am working through the licensing terms right now and unless I am miss understanding them the costs are not impossible at the small scales we will be using.

Wondering if JPEG XS support in UG would be possible, as it appears Intel have published their library to support encode and decode. (https://github.com/OpenVisualCloud/SVT-JPEG-XS) I was also going to inquire with IntoPIX for a quote for their SDK as well. This might be considered an alternative to Comprimato J2K support.

[TheSashmo]

I am in full support of this as well. I've been in talks with them as well and maybe there is a way to share the expense.

[alatteri]

1. What about using High-Throughput JPEG 2000. I believe it is fully open source, and I have a tertiary connection to the developer who expressed interest in getting it in UG in a previous conversation. https://jpeg.org/jpeg2000/htj2k.html

2. What is the end goal here?

3. What are the desires for the type of hardware on the encoder and decoder?
   a) does size and cost matter
   b) are you trying to keep the receiver small and cheap (Intel/Asus NUC?). If so, then Nvidia GPU probably not possible on receiver
   c) if receiver can be 1u rack server, more options
   d) we have 1u encoders with Ryzen 7950x and Nvidia GPU. Gives us flexibility. See this: https://youtu.be/w0O2OcA49F0?si=aZzShWjDlsd6PD9A

4. What resolution and color depth are you expecting?

5. What are the bandwidth limitations?

6. Why not just use x265 software encoding and guarantee yourself a 12bit RGB444 stream?

7. Or use NVENC-HEVC and get a 10bit YUV444 stream? Although in previous testing I've been unable to decode NVENC-HEVC in realtime using software decoder. Only GPU decoder has worked for me, but its been probably 2 years since I last tried.

[TheSashmo]

1. If I recall XS vs. 2000 has a 3:1 ratio better compression
2. Lower bitrates at the same quality in a intra frame compression opposed to using a inter frame compressions
3. Supposedly it can work withou hardware acceleration i.e. NUC could work (I saw a demo of RGB compression and it works)
4. My understanding it will go all the way to 4:4:4 RGB (I could be 100% wrong)
5. 100Mbps XS is comparable to 300Mbps J2000
6. Intra frame is better than inter frame
7. Same as 6.

[alatteri]

What are your resolution requirements?

[TheSashmo]

Starting at 1920x1080i then 1920x1080p and then up to 4k and higher, but the main use would be 1920x1080i from my use cases

[alatteri]

at HD/2k resolution, x265 works great on any recent hardware for both encoder and receiver. What do you see as the disadvantages of inter-frame here?
4K x265 is more challenging but can be done.

If there are no bandwidth constraints, I guess intra-frame is better in terms of packet loss resilience, possibly latency also, but I've done no testing.

[TheSashmo]

Sorry for the copy paste:

Intra-frame Compression
Definition: Compresses each frame of video individually, treating each frame as a separate still image.
Method: Only exploits spatial redundancy within a single frame. Techniques like JPEG compression are applied to each frame independently.
Advantages:
High-quality image retention within each frame.
Better suited for editing because each frame is self-contained.
Disadvantages:
Larger file sizes compared to inter-frame compression.
Higher data rates, requiring more storage and bandwidth.
Usage: Commonly used in editing codecs like ProRes, DNxHD, and still image formats like MJPEG.

Inter-frame Compression
Definition: Compresses video by storing only the differences between consecutive frames.
Method: Uses temporal redundancy by referencing previous (and sometimes future) frames to reduce the amount of data stored. It uses keyframes (I-frames) and difference frames (P-frames, B-frames).
Advantages:
Significantly reduces file size and required bandwidth.
More efficient for storage and streaming.
Disadvantages:
Can introduce artifacts, especially during fast motion.
More complex decoding process, which can make it less ideal for editing.
Usage: Used in delivery codecs like H.264, H.265 (HEVC), and VP9, which are commonly found in streaming and broadcasting.

[alatteri]

I understand the technicalities, but how does either affect YOU in this application?

[TheSashmo]

Quality vs bitrate vs network that is being used to deliver the video.

[alatteri]

right so if bitrate is not a concern, I understand. For our application, since most of our endpoints are residential internet, we try to stay around the 40mbps range for all resolutions.

Do you have a latency tolerance too?

[alatteri]

according to this chart from here: https://fastcompression.medium.com/jpeg-xs-modern-visually-lossless-low-latency-lightweight-codec-2f9299407d8e

j2K has a better PSNR than XS.

From several articles, it seems XS was designed for low complexity and latency, but compression efficiency was secondary, so I believe the previous information to be inaccurate and maybe even inverse. 100Mbps XS is comparable to 300Mbps J2000

There are some additional alternative codecs for JPEG** in FFMPEG
https://www.fastcompression.com/products/products.htm

[alatteri]

but those bandwidth requirements would be not ideal for public internet streaming....that would be hundreds of MB/s.

[TheSashmo]

I never said internet. But I agree. 300Mpbs or 100mbps is no bueno over the internet. But much much better for local network.

[alatteri]

what is the use case?

[alatteri]

I think you miss reading that chart. Look at between 2-4 the blue dot alone in that box. You can see its between 2-4 : 1 compression ratio based on PSNR. Its virtually identical.

actually, the J2K psnr goes straight up, since at the 2-3 compression ratio, it is apparently lossless according to the article.
The highest point of JPEG 2000 curve (with infinite PSNR) shows compression ratio of reversible algorithm

[TheSashmo]

Yes thats correct. J2K is apparently lossless and Jpeg XS is what is planned to replace it.

[michaeldsmith]

@TheSashmo I wanted to chime in here, I think you may not yet have the background to understand the information you are reading and reporting back on this thread. I would encourage you to keep reading and learning about compression technology, it is interesting stuff.

JPEG2000 has better quality per bit than JPEG-XS. The downside of the original JPEG2000 (standardized more than 20 years ago) is that is computationally intensive (thus requiring GPU implementations for realtime performance or many CPU cores). JPEG-XS was designed to be a low-complexity low-memory/low-latency codec but is significantly worse from a compression efficiency (quality per bit) point of view than JPEG2000, so that means, for example, if you like the quality of JPEG2000 at 100Mbs data rate, you would need say 140Mbs data rate when using JPEG-XS to achieve the same quality with the same imagery as JPEG2000 operating at 100Mbs.

High-Throughput JPEG2000 (HTJ2K) is an enhancement to JPEG2000 that was published in 2019, it replaces the slow entropy coder in JPEG2000 with a fast one, but keeps everything else from JPEG2000. The compression efficiency is worse than JPEG2000, but not as
bad as JPEG-XS. There are different configurations of HTJ2K that reduce the memory/latency further at further penalties to compression efficiency. I co-authored the attached paper studying the quality and latency tradeoffs between JPEG2000, JPEG-XS, HTJ2K and VC-2.

jmi-edwards-3066183-x.pdf

One other thing to consider is licensing and patent royalties. Patent royalties are demanded with JPEG-XS but are not required with JPEG2000 and HTJ2K, because JPEG2000 and HTJ2K are royalty-free standards. You can find many open-source implementations of JPEG2000 and HTJ2K but there is only one for JPEG-XS. There are also commercial implementations of JPEG2000, HTJ2K and JPEG-XS.

An excellent HTJ2K open source implementation is OpenJPH, available here: https://github.com/aous72/OpenJPH

HTJ2K decoders have also been included in recent releases of popular open-source packages OpenJPEG and FFMPEG

[alatteri]

Thanks for the great info @michaeldsmith.

One question, does HTJ2K always work within YUV color space, or can it compress 12bit RGB 4:4:4 natively as RGB? Would the same go for XS too?

[TheSashmo]

Thanks @michaeldsmith I will keep reading up on it. What baffling me, is why the broadcast industry has/is moving more towards XS as the better solution consider that XS is in the 2110-7 and moving towards 2210-8 too. Based on what you are saying HTJ2K is the better play. Maybe its just timing, but I will keep my eyes open on this.

I haven't searched for it, but is there any data sheets comparing x264/x265/XS/J2K/HTJ2K?

[jonathan-catt]

Thank you all for the thoughtful discussion so far. I think it’s important to refocus on our core intentions and see if we can find some common ground. In our lab, we have two main use cases:

1. Collaborative Viewing for Clients: We need to deliver low-latency video of sufficient quality for clients to view content remotely, ideally in a “plug and play” setup with lower-powered hardware. However, even with lower-powered devices, the compression requirements remain high. We’re looking for a solution that supports native RGB 4:4:4, with at least 10-bit color depth, as anything less tends to introduce banding in gradients, which our clients find unacceptable.

2. Remote Operations for Operators: For our operators working remotely from satellite locations, we need to deliver a very high-quality, high-bitrate video signal. This typically involves UHD/4K with 12-bit 4:4:4 color, as the content will be viewed in DI theaters and grading suites. Latency must remain minimal since operators control their systems via technologies like RGS or Teradici. Any noticeable delay between screen updates will cause significant frustration.

UltraGrid has been ideal in terms of latency—it outperforms anything else we've tried. However, finding the right compression format for each use case has been a challenge. We've primarily relied on HEVC with Nvidia hardware for encoding and decoding, but HEVC's complexity, especially above HD resolutions, and the limitations of hardware-accelerated libraries (e.g., lack of support for greater than 4:2:0 or 8-bit formats) are problematic. While these constraints make sense for consumer devices, where delivery formats typically don’t exceed those specs, they fall short for professional use. We’ve built systems supporting 4K 12-bit 4:4:4 using Nvidia NVENC/NVDEC and dedicated SDI hardware, and at around 100Mb/s bitrate in 4K, the quality is perceptively very good. However, miniaturizing that into a “set-top” style device for client use has proven challenging.

On the other hand, I’d love to leverage J2K for our high-quality satellite workflows, but licensing the Comprimato SDK has been financially prohibitive at our current scale.

I don’t expect a one-size-fits-all solution for both use cases, which is why I suggested exploring JPEG XS. From what I’ve read, it’s designed to be less complex while still supporting higher bit depths and pixel formats. I reviewed the patent licenses, and while I’m no legal expert, the costs for small-scale usage seem fairly reasonable (though I could be misinterpreting the published rates). That said, I still need to run more tests to evaluate how a 30-50 Mb/s JPEG XS stream performs in HD. At that bitrate, the quality may not be sufficient for our needs. However, it may prove more useful for our Remote Operations use case, where we typically maintain 100-200 Mb/s network connections over WAN. For our Collaborative Viewing use case, we may still need to rely on HEVC, given the stricter bandwidth constraints, which is fine, but I wish there were better pix_format support without relying solely on software. C'est la vie!

I’ll admit, I don’t know much about HTJ2K, but based on @michaeldsmith post, it seems like an interesting possibility as well.

[bluepenguinpost]

My vote would go to HTJ2K. It's becoming the new standard image format for my clients.

[alatteri]

Hi @jonathan-catt

We seems to have some cross over in use cases. One thing though, there is some generic terminology here such as "low latency". I think it would be helpful to have some quantifiable metrics defined.

For us, our average latency glass-to-glass for 12bit RGB 444 2KDCI@24fps SDI is about 14-16 frames within the LA Metro area.
We encapsulate the output of UG into SRT and then use ZeroTier VPN on top of that. This adds some of the latency. With the recent enhancements to FEC in UG when using compressed formats, I'll be testing skipping the SRT step soon.

Some people consider that low latency, other might not. That is using x265 CPU encoding at around 25-30mbps. This same signal gets projected at times on a large cinema screen during color grading with zero complaints about image quality. In theory, x265 is keeping it 12bit RGB native throughout.

We can encode 12bit RGB444 UHD@24fps using x265 on Ryzen7950X, although it requires some additional x265 params that incur some additional latency, making it glass-to-glass about 18-21 frames.

I'm unsure how much of the latency is coming from the different steps in the process:
capture->encode->tranmission->decode->display
Switching to x264 10bit YUV444, which is easy for modern processors, doesn't seem to lower latency meaningfully.

Our encoders are 1U Asrock Rack Ryzen machines.

Our decoders are all Intel NUCi7, ranging from Gen11-13.
For endpoints that need SDI out, we connect them to a BMD UltraStudio 2K or 4K mini, otherwise we use the native 10bit HDMI port on NUC, which is actually very color accurate. Gen13 can decode x265-HEVC 4K in realtime, albeit incurring a high load.

As stated in the other thread, I haven't tried in ~2 years using NVENC-HEVC encoder, an decode on NUC in software. Last time I tried, it didn't work realtime. I've only ever been able to decode NVENC using a GPU, not CPU.

There are many topics here, from compression, transmission, appliance configuration. I'm happy to chat about all of them, but I do think we should segregate each to its own discussion.

If @michaeldsmith would be interested in getting HTJ2K into UG, that would be awesome. Any interest @MartinPulec

[TheSashmo]

I'll second that nomination.