Liquidsoap 2.3.0

Liquidsoap 2.3.0

⚠️ Wanna help us know our users and plan the next release cycle? Fill up this form ⚠️

We are stoked to announce the release of liquidsoap 2.3.0! 🎉 ✨

This release brings a lot of exciting changes. As usual, we recommend to first test it in a staging environment before deploying it to production. Migration notes for this release can be found here.

A lot of hard engineering work went into this release! The largest changes should that happened during this cycle were under the hood:

Complete re-implementation of the streaming cycle and clocks

This change allowed us to prepare the codebase to use OCaml's new multicore support in future releases and get rid of design decisions that were 20 years old now. Back then, we had no idea how things would evolve! In particular, we had to account for new form of content such as raw video and encoded content.

Instead of filling a partial frame during the streaming cycle, we are now manipulating chunks of immutable content, concatenating them to build a frame on each streaming cycle. This is much more inline with the logic underlying video and encoded content where pieces of content can happen at random time, according to their presentation timestamp (PTS) or decoder timestamp (DTS).

Abstracting content this way has allowed us to finally get rid of frame breaks doubling as both track marks and partial filling marks. With the new content model, track marks and metadata are first-class content and part of the frame generation.

If you have ever encountered any of the infamous #get_frame didn't add exactly one break! fatal error, you know what we mean. This early design decision had plagued us for a while and it's wonderful to see it gone!

Along with these changes, we also reimplemented from scratch our notion of clocks, the abstract API that runs the main streaming cycles.

While clocks are still a pretty complex topic (self-sync, active vs. passive etc), we hope that this rewrite will clarify things. It was a good opportunity to clean up our implementation and use the experience gained about it in the recent years.

With these changes, clocks are now surfaced to the user via a clock method on each source. This makes it possible to directly access source clocks, animate them in your script, etc. We've also started to support synchronous streaming scripts, which should make it possible to use liquidsoap as a CLI for offline processing.

Script caching and memory optimization

Another important new feature with this release is the ability to cache the standard library and user scripts. This allows liquidsoap to run much faster. More details are available in this blog post

Incidentally, script caching also has very important benefits for memory usage. This is documented here.

This feature is probably the most important one of this release for most users!

Internal autocue

Autocue, the method for computing the perfect crossfade has also been polished in this release. We have now switched to our internal implementation, which is available when enabling our ffmpeg support.

You should be able to enable it by adding this to your script:

enable_autocue_metadata()

With this, all requests should resolve with the appropriate metadata so that any crossfade produces perfect transitions!

Remember that this feature requires CPU-heavy computations on each track!

A lot of testing went into this feature so we are really thankful to our users for the support and patience!

Other new features

Here's a list of other notable changes in this release:

• Frame duration was dropped to 0.02s. This should enhance streams latency and was made possible thanks to the new streaming implementation. See #4033 for a discussion about the trade-offs involved with this setting.
• New support for video size-independent placement of video elements. This makes it possible to design video streams with different elements whose placement is independent of the final video resolution. See this blog post for an example of it.
• Support for NDI output. We always try to extend the scope of applications that can be interfaced with liquidsoap. In the future, we would like to add support for NDI inputs and extend the supported formats.

The rest of the changelog is included below.

As with each release, we want to thank all our users for their patience, early testing and collaboration to make this release possible. It's a testament to their help that we are able to ship this release with such a level of confidence despite all the major changes that happened in the code base.

Special thanks go to: @vitoyucepi, @gAlleb, @RM-FM, @Moonbase59, @Russsgithub, @gilou, Xogium, the whole AzuraCast and Radio France team and everyone who helped reporting, fixing and confirming bugs.

Thanks y'all!

2.3.0 (2024-11-27)

New:

• Rewrote the streaming API to work with immutable frame content. This
  should greatly impact impredictable side-effect of the previous models w.r.t.
  track marks, content sharing and more. This also impacts multiple operators
  behavior. Mostly, things should be roughly the same with differences around
  behaviors related to track marks (source.on_track and etc). (#3577)
• Added script caching layer for faster script startup time. See: https://www.liquidsoap.info/blog/2024-06-13-a-faster-liquidsoap/ for details (#3924, #3949, #3959 and #3977)
• Rewrote the clock/streaming loop layer. This prepares our streaming system to
  support multicore when the OCaml compiler is mature enough to allow it. Clocks
  are now attached to sources via their clock methods. Returned value is a stripped
  down clock variable. Users can use the clock function to retrieve the full
  methods, e.g. s = sine(); c = clock(s.clock). This value has advanced functions
  for clock control such as start/stop, ticks and self_sync to check for
  self-sync. (#3781)
• Allow frames duration shorter than one video frames, typically values under 0.04s.
  Smaller frames means less latency and memory consumption at the expense of
  a higher CPU usage (#3607)
• Change default frame duration to 0.02s (#4033)
• Optimized runtime (#3927, #3928, #3919)
• Added NDI output support (#4181)
• Added finally to execute code regardless of whether or not an exception is raised
  (see: #3895 for more details).
• Added support for Spinitron submission API (#4158)
• Removed gstreamer support. Gstreamer's architecture was never a good fit for us
  and created a huge maintenance and debugging burden and it had been marked as
  deprecated for a while. Most, if not all of its features should be available using
  ffmpeg. (#4036)
• Removed taglib support. It is superseded by the internal ocaml-metadata module
  and taglib, with its dependency on the C++ runtime library, has been causing issues
  with binary builds portability and crashes with the (not yet supported) OCaml 5
  compiler. (#4087)
• Added video.canvas to make it possible to position video elements independently
  of the rendered video size (#3656, blog post)
• Added cover manager from an original code by @vitoyucepi (#3651)
• Added non-interleaved API to %ffmpeg encoder, enabled by default when only
  one stream is encoded.
• Allow trailing commas in record definition (#3300).
• Added metadata.getter.source.float (#3356).
• BREAKING: Added duration and ticks to metadata available when computing HLS segment names (#4135)
• Added optional main_playlist_writer to output.file.hls and
  derivated operator (#3484)
• Added is_nan, is_infinite, ceil, floor, sign and round (#3407)
• Added %track.drop to the %ffmpeg encoder to allow partial encoding
  of a source's available tracks (#3480)
• Added let { foo? } = ... pattern matching (#3481)
• Added metadata.replaygain method to extract unified replay gain value from metadata (#3438).
• Added metadata.parse.amplify to manually parse amplify override metadata.
• Added compute parameter to file.replaygain to control gain calculation (#3438).
• Added compute parameter to enable_replaygain_metadata to control replay gain calculation (#3438).
• Added copy: protocol (#3506)
• Added file.touch.
• Added support for sqlite databases (#3575).
• Added string.of_int and string.spaces.
• Added list.assoc.nullable.
• Added source.cue (#3620).
• Added string.chars (#4111)
• Added atomic file write operations.
• Added new macos_say speech synthesis protocol. Make it the default implementation for the say:
  protocol on macos.
• Added settings.request.timeout to set the request timeout globally.

Changed:

• Reimplemented request.once, single and more using source.dynamic. Removed experiment
  flag on source.dynamic. The operator is considered stable enough to define advanced sources
  but the user should be careful when using it.
• Mute SDL startup messages (#2913).
• int can optionally raises an error when passing nan or infinity, int(infinity)
  now returns max_int and int(-infinity) returns min_int. (#3407)
• Made default font a setting (#3507)
• Changed internal metadata format to be immutable (#3297).
• Removed source.dump and source.drop in favor of safer request.dump and request.drop.
  source.{dump, drop} can still be implemented manually when needed and with the proper
  knowledge of what's going on.
• Allow a getter for the offset of on_offset and dropped the metadata
  mechanism for updating it (#3355).
• string.length and string.sub now default to utf8 encoding (#4109)
• Disable output paging when TERM environment variable is not set.
• Allow running as root user inside docker container by default (#3406).
• Run check_next before playlist's requests resolutions (#3625)
• Set force to true by default in file.copy to make operator behave
  as expected.
• BREAKING: Float comparison now follows the expected specs, in particular: nan == x is always false and
  nan != x is always true. Use float.is_nan to test if a float is nan.
• BREAKING: replaygain no longer takes ebu_r128 parameter (#3438).
• BREAKING: assume replaygain_track_gain always stores volume in dB (#3438).
• BREAKING: protocols can now check for nested static uri. Typically, this means
  that requests for an uri of the form: annotate:key="value",...:/path/to/file.mp3
  is now considered infallible if /path/to/file.mp3 can be decoded.
• Added parents option of file.mkdir (#3600, #3601).
• Added forced_major_collections record field to the result of runtime.gc.stat() and
  runtime.gc.quick_stat() (#3783).
• Changed the port for the built-in Prometheus exporter to 9599 (#3801).
• Set segments_overheader in HLS outputs to disable segments cleanup altogether.
• Added support for caching LV2 and LADSPA plugins (#3959).
• Pulseaudio input and output now restart on pulseaudio errors (#4174).

Fixed:

• Fixed type generalization on values returned from function applications. Most notably,
  this should help with HTTP endpoint registration (#3303, fixed in #4030)