Render the audio

[padenot]

This works well for me in Chrome 94 beta with Web Codecs enabled on my Linux box, here's how it goes:

• A SharedArrayBuffer-based ring buffer is given to a very simple AudioWorkletProcessor (that reads from it) and the control thread (for now the main thread, but we'll be able to move that to a Web Worker)
• The AudioRenderer pre-buffers some audio and writes the PCM to this ring buffer, on initialization.
• When the AudioContext starts, the AudioWorkletProcessor starts to read from this ring buffer to output the audio.
• The AudioRenderer sees that the ring buffer becomes empty and crosses its threshold for decoding -- it then proceeds to decode some compressed audio and pushes the PCM to the ring buffer. If the threshold isn't reached, it simply re-schedules itself in the future.
• Pausing the AudioRenderer is made via suspending the AudioContext, so pause/play is supported
• A volume slider has been added: this is just a GainNode and an <input type=range> -- we get smooth volume ramps for free :-).
• The clock is simply the AudioContext.currentTime, offset by the output latency, converted to microseconds.

But it is only a first cut, this is what I'm planning to fix:

• Only the left channel is rendered for now, because that was easier enough to get something going. Not sure if Chrome has all the copyTo variants already, and [AllowShared], but in any case I have code lying around to interleave/deinterleave if necessary
• There is a copy we should skip in the decoder output callback
• A/V sync is not as good as it could be because of the lack of AudioContext.outputLatency in Chrome, see https://blog.paul.cx/post/audio-video-synchronization-with-the-web-audio-api/ for details. For now, I picked a number that was kind of OK for my Linux workstation with my sound card, but YMMV. We'll do something fancier, per platform, in a second step, so it looks good
• Increase the latency of the AudioContext: no need for extreme low latency here.

All the code is in PR is from me originally so no license issue. The ring buffer and the server are from https://github.com/padenot/ringbuf.js, but I've just copied/tweaked the necessary files for this demo.

[padenot]

Only the left channel is rendered for now, because that was easier enough to get something going. Not sure if Chrome has all the copyTo variants already, and [AllowShared], but in any case I have code lying around to interleave/deinterleave if necessary

I was using the wrong google-chrome-unstable it seems like, I took another one (95) and it doesn't throw anymore.

[chcunningham]

Looks great. I had some questions, but feel free to submit whenever you're happy.

[chcunningham]

@hoch FYI. bumping https://bugs.chromium.org/p/chromium/issues/detail?id=785409

[guest271314]

If I understand the requirement correctly this https://guest271314.github.io/webcodecs/ is how I serialize and deserialize audio with WebCodecs. It is non-trivial resampling the fixed 48000 sample rate that Chromium has hardcoded, especially when the input file is 22050, 1 channel, and I never set Opus encoder sample rate to 48000.

[guest271314]

The AudioRenderer sees that the ring buffer becomes empty and crosses its threshold for decoding -- it then proceeds to decode some compressed audio and pushes the PCM to the ring buffer. If the threshold isn't reached, it simply re-schedules itself in the future.

The rescheduling part can be problematic re glitches and gaps between schedules. One option is to utilize WebAssembly.Memory.grow() to dynamically grow the underlying buffer - within the 4GB Chromium limitation - to write and read to a single contigous block of memory, something like

const initial = (384 * 512 * 3) / 65536; // 3 seconds
const maximum = (384 * 512 * 60 * 60) / 65536; // 1 hour
let started = false;
let readOffset = 0;
let init = false;
const memory = new WebAssembly.Memory({
  initial,
  maximum,
  shared: true,
});
console.log(memory.buffer.byteLength, initial / 65536);
// ...
await readable.pipeTo(
  new WritableStream(
    {
      start() {
        console.log('writable start');
      },
      async write(value, controller) {
        console.log(value, value.byteLength, memory.buffer.byteLength);
        if (readOffset + value.byteLength > memory.buffer.byteLength) {
          console.log('before grow', memory.buffer.byteLength);
          memory.grow(3);
          console.log('after grow', memory.buffer.byteLength);
        }
        let uint8_sab = new Uint8Array(memory.buffer);
        let i = 0;
        if (!init) {
          init = true;
          i = 44;
        }
        for (; i < value.buffer.byteLength; i++, readOffset++) {
          if (readOffset + 1 >= memory.buffer.byteLength) {
            console.log(`memory.buffer.byteLength before grow() for loop: ${memory.buffer.byteLength}.`);
            memory.grow(3);
            console.log(`memory.buffer.byteLength after grow() for loop: ${memory.buffer.byteLength}`);
            uint8_sab = new Uint8Array(memory.buffer);
          }              
          uint8_sab[readOffset] = value[i];
        }
        if (!started) {
          started = true;
          aw.port.postMessage({
            started: true,
          });
        }
      },
      close() {
        console.log('writable', readOffset, memory.buffer.byteLength);
        aw.port.postMessage({
          readOffset,
        });
      },
    }
  )
);

[padenot]

What's left to do is to check A/V sync on various OSes, I have only been testing on a Linux desktop with a USB DAC.