Choppy camera sound in HomeKit

[VladNord]

I have Hikvision camera with built-in microphone.
When I watch the stream directly from go2rtc- flawless, but exported to HomeKit camera has choppy sound, picture is ok
Sound codec g711ulaw (tried another- no luck)

Here is config:

Thanks for help!

streams:
  home_cam_outdoor:
    - rtsp:/*****/ISAPI/Streaming/Channels/101
    - ffmpeg:home_cam_outdoor#audio=opus  

homekit:
  home_cam_outdoor:            
    pin: 12345678         
    name: home_cam_outdoor      

[AlexxIT]

Known problem

[civita]

It seems like this is related to incorrect timestamp increment (ts incr) of Opus RTP packet. According to RFC 7587, ts incr of a 48k sampling rates and 20ms frame size will be 960, or ts incr = frame size(ms) * sampling rate(k). I'm adding this into srtp/session.go. However, depends on connection type of iOS devices (Cellular/Wi-Fi), frame size will be other than 20 (60 on Cellular connection). Mismatched frame size results in no sound.

[civita]

Update: I am able to get HomeKit Opus Audio working on Cellular connection (frame size 60ms) by repacketizing three 20ms frames into one packet (with some hard-coded codes based on RFC 6716). This can be improved if ffmpeg args (-frame_duration in particular) can be modified after SelectedStreamConfig is retrieved from iOS devices. These only work if Opus producer is ffmpeg.

[AlexxIT]

As said in the RFC 7587 - your timestamp wrong. But maybe Apple also doesn't respect this RFC.

   Opus supports 5 different audio bandwidths, which can be adjusted
   during a stream.  The RTP timestamp is incremented with a 48000 Hz
   clock rate for all modes of Opus and all sampling rates.  The unit
   for the timestamp is samples per single (mono) channel.  The RTP
   timestamp corresponds to the sample time of the first encoded sample
   in the encoded frame.  For data encoded with sampling rates other
   than 48000 Hz, the sampling rate has to be adjusted to 48000 Hz.

[civita]

You are correct. I misinterpreted the last sentence (For data encoded with sampling rates other than 48000 Hz, the sampling rate has to be adjusted to 48000 Hz) as to add a factor of sampling rate into ts incr. After some research, it looks like apple doesn't follow RFC 7587 for Opus audio according to Hap spec (11.2.1 Streaming Requirements). Instead, timestamp is based on RFC 3550. If I understand correctly, it uses "sampling period" as timestamp increment, so 48k sampling rate in 20ms frame will be: 0.02(s)*48000(1/s)=960. (somehow matches frame size(ms) * sampling rate(k))

[AlexxIT]

I have tested Scrypted (because I don't have HomeKit camera with OPUS support):

• Sample rate: 8000, RTPTime: 20, Config: 19, Code: 0, Time inc: 160
• Sample rate: 16000, RTPTime: 20, Config: 23, Code: 0, Time inc: 320
• Sample rate: 24000, RTPTime: 20, Config: 27/23, Code: 0, Time inc: 480
• Sample rate: 8000, RTPTime: 40, Config: 19, Code: 2, Time inc: 320
• Sample rate: 8000, RTPTime: 60, Config: 19, Code: 3, Time inc: 480
• Sample rate: 24000, RTPTime: 60, Config: 27/23, Code: 3, Time inc: 1440

[AlexxIT]

And test for FFmpeg:

• Sample rate: 8000, Frame duration: 20, Config: 19, Code: 0, Time inc: 960
• Sample rate: 16000, Frame duration: 20, Config: 23, Code: 0, Time inc: 960
• Sample rate: 24000, Frame duration: 20, Config: 27, Code: 0, Time inc: 960
• Sample rate: 8000, Frame duration: 40, Config: 19, Code: 2, Time inc: 1920
• Sample rate: 8000, Frame duration: 60, Config: 19, Code: 3, Time inc: 2880
• Sample rate: 24000, Frame duration: 60, Config: 27, Code: 3, Time inc: 2880

[civita]

I have tested Scrypted (because I don't have HomeKit camera with OPUS support):

• Sample rate: 8000, RTPTime: 20, Config: 19, Code: 0, Time inc: 480
• Sample rate: 16000, RTPTime: 20, Config: 23, Code: 0, Time inc: 480
• Sample rate: 24000, RTPTime: 20, Config: 23, Code: 0, Time inc: 480
• Sample rate: 8000, RTPTime: 40, Config: 19, Code: 2, Time inc: 480
• Sample rate: 8000, RTPTime: 60, Config: 19, Code: 3, Time inc: 480
• Sample rate: 24000, RTPTime: 60, Config: 23, Code: 3, Time inc: 480

Are you using Scrypted as HomeKit camera input in go2rtc? No sure if constant 480 ts incr is due to this timekeeper.

And test for FFmpeg:

ffmpeg ones look reasonable since it follows RFC 7587:

+---------+-----------------+-----+-----+-----+-----+------+------+
|   Mode  |        fs       | 2.5 |  5  |  10 |  20 |  40  |  60  |
+---------+-----------------+-----+-----+-----+-----+------+------+
| ts incr |       all       | 120 | 240 | 480 | 960 | 1920 | 2880 |
|         |                 |     |     |     |     |      |      |
|  voice  | NB/MB/WB/SWB/FB |  x  |  x  |  o  |  o  |  o   |  o   |
|         |                 |     |     |     |     |      |      |
|  audio  |   NB/WB/SWB/FB  |  o  |  o  |  o  |  o  |  x   |  x   |
+---------+-----------------+-----+-----+-----+-----+------+------+

I haven't tried Scrypted or official HomeKit cameras before, and in my config I use generic rtsp camera and add audio=opus/16000 in order to get audio work in my iOS device.

streams:
  test:
    - ffmpeg:rtsp://...:...@.../...
    - ffmpeg:test#video=copy#audio=opus/16000
homekit:
  test:
    pin: ...
    pairings:
      - client_id=...

[AlexxIT]

You was right about "timekeeper". Thanks! It was made for AAC-ELD.
I have changed previous message about Scrypted.

PS. Interesting. Scrypted with Sample Rate 24000 starts with conf 27, but continue with conf 23.

08:36:12.285 DBG [RTP/OPUS] conf=27 code=0 size=    78 ts=2108177848 dt=    0 pt=110 ssrc=11318286 seq=2816 mark=true
08:36:12.313 DBG [RTP/OPUS] conf=27 code=0 size=    65 ts=2108178328 dt=  480 pt=110 ssrc=11318286 seq=2817 mark=true
08:36:12.313 DBG [RTP/OPUS] conf=27 code=0 size=    49 ts=2108178808 dt=  480 pt=110 ssrc=11318286 seq=2818 mark=true
08:36:12.350 DBG [RTP/OPUS] conf=27 code=0 size=    53 ts=2108179288 dt=  480 pt=110 ssrc=11318286 seq=2819 mark=true
08:36:12.351 DBG [RTP/OPUS] conf=27 code=0 size=    68 ts=2108179768 dt=  480 pt=110 ssrc=11318286 seq=2820 mark=true
08:36:12.391 DBG [RTP/OPUS] conf=27 code=0 size=    64 ts=2108180248 dt=  480 pt=110 ssrc=11318286 seq=2821 mark=true
08:36:12.391 DBG [RTP/OPUS] conf=27 code=0 size=    53 ts=2108180728 dt=  480 pt=110 ssrc=11318286 seq=2822 mark=true
08:36:12.432 DBG [RTP/OPUS] conf=27 code=0 size=    63 ts=2108181208 dt=  480 pt=110 ssrc=11318286 seq=2823 mark=true
08:36:12.432 DBG [RTP/OPUS] conf=27 code=0 size=    63 ts=2108181688 dt=  480 pt=110 ssrc=11318286 seq=2824 mark=true
08:36:12.472 DBG [RTP/OPUS] conf=23 code=0 size=    53 ts=2108182168 dt=  480 pt=110 ssrc=11318286 seq=2825 mark=true
08:36:12.472 DBG [RTP/OPUS] conf=23 code=0 size=    55 ts=2108182648 dt=  480 pt=110 ssrc=11318286 seq=2826 mark=true
08:36:12.512 DBG [RTP/OPUS] conf=23 code=0 size=    50 ts=2108183128 dt=  480 pt=110 ssrc=11318286 seq=2827 mark=true
08:36:12.514 DBG [RTP/OPUS] conf=23 code=0 size=    46 ts=2108183608 dt=  480 pt=110 ssrc=11318286 seq=2828 mark=true
08:36:12.552 DBG [RTP/OPUS] conf=23 code=0 size=    63 ts=2108184088 dt=  480 pt=110 ssrc=11318286 seq=2829 mark=true
08:36:12.552 DBG [RTP/OPUS] conf=23 code=0 size=    48 ts=2108184568 dt=  480 pt=110 ssrc=11318286 seq=2830 mark=true
08:36:12.593 DBG [RTP/OPUS] conf=23 code=0 size=    56 ts=2108185048 dt=  480 pt=110 ssrc=11318286 seq=2831 mark=true
08:36:12.594 DBG [RTP/OPUS] conf=23 code=0 size=    51 ts=2108185528 dt=  480 pt=110 ssrc=11318286 seq=2832 mark=true

[AlexxIT]

https://github.com/AlexxIT/go2rtc/releases/tag/v1.8.5