Two-way audio working?

[borgqueenx]

Hey there yi hacks! I bought 5 dome u pro cameras tbis black friday and wondering if we could use two-way audio on the device. I found two discussions regarding this that say we can use a text to speech service and upload a wav file, but nothing about using a microphone to talk live while watching the (rtsp?) Stream.
Is this now possible or has it been found unfortunately not possible?
Thanks!

[roleoroleo]

If you enable sound in the web gui, the rtsp will contain an audio channel.
You can get an audio only stream using the url rtsp://IP-CAM/ch0_2.h264

About the speaker, at the moment you can use it in two ways:

• Sending a raw audio stream to the web service
• Sending a text to the web service (requires a tts engine: https://github.com/roleoroleo/yi-hack-utils/tree/main/Allwinner-v2)

Check the wiki for the syntax:
https://github.com/roleoroleo/yi-hack-Allwinner-v2/wiki/Web-services-description#audio

These features are integrated in the ha custom component and you can use them from ha as a service.

[roleoroleo]

Another use, there is a hidden web page:
http://IP-CAM/?page=speak

[borgqueenx]

If we can upload a wav, wouldnt it be possible to insert a video stream on that page and a microphone button, that then saves in wav and immediately pushes it to the camera? Probably not so simple to include this in a go2rtc stream(i see that go2rtc is not used?), but it would already be a improvement :) Sent from Nine

…

________________________________ Van: roleo ***@***.***> verzonden: woensdag 29 november 2023 16:49 Aan: roleoroleo/yi-hack-Allwinner-v2 Cc: borgqueenx; Author Onderwerp: Re: [roleoroleo/yi-hack-Allwinner-v2] Two-way audio working? (Issue #762) Another use, there is a hidden web page: http://IP-CAM/?page=speak — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[AlexxIT]

Dahua cameras support multipart POST for playing audio files
AlexxIT/go2rtc#52 (comment)

I can try to support something like this. Multipart uses for sending multiple audio chunks to single POST connection.

[roleoroleo]

Please, explain me better.

EDIT

Actually the speaker.sh cgi handles a multipart post of this type:


Where the content is a raw file S16LE / 16 KHz, 16 bit, mono (or a wav file with the same parameters).

[AlexxIT]

Does it support multiple multipart files in same connection? One by one with little delay between.

[roleoroleo]

No, at the moment.

[roleoroleo]

Maybe I could implement it server side (within the cam).
But is there a JavaScript example to send multipart/x-mixed-replace data?
I found many examples on how to receive streaming from a server but nothing to upload.

[AlexxIT]

I don't know of any such examples.
And haven't implemented anything similar for sending data yet. I've only done it for receiving data.

[Dotatay]

The idea is so cool. I’m not good at JavaScript, but I tried to get an example from Codex:

const audioFile = new File(["audio.raw"], "audio.raw");
const buffer = await audioFile.readAsArrayBuffer();

fetch("https://example.com/upload", {
  method: "POST",
  headers: {
    "Content-Type": "audio/x-raw, format=S16LE, rate=16000, channels=1"
  },
  body: [
    {
      name: "audio",
      data: new Uint8Array(buffer)
    }
  ]
});

---

Second example:

let formData = new FormData();
formData.append("audio", new Blob([new Uint8Array(buffer)], {type: "audio/x-raw, format=S16LE, rate=16000, channels=1"}));

fetch("https://example.com/upload", {
  method: "POST",
  body: formData
})
.then(response => response.json())
.then(result => console.log(result))
.catch(error => console.log('error', error));

[kyutov]

@roleoroleo - is it possible to stream audio through the speaker of the camera? I have two h60ga cameras. I'm trying to achieve this Smart Baby Monitor. In short what I want to achieve it to capture the sound through ffmpeg_noise sensor in home assistant from the RTSP audio stream of the first camera, to upload it to Icecast 2 server and then to stream it through the speaker of the second camera when home assistant detects baby cry in the RTSP audio stream.
If it is not possible at the moment to stream audio through the speaker of the camera, will it possible to add such functionality to the hack?

[roleoroleo]

You can send an audio file to the cam but a stream is not supported.
About capture, you can use the rtsp stream, there is an only audio stream.

[kyutov]

Yes, I have already managed to capture the audio through the audio stream of the camera. Unfortunately the output creates audio file, which is continually feeded from the audio stream so there is no way to have a completed file which can be sent to the camera.
According to you will it be possible to add streaming functionality to the hack?

[roleoroleo]

Take a look to this repo:
https://github.com/roleoroleo/yi-hack_rtp_streamer

[kyutov]

Wow, this is great! Thank you!

I have tried both streaming ways (streamer/received and streaming using ffmpeg) and both of them are working. While using streamer/received I cannot control the volume and it is too low, but when I stream using ffmpeg I can control it so I got really good result.

I need to find out a way to start and stop the streamer and the receiver from Home Assistant. Will it be hard to add such functionality?

[roleoroleo]

You could create a ha script to run a remote command through ssh.

[kyutov]

Yes, I will go in this direction. Thanks for the suggestion and for the rtp streamer which you have created 👍

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.