Improve sound support for external controller (#6591) (#6605)

* Improve sound support for external controller (#6591) Stream sound data from controller to Webots if the sound file path is accesible by the controller process. This allows external controller to use sounds even if the file path is not accessible from Webots. * Controller: Try to load sound data on controller side and stream it to webots as part of the C_SPEAKER_PLAY_SOUND message. (speaker.c) * Webots: Read controller uploaded sound data in C_SPEAKER_PLAY_SOUND message (WbSpeaker). Use former file based load method if no data was streamed from controller. * Streamed sound data is cached inside WbSpeaker instance as controller provides the data only on first use of a sound file. * Updated speaker.md to explain when wb_speaker_play_sound() streams loads the sound data itself already. * Prevent copy/assignment of WbSoundClip as it now hold a heap allocated member (mDevice). * WbSpeaker.cpp - fix source indentation * Fix comment typo in speaker.c * Address cppcheck issues * Added changelog entry for this PR Fixed ubuntu tester src formatting issue. * Fix formatting issue in WbSpeaker.cpp * Fixed bug radar.py (#6606) * fix bug radar.py Fixed the bug that when using Python to write a controller, using getTargets() could not correctly obtain multiple target data detected by radar * Fixed the bug in obtaining radar detection target information#6606 Fixed the bug that when using Python to write a controller, using getTargets() could not correctly obtain multiple target data detected by radar --------- Co-authored-by: Olivier Michel <Olivier.Michel@cyberbotics.com> * Added changelog entry for this PR Fixed ubuntu tester src formatting issue. * Addressed code review comments #6605 * Incorporated review feedback Added improvement provided by CoolSpy3. * Update src/controller/c/speaker.c --------- Co-authored-by: Olivier Michel <Olivier.Michel@cyberbotics.com> Co-authored-by: Gabryel Reyes <66941456+gabryelreyes@users.noreply.github.com> Co-authored-by: lonely-poppy <77427326+lonely-poppy@users.noreply.github.com>
cyberbotics · Aug 5, 2024 · 50ef671 · 50ef671
1 parent 1200531
commit 50ef671
Show file tree

Hide file tree

Showing 7 changed files with 149 additions and 32 deletions.
diff --git a/docs/reference/changelog-r2024.md b/docs/reference/changelog-r2024.md
@@ -17,4 +17,5 @@ Released on December **th, 2023.
     - Fixed crashes (with some graphics cards) caused by references to unused GLSL uniforms ([#6587](https://github.com/cyberbotics/webots/pull/6587)).
     - Fixed `Brake`s added to the second axis of a `Hinge2Joint` being applied to the non-transformed axis ([#6584](https://github.com/cyberbotics/webots/pull/6584)).
     - Fixed invalid absolute sound file path resulted in crash ([#6593](https://github.com/cyberbotics/webots/pull/6593))
+    - Fixed Speaker relative sound file path not working with external controller ([#6605](https://github.com/cyberbotics/webots/pull/6605)).
     - Fixed the bug that when the language is Python, getTargets() cannot correctly obtain the multi-target data detected by the radar ([#6606](https://github.com/cyberbotics/webots/pull/6606))
diff --git a/docs/reference/speaker.md b/docs/reference/speaker.md
@@ -111,9 +111,11 @@ Finally, the boolean `loop` argument defines if the sound will be played only on
 
 It is possible to change the volume, pitch, balance, and loop parameters of a sound currently playing by calling again the `wb_speaker_play_sound` function with the same speakers and `sound` arguments.
 
-> **Note**: The path to the sound file should be defined either absolutely or relatively.
-If defined relatively, it will be searched first relatively to the robot controller folder.
-If not found there and if the robot is a PROTO, it will be searched relatively to the PROTO folder of the robot.
+> **Note**: The path to the sound file can be defined either absolutely or relatively.
+If the file path is accessible by the robot controller either absolutely or relative to its working directory, it is loaded by the controller and its contents are streamed to Webots.
+If the file is not found by the controller, the given path is passed to Webots.
+Webots will first search for the path relative to the robot controller folder.
+If the file is not found relative to the controller folder and the robot is a PROTO, Webots will search for the sound file relative to the folder of the PROTO file.
 
 ---
 

diff --git a/src/controller/c/speaker.c b/src/controller/c/speaker.c
@@ -40,6 +40,8 @@ struct Sound {
   bool need_update;
   bool need_stop;
   bool is_playing;
+  unsigned char *upload_data;  // sound data for uploading to webots
+  int upload_size;             // size of sound data
   struct Sound *next;
 };
 
@@ -257,6 +259,18 @@ static void speaker_write_request(WbDevice *d, WbRequest *r) {
         else
           request_write_char(r, 0);
       }
+
+      request_write_int32(r, sound->upload_size);
+      if (sound->upload_size) {
+        /* Sound data available to stream, send it one time and discard it.
+         * Webots will cache the data inside the device.
+         */
+        request_write_data(r, sound->upload_data, sound->upload_size);
+        free(sound->upload_data);
+        sound->upload_data = NULL;
+        sound->upload_size = 0;
+      }
+
       sound->need_update = false;
       sound = sound->next;
     }
@@ -298,6 +312,43 @@ static void speaker_toggle_remote(WbDevice *d, WbRequest *r) {
   // nothing to do.
 }
 
+/* Try to read sound file contents if it is found by given path. */
+static void speaker_try_load_sound_local(Sound *sound, const char *sound_file_name) {
+  FILE *fp;
+
+  sound->upload_data = NULL;
+  sound->upload_size = 0;
+
+  if ((fp = fopen(sound_file_name, "rb")) != NULL) {
+    if (fseek(fp, 0L, SEEK_END) == 0) {
+      const long file_size = ftell(fp);
+      if (file_size != -1) {
+        if (fseek(fp, 0L, SEEK_SET) == 0) {
+          size_t consumed = 0U;
+          sound->upload_data = malloc(file_size);
+          sound->upload_size = (int)file_size;
+          while (consumed < file_size) {
+            const size_t read = fread(&sound->upload_data[consumed], sizeof(unsigned char), file_size - consumed, fp);
+            if (ferror(fp)) {
+              free(sound->upload_data);
+              sound->upload_data = NULL;
+              sound->upload_size = 0;
+              break;
+            }
+            consumed += read;
+          }
+        }
+      }
+    }
+
+    if (sound->upload_data == NULL) {
+      fprintf(stderr, "Warning: %s() failed to read sound file '%s' :", __FUNCTION__, sound_file_name);
+      perror(NULL);
+    }
+    fclose(fp);
+  }
+}
+
 static void speaker_play_sound(WbDevice *device, const char *sound_name, double volume, double pitch, double balance, bool loop,
                                int side) {
   Sound *sound = speaker_get_sound_if_exist(device, sound_name);
@@ -309,6 +360,8 @@ static void speaker_play_sound(WbDevice *device, const char *sound_name, double
     memcpy(sound->sound_file, sound_name, l);
     sound->next = ((Speaker *)device->pdata)->sound_list;
     ((Speaker *)device->pdata)->sound_list = sound;
+
+    speaker_try_load_sound_local(sound, sound_name);
   }
   sound->volume = volume;
   sound->pitch = pitch;

diff --git a/src/webots/nodes/WbSpeaker.cpp b/src/webots/nodes/WbSpeaker.cpp
@@ -27,6 +27,7 @@
 
 #include "../../controller/c/messages.h"
 
+#include <QtCore/QBuffer>
 #include <QtCore/QDataStream>
 #include <QtCore/QDir>
 
@@ -61,6 +62,10 @@ WbSpeaker::~WbSpeaker() {
   }
   mSoundSourcesMap.clear();
   mPlayingSoundSourcesMap.clear();
+
+  foreach (QByteArray *buffer, mStreamedSoundDataMap)
+    delete buffer;
+  mStreamedSoundDataMap.clear();
 }
 
 void WbSpeaker::postFinalize() {
@@ -79,21 +84,8 @@ void WbSpeaker::handleMessage(QDataStream &stream) {
       int numberOfSound = 0;
       stream >> numberOfSound;
       for (int i = 0; i < numberOfSound; ++i) {
-        short size;
-        short side;
-        double volume;
-        double pitch;
-        double balance;
-        unsigned char loop;
-        stream >> size;
-        char soundFile[size];
-        stream.readRawData(soundFile, size);
-        stream >> volume;
-        stream >> pitch;
-        stream >> balance;
-        stream >> side;
-        stream >> loop;
-        playSound(soundFile, volume, pitch, balance, (bool)loop, (int)side);
+        SoundPlayData playData(stream);
+        playSound(playData);
       }
       return;
     }
@@ -194,18 +186,28 @@ void WbSpeaker::playText(const char *text, double volume) {
   }
 }
 
-void WbSpeaker::playSound(const char *file, double volume, double pitch, double balance, bool loop, int side) {
-  QString filename = QString(file);
-  QString key = filename;
-  if (side == -1)
+void WbSpeaker::playSound(SoundPlayData &playData) {
+  const QString filename(playData.file());
+  QString key(filename);
+  if (playData.side() == -1)
     key += "_left";
-  else if (side == 1)
+  else if (playData.side() == 1)
     key += "_right";
 
+  // Controller streamed sound data available ?
+  if (playData.rawLength() && !mStreamedSoundDataMap.contains(filename))
+    mStreamedSoundDataMap[filename] = new QByteArray(playData.rawData());
+  QByteArray *cachedSoundData = NULL;
+  if (mStreamedSoundDataMap.contains(key))
+    cachedSoundData = mStreamedSoundDataMap[key];
+
   if (!mSoundSourcesMap.contains(key)) {  // this sound was never played
     QString path;
+    // check if sound data was streamed from controller
+    if (cachedSoundData)
+      path = filename;
     // check if the path is absolute
-    if (QDir::isAbsolutePath(filename) && QFile::exists(filename))
+    if (path.isEmpty() && QDir::isAbsolutePath(filename) && QFile::exists(filename))
       path = filename;
     // check if the path is relative to the controller
     if (path.isEmpty() && QFile::exists(mControllerDir + filename))
@@ -221,7 +223,18 @@ void WbSpeaker::playSound(const char *file, double volume, double pitch, double
     WbSoundSource *source = WbSoundEngine::createSource();
     updateSoundSource(source);
     const QString extension = filename.mid(filename.lastIndexOf('.') + 1).toLower();
-    const WbSoundClip *soundClip = WbSoundEngine::sound(path, extension, NULL, balance, side);
+
+    // Use controller streamed sound data if available.
+    QBuffer *device = NULL;
+    if (cachedSoundData) {
+      device = new QBuffer(cachedSoundData);
+      device->open(QBuffer::ReadOnly);
+    }
+
+    const WbSoundClip *soundClip = WbSoundEngine::sound(path, extension, device, playData.balance(), playData.side());
+    delete device;
+    device = NULL;
+
     if (!soundClip) {
       this->warn(tr("Impossible to play '%1'. Make sure the file format is supported (8 or 16 bits, mono or stereo wave).\n")
                    .arg(filename));
@@ -237,9 +250,9 @@ void WbSpeaker::playSound(const char *file, double volume, double pitch, double
     mPlayingSoundSourcesMap[key] = source;
     if (!source->isPlaying())  // this sound was already played but is over
       source->play();
-    source->setLooping(loop);
-    source->setGain(volume);
-    source->setPitch(pitch);
+    source->setLooping(playData.loop());
+    source->setGain(playData.volume());
+    source->setPitch(playData.pitch());
     if (WbSimulationState::instance()->isPaused() || WbSimulationState::instance()->isStep())
       source->pause();
   }
@@ -280,3 +293,24 @@ void WbSpeaker::updateSoundSource(WbSoundSource *source) {
   source->setVelocity(linearVelocity());
   source->setDirection(rotationMatrix() * WbVector3(0, 1, 0));
 }
+
+WbSpeaker::SoundPlayData::SoundPlayData(QDataStream &stream) : mFile(), mRawData() {
+  short size;
+  unsigned char loopByte;
+  stream >> size;
+  char soundFile[size];
+  stream.readRawData(soundFile, size);
+  mFile = QString(soundFile);
+
+  stream >> mVolume;
+  stream >> mPitch;
+  stream >> mBalance;
+  stream >> mSide;
+  stream >> loopByte;
+  mLoop = (bool)loopByte;
+  stream >> mRawLength;
+  if (mRawLength) {
+    mRawData.resize(mRawLength);
+    stream.readRawData(mRawData.data(), mRawLength);
+  }
+}
diff --git a/src/webots/nodes/WbSpeaker.hpp b/src/webots/nodes/WbSpeaker.hpp
@@ -20,6 +20,7 @@
 #include <QtCore/QMap>
 
 class WbSoundSource;
+class QDataStream;
 
 class WbSpeaker : public WbSolidDevice {
   Q_OBJECT
@@ -39,12 +40,39 @@ class WbSpeaker : public WbSolidDevice {
   void postPhysicsStep() override;
 
 private:
+  // private data types
+
+  // Data from wb_speaker_play_sound() API
+  class SoundPlayData {
+  public:
+    explicit SoundPlayData(QDataStream &);
+
+    const QString &file() const { return mFile; }
+    double volume() const { return mVolume; }
+    double pitch() const { return mPitch; }
+    double balance() const { return mBalance; }
+    bool loop() const { return mLoop; }
+    short side() const { return mSide; }
+    int rawLength() const { return mRawLength; }
+    QByteArray &rawData() { return mRawData; }
+
+  private:
+    QString mFile;
+    double mVolume;
+    double mPitch;
+    double mBalance;
+    bool mLoop;
+    short mSide;
+    int mRawLength;
+    QByteArray mRawData;
+  };
+
   // private functions
   WbSpeaker &operator=(const WbSpeaker &);  // non copyable
   WbNode *clone() const override { return new WbSpeaker(*this); }
   void init();
 
-  void playSound(const char *file, double volume, double pitch, double balance, bool loop, int side);
+  void playSound(SoundPlayData &playData);
   void playText(const char *text, double volume);
   void stop(const char *sound);
   void stopAll();
@@ -53,6 +81,7 @@ class WbSpeaker : public WbSolidDevice {
 
   QMap<QString, WbSoundSource *> mSoundSourcesMap;
   QMap<QString, const WbSoundSource *> mPlayingSoundSourcesMap;
+  QMap<QString, QByteArray *> mStreamedSoundDataMap;
 
   QString mControllerDir;
   QString mEngine;

diff --git a/src/webots/sound/WbSoundClip.cpp b/src/webots/sound/WbSoundClip.cpp
@@ -28,7 +28,7 @@
 #include <AL/al.h>
 #endif
 
-WbSoundClip::WbSoundClip() : mDevice(NULL), mBuffer(0), mSide(0), mBalance(0.0) {
+WbSoundClip::WbSoundClip() : mBuffer(0), mSide(0), mBalance(0.0) {
 }
 
 WbSoundClip::~WbSoundClip() {
@@ -46,7 +46,6 @@ void WbSoundClip::load(const QString &filename, const QString &extension, QIODev
   mFilename = wave.filename();
   mSide = side;
   mBalance = balance;
-  mDevice = device;
   load(&wave);
 }
 

diff --git a/src/webots/sound/WbSoundClip.hpp b/src/webots/sound/WbSoundClip.hpp
@@ -38,7 +38,6 @@ class WbSoundClip {
 
 protected:
   QString mFilename;
-  QIODevice *mDevice;
   unsigned int mBuffer;
   int mSide;  // 0: both sides, -1: left only, 1: right only
   double mBalance;