[Resolved] Timing problems on RPi4 (buffer overflows, icecast stalling) without optimization

[thowe-switzerland]

I am currently testing the unstable branch intensively on a Raspberry Pi 4B 4GB under Raspberry Pi OS.

I noticed in several places that the processing of the samples and the generation of e.g. the Icecast streams can lead to timing problems. This manifests itself in buffer overflows and stalling Icecast streams. Although the 4 cores are only used about 30% each.

I have read on the web that other users with real time audio applications on the RPi 4 have experienced similar problems.

I have significantly less problems when I build the Real Time RT kernel myself or when I use Ubuntu Server 64 bit. Especially the standard Raspberry Pi OS kernel seems to be affected. At least for me.

Since it's still most convenient to use Raspberry Pi OS, I tried to get a handle on the problem using task scheduler optimizations. And indeed everything runs much smoother with the following service file, where I choose the Round Robin scheduler with high priority, which is proven for Real Time.

[Unit]
Description=SDR AM/NFM demodulator
Documentation=https://github.com/szpajder/RTLSDR-Airband/wiki
Wants=network.target
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/rtl_airband -Fe
CPUSchedulingPolicy=rr
CPUSchedulingPriority=80
# The program may exit only due to startup failure (eg. misconfiguration)
# or due to failure of all SDR devices (eg. disconnection). In either case,
# there is no point to restart it, because it would fail once again.
Restart=no

[Install]
WantedBy=multi-user.target

Am I alone with my observations?

[charlie-foxtrot]

I'm running on an RPi 4 using the minimal Raspberry Pi 32-bit OS and not seeing a problem. I'm using 3 dongles (2.4 MHz bandwidth each) to process a total of 72 NFM frequencies and feed 7 mixers, each with an icecast stream.

I compile with -DPLATFORM=native and have the following options enabled at the top of my rtl_airband.conf:

multiple_demod_threads = true;
multiple_output_threads = true;

The larger bandwidth of the RSP1As may be slowing you down, I'm curious if enabling multiple demodulation threads will help.

[thowe-switzerland]

Thanks for your info and the two global flags! Your experience shows that it can work.

I have now commented out the rr scheduler in the service file for a test and set the two flags in the rtl_airband.conf. I am curious about the result and let you know.

The flag multiple_demod_threads = true; I already knew and had tested - without noticeable improvement. I am very curious if the multiple_output_threads = true; helps at all.

With RTL sticks I never have such problems. Everything always runs smoothly. The problems occur rather with SoapySDR or the higher bandwidths that come with it. But since the RPi 4 is quite performant and the CPU is far from busy, I wouldn't expect the problems yet. Something seems to cause "friction".

[thowe-switzerland]

I sporadically monitored the Icecast stream today with the new settings. The number of "hickups" seems to be much smaller with these settings, even without special task scheduling. A step in the right direction. For example, I never noticed that the stream was completely stopped. This is an improvement. However, I sporadically have the effect that fractions of a second are missing or a fraction of a second is played twice - a kind of short stutter. This happens about 5 times per hour.

I'm still trying to figure out what's causing minor timing problems. But it's getting better and better. :-)

[thowe-switzerland]

I just can't get rid of the timing issues on the mixer. I have attached an example which contains two MP3 files. The effect is that whole syllables, even words, are missing. Instead of "Turn left heading..." only "..... ding" can be heard in the mixer. Over the Icecast stream it came exactly the same:

mixer-problem.zip

1: Direct MP3 recording of the channel
2: Indirect MP3 recording of the channel via mixer

Currently I have both options for multithreading set to true, the TaskScheduler set to rr (round robin) with high prio. RTLSDR airband is compiled for native platform this time. The problem is the same for other (more conservative) settings.

Here are some statistics:

# HELP buffer_overflow_count Number of times a device's buffer has overflowed.
# TYPE buffer_overflow_count counter
buffer_overflow_count{device="0"}       0
buffer_overflow_count{device="1"}       0

# HELP output_overrun_count Number of times a device or mixer output has overrun.
# TYPE output_overrun_count counter
output_overrun_count{device="0"}        328
output_overrun_count{device="1"}        293
output_overrun_count{mixer="0"} 216
output_overrun_count{mixer="1"} 223

# HELP input_overrun_count Number of times mixer input has overrun.
# TYPE input_overrun_count counter
input_overrun_count{mixer="0",input="0"}        46
input_overrun_count{mixer="0",input="1"}        63
input_overrun_count{mixer="1",input="0"}        48
input_overrun_count{mixer="1",input="1"}        58

Hardware: RPi 4, 4GB
OS: Current Raspberr Pi OS 32 Bit
Samsung 16GB CF Card

I am wondering what this could be and what I can do to locate the problem.

[charlie-foxtrot]

It may just be too many bits to process in time. Try disabling one of your two devices and see if cutting down the load causes less drops

[thowe-switzerland]

Thank you. I will try this.

I currently just find it hard to imagine that the RPi 4 is already at its limit. All cores are 25% to 30% utilized, the load around 1 with 4 cores!

[charlie-foxtrot]

From the other conversation, if you're running an SDR with 5.0 MSPS at 14 bit resolution, thats a lot of bits. The initial FFT computation and demodulation is done in one thread (therefore one core) per input device. You could have 3 cores idle and 1 core max'ing out and not able to keep up with the incoming data stream.

[thowe-switzerland]

The 5 MSPS was only a temporary test to force the internal RF bandpath filter of the RSP1A from 1.5 to 5 MHz.

Currently, both RSP1As run with 3.0 MSPS each. So each only 0.5 MSPS more than the RTL sticks. And if you can run multiple sticks with an RPi 3, then the 2 x 3.0 MSPS on an RPi 4 should be fine. Isn't it?

The output of top also shows that the 4 cores are primarily idle.

I use many low-power computers: RPi1, 2, 3, 4, Tinker Board, APU2. Based on my experience, the RPi 4 would have to suffice if there wasn't a hitch somewhere. Already only your successful use case shows that clearly more lies in it. With FM function, the load increases significantly (I only have AM, which is easier to demodulate), and then you have many more channels and mixers than I do.

And my setup works fine over long periods of time. Such a swallow in the mixer happens maybe every 5 to 10 minutes.

Something is still suboptimal with me. Maybe even caused by myself. Some stupid little thing. I stay on it ;-)

[charlie-foxtrot]

I'm using the rtl-sdr.com dongles that are 8-bit so that is another difference, but I agree that NFM and the higher fft_size I'm running would be more taxing than yours.

There is some profiling support that you could try and may be informative. To use it compile with -DPROFILING=TRUE, set an environment variable CPUPROFILE_FREQUENCY=1000, run rtl_airband for a while then generate profiling results withgoogle-pprof --text --lines rtl_airband ./rtl_airband.prof

[thowe-switzerland]

Thanks! I'll do that too.

[charlie-foxtrot]

Another thing to try to see if it’s a load issue is over clock the pi

[thowe-switzerland]

I have now turned off all outputs to the SD card (MP3, RAW) for test purposes. Only one of the mixers I output as Icecast and as MP3. With this I want to find out if the output to the SD card could be the problem.

[thowe-switzerland]

At least in my case, it appears that the output to the SD card caused all the timing problems.

After switching to an output to the ramdisk, no more problems occur. I therefore set the title to Resolved.

Thanks for the support!