Long time taken between calling sem_post() and getting out of sem_wait()

[Indra5196]

Required information

OS: Linux
Machine: QEMU x86-64
Host OS: Ubuntu 18.04 LTS

Compiler version:
E.g. GCC 7.4.0

Observed result or behaviour:
I tried transferring field data (uint32) between two processes. Receiver callback was registered at the subscriber using a listener. The time taken in the data transfer was very huge (100ms). On debugging, I found that huge time is consumed between sem_post system call (called after pushing the chunk on the receiver queue through the notify method) and the release of sem_wait system call on the corresponding semaphore. Please refer to the screenshot attached. (Have a look at "After iox_sem_post call" and "After iox_sem_wait call" time stamps.)

Note: All the timestamps are in Nanoseconds

Expected result or behaviour:
Time between the system calls should have been instantaneous (in order of Microseconds or less)

[elfenpiff]

@Indra5196 Can you please tell us more about your setup.

Was it like two applications, one with the publisher and another with a subscriber inside the listener and the subscriber required ~100ms until it received the message? With what rate did you send the data?

If not then a short info with:

1. How many applications where running.
2. How many publishers/subscribers/topics were active in the system and in which applications
3. What kind of data did you send, only uint32_t or were there some other active topics
4. With which rate did you send the data?
5. How many CPU cores does your system have.
   would be excellent so that we can recreate your problem and start looking for the cause.

[elBoberido]

@Indra5196 as it seems you are running iceoryx in QEMU. Could you maybe check whether sem_post and sem_wait without all the abstractions added by iceoryx also behaves the same, i.e. running two processes and calling sem_post and sem_wait on a named semaphore?

[Indra5196]

@elfenpiff here is the required info

1. How many applications where running

two, one as publisher, and one as subscriber

3. What kind of data did you send, only uint32_t or were there some other active topics

uint32_t along with a userHeader (a struct having two uint64_t fields in it, used for validation in CM)

4. With which rate did you send the data?

We are pausing for 2 seconds before sending each chunk of data. We are sending it over a 1kb chunk for 10 iterations, then over 2kb chunk for 10 iterations and so on up till 256kb. We wanted to take averages for each chunk size and calculate performance of iceoryx when integrated with CM

5. How many CPU cores does your system have.

My system uses intel i3 7th gen processor (dual core). However, QEMU emulates a pentium 2, which is a single core processor

[Indra5196]

as it seems you are running iceoryx in QEMU. Could you maybe check whether sem_post and sem_wait without all the abstractions added by iceoryx also behaves the same, i.e. running two processes and calling sem_post and sem_wait on a named semaphore?

Yes I've already did that, and the time taken is in the order of micro-seconds. I've also tried running iceoryx examples for recording the time between these system calls, and that too is in micro-seconds.

So far, I've realized that there might be something wrong in my CM implementation, but I am kind of clueless regarding what exactly could trigger a bottleneck in the execution of these system calls....

[elBoberido]

@Indra5196 the next you could try to use the posixCall abstraction to call sem_post and sem_wait. With his we can exclude another source for errors but I fear the issue is either placing the semaphore in shared memory or running iceoryx on an emulated 32 bit CPU. Some of the code was created with the assumption to run on 64 bit CPUs. Is a warning printed to the console when you run iceoryx on QEMU?

[elfenpiff]

@Indra5196 Is this issue still active or can we close it?

[mossmaurice]

@Indra5196 @hemalbavishi Closing this issue, feel free to re-open if you have further questions.