20-60 second delays in actionlib.SimpleActionClient.wait_for_server()

[lindzey]

(Submitted here based on a suggestion to my ROS answers question).

I am sporadically seeing actionlib’s wait_for_server() take tens of seconds to return when starting a new client node, even when the server node is already up, running, and responding to goals from other nodes.

It’s disconcerting to have this sometimes return instantly, and sometimes require up to a minute.

Is this expected? Has anybody seen this before? Does it reflect some error in how we’ve configured the system or how we're using actionlib?

---

We are running Indigo on Ubuntu 14.04, using the python implementation of SimpleActionClient/SimpleActionServer, and a single node implements multiple servers.

Unfortunately, I have been unable to reproduce this delay in simulation or on the bench. So far, it has only (sporadically) shown up when testing in the field. I don’t have enough samples to say for sure, but the delays might get longer the longer we’re operating. Additionally, I have not seen this the first time a client connects to the server; it seems to happen on subsequent connections.

Based on the SimpleActionClient’s and SimpleActionServer’s log files, I think the issue is that the /cancel and /goal topics aren’t connecting immediately: note the 20 second delay between 13:51:55 -> 13:52:15.

If I understand the logs, the problem was that rosmaster needed to push out updates regarding who was publishing/subscribing in order for the SimpleActionServer to properly set up inbound connections.

I’m only seeing it in this particular node, and only for inbound. (While waiting for the client to start, we used rostopic echo to create new connections on other topics.)

---

Here are what I think are the relevant bits of the logs:

• Server (/motions)
• Client (/planner) (Note that there was already another client node connected to the server at this time, on the same topics.
• master.log (I removed +PUB, +SUB, publisherUpdate notifications for other topics, but left in all errors)

[djh329]

Was the cause of this ever discovered?

[lindzey]

Not that I ever heard of. We didn't have resources at the time to dig deeper, and I'm no longer with that team.

[lindzey]

No, we weren't using Docker.

…

On Sat, Feb 8, 2020 at 2:55 PM Daniel Hirsch ***@***.***> wrote: Are you running within a Docker container by chance? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#93?email_source=notifications&email_token=AAOVW4SPTATGBLRKZ6MR76DRB4E4TA5CNFSM4EKWRMJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELF2GWI#issuecomment-583770969>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOVW4S7Z62Q7QRG3FLA7DDRB4E4TANCNFSM4EKWRMJA> .

[guiolpei]

Any news on this?

We are experiencing the similar problem. Works perfectly in simulated environment, but delays appear when testing on real setup.

First run works without delays, but from the second time on these same delays occur.

[djh329]

I was able to solve this by adjusting File ULimits. I was running inside Docker. I found the solution here: moby/moby#38814

[gavanderhoorn]

@djh329: so what was the solution? Lowering or raising the limits? From some of the comments on the issue you link it seems lowering would reduce accounting delay, but issues with ulimits are typically 'solved' by raising them.

[guiolpei]

@djh329: Unfortunately we are not using Docker.

[djh329]

We lowered them to 1024 in our case

[guiolpei]

We were defining the SimpleActionClient inside the control loop, causing it to be created and destroyed in each execution.

We managed to solve this problem by defining it as a global variable and keeping it alive during the whole process.