Special handling of "cannot assign requested address" errors may be needed?

[sjmudd]

I noticed in some logging the following errors:

2017-06-12 02:33:09 ERROR dial tcp 10.1.2.3:3306: connect: cannot assign requested address
2017-06-12 02:33:09 CRITICAL dial tcp 1.1.2.3:3306: connect: cannot assign requested address
2017-06-12 02:33:09 ERROR dial tcp 10.1.2.3:3306: connect: cannot assign requested address

This happened for a number of destination mysql servers that orchestrator was expected to discover.

It seems that the cause of this is that I’ve run out of local ports to make a tcp connection.

I am not sure how orchestrator treats this error but this error is NOT a remote server error but a local network (congestion) problem. I think it is likely that orchestrator may consider this a remote server failure, which if incorrect might lead it to try to recover the server when actually it is a local problem. In fact I did notice orchestrator reporting a number of issues last night related to this (output below is from an alerting interface but the output has been somewhat anonymised):

02:34 : orchestrator [critical]: server1 PROBLEM DETECTED: Intermediate master cannot be reached by orchestrator and all of its replicas are unreachable
02:34 : orchestrator [critical]: server2 PROBLEM DETECTED: Master cannot be reached by orchestrator and none of its replicas is replicating
02:34 : orchestrator [critical]: server3 PROBLEM DETECTED: Intermediate master cannot be reached by orchestrator and all of its replicas are unreachable

My thoughts are:

• if this is considered as a remote server problem that should be changed
• it might be convenient to generate a counter which holds the number of times this happens so we can monitor it.
• it may be worth adding a documentation note on this problem and provide some pointers as to what it is and provide suggestions on how to solve it. Probably some generic comments are sufficient.

[sjmudd]

I've added a patch to the init script which mentions this issue. That might help and using /etc/orchestrator_profile can fix my environment.

That said it seems good to avoid triggering false positives with this specific error message and that still needs to be handed.

[sjmudd]

See: also #210 which tries to alleviate this issue in the backend connections.