ESP8266 Doesn't respond to ARP requests

[marrold]

My ESP8266 regularly stops responding to ARP requests. I raised this in #6873 but I was asked to troubleshoot with the Tasmota Devs first (Github Issue) and provide a MCVE example. Tasmota ruled out an issue with their code, but did identify that this may be an interoperability issue between the ESP8266 and Mikrotik access points.

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP8266
• Core Version: 2.6.2
• Development Env: Arduino IDE
• Operating System: MacOS

Problem Description

I have discovered if If wifi_set_sleep_type is light or modem ARP requests will consistently fail. When set to none I reliably get responses to ARP requests. This appears to affect ESP8266 devices connected to Mikrotik Access Points.

ARP responses when sleep type set to none: (1 lost is acceptable IMO)

sudo nping --arp-type ARP 10.0.130.109 -c 20
...
SENT (19.0551s) ARP who has 10.0.130.109? Tell 10.0.130.102
RCVD (19.0663s) ARP reply 10.0.130.109 is at 5C:CF:7F:88:E3:2F
...
Raw packets sent: 20 (840B) | Rcvd: 19 (532B) | Lost: 1 (5.00%)

ARP responses when sleep type set to light:

sudo nping --arp-type ARP 10.0.130.109 -c 20
...
SENT (1.0130s) ARP who has 10.0.130.109? Tell 10.0.130.102
RCVD (1.0276s) ARP reply 10.0.130.109 is at 5C:CF:7F:88:E3:2F
...
Raw packets sent: 20 (840B) | Rcvd: 5 (140B) | Lost: 15 (75.00%)

ARP responses when sleep type set to modem:

sudo nping --arp-type ARP 10.0.130.109 -c 20
...
SENT (1.0130s) ARP who has 10.0.130.109? Tell 10.0.130.102
RCVD (1.0276s) ARP reply 10.0.130.109 is at 5C:CF:7F:88:E3:2F
...
Raw packets sent: 20 (840B) | Rcvd: 3 (84B) | Lost: 17 (85.00%)

MCVE Sketch

#include <ESP8266WiFi.h>

#ifndef STASSID
#define STASSID [YOUR SSID]
#define STAPSK  [YOUR PASSWORD]
#endif

const char* ssid     = STASSID;
const char* password = STAPSK;

void setup() {
  Serial.begin(115200);

  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  wifi_set_sleep_type(NONE_SLEEP_T);
  //wifi_set_sleep_type(LIGHT_SLEEP_T);
  //wifi_set_sleep_type(MODEM_SLEEP_T);

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  Serial.println("I'm alive");
  delay(30000);
}

[TD-er]

In ESPEasy I added a send Gratuitous ARP option to overcome this issue. (hide the symptoms actually)
It is not limited to MikroTik AP's, also with Fritzbox I noticed these issues.

[kugelkopf123]

I can confirm that! (Fritzbox)

_{Sent with GitHawk}

[ascillato]

In ESPEasy I added a send Gratuitous ARP option to overcome this issue. (hide the symptoms actually)
It is not limited to MikroTik AP's, also with Fritzbox I noticed these issues.

@TD-er cool, can you share the snippet of code for that. May be it is worth to add it directly to the core because is needed when using sleep. Thanks

[TD-er]

@ascillato Well it is already shared as you may know ;)
See: https://github.com/letscontrolit/ESPEasy/search?q=gratuitous&unscoped_q=gratuitous

I think this is where most of the magic happens:
https://github.com/letscontrolit/ESPEasy/blob/0c6e6ca915e5440de54213246dc8c1a8e6ca9cce/src/Networking.ino#L15-L39

https://github.com/letscontrolit/ESPEasy/blob/0c6e6ca915e5440de54213246dc8c1a8e6ca9cce/src/Networking.ino#L938-L962

Some of the magic lies in how often I send it.
There is a dynamic interval to send these packets.
This interval does start low and gradually increases over time.
It is sent immediately when some kind of connect attempt fails.
This can be a DNS lookup, a NTP request, or any other wifi client connect attempt.
Such an immediate interfering action does also reset the interval timer to the lowest value.,

The Gratuitous ARP packet is also sent right after the connection to WiFi is fully active (got IP event + few 100 msec)

This is still not a fix for the problem, as you sometimes experience when the first try to connect to a node after some time does take a few seconds.
Especially with my "eco" mode enabled (calling delay() when my scheduler has nothing to do) this problem is more apparent.

[ascillato]

Great, thanks. That is simple to add, but IMHO this should be managed inside the core. What do you think? @d-a-v @devyte @earlephilhower
Is there a chance to be added inside wifi client library? Or must be managed by the sketches/projects ?

[TD-er]

Well, Gratuitous ARP is not a fix for the problem, it is to hide the symptoms.
It is an answer to a question nobody asked, just to make sure all hosts, switches and AP's in the network keep it in their MAC tables.
The real problem is that when the question is asked, it may not be answered and that's core stuff, not part of this repo.

[ascillato]

Searching a bit more, this issue has been discussed before but not added to the core. There are workarounds inside the projects.

#5998

xoseperez/espurna#1877

[ascillato]

The real problem is that when the question is asked, it may not be answered and that's core stuff, not part of this repo.

Exactly. But this only happens if using sleep. So, the SDK don't answer it because the ARP request is being done while the device is sleeping (request outside the DTIM time) ?
Or because the SDK ignores ARP request when it wakes up? I don't know this part of the wifi protocol.

And the weird thing is that this issue is not happening in other routers. Just in mikrotiks and fritzbox AFAIK

[d-a-v]

#6484 made the FW much more stable, but this issue is still beyond our control.

We can add a gratuitous ARP trigger API. I think only STA interface is relevant.
We can make it recurrent and transparently called based on some time interval.
That would help projects not using tasmota/esp-easy-mega/espurna projects that are already implementing gratuitous ARP packets.

[TD-er]

I think only STA interface is relevant.

That's right. With AP mode active the WiFi is not put to sleep.

However, it would be a welcome addition to send out the Gratuitous ARP as soon as a connection is made.
I use wifi events, but as far as I know Tasmota doesn't.
So for those that rely on the reconnect of the core, it is unknown when a reconnect takes place and may be hard to implement it.

[marrold]

Have any of you been able to identify the low level cause of the issue? My guess is some interoperability issue with the power saving negotiation on these vendors.

I plan on sniffing the WiFi later and seeing if I can spot anything of interest.

[TD-er]

I don't think it is limited to these brands (Fritzbox/MikroTik).

But what I do find odd and still have no explanation for is this.
ARP packets (and also UDP packets) may get lost when the ESP is in some kind of sleep mode.
However, a ping will always be answered. It may take up-to 900 msec, for the first answer, but it will be answered.
After the first ping, the ESP will up its current consumption and the replies come in much faster.

This looks like the AP does try to send out the ping packet more often (or the ESP does receive it while dormant?) and packets that don't expect a reply like ARP or UDP will not be attempted again.
If it is indeed the AP that does these re-transmits of the packet, then there may be a difference between vendors on what packets will be re-transmitted and which don't.
Maybe also the AP does know it has some MAC/IP combination connected, so it may answer if the connected device doesn't?

[d-a-v]

Have any of you been able to identify the low level cause of the issue?

Not to my knowledge

I plan on sniffing the WiFi later and seeing if I can spot anything of interest.

Please have a look to #2330 (I hope you'll not see me as evil for suggesting)

#6889 is aimed at anyone concerned with this issue.
Please review

[devyte]

my suspicion is this is the sleep level. Up to sdk 3, sleep level of the wifi can't be set and is fixed at max. That level is not appropriate for sending packets to the esp and then waiting for a response, but is rather meant for the esp initiating communications.
Starting with sdk (pre) 3, the sleep level can be chosen between min and max. Level min is meant for the esp serving.
There was an explanation I saw about this somewhere.

[TD-er]

Up to sdk 3, sleep level of the wifi can't be set and is fixed at max.

I got the feeling the level was not fixed, but somewhat dynamic, based on rules we don't know and can't control.
If you look at the power consumption of the ESP, you see it uses several levels of power consumption, which are somewhat based on the amount of activity in the past N seconds.
For example running transmissions every 5 seconds will result in a higher minimum current compared to only active for a short interval every minute.

Still having set the sleep mode to no sleep is also no guarantee it will remain awake.
If you call a lot of times the delay() function, the power consumption will go down and also the number of missed packets will go up.

[marrold]

ARP packets (and also UDP packets) may get lost when the ESP is in some kind of sleep mode.
However, a ping will always be answered. It may take up-to 900 msec, for the first answer, but it will be answered.

Unicast and Broadcast / Multicast traffic is handled differently by the Access Point as I understand it, which may explain this discrepancy. If I set multicast-helper=full ARP replies are much more reliable. This is a Mikrotik specific setting which converts Broadcast/Multicast traffic to unicast. IMO this is more of a workaround than a fix.

Regarding the Gratuitous ARP suggestion, if this is a fault with the AP vendors then I think this is acceptable. If it's masking an issue within the ESP8266 core then perhaps it's not such a great idea.

[TD-er]

I changed the multicast-helper in my MikroTik AP and it does feel a lot more snappy.
Does such a setting also exist for other AP's, like OpenWRT?

[marrold]

multicast_to_unicast seems to be available for OpenWRT. But it's a work around / bodge that might break other things and I think it means a Broadcast packet is re-transmitted to each client, so there is additional overhead.

I've connected an ESP8266 to an open wifi network and done some packet captures with airodump-ng. I can see the ESP8266 notifying the AP it's going to sleep and periodically waking up and notifying the AP. However the pcap only shows a single frame for a broadcast and doesn't show a retransmission when the ESP8266 wakes up. I suspect this is down to airodump-ng or the parameters I'm using.

If anyone familiar with low level Wifi troubleshooting could give me some hints that would be appreciated.

[marrold]

I've been doing more testing and observed the same issue with missing ARP responses across other devices that are using Wifi Power Save features, namely Huawei P30 and a Samsung Galaxy 9 Android Phones.

If some other Mikrotik users in this thread could test the theory with nping that would be great:
sudo nping --arp-type ARP [IP Address of Phone -c 20

As it stands this doesn't look like an issue with the ESP8266 Arduino Core so will wait for some further feedback and then close.

[TD-er]

Well it may be less of an issue on mobile phones, as they do normally not run services which must be reachable from the network.
Also for TCP, you get an acknowledgement, so it is known whether the device was reached.
For UDP this is more of an issue, but I don't know if UDP is used a lot on mobiles (and for what?)

But since you're sniffing for ARP, can you see if those mobiles send out gratuitous ARP packets?
Can you see a pattern when and how often they send it?

[marrold]

Well it may be less of an issue on mobile phones, as they do normally not run services which must be reachable from the network.

Yeah, this is why I have not noticed before. But regardless, the AP should buffer the ARP requests and re-transmit them when the device wakes up and requests them.

I do not see these Android devices sending Gratiutous ARPs. Interestingly ESPHome devices are also sending Gratiutous ARPs but I cannot see in their code where it's being sent.

[marrold]

I've created a post on the Mikrotik user forum - https://forum.mikrotik.com/viewtopic.php?f=2&t=154613

As this doesn't appear to be an issue in the core I'll close it. Thanks all for your assistance and comments.

In the meantime I think the Gratuitous ARP workaround @d-a-v has submitted in #6889 is probably the best solution for these misbehaving Access Points.

[TD-er]

But regardless, the AP should buffer the ARP requests and re-transmit them when the device wakes up and requests them.

"Should" as in "would be nice" or "according to some standard" ?
I think that it would be a great addition to the standard if it isn't defined right now.
But on the other hand, what is the timeout for reacting on an ARP request?
And what is the TTL of ARP data? Apparently Cisco does use 4 hours as default, but I cannot find what would be a reasonable TTL for this kind of information. It would make sense for an AP to cache it and be the one to react if the node is still connected but not replying to ARP requests it should reply to.

[marrold]

According to some standard (Legacy Power Save / UAPSD ), although I will admit I am not an expert. And there are still unanswered questions like how long it should be buffered for before dropping.

It would make sense for an AP to cache it and be the one to react if the node is still connected but not replying to ARP requests it should reply to.

I did see something that implied Aruba APs would respond to ARP requests on behalf of connected clients but I guess this is vendor specific.

[TD-er]

I was thinking about this issue with ARP for a very specific use case.

For example, you have 2 access points and want to switch to "the other AP".
Since these are separate units in the network, you can be pretty sure they are not connected to the same switch port (and we don't have hubs anymore).

So one second you are connected to AP1 and the switches in the network know where to route packets to for your ESP node.
Now you disconnect and re-connect to AP2.
You send out a DHCP request, but where does the answer end up? Not on the just connected node.

That's exactly what I am seeing in my network, when I intentionally switch to the other AP.

Is there a way to "de-register" any existing ARP cache on the network?
The problem is, you don't have an IP yet, so you can't send out a Gratuitous ARP packet.

[TD-er]

Is there a way to "de-register" any existing ARP cache on the network?
The problem is, you don't have an IP yet, so you can't send out a Gratuitous ARP packet.

Just to quote myself.
I was thinking. The reply to a DHCP request is an unicast reply, which is point-to-point.
So it should announce itself before making the DHCP request, by sending out a gratuitous ARP packet.
But you don't have an IP at that time, so what if we send out a gratuitous ARP request with IP set to 0.0.0.0 ?
Then at least all layer 2 network stuff knows at what port to find the MAC address.
Would that be a workable solution?

[marrold]

You send out a DHCP request, but where does the answer end up? Not on the just connected node

I'm not following. As soon as you send a DHCP request the switch will learn the MAC address.

On the other hand, I don't think you'd send a DHCP request if you're roaming between APs with the same SSID. So you may be right that immediately after roaming the switch will assume you're connected to the old IP until a packet is sent.

I think more enterprise level access points handle this stuff for you.

[TD-er]

When I switch AP, I do perform a disconnect and a connect to the other AP, so there will be a DHCP request.
Also I noticed there is something really fishy happening on the ESP here. It can still be something specific to the MikroTik AP, but every now and then the DHCP does appear to be successful (reply with event of "got IP" is ridiculous fast, sometimes less than 10 msec), but still there is no routing possible until after some time sending out gratuitous ARP packets.
The AP claims the ESP is still connected and it also reports activity, but ping, nor any other IP traffic is possible.
The net-if in the ESP claims to have all IP settings setup, link is up etc.
Only after some time it is possible to get some traffic to and from the ESP.

[d-a-v]

@TD-er
I have noticed the same issue when I switch an ethernet connected computer from one switch to another (in a large network).
I think it is an issue between the switches (or in your case, your APs). They need to talk each other (I don't remember if there is a standard protocol or only a proprietary one) or wait for the expiration of their local ARP table which is falsely trusted.
The computer (or the ESP) can't do anything about it.

[TD-er]

Well I was thinking, what if we send out a gratuitous ARP packet with IP address 0.0.0.0, right before sending out a DHCP request (so right after we get an connection).
Then the switch would already see the ARP packet on another port as expected.

[d-a-v]

Ah right.
Yes, so the loop from #6889 could be that instead:

  for (netif* interface = netif_list; interface != nullptr; interface = interface->next)
         if (
               (interface->flags & NETIF_FLAG_LINK_UP) 
   /////    && (interface->flags & NETIF_FLAG_UP)
#if LWIP_VERSION_MAJOR == 1
            && interface == eagle_lwip_getif(STATION_IF) /* lwip1 does not set if->num properly */
    /////   && (!ip_addr_isany(&interface->ip_addr))
#else
            && interface->num == STATION_IF
   //////   && (!ip4_addr_isany_val(*netif_ip4_addr(interface)))
#endif
        ) { ...

Is that what you tried ?

[TD-er]

Have not tried it with the commented-out code, but I can try for sure (will try this evening, have to pick up my daughter soon)

[beren12]

This has been affecting me on Tasmota with a FreeBSD router OPNsense/PFSense as well. BSD has a 20min arp cache timeout and if it's not refreshed the entry is removed. This is slightly different than linux which seems to keep it but just mark it stale. It's pretty important for networking to have working ARP replies. That the WiFI is actively talking to other machines but not replying to an arp request is a pretty big bug. Is there any chance of this getting fixed?

Tasmota has a workaround to hide it where you can send an arp broadcast on a timer but that doesn't fix it 100%.

@TD-er did you get a chance to test that change?

[beren12]

Also, without a specific arp entry My AP (which is not my router) could not ping my Tasmota light even though it had a valid DHCP lease and an active connection to my MQTT broker

[d-a-v]

and if it's not refreshed the entry is removed

Gratuitous arp should help to solve that. I don't know if it is enabled by default in tasmota.

Tasmota has a workaround to hide it where you can send an arp broadcast on a timer but that doesn't fix it 100%.

Did you try with tcpdump to check these packets are really received ?

[TD-er]

The way I use Gratuitous Arp is a bit more elaborate compared to the fix offered here.
But still it is no guarantee to work.

About the 2nd issue you mentioned, that may be a different bug.
A ping will always work, but it may take a while for the first ping to get a reply... at least with Gratuitous ARP enabled.

[beren12]

A ping doesn't work if there's no arp reply. You can try with sudo nping --arp-type ARP 10.0.0.175 -c 20 and unless you're super lucky that the ESP wakes, it'll never reply even if it's doing network traffic. This violates https://tools.ietf.org/html/rfc894:
"Each host's implementation must know its own Internet address and respond to Ethernet Address Resolution packets appropriately"
and also https://tools.ietf.org/html/rfc826 states
"Periodic broadcasting is definitely not desired" talking about ARP.

When the ESP isn't replying to ARP requests it's not in compliance with the RFCs. What could be done to fix it? It's not in a deep sleep or anything

[devyte]

@beren12 said:

What could be done to fix it?

I thought this arp thing was already addressed since the ets_intr_lock thing. If that's not the case, then the official answer from us core maintainers here in this repo, pending some new hint that hasn't shown up in the already extensive investigation and work already done on this matter, is "reverse engineer Espressif's closed source libraries, starting with the wifi lib".
As I understand the problem, and from what I remember, when the esp8266 stops responding to arp requests, the requests actually never reach lwip (lwip is our own integration, so we have visibility there once packets are passed in from the wifi layer), which is where the arp request is actually processed, the reply is formulated, and then passed back out. That means that either the arp request isn't accepted as valid packets at the wifi level and is ignored, or the wifi layer somehow fails to pass it in to lwip for whatever reason.

The wifi lib is closed source and belongs to Espressif, who won't open source it. So without reverse engineering that code to gain visibility into what's happening, there is no solution.
Or you could try to find some hint of some other possible cause, that is what happened with the ets_intr_lock thing after all, but this has been investigated for a while now, so 🤷‍♂️

[TD-er]

You can also try yourself to switch between SDK builds to see if it makes a difference for your board .
Sadly, there is an unknown difference between SDK builds, which make a difference on some builds and not on others.

I suspect (as in no proof) there is a slight timing difference between nodes.
These timings can be caused by just about anything, like different quality crystals, flash speed, etc.
Even a random factor introduced by linking order at compile time may be attributing. (or there is some other bug why one build is working perfect and another build where a completely unrelated function has changed results in an unusable build, or even compiling on another platform may yield different results)

Anyway, if the node is running normally (with or without sleep state specifically set), the power consumption will differ based on the load of the node. That does not sound very odd, but the ESP's power consumption doesn't really change a lot between computing pi or performing busy waits. The thing that really makes a difference is the WiFi module.
You can test yourself, by running a node on a power supply which shows the current used with at least 10 mA resolution (1 mA resolution is preferred)
After a while, you can see the power consumption will drop significantly when there is little to none WiFi activity.
Just send a ping to it and after the first ping has been replied, the power will go up again to similar amount as when just connected to WiFi.
If you run a lot of calls to delay() the power consumption drops again rather quickly.
In ESPEasy, that's what I do in the scheduler I use, when the "Eco" mode is selected in the settings of ESPEasy. If I know how long it takes to perform the next action, I call delay for max. a few msec.
Apart from lowering power consumption significantly, the node becomes less responsive (even while sending out Gratuitous ARP frequently).

What I think is happening...
When in this lower power mode, the ESP is not waking up on every beacon interval of the access point.
This is normal behavior for all WiFi appliance and is related to the DTIM interval on the access point.
DTIM (Delivery Traffic Indication Map/Message) is an extra message sent after the normal beacon broadcast of the access point.
The general idea is the access point will buffer multicast/broadcast messages until the connected device can collect those messages.
The DTIM interval X, which can be set in some access points, is typically between 1...3, which means those buffered packets are kept for X beacon intervals.
It looks like the ESP is switching its own interval of waking up without taking into account the offered DTIM range of the access point.
For example, if the access point allows for a DTIM of 3, but the ESP does wake up every 4 beacon intervals, then it may miss those ARP requests (and also UDP traffic).
What I'm not 100% sure of, is whether the DTIM interval on the access point means the max. buffer time, or it is actually sending those DTIM messages at that rate. I think it is the latter, so with a DTIM of 3 on the access point and 4 on the ESP, then both will be in sync every 12 beacon intervals. (roughly 1.2 seconds)

Devices entering sleep mode can notify the access point and thus forcing a higher DTIM interval for all connected devices.
If the ESP is missing (or ignoring) those DTIM changes, you may run into issues like these.

Apart from DTIM, there is another metric which may attribute to these issues.
The beacon interval itself. Typically the beacon interval is 100 time units of 1.024 msec (thus 102.4 msec)
This can be different (and sometimes also set by the user) per brand or model of access points.
But the most common setting is 102.4 msec.
If the clock of the ESP is not really accurate, or the ESP is running blocking code, it may miss out on those beacon intervals.
It is very well possible this is one of those settings that make a difference between SDK builds, as it can differ per module what time is needed to start the WiFi and have a stable radio. Depending on the chip/antenna/power supply/etc.
Maybe also the actual beacon interval used by the access point is fluctuating a bit, which can cause some jitter in the timings out of the tolerances used by the ESP to expect a new beacon.

This is what I think, may be factors in this issue.

Edit:
One other note on this.
By turning on the AP mode in the ESP, you effectively force the WiFi radio to be powered on continously and thus also eliminating the chance the radio needs to power on to listen to a new beacon.

[d-a-v]

There is also WiFi.setSleepMode(WIFI_NONE_SLEEP).

[TD-er]

There is also WiFi.setSleepMode(WIFI_NONE_SLEEP).

Which (sadly) does not guarantee the power consumption will not go down after a while.