Arduino esp8266 build fails

[Pfannex]

Hi,

I'm using the Arduino IDE 1.8.7 with the latest esp8266 Version 2.4.2 including lwIP2.
Board: WeMos D1 R1

I tried to compile this minimal sketch:

#include "ESP8266WiFi.h"
#include <w5500-lwIP.h>
void setup() {}
void loop() {}

build fails with:

Arduino: 1.8.2 (Windows 10), Board: "WeMos D1 R1, 80 MHz, Flash, 4M (1M SPIFFS), v2 Lower Memory, Disabled, None, Only Sketch, 921600"

In file included from C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/lwip/arch.h:48:0,

                 from C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/lwip/debug.h:40,

                 from C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/lwipopts.h:3004,

                 from C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/lwip/opt.h:51,

                 from C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/lwip/netif.h:40,

                 from C:\Users\pf\Documents\Arduino\libraries\W5500lwIP\w5500-lwIP.cpp:8:

C:\Users\pf\Documents\Arduino\libraries\W5500lwIP\w5500-lwIP.cpp: In member function 'err_t Wiznet5500lwIP::start_with_dhclient()':

C:\Users\pf\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.2/tools/sdk/lwip2/include/arch/cc.h:124:24: error: 'ethernet_input_LWIP2' was not declared in this scope

 #define ethernet_input ethernet_input_LWIP2

                        ^

C:\Users\pf\Documents\Arduino\libraries\W5500lwIP\w5500-lwIP.cpp:53:66: note: in expansion of macro 'ethernet_input'

     if (!netif_add(&_netif, &ip, &mask, &gw, this, netif_init_s, ethernet_input))

                                                                  ^

exit status 1
Fehler beim Kompilieren für das Board WeMos D1 R1.

How can I compile with lwip2?

regards
Pf@nne

[borisneubert]

Same issue here (minimal sample project referenced there).

David, you probably have a working project. Would you mind sharing the code with us?

Thank you and kind regards,
Boris

[d-a-v]

Will answer later today. Thanks for your interest.
This library should be moved directly into the core don't you think ?

[d-a-v]

Fix pushed

[borisneubert]

Thank you. I confirm successful compilation. Still need to test with actual hardware.

Benefits of library being in the core:

• W5500 module is preferred ethernet adapter (raw driver should made use of full potential of the chip to offload CPU, not only PHY layer)
• Being part of the standard greatly widens potential user base which is useful for getting feedback on issues and enhancements.
• We prefer standard over non-standard libraries as they tend to be more likely maintained for a long time

Thus yes, library should be moved to core.

[Pfannex]

Hi David,

thanks for fixing!

Can you give us a short example code?
We are using a ESP8266 WeMos D1 R1with a W5500-board like this:
https://www.ebay.de/itm/W5500-Ethernet-Network-Modul-TCP-IP-51-STM32-SPI-Schnittstelle-Fur-Arduino-New/272461200469?hash=item3f6ff3fc55:g:p~AAAOSwB09YNpwB&redirect=mobile

Please share also the wire connection with us, especially the CS pin.
We use GPIO4(D2) for CS. Because of the PullUp/GPIO15 problem.

Regards
Pf@nne

[d-a-v]

I use the same board ethernet board and wemos D1 mini lite.
I use D0 for CS.

Here's a stripped example.

// D0 CS SDCARD
// D8 CS ethernet
// D3 MOS
// D2 INPUT WATER
// D4 1-WIRE temp

#include <ESP8266WiFi.h>

#include <w5500-lwIP.h>

Wiznet5500lwIP eth(D0);

void setup ()
{
  // setup code
  // ...

  eth.setDefault(); // use ethernet for default route
  eth.begin(); // default mtu & mac address
}

void loop ()
{
  eth.loop(); // call this at least once per loop and around delay()/yield() calls

  // user code
}

[Pfannex]

Hi David,

thanks for your simpleCode, your code works well with the W5500.......
W5100 doesn´t work, begin() returns 0.

#include <Arduino.h>
#include "ESP8266WiFi.h"
#include <w5500-lwIP.h>

// Arduino Pin 4 = Wemos Pin D2
#define CSPIN 4

Wiznet5500lwIP eth(CSPIN);
byte mac[] = {0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x02};

void setup() {

  Serial.begin(115200);
  Serial.println("");
  Serial.println("start");

  eth.setDefault(); // use ethernet for default route
  int present = eth.begin(mac);
  Serial.println("");
  Serial.println("present= " + String(present, HEX));

  while (!eth.connected()) {
    eth.loop();
    Serial.print(".");
    delay(1000);
  }
}

void loop() {
   eth.loop();
   Serial.println(eth.localIP());
   delay(1000);
}

[Pfannex]

Hi David,

we are not able to bring a simple NTPClient or WebServer to work. We get an IP address but the NTPClient or WebServer do not connect/cannot be contacted as if the did not bind to the interface.

Would you mind providing a simple example for that, please?
Like this: https://github.com/blackketter/NTPClient

Regards,
Pf@nne

[borisneubert]

I guess the issue is that either the services do not bind to the interface (although they bind to IPANY in their code) or that some additional define is missing besides PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY

[d-a-v]

There are number of small issues to solve.
One of them is that incoming packets are processed when ::loop() is called, so

• it must be called quite often
• this is totally not compatible with delay() in loop() which is the usual arduino way
  (and the NTPCLient example)

So we should be using some sort of timer to call ::loop() or it should automatically be called when yield() is called (that would be after integration into the arduino core).

Before that, we do what we can, NTPClient example could then be:

#include <NTPClient.h>
// change next line to use with another board/shield
#include <ESP8266WiFi.h>
//#include <WiFi.h> // for WiFi shield
//#include <WiFi101.h> // for WiFi 101 shield or MKR1000
#include <WiFiUdp.h>
#include <w5500-lwIP.h>

#define WIFI 0

#if WIFI
const char *ssid     = "SSID";
const char *password = "PSK";
#endif

Wiznet5500lwIP eth(D0); // <- use your CS pin
WiFiUDP ntpUDP;

// By default 'time.nist.gov' is used with 60 seconds update interval and
// no offset
NTPClient timeClient(ntpUDP);

// You can specify the time server pool and the offset, (in seconds)
// additionaly you can specify the update interval (in milliseconds).
// NTPClient timeClient(ntpUDP, "europe.pool.ntp.org", 3600, 60000);

unsigned long sectest = 0;
unsigned long now = 0;
bool everysec ()
{
  now = millis();
  if ((now - sectest) > 1000)
  {
    sectest = now;
    return true;
  }
  return false;
}


void setup() {
  Serial.begin(115200);

#if WIFI
  WiFi.begin(ssid, password);

  while ( WiFi.status() != WL_CONNECTED ) {
    delay ( 500 );
    Serial.print ( "." );
  }
#else
  WiFi.mode(WIFI_OFF);
#endif

  if (eth.begin())
  {
    eth.setDefault();
    Serial.println("eth started");
    while (!eth.connected())
    {
      eth.loop();
      if (everysec())
        Serial.print(".");
    }
    Serial.println();
  }
  else
    Serial.println("could not start ethernet");
  Serial.println("starting");
  timeClient.begin();
}

void loop() {
  eth.loop();
  if (everysec()) //   delay(1000);
  {
    timeClient.update();

    Serial.print(timeClient.getFormattedTime());
    Serial.print(" eth-conn=");
    Serial.print(eth.connected());
    Serial.print(" eth-ip=");
    Serial.print(eth.localIP());
    Serial.print(" wifi-ip=");
    Serial.println(WiFi.localIP());
  }
}

[borisneubert]

Thank you very much, David, for looking into this and for providing the example. I still have to look at it in detail.

My W5500 arrived by mail order. Thus, yesterday night, I extended my example code.

The AsyncWebserver example worked right out of the box and is reachable both via WiFi and Ethernet.

NTP client example did not work right away and now does only with the IP address of the NTP server, not with its FQDN. I will rewrite it according to your example, though.

I monitored network traffic with wireshark yesterday. It looks like I am missing some packets in wireshark (I see no name resolution). At least I see issues with ARP. Ethernet has difficulties to get the MAC address of the gateway, sending ARP requests 5 times but receiving no response. I particular, the overall process hangs for seconds, which coincides with 4 to 5 seconds latencies of ping. This is compatible with your reasoning that eth.loop() is not called as often as required. This is like fishing in murky waters.

What can I do to foster integration of the library into the core?

Is it instead reasonable to do it in a minimal fork of the relevant Arduino Iibrary if the integration into the core does not happen or takes too much time?

Cheers
Boris

[borisneubert]

@d-a-v We created a full real-world example with AsyncWebserver, NTP Client and MQTT Client. Observations:

• AsyncWebserver + NTP Client + MQTT Client work over WiFi (with Ethernet turned off)
• AsyncWebserver + NTP Client work over WiFi plus Ethernet
• AsyncWebserver + NTP Client + MQTT Client do not work at all if WiFi and Ethernet are both turned on, no matter if Ethernet is the default interface or not

This is likely due to Ethernet getting stuck if the MQTT Client tries to connect without calling eth.loop(). This is consistent with your considerations about ::loop() to be called with yield().

We now have a real-world example for testing a future version of the Arduino library with integrated W5500 ethernet.

We are keen to support the integration into the core with testing. @Pfannex is experienced with low-level device programming and could support the addition of status functions and the like into the W5500 device class.

@d-a-v How do you like to proceed and how can we help?

[d-a-v]

Here's what I propose for a try:

Add this inside core/esp8266/coredecls.h:

void eth_loop (void);

Change core/esp8266/core_esp8266_main.cpp ~L97

extern "C" void __yield() {
    if (cont_can_yield(g_pcont)) {
        esp_schedule();
        esp_yield();
    }
    else {
        panic();
    }
}

to

extern "C" void __yield() {
    if (cont_can_yield(g_pcont)) {
        esp_schedule();
        esp_yield();
        eth_loop();   // <--- addition
    }
    else {
        panic();
    }
}

And somewhere in your sketch, code eth_loop:

extern "C" void eth_loop ()
{
    // first, check if ethernet is initialized,
    // then call your eth.loop()
}

The cont_can_yield() test ensures the call is not done while being in an ISR.
With this, I guess you can try and comment out all your previous calls to eth.loop().

Thanks for your reports and for helping this project !

[Pfannex]

Hi David,

thanks for your quick response!
I added your changes to the core:

• W5500-example.piolibdeps\Arduino\cores\esp8266\coredecls.h
• W5500-example.piolibdeps\Arduino\cores\esp8266\core_esp8266_main.cpp

Unfortunately this got no effect.

What does

extern "C" void eth_loop ()
{
    // first, check if ethernet is initialized,
    // then call your eth.loop()
}

mean?

You can find our PlatformIO repo here:
https://github.com/borisneubert/W5500-example/tree/MQTT

[borisneubert]

@Pfannex Just start with

#define CSPIN 4
Wiznet5500lwIP eth(CSPIN);
int present = 0;
...
present = eth.begin();

and put

extern "C" void eth_loop ()
{
    if(present) eth.loop();
}

somewhere in your sketch such that it get linked into the executable and gets called from __yield().

[d-a-v]

@Pfannex Is this now working ?

[Pfannex]

@d-a-v ,

sorry, we can't give you a positive response......but we have some questions.
For our development we are using WIN10 with ATOM / platformIO.

1. add eth_loop and call it from the core
   First we searched for the right directory which holds the core files.
   We found 2 possible directories:

   • platformIO home
     C:\Users\pf.platformio\packages\framework-arduinoespressif8266\cores\esp8266\
   • platformIO Sketch repository
     h:..\W5500-example.piolibdeps\Arduino\cores\esp8266\

2. We find out, that the platformIO home directory includes the files that will be used for compiling our sketch. We don't really know why the core also exists in our sketch repo.
   After looking into the files we also ascertain that they had different code.

platformIO home - coredecls.h

#ifndef __COREDECLS_H
#define __COREDECLS_H

#ifdef __cplusplus
extern "C" {
#endif

// TODO: put declarations here, get rid of -Wno-implicit-function-declaration

extern bool timeshift64_is_set;

void tune_timeshift64 (uint64_t now_us);
void settimeofday_cb (void (*cb)(void));
void eth_loop (void);

#ifdef __cplusplus
}
#endif

#endif // __COREDECLS_H

platformIO Sketch repository - coredecls.h

#ifndef __COREDECLS_H
#define __COREDECLS_H

#ifdef __cplusplus
extern "C" {
#endif

// TODO: put declarations here, get rid of -Wno-implicit-function-declaration

#include <stdint.h>
#include <cont.h> // g_pcont declaration

extern bool timeshift64_is_set;

void esp_yield();
void esp_schedule();
void tune_timeshift64 (uint64_t now_us);
void settimeofday_cb (void (*cb)(void));
void disable_extra4k_at_link_time (void) __attribute__((noinline));

uint32_t sqrt32 (uint32_t n);

#ifdef __cplusplus
}
#endif

#endif // __COREDECLS_H

3. After adding your code the new compiler did not found the declaration void eth_loop (void); in coredecls.h, so we declared the function directly in core_esp8266_main.cpp.

So far the code is compilable but does not work as expected.
We expected, that eth_loop() in our sketch is called as often as possible by the core. Actual eth_loop() is called sporadic and nor very often.

Finally we need to find out why we got different versions of the core files and why eth_loop()is called sporadic and nor very often.

[Pfannex]

While showering I knew why we have two core directories. The sketch internal core is loaded by platformIO because we listed it in the platformio.ini.
This Core must be actual because it will be automatically refreshed.

[common]
lib_deps =
  https://github.com/esp8266/Arduino

Thus it looks our Arduino ESP8266-core platformIO core is not up to date.
That might be also the reason that the declaration in coredecls.h is not found.

I try to update the core and test it again.

So stay tuned.....

[d-a-v]

platformIO Sketch repository - coredecls.h

This one is more up to date than the first.

I went to watch PubSubCLient sources that you use.

Whereas there is a yield() in that line inside a while loop:
https://github.com/knolleary/pubsubclient/blob/26ce89fa476da85399b736f885274d67676dacb8/src/PubSubClient.cpp#L225

There is no yield() call in this while loop:
https://github.com/knolleary/pubsubclient/blob/26ce89fa476da85399b736f885274d67676dacb8/src/PubSubClient.cpp#L192

Maybe you can try and add that call there and see what happens.
It is important because is seems to be waiting for a request sent just above. So not calling eth_loop() / yield() specifically there will prevent packets to be received.

[d-a-v]

For the record, the second loop calls client->connected() which calls optimistic_yield() which calls yield() but I'm not sure how often (and I suspect only once until the main arduino loop() ends).
So yield() in the second loop above is worth a try.

You can also add a Serial.print('something') or a digitalWrite(1-digitalRead(LED_BUILTIN)) in your eth_loop() and check whether blink-alive is not dead.

[Pfannex]

Hi David,

after installing the latest core I add eth_loop() to esp8266/core_esp8266_main.cpp. The declaration in coredecls.h causes a error.
So we moved the declaration also to esp8266/core_esp8266_main.cpp:

extern "C" void call_user_start();
extern void loop();
extern void setup();
extern void (*__init_array_start)(void);
extern void (*__init_array_end)(void);
extern "C" void eth_loop (void);

I also add a "print" into the eth_loop().

extern "C" void eth_loop (){
    // first, check if ethernet is initialized,
    // then call your eth.loop()
    //Serial.println("eth.connected()");
    //eth.loop();

  if (present){
    Serial.println("eth.loop()");
    eth.loop();
  }
}

For the moment MQTT is still disabled and all eth.loop() in our Code were removed.
The modification should keep the eth alive, but it did not.
The AsyncWebServer is only reachable after reinsert a eth.loop() into the main loop().

Printing the "eth.loop()" occurs still very sporadically.
So this solution doesn't seem to work.

I think we need to find a way to call the eth.loop() as quite often as possible.

[Pfannex]

While testing the "plug-flag" I noticed that a few seconds after disconnecting the eth plug eth_loopwas called quite often.

[d-a-v]

Then we would use a timer like in this example

datasheet:

Function Enable a millisecond timer.
Prototype void os_timer_arm (
os_timer_t *ptimer,
uint32_t milliseconds,
bool repeat_flag
)
os_timer_t *ptimer: timer structure.
uint32_t milliseconds: timing; unit: millisecond.

If system_timer_reinit has been called, the timer value allowed ranges from 100
to 0x689D0.
If system_timer_reinit has NOT been called, the timer value allowed ranges from
5 to 0x68D7A3.
bool repeat_flag: whether the timer will be invoked repeatedly or not.
Return
none

(note: system_timer_reinit() is not called)

[Pfannex]

After implementing the timer and adding the yield() @L192, the MQTT.Client was able to connect to the MQTT-Broker.

So far a step into the right direction....!

But the MQTT.client doesn`t work clean.
The client dissconect every 3 seconds and reconnects successfully. Disconnecting and connecting happend very fast so the MQTT.client works as exspected.

At all this can not be the final solution.

For me, it looks like the TCP-connection to the MQTT-Broker is disconnecting after 3 seconds.

Can we run further tests?

[borisneubert]

I experimented with the timer variant as well but was not so bold as to call eth.loop() more often than every 100 ms. 10 calls per second are not sufficient: getting an Pv4 address from the DHCP server takes very long and in most cases forever, NTP and MQTT clients do not connect at all.

Increasing the frequency up to one call every millisecond yielded the following results:

• I immediately get an IPv4 address from the DHCP server.
• NTP client immediately gets time from NTP server.
• ASyncWebServer works.
• MQTT loses connection every 3 seconds and then reconnects, as @Pfannex observed.

Polling the ethernet every millisecond seems excessive to me. I wonder if one was not better off with using the hardwired IP stack in the Wiznet W5500 module and the TX and RX buffers to offload the work to the chip. There is a wealth of information at the Wiznet W5500 web page, including the official library at github and a Wiki.

[d-a-v]

I wonder if one was not better off with using the hardwired IP stack in the Wiznet W5500 module

The Ethernet (edit:+TCP/IP) driver is also already in the esp8266/arduino core, for the W5100. It is possibly compatible with W5500.
(but this is not the purpose of this repository)

MQTT loses connection every 3 seconds and then reconnects, as @Pfannex observed.

Calling the loop too often is not good - I agree - but at least would make everything work, so I need to run tests myself too, so I can watch what's hapenning.

There's also the interrupt pin that could be used (wired + software-honored).

[borisneubert]

Calling the loop too often is not good - I agree - but at least would make everything work, so I need to run tests myself too, so I can watch what's hapenning.

Thank you, David. For starters, the repository W5500-example contains the latest sketch from our experiments in the MQTT branch, corresponding to previous report.

[Pfannex]

If needed please use our MQTT-branch

[borisneubert]

I wonder if one was not better off with using the hardwired IP stack in the Wiznet W5500 module

The Ethernet (edit:+TCP/IP) driver is also already in the esp8266/arduino core, for the W5100. It is possibly compatible with W5500.
(but this is not the purpose of this repository)

Maybe we are looking in the wrong place anyway? Our use case is to have built-in WiFi adapter and external W5500 LAN adapter sitting in the same TCP/IP stack such that higher-level services and protocols (FTP, NTP, MQTT, HTTP) can be accessed via both interfaces. We had the experience that the Arduino libraries allow to either use WiFi or Ethernet as two completely separate TCP/IP stacks. But I might be mistaken.

[andrethomas]

How is lwip currently integrated into the Arduino core - I'm guessing its not the one coming with the NONOSSDK so if not then we can add to it not so?

[d-a-v]

https://github.com/esp8266/Arduino/tree/master/tools/sdk/lwip2

[d-a-v]

Please see last commit