Understanding lwip / ethernet

[MassiPi]

Hello,
i'm trying to use a W5500 module on esp8266 and i'm struggling to understand structurally what libraries to use.
Example: the connection is "working" just including the <W5500lwIP.h> library that, as i understand, is based directly on lwip.
So why would i need the Ethernet module?
Briefly, i can't understand the relationship between modules, if i can use also Ethernet, EthernetUDP, but maybe also WIFIudp, together with the W5500lwip
The first problem i found is that i could not resolve dns names (inside UDP packages), because based on functions of other modules, but the connection itself is working because i could reproduce the same NTP client with plain lwip functions, so i'd like to know how to use "common" ethernet function, based on W5500lwip module
Thanks!

[JAndrassy]

did you see the example of the W5500lwIP library?

for DNS this could help #8076

[d-a-v]

So why would i need the Ethernet module?

Do you mean "Ethernet Library" which is a driver + a library (I suppose that module = hardware) ?

There are two drivers in this repository. The legacy Ethernet library (that is a submodule pointing to https://github.com/arduino-libraries/Ethernet.git) and the other one compatible with lwIP.
With the first, esp8266 specific network goodies are not available.
With the second, which appeared in release 3.0.0, everything works just like with the WiFi interface (dhcp, ntp, webserver, webclient, mdns, etc and of course WiFiClient, WiFiClientSecure and WiFiUdp classes). This driver-library is at early stage and some adjustments may be needed as @JAndrassy states above.

so i'd like to know how to use "common" ethernet function, based on W5500lwip module

The two drivers are not inter-operable. It's one or the other.
Legacy Ethernet library embeds a the driver and an independent network library, they are not lwIP-aware.
The other is only a driver, and the library is already here because it is shared with what's already here for WiFi, lwIP being the common glue.

[MassiPi]

Do you mean "Ethernet Library" which is a driver + a library (I suppose that module = hardware) ?

Yes, sorry, you got it right :)

There are two drivers in this repository. The legacy Ethernet library (that is a submodule pointing to https://github.com/arduino-libraries/Ethernet.git) and the other one compatible with lwIP.

Thanks A LOT with the description of the differences between the Ethernet and the lwIP. Indeed, with the latter i could use async webserver and OTA, but i got stuck with EZtime because, i found out later, the NTP part was not working.
I tried to look just at that and i found out that with wifiudp the WiFi.hostByName() was not giving a result (blocking the beginPacket function for seconds), but it was working if passing directly the ip.
But as far as i have understood, the fix linked by @JAndrassy is related only with the manual config of the connection, where on the other side if i access the DNS server of the connection with lwip function (dns_getserver(0)) i get the correct ip (dhcp passed).
btw, i really could not get this information, as clear as you gave me, in the last week, so thanks a lot!
But now i can't understand why i can't get WifiUDP to work.
i hope i did something wrong, but if i got it right i should just initialize the W5500 lwip connection, set it as default and everything should work.. and i did! or at least, i tried :) But this script is not working (not resolving the address..)

#include <SPI.h>
#include <WiFiUdp.h>
#include <W5500lwIP.h>
#include "lwip/dns.h"

byte mac[] = {0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x02};

unsigned int localPort = 8888;       // local port to listen for UDP packets
const char timeServer[] = "time.nist.gov"; // time.nist.gov NTP server
const int NTP_PACKET_SIZE = 48; // NTP time stamp is in the first 48 bytes of the message
byte packetBuffer[NTP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets

// A UDP instance to let us send and receive packets over UDP
Wiznet5500lwIP eth(D1);
WiFiUDP Udp;
ip_addr_t ip_Addr;

void dns_found_cb(const char *name, const ip_addr_t *ipaddr, void *callback_arg)
{
  Serial.print("Name ");
  Serial.println(name);
  Serial.print("ip ");
  Serial.println(ip4addr_ntoa(ipaddr));
}

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  SPI.begin();
  SPI.setBitOrder(MSBFIRST);
  SPI.setDataMode(SPI_MODE0);
  SPI.setFrequency(4000000);

  eth.setHostname("TEST");
  eth.setDefault(); // use ethernet for default route
  int present = eth.begin(mac);
  if (!present) {
    Serial.println("no ethernet hardware present");
    while(1);
  }
  
  Serial.print("connecting ethernet");
  while (!eth.connected()) {
    Serial.print(".");
    delay(1000);
  }
  
  Serial.println("Connected");

  Serial.print("Starting udp receiver with code ");
  Serial.println(Udp.begin(localPort));
}

void sendNTPpacket(const char * address);

void loop() {
  //dns_gethostbyname(timeServer, &ip_Addr, dns_found_cb, NULL );
  //Serial.println(ip4addr_ntoa(&ip_Addr));
  //Serial.println(ip4addr_ntoa(dns_getserver(0)));

  delay(1000);

  Serial.print("Sending packet @ millis ");
  Serial.println(millis());
  sendNTPpacket(timeServer); // send an NTP packet to a time server
  //sendNTPpacket(ip4addr_ntoa(&ip_Addr));
  Serial.print("Sent: ");
  Serial.println(millis());
  // wait to see if a reply is available
  delay(1000);
  if (Udp.parsePacket()) {
    Serial.println("Received packet");
    // We've received a packet, read the data from it
    Udp.read(packetBuffer, NTP_PACKET_SIZE); // read the packet into the buffer

    // the timestamp starts at byte 40 of the received packet and is four bytes,
    // or two words, long. First, extract the two words:
    unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
    unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);
    // combine the four bytes (two words) into a long integer
    // this is NTP time (seconds since Jan 1 1900):
    unsigned long secsSince1900 = highWord << 16 | lowWord;
    Serial.print("Seconds since Jan 1 1900 = ");
    Serial.println(secsSince1900);

    // now convert NTP time into everyday time:
    Serial.print("Unix time = ");
    // Unix time starts on Jan 1 1970. In seconds, that's 2208988800:
    const unsigned long seventyYears = 2208988800UL;
    // subtract seventy years:
    unsigned long epoch = secsSince1900 - seventyYears;
    // print Unix time:
    Serial.println(epoch);

    // print the hour, minute and second:
    Serial.print("The UTC time is ");       // UTC is the time at Greenwich Meridian (GMT)
    Serial.print((epoch  % 86400L) / 3600); // print the hour (86400 equals secs per day)
    Serial.print(':');
    if (((epoch % 3600) / 60) < 10) {
      // In the first 10 minutes of each hour, we'll want a leading '0'
      Serial.print('0');
    }
    Serial.print((epoch  % 3600) / 60); // print the minute (3600 equals secs per minute)
    Serial.print(':');
    if ((epoch % 60) < 10) {
      // In the first 10 seconds of each minute, we'll want a leading '0'
      Serial.print('0');
    }
    Serial.println(epoch % 60); // print the second
  } else {
    Serial.println("Nothing..");
  }
 }

// send an NTP request to the time server at the given address
void sendNTPpacket(const char * address) {
  // set all bytes in the buffer to 0
  memset(packetBuffer, 0, NTP_PACKET_SIZE);
  // Initialize values needed to form NTP request
  // (see URL above for details on the packets)
  packetBuffer[0] = 0b11100011;   // LI, Version, Mode
  packetBuffer[1] = 0;     // Stratum, or type of clock
  packetBuffer[2] = 6;     // Polling Interval
  packetBuffer[3] = 0xEC;  // Peer Clock Precision
  // 8 bytes of zero for Root Delay & Root Dispersion
  packetBuffer[12]  = 49;
  packetBuffer[13]  = 0x4E;
  packetBuffer[14]  = 49;
  packetBuffer[15]  = 52;

  // all NTP fields have been given values, now
  // you can send a packet requesting a timestamp:
  Udp.beginPacket(address, 123); // NTP requests are to port 123
  Udp.write(packetBuffer, NTP_PACKET_SIZE);
  Udp.endPacket();
}

But if I uncomment the external resolving of the dns name, it works.
What am i doing wrong?

And please, how the hell should i change the hostname?? i tried putting the eth.setHostname("NAME") almost EVERYWHERE in the script, no result at all :)

Thanks a lot!

[Villy-web]

And please, how the hell should i change the hostname?? i tried putting the eth.setHostname("NAME") almost EVERYWHERE in the script, no result at all :)

Did you manage to solve this?

[MassiPi]

I'm having a hard time with this.. I'm trying to see what's under, but i really can't understand why i can't call a simple WiFi.hostByName() (always error -5), but i can call directly the dns_gethostbyname() THAT IS USED BY THE FIRST!

Simpler script (the problem is just dns, the udp part is not needed):

#include <ESP8266WiFi.h>
#include <SPI.h>
#include <W5500lwIP.h>
#include "lwip/dns.h"

byte mac[] = {0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x02};

const char dnsHostOne[] = "google.com"; // time.nist.gov NTP server
const char dnsHostTwo[] = "google.us"; // time.nist.gov NTP server
const char dnsHostThree[] = "google.co.uk"; // time.nist.gov NTP server

Wiznet5500lwIP eth(D1);
ip_addr_t ip_Addr;

void dns_found_cb(const char *name, const ip_addr_t *ipaddr, void *callback_arg)
{
  Serial.print("Name ");
  Serial.println(name);
  Serial.print("ip ");
  Serial.println(ip4addr_ntoa(ipaddr));
}

void setup() {
  WiFi.mode(WIFI_OFF);
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  SPI.begin();
  SPI.setBitOrder(MSBFIRST);
  SPI.setDataMode(SPI_MODE0);
  SPI.setFrequency(4000000);

  eth.setDefault(); // use ethernet for default route
  eth.setHostname("TEST"); //THIS ALWAYS FAILS
  int present = eth.begin(mac);
  if (!present) {
    Serial.println("no ethernet hardware present");
    while(1);
  }
  
  Serial.print("connecting ethernet");
  while (!eth.connected()) {
    Serial.print(".");
    delay(100);
  }
  
  Serial.println("Connected");
}

void loop() {
  Serial.println("Starting First.");
  while ( dns_gethostbyname(dnsHostOne, &ip_Addr, dns_found_cb, NULL ) != ERR_OK ){
    delay(1);
  }
  Serial.print("First over: ");
  Serial.println(ip4addr_ntoa(&ip_Addr));
  Serial.println("Starting Second.");
  while ( dns_gethostbyname_addrtype(dnsHostTwo, &ip_Addr, dns_found_cb, NULL, LWIP_DNS_ADDRTYPE_DEFAULT ) != ERR_OK ){
    delay(1);
  }
  Serial.print("Second over: ");
  Serial.println(ip4addr_ntoa(&ip_Addr));

  IPAddress tempAddress;
  Serial.println("Direct lookup: ");
  WiFi.hostByName(dnsHostThree, tempAddress);
  Serial.println(ip4addr_ntoa(tempAddress));

  delay(5000);
}

Debug info (well.. all i could get) (and setting hostname is always impossible for me..)

WiFi.hostname(TEST): wifi_station_set_hostname() failed
connecting ethernet..Connected
Starting First.
Name google.com
ip 142.250.180.110
Name google.com
ip 142.250.180.110
First over: 142.250.180.110
Starting Second.
Name google.us
ip 142.250.184.100
Second over: 142.250.184.100
Direct lookup:
[hostByName] request IP for: google.co.uk
[hostByName] Host: google.co.uk lookup error: -5!
255.255.255.255

Using platformio with upstream version of esp8266 arduino core

Any hint? How can i debug this further?
Thanks!

[d-a-v]

Can you try this:

diff --git a/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp b/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp
index 19da592c..aa8f183d 100644
--- a/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp
+++ b/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp
@@ -621,7 +621,8 @@ int ESP8266WiFiGenericClass::hostByName(const char* aHostname, IPAddress& aResul
         aResult = IPAddress(&addr);
     } else if(err == ERR_INPROGRESS) {
         _dns_lookup_pending = true;
-        delay(timeout_ms);
+        for (int i = 0; i < timeout_ms && _dns_lookup_pending; i++)
+            delay(1);
         // will resume on timeout or when wifi_dns_found_callback fires
         _dns_lookup_pending = false;
         // will return here when dns_found_callback fires

[d-a-v]

Reference: #6212 (comment)
This PR from @emelianov should have been updated and integrated into the core to solve that kind of issue, but it has been hindered on its way.
I hope it will be fixed from inside the core soon.

[MassiPi]

Sorry, i was not able to find that post..
i'll try the modification as soon as possible (later today, i hope), at least it seems it is not just me going crazy :)

[MassiPi]

Hello,
the first proposal is only partially working, since the first lookup in any case takes up to timeout_ms
I then found out that the linked comment is different, but in the meantime i got to the same result adding in the callback an explicit assignment as "false" to the pending variable (isn't this logic? the callback is called as the lookup is no more pending..)

void wifi_dns_found_callback(const char *name, const ip_addr_t *ipaddr, void *callback_arg)
{
    (void) name;
    if (!_dns_lookup_pending) {
        return;
    }
    if(ipaddr) {
        (*reinterpret_cast<IPAddress*>(callback_arg)) = IPAddress(ipaddr);
		_dns_lookup_pending = false;
    }
    esp_schedule(); // break delay in hostByName
}

Btw yes, thank you a lot, this solves the issue!
I assume we'll need to add the same non-blocking delay also @ line 674 of the same file..

Edit: ok, i found the PR you were referring to, it's perfect :)

[d-a-v]

Can you try #8317 which aims at replacing #6212 ?