2.0.3 RC1 issues with ESPAsyncWebServer Processor #6502

dsilletti · 2022-03-30T19:55:49Z

Board

ESP32 Dev Module

Device Description

DevKitC

Hardware Configuration

SPI connected Display

Version

latest master

IDE Name

Arduino IDE

Operating System

Mac OS 12.3

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

115200

Description

after the upgrade to 2.0.3 RC1 the ESPAsyncWebServer Processor is not working as expected, is cutting the output in more points without any placeholders involved. Downgrading to 2.0.2 fix the issue.

I was already using the github master repo till approximately one month ago with no issues, I upgraded today to the latest master and the same code is not working anymore. Downgrading to 2.0.2 fix the issue.

Sketch

server.on("/status", HTTP_GET, [](AsyncWebServerRequest *request){
    if(!request->authenticate(http_username, http_password)) {
      return request->requestAuthentication();
    }
    request->send(LittleFS, "/status.html", String(), false, processor);
}).setFilter(ON_STA_FILTER);

Output expected (as per static html file served)

<select name="selectedTA" id="selectedTA">
  <option value="0">100A - 50mA</option>
  <option value="1">250A - 0.1A</option>
  <option value="2">120A - 40mA</option>
  <option value="3">400A - 0.1A</option>
</select>

Output produced

<select name="selectedTA" id="selectedTA">
   <option value="0">100A - 50mA</option>
   <option value="1">250A - 0.1A</option>
   <option value="2">120A - 40mA</option>
   <option value="3a.append("timezone", timezone);
   data.append("tz", tz);
   ... (cutted at least 20-30 lines with no reason after value="3)

Other Steps to Reproduce

try to serve a web page (4-5Kb) with processor (in my case from LittleFS), it should cut parts of the output in one or more point, with no placeholders involved in that points.

I have checked existing issues, online documentation and the Troubleshooting Guide

I confirm I have checked existing issues, online documentation and Troubleshooting guide.

The text was updated successfully, but these errors were encountered:

dsilletti · 2022-03-30T21:01:36Z

I tried to revert to previous commits to find the issue.

I found the commit that is breaking the code:
Edited VFSFileImpl::read to use both read/fread 7b89b39

please fix it or remove it.

reverting to the previous commit 905f8f2 Warns about SSP only available for ESP32 (#6455) fixed finally the issue and the web page is served full.

me-no-dev · 2022-03-30T21:04:57Z

@P-R-O-C-H-Y take a look please :)

@dsilletti thanks for digging in! Will get fixed!

Jason2866 · 2022-03-31T15:20:13Z

We run in issues to. See #6507 Removing this PR did solve too.

P-R-O-C-H-Y · 2022-04-04T15:04:09Z

Hi @dsilletti,
Can you provide for me sketch that I can just easily run? To test your case. The more test cases I have the better it is to fix the bug. Thanks

dsilletti · 2022-04-05T09:19:43Z

Hi @P-R-O-C-H-Y,
basically my sketch is using EspAsyncWebserver to serve html files from the LITTLEFS, using also the processor to replace some placeholders with some strings.
I think the problem is with reading the file, the output most of the time is missing large chunks, even in multiple parts of the html file, the missing chunks are always the same (probably based on the buffer size you configured in the PR).
Only 10-20% of the requests are successful with full html served.
I will try to provide a short sketch that does just this, with an example html page to upload to the SPIFFS/LITTLEFS partition.

P-R-O-C-H-Y · 2022-04-05T09:34:29Z

Hi @P-R-O-C-H-Y,
basically my sketch is using EspAsyncWebserver to serve html files from the LITTLEFS, using also the processor to replace some placeholders with some strings.
I think the problem is with reading the file, the output most of the time is missing large chunks, even in multiple parts of the html file, the missing chunks are always the same (probably based on the buffer size you configured in the PR).
Only 10-20% of the requests are successful with full html served.
I will try to provide a short sketch that does just this, with an example html page to upload to the SPIFFS/LITTLEFS partition.

@dsilletti
Can you test your sketch with this fix #6536 ? If you will still have some issues, you can provide some sketch for testing :)

dsilletti · 2022-04-05T10:34:51Z

Hi @P-R-O-C-H-Y ,
I just tested it and unfortunately is not fixing the issue.

The behaviour now is different, before the chunks were just missing, now they are added, repeated in a part of the file where they should not be.

On a 243 lines html page, before I had just 194 lines, now I have 294 lines with a repeated chunk pasted in the middle of some other lines.

P-R-O-C-H-Y · 2022-04-05T10:52:20Z

Hi @dsilletti,
Thanks for testing. There is still work in progress on the PR. We are discussing changes here #6507 so you can read about that more. Some users are creating test cases which helps me a lot to make everything work perfectly :)

If you can provide the sketch it will be awesome :)

P-R-O-C-H-Y · 2022-04-05T11:14:20Z

@dsilletti Updated PR, can you test if its still the same? Thank you

dsilletti · 2022-04-05T12:19:04Z

Still the same, please try with this test sketch and test html file.

When you will see the source of the webpage served (using your last PR), from line 189 to line 230 you will find a chunk of the previous part of the html file.

I added also the OTA to the sketch so you will be also able to create the LittleFS.bin file with mklittlefs and upload it from the /ota web page.

#include <WiFi.h>
#include <ESPAsyncWebServer.h>
#include <LittleFS.h>
#include <Update.h>

AsyncWebServer server(80);

// Replace with your network credentials
const char* ssid = "YOUR_SSID";
const char* password = "YOUR_PASSWORD";


// Replaces placeholder with value
String processor(const String& var){
  Serial.println(var);
  if (var == "IP_ADDRESS") return WiFi.localIP().toString();
  else if (var == "SUBNET_MASK") return WiFi.subnetMask().toString();
  else if (var == "GATEWAY") return WiFi.gatewayIP().toString();
  else if (var == "DNS") return WiFi.dnsIP().toString();
  else return String();
}

void setup() {

  Serial.begin(115200);

  // Initialize LittleFS
  if(!LittleFS.begin(true)){
    Serial.println("An Error has occurred while mounting LittleFS");
    return;
  }

  // Connect to Wi-Fi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  Serial.println(WiFi.localIP().toString());

  // Route for root / web page
  server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(LittleFS, "/index.html", String(), false, processor);
  });

  
  // OTA Firmware Update
  server.on("/ota", HTTP_GET, [](AsyncWebServerRequest *request) {
    Serial.println("OTA Page");
    String page = F("<form method='POST' enctype='multipart/form-data'>"
      "<input type='file' accept='.bin' name='update' >"
      "<button type='submit'>Upload</button>"
      "</form>");
    request->send(200, "text/html", page);
  });
  
  server.on("/ota", HTTP_POST, [](AsyncWebServerRequest *request) {},[](AsyncWebServerRequest *request, const String& filename, size_t index, uint8_t *data, size_t len, bool final) {
   String body;
   size_t content_len;
   if (!index){
      Serial.println("Update");
      content_len = request->contentLength();
      // if filename includes LittleFS, update the LittleFS partition
      int cmd = (filename.indexOf("LittleFS") > -1) ? U_SPIFFS : U_FLASH;
      if (!Update.begin(UPDATE_SIZE_UNKNOWN, cmd)) {
        Update.printError(Serial);
      }
    }
    if (Update.write(data, len) != len) {
      Update.printError(Serial);
    }
    if (final) {
      Serial.println("Uploaded");
      if (!Update.end(true)){
        Update.printError(Serial);
        body = F("Update Failed!");
        request->send(200, "text/html", body);
        ESP.restart();
      }
      
      else {
        Serial.println("Update complete");      
        body = F("Update Successful!");
        request->send(200, "text/html", body);
        ESP.restart();
      }
    }
  });


  // Start server
  server.begin();
}



void loop() {

}

index.html test file:

<!DOCTYPE html>
<html>
<head>
   <meta name='viewport' content='width=device-width,initial-scale=1.0,minimum-scale=1.0'>
   <title>TEST</title>
   <meta charset='UTF-8'>
   <link href="css/style.css" rel="stylesheet" />
</head>
<body>

   <input class="toggle-menu" id="toggle-menu" type="checkbox" />
   <div class="header">
      <a href="/">
         <div class="logo">
            <img src="img/logo.webp" width="50px" height="50px">
         </div>
         <div class="logotext"><h1>test</h1></div>
      </a>
      <label class="button-toggle-menu" for="toggle-menu"><span></span><span></span><span></span></label>
   </div>

   <div class="content">

      <div class="main-content">
         <div class='card'>
            <div class="title">Status</div>
            <hr/>
            <div class="row">
               <table>
                  <tr>
                     <td>Date and Time</td>
                     <td id="datetime"></td>
                  </tr>
                  <tr>
                     <td>Hostname</td>
                     <td>%HOSTNAME%</td>
                  </tr>
                  <tr>
                     <td>Uptime</td>
                     <td id="uptime"></td>
                  </tr>
               </table>
            </div>


            <div class="title">Wi-Fi</div>
            <hr/>
            <div class="row">
               <table>
                  <tr>
                     <td>Status</td>
                     <td>%WIFI_STATUS%</td>
                  </tr>
                  <tr>
                     <td>Wi-Fi connected</td>
                     <td id="wifiuptime"></td>
                  </tr>
                  <tr>
                     <td>MAC Address</td>
                     <td>%MAC_ADDRESS%</td>
                  </tr>
                  <tr>
                     <td>SSID</td>
                     <td>%SSID%</td>
                  </tr>
                  <tr>
                     <td>RSSI</td>
                     <td id='rssi'>%RSSI%</td>
                  </tr>
                  <tr>
                     <td>DHCP</td>
                     <td>%DHCP%</td>
                  </tr>
                  <tr>
                     <td>IP address</td>
                     <td>%IP_ADDRESS%</td>
                  </tr>
                  <tr>
                     <td>Subnet mask</td>
                     <td>%SUBNET_MASK%</td>
                  </tr>
                  <tr>
                     <td>Default gateway</td>
                     <td>%GATEWAY%</td>
                  </tr>
                  <tr>
                     <td>DNS</td>
                     <td>%DNS%</td>
                  </tr>
               </table>
            </div>


            <div class="title">MQTT</div>
            <hr/>
            <div class="row">
               <table>
                  <tr>
                     <td>Status</td>
                     <td id="mqttstatus"></td>
                  </tr>
                  <tr>
                     <td>MQTT connected</td>
                     <td id="mqttuptime"></td>
                  </tr>
                  <tr>
                     <td>Broker</td>
                     <td>%MQTT_BROKER%</td>
                  </tr>
                  <tr>
                     <td>Version</td>
                     <td>3.1.1</td>
                  </tr>
               </table>
            </div>

            <div class="title">Chipset</div>
            <hr/>
            <div class="row">
               <table>
                  <tr>
                     <td>Status</td>
                     <td>%ADE_STATUS%</td>
                  </tr>
                  <tr>
                     <td>Version</td>
                     <td></td>
                  </tr>
                  <tr>
                     <td>Algorithms Version</td>
                     <td></td>
                  </tr>
               </table>
            </div>


            <div class="title">Resources</div>
            <hr/>
            <div class="row">
               <table>
                  <tr>
                     <td>CPU</td>
                     <td>%CPU%</td>
                  </tr>
                  <tr>
                     <td>Flash</td>
                     <td>%FLASH%</td>
                  </tr>
                  <tr>
                     <td>Heap</td>
                     <td>
                     </td>
                  </tr>
                  <tr>
                     <td>Sketch</td>
                     <td>
                        <div class="meterbar">
                          <div class="value" style="width:%SKETCH%&#37;"></div>
                        </div>
                     </td>
                  </tr>
               </table>
            </div>

         </div>

         <div class="footer"></div>
      </div>
   </div>

   <script>
      function uptime(timestamp) {
         var secs=timestamp;
         var formattedTime = "";
         var mins=Math.floor(secs/60); //convert seconds to minutes
         var hours=Math.floor(mins/60); //convert minutes to hours
         var days=Math.floor(hours/24); //convert hours to days
         secs=Math.floor(secs-(mins*60)); //subtract the coverted seconds to minutes in order to display 59 secs max 
         mins=Math.floor(mins-(hours*60)); //subtract the coverted minutes to hours in order to display 59 minutes max
         hours=Math.floor(hours-(days*24)); //subtract the coverted hours to days in order to display 23 hours max

         if (days>0) {
            formattedTime += days;
            formattedTime +=" days ";
         }
         if (hours>0) {
            formattedTime += hours;
            formattedTime +=" hours ";
         }
         if (mins>0) {
            formattedTime += mins;
            formattedTime +=" min ";
         }
         formattedTime +=secs;
         formattedTime +=" sec";



         return formattedTime;
      }

      setInterval(function(){
         var xhttp=new XMLHttpRequest();
         xhttp.onreadystatechange=function(){
            if(this.readyState==4 && this.status==200){            
               var data=JSON.parse(this.responseText);
               for(var key of Object.keys(data)){
                  document.getElementById(key).innerHTML=data[key];
               }    
            }
         };
         xhttp.open('GET','/status.json',true);
         xhttp.send();
      },1000);
   </script>

</body>
</html>

P-R-O-C-H-Y · 2022-04-05T14:46:00Z

@dsilletti I will test that, thank you for great sketch.
I have added another commit to PR, can you test with latest changes please? TY

dsilletti · 2022-04-05T15:02:19Z

@P-R-O-C-H-Y You are welcome! Tested now, still the same issue.

P-R-O-C-H-Y · 2022-04-12T11:27:26Z

Hi @dsilletti, i have linked PR to this issue which will fix the problems. It will revert the commit which was breaking the FS.

dsilletti added the Status: Awaiting triage Issue is waiting for triage label Mar 30, 2022

me-no-dev added Type: Bug 🐛 All bugs Priority: High 🗻 Issues with high priority which needs to be solved first. Type: Regression Result of unforeseen consequences of a previous change and removed Status: Awaiting triage Issue is waiting for triage labels Mar 30, 2022

VojtechBartoska added this to Arduino ESP32 Core Project Roadmap Mar 31, 2022

VojtechBartoska added this to the 2.0.3 milestone Mar 31, 2022

VojtechBartoska assigned P-R-O-C-H-Y Mar 31, 2022

VojtechBartoska moved this to Todo in Arduino ESP32 Core Project Roadmap Mar 31, 2022

Jason2866 mentioned this issue Apr 1, 2022

Revert "Edited VFSFileImpl::read to use both read/fread" #6516

Closed

VojtechBartoska moved this from Todo to Under investigation in Arduino ESP32 Core Project Roadmap Apr 6, 2022

VojtechBartoska mentioned this issue Apr 7, 2022

ESP32-CAM test crashes in 2.0.2- OK in 1.0.6 #6526

Open

1 task

Jason2866 mentioned this issue Apr 10, 2022

ESP32 is crashing when WiFi is connecting / BLE is initializing #6541

Closed

1 task

P-R-O-C-H-Y mentioned this issue Apr 12, 2022

BUGFIX: FS read + speed improvements for SD #6569

Merged

VojtechBartoska moved this from Under investigation to In Review in Arduino ESP32 Core Project Roadmap Apr 20, 2022

me-no-dev closed this as completed in #6569 Apr 21, 2022

Repository owner moved this from In Review to Done in Arduino ESP32 Core Project Roadmap Apr 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.0.3 RC1 issues with ESPAsyncWebServer Processor #6502

2.0.3 RC1 issues with ESPAsyncWebServer Processor #6502

dsilletti commented Mar 30, 2022 •

edited

Loading

dsilletti commented Mar 30, 2022

me-no-dev commented Mar 30, 2022

Jason2866 commented Mar 31, 2022 •

edited

Loading

P-R-O-C-H-Y commented Apr 4, 2022

dsilletti commented Apr 5, 2022 •

edited

Loading

P-R-O-C-H-Y commented Apr 5, 2022 •

edited

Loading

dsilletti commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

dsilletti commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

dsilletti commented Apr 5, 2022 •

edited

Loading

P-R-O-C-H-Y commented Apr 12, 2022

2.0.3 RC1 issues with ESPAsyncWebServer Processor #6502

2.0.3 RC1 issues with ESPAsyncWebServer Processor #6502

Comments

dsilletti commented Mar 30, 2022 • edited Loading

Board

Device Description

Hardware Configuration

Version

IDE Name

Operating System

Flash frequency

PSRAM enabled

Upload speed

Description

Sketch

Output expected (as per static html file served)

Output produced

Other Steps to Reproduce

I have checked existing issues, online documentation and the Troubleshooting Guide

dsilletti commented Mar 30, 2022

me-no-dev commented Mar 30, 2022

Jason2866 commented Mar 31, 2022 • edited Loading

P-R-O-C-H-Y commented Apr 4, 2022

dsilletti commented Apr 5, 2022 • edited Loading

P-R-O-C-H-Y commented Apr 5, 2022 • edited Loading

dsilletti commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

dsilletti commented Apr 5, 2022

P-R-O-C-H-Y commented Apr 5, 2022

dsilletti commented Apr 5, 2022 • edited Loading

P-R-O-C-H-Y commented Apr 12, 2022

dsilletti commented Mar 30, 2022 •

edited

Loading

Jason2866 commented Mar 31, 2022 •

edited

Loading

dsilletti commented Apr 5, 2022 •

edited

Loading

P-R-O-C-H-Y commented Apr 5, 2022 •

edited

Loading

dsilletti commented Apr 5, 2022 •

edited

Loading