ESPhttpUpdate Error[12] Signature verification failed

[seism0saurus]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [D1 Mini]
• Core Version: [git fff12e3]
• Development Env: [Arduino IDE 1.8.13]
• Operating System: [Debian stable]

Settings in IDE

• Module: [Lolin(Wemos) D1 mini r2 & mini]
• Flash Size: [4MB (2MB FS 1MB OTA)]
• lwip Variant: [v2 Lower Memory]
• CPU Frequency: [80Mhz]
• Upload Using: [First SERIAL than OTA]
• Upload Speed: [921600]
• Debug Level: [HTTP_UPDATE]
• They are the default settings after selecting the D1 mini as board

Problem Description

Since I used the latest version from the master branch the autosigning of the bin files does no longer work correctly. It worked with the last release. I switched to the git version because a bugfix for a component is not released yet.

I made a simplified MCVE with only WLAN and ESP8266httpUpdate to test that problem.

• create a project folder in your Arduino IDE folder
• You need a keypair in the project folder to enable the signing:

  openssl genrsa -out private.key 2048
  openssl rsa -in private.key -outform PEM -pubout -out public.key

• You have to set your WLAN credentials and the ip of your development pc and compile the scratch
• Upload it to the D1 mini
• Export the scratch as bin with the Arduino IDE feature
• Start a small server to deliver the bin file

  { echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <MCRE.ino.d1_mini.bin)\r\n\r\n"; cat MCRE.ino.d1_mini.bin; } | nc -l 9090

• Restart the D1 mini and watch the serial output. The update will fail because it can't verify the signature
• In the export log was also a signed file mentioned. Copy it

  cp /tmp/arduino_build_113468/MCRE.ino.bin.signed MCRE.ino.d1_mini.bin.signed

• Start a small server to deliver the signed bin file

  { echo -ne "HTTP/1.0 200 OK\r\nContent-Length: $(wc -c <MCRE.ino.d1_mini.bin.signed)\r\n\r\n"; cat MCRE.ino.d1_mini.bin.signed; } | nc -l 9090

• Restart the D1 mini and watch the serial output. The update will fail again because it can't verify the signature

Is it intentional that the exported bin is not the signed bin? I can see that the signed bin from the tmp folder is 260 bytes bigger. I had expected that the signed bin would be exported, since it already detected, that it should be signed.

The main problem is that the signature verification does not work anymore. I searched through the documentation if i missed any change but could not find anything. Could you please help me with this MCVE?

MCVE Sketch

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>

// Please replace the ip with the ip of your computer and make port 9090 accessable so that netcat can deliver the update
String ipOfThisComputer = "192.168.179.222";

// Please replace with the WLAN credentials, so that your computer and the ESP are in the same network
String ssid = "mywlan";
String wlanPassword = "mypassword";
WiFiClient client;

void setup() {
  delay(1000);
  Serial.begin(9600);
  Serial.println(F("Serial inteface for debugging is initialized with 9600 baud."));
  
  initWlan();

  ESPhttpUpdate.setLedPin(LED_BUILTIN, LOW); 
  t_httpUpdate_return ret = ESPhttpUpdate.update(client, ipOfThisComputer, 9090, "/");
  handleUpdateResult(ret);
}

void loop() { 
  delay(1000);
  Serial.println(F("Sleeping..."));
}

void initWlan(){
  Serial.print(F("Connecting to "));
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, wlanPassword);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(F("."));
  }
  Serial.println(F("WiFi connected"));
}

void handleUpdateResult(t_httpUpdate_return ret){
  String logMessage;
  switch (ret) {
    case HTTP_UPDATE_FAILED:
      logMessage = "Firmware update canceled. Error (";
      logMessage += ESPhttpUpdate.getLastError();
      logMessage += "): ";
      logMessage += ESPhttpUpdate.getLastErrorString().c_str();
      break;
    case HTTP_UPDATE_NO_UPDATES:
      logMessage = "Already using the latest firmware. No update needed";
      break;
    case HTTP_UPDATE_OK:
      logMessage = "Firmware update done";
      break;
  }
  Serial.println(logMessage);
}

Debug Messages

[httpUpdate] Header read fin.
[httpUpdate] Server header:
[httpUpdate]  - code: 200
[httpUpdate]  - len: 342964
[httpUpdate] ESP8266 info:
[httpUpdate]  - free Space: 1748992
[httpUpdate]  - current Sketch Size: 346080
[httpUpdate] runUpdate flash...
sleep disable
[httpUpdate] Update.end failed! (ERROR[12]: Signature verification failed)
[httpUpdate] Update failed
Firmware update canceled. Error (12): Update error: ERROR[12]: Signature verification failed
pm open,type:0 0

[mcspr]

Weird, I remember testing specific issue :/
There were only two updates to signing code - #8507 (shipped with an incorrect length offset calc) and #8545 (my attempted fix)

[seism0saurus]

I will roll back before the merge and try it with that version.

[mcspr]

Debug Level: [HTTP_UPDATE]

Based on the PR above, DEBUG_UPDATER might also be useful

[seism0saurus]

Thanks for the super fast reply. I set the log level to DEBUG_UPDATER.

The log with the unsigned bin that is created via the export shows, that no signature was found and he assumes the whole bin is the signature. So the length does not match.

sleep disable
[begin] roundedSize:       0x00054000 (344064)
[begin] updateEndAddress:  0x00200000 (2097152)
[begin] currentSketchSize: 0x00055000 (348160)
[begin] _startAddress:     0x001AC000 (1753088)
[begin] _currentAddress:   0x001AC000 (1753088)
[begin] _size:             0x00053AB0 (342704)
Header: 0xE9 2 2 40
[Updater] expected sigLen: 256
[Updater] sigLen from flash: 587202560
ERROR[12]: Signature verification failed
Firmware update canceled. Error (12): Update error: ERROR[12]: Signature verification failed

The signed bin from the tmp directory has a signature with correct length. The SHA256 checksum is also correct. Therefore the error should be in the signature content?

sleep disable
[begin] roundedSize:       0x00054000 (344064)
[begin] updateEndAddress:  0x00200000 (2097152)
[begin] currentSketchSize: 0x00055000 (348160)
[begin] _startAddress:     0x001AC000 (1753088)
[begin] _currentAddress:   0x001AC000 (1753088)
[begin] _size:             0x00053BB4 (342964)
Header: 0xE9 2 2 40
[Updater] expected sigLen: 256
[Updater] sigLen from flash: 256
[Updater] Adjusted size (without the signature and sigLen): 342704
[Updater] Computed Hash: 81 2e 43 8a 32 53 ee 1e b1 5b ae 95 64 64 8b d1 7b 26 1e d0 69 8a ef 5d 80 4b f9 63 16 a6 61 5b
[Updater] Received Signature: 3a a8 1f d2 82 ae 4c dc 8f e4 74 74 96 38 6b 69 7d df bb 3e fc 26 94 3e db cf ed 74 3d d7 fa dd 35 b8 01 83 52 4f 18 ef 6c 27 52 dd 5d 57 5f 7f 12 c4 9f 5a 4a 2c 00 76 21 0e 18 23 58 16 d4 33 12 e5 0a 15 c5 1b be 4b 8f 01 ac 07 e8 0b be a8 a7 96 e2 01 9c 78 fe 33 c2 c8 f1 db 4b 1b e5 86 5b 63 59 87 45 40 ca c8 30 d1 2d 67 4f 93 67 3e aa 77 1e 93 1c b2 e1 54 1d 1d 41 df 86 c5 e9 57 b9 dd 13 19 d8 30 47 86 99 e8 d7 35 3d 43 28 1c ad 9d 52 9d 8b 10 f4 dd b5 75 d1 50 43 ab 98 a9 33 1c 75 32 fd 36 73 bd f6 0f 9f 68 7e e5 7c db c5 d1 1e 9d a5 72 be 0a f2 b2 14 8f ce e8 7d 72 77 07 d5 02 2d d6 cf 0c 1c 56 a3 30 1b a3 1d 52 cf 64 fc 24 54 7c 71 c7 38 57 c0 3b 76 27 e7 69 a3 a3 62 bf 5d fe 5d ae e9 a0 36 2e 05 ff 0a 38 e1 c4 04 e4 67 0c ea 87 28 a8 82 e5 1c a6 01 0e
Epm open,type:0 0
RROR[12]: Signature verification failed
Firmware update canceled. Error (12): Update error: ERROR[12]: Signature verification failed

Tomorrow I'm free and will check the signature and the commits.

[mcspr]

Thanks! memcmp operation looks most suspect atm, hash->len() may not be equal to sizeof(vrf). Will try to test this as well
(...and for the sigLen weirdness, too...)

[seism0saurus]

I switched to commit 313b3c0 and got the following result with the automatically signed bin.

sleep disable
[begin] roundedSize:       0x00055000 (348160)
[begin] updateEndAddress:  0x00200000 (2097152)
[begin] currentSketchSize: 0x00055000 (348160)
[begin] _startAddress:     0x001AB000 (1748992)
[begin] _currentAddress:   0x001AB000 (1748992)
[begin] _size:             0x00054724 (345892)
Header: 0xE9 2 2 40
[Updater] sigLen: 256
[Updater] Adjusted binsize: 345892
[Updater] Computed Hash: 17 03 b3 db d2 39 3e 6a 55 5d 1c 32 86 7f 48 73 10 b0 b5 3c 0c fb 14 46 26 b1 fd 96 29 e8 6d 06
[Updater] Received Signature: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
Epm open,type:0 0
RROR[12]: Signature verification failed
Firmware update canceled. Error (12): Update error: ERROR[12]: Signature verification failed

With commit a0c7a85 it is the following output.

sleep disable
[begin] roundedSize:       0x00055000 (348160)
[begin] updateEndAddress:  0x00200000 (2097152)
[begin] currentSketchSize: 0x00055000 (348160)
[begin] _startAddress:     0x001AB000 (1748992)
[begin] _currentAddress:   0x001AB000 (1748992)
[begin] _size:             0x000547F4 (346100)
Header: 0xE9 2 2 40
[Updater] expected sigLen: 256
[Updater] sigLen from flash: 256
[Updater] Adjusted size (without the signature and sigLen): 345840
[Updater] Computed Hash: ff af 2e 0a 97 36 da 6b ff 1e eb 1f 42 40 06 6b 51 a9 7d 36 29 5a 30 0b 30 d4 eb 7d 79 e9 0b 84
[Updater] Received Signature: 8e f1 8b 0d 83 fb 93 7c aa 35 8b 9d 3b ef 1b 2d da 4e 61 0b 93 3e ec 3d 33 b0 ef 66 20 1f b8 90 b5 82 8b cb a1 bc d6 e3 5c 05 c5 78 23 86 1b e3 a3 fe d5 35 7d 46 17 57 98 4c 73 00 d0 54 4d 37 e0 f8 d6 45 6b 52 31 8c f6 fe f1 ae 26 f1 4f 6e d3 56 dd 54 f3 d3 4f b3 aa fe b0 a9 70 23 2a f8 83 14 ec 5b 44 29 c8 65 74 23 87 8a 7e 45 f8 33 ca 38 db 3e 04 7b 02 17 21 03 87 73 bc df 51 1b 63 3a 67 d8 c0 b8 8f 9b ac ac 6b ba 30 69 a6 2d 97 84 49 a2 3e 1c b5 71 56 27 d5 70 59 63 85 4e a9 44 d6 08 87 0b 88 42 42 8a 9a 83 a9 87 29 e2 0e 57 d0 45 d8 c7 88 77 ed 66 0f fb 1c ea bb c6 fd 97 6f e6 69 79 be 02 fd 4e bd b9 3f ba c8 61 71 71 40 6e 43 06 b0 0c b4 45 50 20 56 e1 9d 6a 64 6a 48 26 9b 9a 12 fc 43 e9 57 d5 dd b5 8e 0c 68 ef 08 6e 26 29 80 d7 5e 53 0a ff 28 b9 81 b6
ERROR[12]: Signature verification failed
Firmware update canceled. Error (12): Update error: ERROR[12]: Signature verification failed

The signature is back to normal hex and not questionsmarks but invalid.

To find the "bad" commit I started a git bisect.

The tag 3.0.2 was good.

sleep disable
[begin] roundedSize:       0x00055000 (348160)
[begin] updateEndAddress:  0x00200000 (2097152)
[begin] currentSketchSize: 0x00055000 (348160)
[begin] _startAddress:     0x001AB000 (1748992)
[begin] _currentAddress:   0x001AB000 (1748992)
[begin] _size:             0x00054384 (344964)
Header: 0xE9 2 2 40
[Updater] sigLen: 256
[Updater] Adjusted binsize: 344704
[Updater] Computed Hash: 83 0c ea ae 07 2d 4e 07 66 f4 ec d2 10 ba 25 8e 91 62 7a c3 be 7b 69 f5 bf dd e9 d6 f9 d0 a7 ac
[Updater] Received Signature: 94 ee 58 49 a2 b9 39 b7 26 94 96 06 35 7a ee b4 71 b0 34 99 91 54 8f ee 62 27 a3 48 15 53 66 ab 59 b1 80 a9 c9 6d 01 3f 1c 22 8b a5 07 89 07 af 91 72 0e b3 95 3e 73 72 63 0b f5 2f 90 50 82 ac 8b 88 c3 bb 77 a6 2c 4f 37 e0 2b 3b 66 fa b4 93 d6 68 ba 82 6d 2e a1 1c a3 34 da 2f 0f 55 7a c8 42 c3 03 d7 86 47 bc c8 13 7b c7 5e 69 73 14 4a 55 94 03 4f 3f 44 61 c3 da a3 5d c9 46 bc b9 64 9e 98 20 f9 26 a7 d9 bb e8 dc 42 72 d4 e1 a9 ae 46 0a d1 f1 8c 7e cf 29 46 d9 b0 4c 63 6a 24 f6 9c 1c 11 4b 0f a3 d9 ec ac b5 76 8c e9 76 8f 73 76 bc 76 a1 95 82 c7 21 06 d0 93 56 d6 f8 4c 73 81 5f 0a 58 11 3a 9c d2 78 17 d7 90 77 6b f1 70 23 69 e1 3e 1a 0dpm open,type:0 0
 d2 35 c1 ad 49 9d 5e 3d 38 dc d9 6f 4c b7 58 dc 4a 2d 80 00 83 7a 09 fd db 64 ce b3 22 c6 51 75 c9 c8 9c 70 3f 82 6d df f1 61
[Updater] Signature matches
Staged: address:0x001AB000, size:0x00054280
state: 5 -> 0 (0)
rm 0
pm close 7
del if0
usl⸮⸮⸮
SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635
Serial inteface for debugging is initialized with 9600 baud.
Connecting to supertux
fpm close 3 
mode : sta(ac:0b:fb:d9:90:94)
add if0
.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 3
cnt 

The current head is bad.

After some steps the result was commit f78c633. That's the commit from the pull request you linked: #8507
It seems that the fix for it is not complete.

Can I provide any other information to help you fix this? I'm not very experienced with Arduino but can do tests and am familiar with the Linux command line.

[mcspr]

sry, I meant reversal of the change done here
40373f8 (part of the 2nd PR)

[seism0saurus]

I found the part, that let's the check fail in my case and created a pull request.
In 40373f8 the size of the vrf Chararray was changed to fix 64 bytes. This works with 64 byte hashes but if the hash is shorter, the memcmp call fails. I assume, that it reads garbage or data behind the hash from the stack, but that's just my assumption as a non-C developer.
My proposal is to use the actual length of the hash. This works in my case but should be tested or security reasons and other use cases.