Signed updates #3105

madpilot · 2017-04-02T06:47:48Z

Adds the ability to verify the signature of an OTA update.

The developer certificate and signature are bundled at the end of the binary, with two additional uint32_t variables added to the very end to let the updater know where to look for them.

It works, but there is a few rough edges to clean up:

Bundling requires a C program that isn't very smart. I need to convert it to Go or Python, and then ideally integrate it with the Arduino IDE
Only MD5 signatures are supported. It would be fairly easy to add SHA1 or ideally SHA256.
Basically no documentation or formal tests 😄 - I'll fix that in the next few days - I just wanted to open this PR to get feedback.

NOTE: The is a feature flag:

#define VERIFY_SIGNATURE 1

Iy's turned on by default in this version. Final version will allow it to be set as a build paramater.

Quick and Dirty documentation for those that know what they are doing

First step is to create a CA, and reference in the source of the sketch, by converting it to a binary header file using xxd:

openssl req -new -x509 -days 3650 -extensions v3_ca -keyout ca.key.pem -out ca.crt.pem
xxd -i ca.crt.pem

Include the file in the Sketch, and using the new addCA function to reference it.

#include "ca.h"

void setup(void) {
  Update.addCA(ca_crt_der, &ca_crt_der_len);
}

Compile and upload the sketch using Serial.

To do an update:

Create a developer certificate:

openssl genrsa -out developer.key.pem 2048
openssl req -out developer.csr.pem -keydeveloper.key.pem -new
openssl x509 -req -in developer.csr.pem -CA ca.crt.pem -CAkey ca.key.pem -CAcreateserial -out developer.crt.pem -days 365

The library definately suppports sha256. It should support sha1 and md5 too.

Generate the signature for the new binary using openssl:

openssl dgst -md5 -sign ~/Projects/ESP8266FileSigning/cert/developer.key.pem -out Sketch.ino.sig Sketch.ino.bin

The mash them together using this C program (I'll convert it to Go or Python shortly)

#include <stdio.h>
#include <stdlib.h>

unsigned char *bundle;

uint32_t size;
uint32_t certificate_size;
uint32_t signature_size;

int main() {
  FILE *f1 = fopen("Sketch.ino.bin", "rb");
  if(f1) {
    fseek(f1, 0, SEEK_END);
    size = ftell(f1);
    rewind(f1);
    printf("Binary file size: %i\n", size);
  } else {
    printf("Unable to open Sketch.ino.bin\n");
    return -1;
  }

  FILE *f2 = fopen("developer.crt.der", "rb");
  if(f2) {
    fseek(f2, 0, SEEK_END);
    certificate_size = ftell(f2);
    rewind(f2);
    printf("Certificate size: %i\n", certificate_size);
  } else {
    printf("Unable to open developer.crt.der\n");
    return -1;
  }

  FILE *f3 = fopen("Sketch.ino.sig", "rb");
  if(f3) {
    fseek(f3, 0, SEEK_END);
    signature_size = ftell(f3);
    rewind(f3);
    printf("Signature size: %i\n", signature_size);
  } else {
    printf("Unable to open Sketch.ino.sig\n");
    return -1;
  }

  printf("Signature size: 0x%x\n", signature_size);
  uint32_t bundle_size = size + certificate_size + signature_size + (2 * sizeof(uint32_t));
  bundle = (unsigned char *)malloc(bundle_size);

  for(int i = 0; i < bundle_size; i++) {
    bundle[i] = 0;
  }

  fread(bundle, size, 1, f1);
  fread(bundle + size, certificate_size, 1, f2);
  fread(bundle + size + certificate_size, signature_size, 1, f3);

  bundle[bundle_size - 4] = signature_size & 0xFF;
  bundle[bundle_size - 3] = (signature_size >> 8) & 0xFF;
  bundle[bundle_size - 2] = (signature_size >> 16) & 0xFF;
  bundle[bundle_size - 1] = (signature_size >> 24) & 0xFF;

  bundle[bundle_size - 8] = certificate_size & 0xFF;
  bundle[bundle_size - 7] = (certificate_size >> 8) & 0xFF;
  bundle[bundle_size - 6] = (certificate_size >> 16) & 0xFF;
  bundle[bundle_size - 5] = (certificate_size >> 24) & 0xFF;

  FILE *f4 = fopen("Bundle.bin", "wb");
  if(f4) {
    fwrite(bundle, bundle_size, 1, f4);
    printf("Bundle size: %i\n", bundle_size);
  } else {
    printf("Unable to save Bundle.bin");
  }

  return 0;
}

… key.

…e last moment, allowing us to ignore the additional signature and certificate

…rified it.

madpilot · 2017-04-02T20:51:30Z

Having a look through some archives, and I've found https://github.com/igrr/axtls-8266 which I assume is what is providing axTLS. I'll look at using the binary blob, rather than re-including my versions of those files.

davisonja · 2017-04-02T21:43:06Z

I've got some completed changes to support storing updater commands in flash, which also include a fixed location for finding the updater commands in flash, grand scheme I'd envisaged that the location of such certificates would be included - either in the flash-command block itself, or with the location of the flash command block (in fact I was looking to put the certificates themselves in the flash-command block since it needs to be preserved across updates and presumably so does the certificate? Or does each update always include certificates?) It's waiting on my finishing a script to generate boards.txt (as an inherent part of putting commands in flash is ensuring things don't overwrite them, which is simplest if we adjust the areas considered available for code and SPIFFs, which implies updates to the Makefiles, and since the info is also in boards.txt one script will generate both). I've been a bit short on time recently, unfortunately :(

…

On Mon, Apr 3, 2017 at 8:51 AM, Myles Eftos ***@***.***> wrote: Having a look through some archives, and I've found https://github.com/igrr/axtls-8266 which I assume is what is providing axTLS. I'll look at using the binary blob, rather than re-including my versions of those files. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A313x3MTAsKZLXS6-f-INADplZhqks5rsApTgaJpZM4MwuCh> .

madpilot · 2017-04-02T21:51:55Z

@davisonja I've been thinking about the certificate storage issue.

I think the certificate used to decrypt the signature should be part of the payload - as they usually have a shortish life (Mine expire after 12 months), and rotating them would be painful, as it would require manual reflashing. They are public anyway, so there isn't really any harm having them as part of the payload.

The CA certificate - I'm torn. Having them in code would effectively update them every update. I'm not sure if this is a good idea or bad idea. It's good from the point of view that you can update them when they expire (though that would probably be a long time in the future).

It's bad in that they could tampered with a bit more easily.

Having them in flash also means you can get them out of source control, which is probably a good thing too.

What is involved in getting them into flash?

davisonja · 2017-04-02T22:02:09Z

I think the first interesting question is how do you update the certificates. There's no technical barrier to the firmware itself updating the certificate (since it's running on the chip it can read/write the flash), and if your system ensure only legitimate code is running then that's an okay thing to do. It shifts the responsibility to the firmware, and could be integrated into the Updater code itself (when checking if new firmware is available, also check if new certificates are available). Is the entire update binary encrypted? Putting things in flash is simple enough, do you have a ballpark size? If you're just appending data files to the binary you might be able to use the ESP tool which currently prepends the bootloader binary to the compiled sketch.

…

On Mon, Apr 3, 2017 at 9:51 AM, Myles Eftos ***@***.***> wrote: @davisonja <https://github.com/davisonja> I've been thinking about the certificate storage issue. I think the certificate used to decrypt the signature should be part of the payload - as they usually have a shortish life (Mine expire after 12 months), and rotating them would be painful, as it would require manual reflashing. They are public anyway, so there isn't really any harm having them as part of the payload. The CA certificate - I'm torn. Having them in code would effectively update them every update. I'm not sure if this is a good idea or bad idea. It's good from the point of view that you can update them when they expire (though that would probably be a long time in the future). It's bad in that they could tampered with a bit more easily. Having them in flash also means you can get them out of source control, which is probably a good thing too. What is involved in getting them into flash? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A4nc6wZD91aHisv6YdZUhedtER8dks5rsBh8gaJpZM4MwuCh> .

madpilot · 2017-04-02T22:07:53Z

The CA cert I've been testing with (just a root, no chaining) is 991 bytes. The signing certificate is 925 bytes. (My original POC with test certs is here: https://github.com/madpilot/ESP8266FileSigning)

I still like passing the signing certificate with the payload - no need to check for a new one - the newest one is always there.

davisonja · 2017-04-02T22:11:10Z

The updates to eboot for storing updater commands in flash (which allows it to survive a power loss for more of the process) has to reserve a flash page, which is mostly 4k. By default it's using around 1k of that, so there's 3k 'spare' which I had hoped could be used for persistent certificate/key storage.

…

On Mon, Apr 3, 2017 at 10:07 AM, Myles Eftos ***@***.***> wrote: The CA cert I've been testing with (just a root, no chaining) is 991 bytes. The signing certificate is 925 bytes. (My original POC with test certs is here: https://github.com/madpilot/ESP8266FileSigning) I still like passing the signing certificate with the payload - no need to check for a new one - the newest one is always there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A4WcQifT9_Fp8IZOvPnEeqhWklHMks5rsBw6gaJpZM4MwuCh> .

madpilot · 2017-04-02T22:16:12Z

3k should be plenty. All I know about the memory layout is this: https://github.com/esp8266/Arduino/blob/master/doc/filesystem.md#flash-layout

I assume you are talking about the OTA section?

…to do certificate validation

madpilot · 2017-04-03T09:23:47Z

I've removed the copied axtls files. By copying the headers files from the https://github.com/igrr/axtls-8266 we can use the binary blob.

We can probably remove the ssl.h file in ESP8266WiFi/src/

madpilot · 2017-04-05T22:35:02Z

@martinayotte suggested using EEPROM which sounds like a good solution - it's persistent, and it can be written to from the command line, making the initial setup fairly easy.

Though it made me think about giving the develooer choices. I think I'll add a helper function that reads the very from EEPROM and another that will read it from SPIFFS, while leaving the existing addCA function.

That way there is a sane default (EEPROM) but the developer can decide to store it elsewhere.

davisonja · 2017-04-05T22:37:36Z

When you say EEPROM, do you mean external hardware that's attached, or the one that's stored in flash? Leaving it up to the user seems like a good idea.

…

On Thu, Apr 6, 2017 at 10:35 AM, Myles Eftos ***@***.***> wrote: @martinayotte <https://github.com/martinayotte> suggested using EEPROM which sounds like a good solution - it's persistent, and it can be written to from the command line, making the initial setup fairly easy. Though it made me think about giving the develooer choices. I think I'll add a helper function that reads the very from EEPROM and another that will read it from SPIFFS, while leaving the existing addCA function. That way there is a sane default (EEPROM) but the developer can decide to store it elsewhere. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_Az-gQvCKp8IJywKJrZ8M2m25iVGnks5rtBcXgaJpZM4MwuCh> .

madpilot · 2017-04-05T22:43:25Z

@davisonj the one that is stored in flash.

…ng clashing version of max() and min()

igrr · 2017-05-08T08:17:47Z

This change looks very good, but i don't think that using a compile-time define (VERIFY_SIGNATURE) to enable verification is a good idea, in Arduino environment. The only way we can expose this "switch" in Arduino is by adding yet another menu option to all the boards. I would like to avoid this, if possible.

Arduino way of implementing such an optional feature (whether anyone likes this or not) is to enable it at run time via some method call. addCA method looks like a good candidate: if you pass a CA, you probably expect the firmware to be signed. This way is also perfectly backwards compatible.

Note that it is still possible to benefit from compile-time code size savings if certificate verification is not used.
For example, if addCA is called, you can set some member of updater (UpdateVerificationTraits _verificationTraits) to an instance of a class which performs certificate verification (_verificationTraits = new CertificateVerification(cert, size);). Then at the time when verification needs to happen, you call _verificationTraits->verify interface method. If addCA is never used in the sketch, all related functionality doesn't get linked in. Similar pattern is used in HTTPClient library, where SSL certificate verification is exposed in the form of a "trait" class.

igrr · 2017-05-08T08:20:38Z

cores/esp8266/Updater.h

@@ -141,18 +148,35 @@ class UpdaterClass {
      return written;
    }

+#ifdef VERIFY_SIGNATURE
+    int addCA(const uint8_t *cert, int *len);


Similar to WiFiClientSecure, suggest exposing a helper template method which loads CA certificate from a file (Stream).

igrr · 2017-05-08T08:21:07Z

cores/esp8266/Updater.h

 #include <Arduino.h>
 #include <flash_utils.h>
 #include <MD5Builder.h>

+#ifdef VERIFY_SIGNATURE
+#include <ssl/crypto_misc.h>
+#include <crypto/crypto.h>


Do we need these headers in user-facing header file?

igrr · 2017-05-08T08:21:49Z

cores/esp8266/Updater.h

@@ -1,10 +1,17 @@
 #ifndef ESP8266UPDATER_H
 #define ESP8266UPDATER_H

+#define VERIFY_SIGNATURE 1


Posted a note about using preprocessor macro as a feature switch above.

igrr · 2017-05-08T08:23:13Z

cores/esp8266/Updater.h

  private:
    void _reset();
    bool _writeBuffer();

    bool _verifyHeader(uint8_t data);
    bool _verifyEnd();

+#ifdef VERIFY_SIGNATURE
+    CA_CERT_CTX *_ca_ctx;


Ok, i see now why the headers were needed. If you go for the approach with VerificationTraits interface (call it some other way if you like), you will not have to expose these internals to the users, introducing axTLS names into the sketch global namespace.

igrr · 2017-05-08T08:24:15Z

cores/esp8266/md5.h

-
-extern void MD5Init (md5_context_t *);
-extern void MD5Update (md5_context_t *, uint8_t *, uint16_t);
-extern void MD5Final (uint8_t [16], md5_context_t *);


Suggest keeping this file — it links to the functions defined in ESP8266 ROM code. MD5 functions which run from ROM code run much faster than the axTLS defined ones which run from flash.

igrr · 2017-05-08T08:26:14Z

tools/sdk/include/crypto/bigint.h

@@ -0,0 +1,99 @@
+/*


Do you think there is a way to submodule these instead? If anything, i have one more use case for using axTLS as a submodule, to enable root certificate store (igrr/axtls-8266#44).

A submodule or sub tree makes a lot of sense here

igrr · 2017-05-08T08:27:03Z

cores/esp8266/MD5Builder.cpp

@@ -11,12 +11,12 @@ uint8_t hex_char_to_byte(uint8_t c)
 void MD5Builder::begin(void)
 {
    memset(_buf, 0x00, 16);
-    MD5Init(&_ctx);
+    MD5_Init(&_ctx);


Likewise, suggest keeping ROM functions for performance reasons.

igrr · 2017-05-08T08:28:46Z

cores/esp8266/Updater.cpp

@@ -23,6 +23,9 @@ UpdaterClass::UpdaterClass()
 , _currentAddress(0)
 , _command(U_FLASH)
 {
+#ifdef VERIFY_SIGNATURE
+  _ca_ctx = (CA_CERT_CTX *)malloc(sizeof(CA_CERT_CTX));


Suggest allocating this only when the feature gets actually used.

madpilot · 2017-05-09T22:27:29Z

Thanks for the feedback @igrr - that should all be doable. I'm thinking some sort of delegator so the developer can drop in different validation schemes. I'll look at making the changes in the coming weeks (though we are just about to have a baby so progress will be slow!)

tuxedo0801 · 2017-07-11T07:29:27Z

ThumbsUp Looking forward for a release of this feature...
Is there some progress? Last update is from 10 May?!

madpilot · 2017-07-11T07:47:35Z

We just had a baby, so progress has slowed somewhat.

corbolais · 2017-11-11T01:32:07Z

@madpilot congratulations! hot times ahead! in every sense ;) hope you 3(?) enjoy life between changing diapers.. :)
Regarding signed updates: I'd be somewhat interested in seeing this progressing.
@igrr any chance this will ripe into an accepted PR anytime soon..-ish?
thanks for all the fish so far!

targence · 2017-12-18T09:34:31Z

Hi, guys!

Thanks for your work, any updates?
Looking forward for a release.

arihantdaga · 2018-07-16T05:39:07Z

Hi Guys...
Are we releasing this amazing feature anytime soon ?
It looks great...

dumarjo · 2018-10-04T19:50:00Z

Hi,

this look amazing!!!!

earlephilhower · 2018-11-09T22:08:57Z

Closing this as it lives on in #5213. Thanks, @madpilot, for the original idea and implementation!

madpilot added 6 commits March 23, 2017 08:15

Adding required axtls libraries. Code compiles. Untested though

991197c

Successfully reads bundled certificate and signature size from the flash

1546732

Succesfully loading the developer certificate from flash

effbb3f

Successfully verifies the developer certificate and decrypts the hash…

0b69fa4

… key.

Allow md5 and sha1 signed certs. Refactored so MD5 hash happens at th…

e5e6747

…e last moment, allowing us to ignore the additional signature and certificate

Successfully parsed binary, verified signature, decrypted hash and ve…

d4e81ed

…rified it.

madpilot mentioned this pull request Apr 2, 2017

Implement firmware update signing and verification #2103

Closed

6 tasks

Adding axtls headers from axtls-esp8266 so I can use the binary blob …

60c1b5b

…to do certificate validation

Adding SDK path to test include path

b819b6c

madpilot added 4 commits April 13, 2017 19:37

Adding object fiels to gitignore

9b880f9

Adding static version of libaxtls so the test suite will work. Removi…

024ef36

…ng clashing version of max() and min()

Merge branch 'master' into signed-updates

68da99c

Adding tests/host/bin to gitignore

74a9b90

igrr requested changes May 8, 2017

View reviewed changes

igrr mentioned this pull request May 10, 2017

WIP: Add a SHA256Builder #3223

Closed

4 tasks

tuxedo0801 mentioned this pull request Jul 11, 2017

ESP8266httpUpdate: How to ensure correct/valid update-file? #3412

Closed

madpilot added 2 commits July 14, 2017 07:27

Merging from esp8266/master

659c291

Saving works a lot better when you actually hit save

84b0005

This was referenced Jul 15, 2017

Subtree axtls #3423

Closed

Subrepo axtls #3424

Closed

Merge branch 'master' into signed-updates

6616f03

earlephilhower self-assigned this Oct 4, 2018

earlephilhower mentioned this pull request Oct 6, 2018

Add cryptographically signed update support #5213

Merged

earlephilhower closed this Nov 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Signed updates #3105

Signed updates #3105

madpilot commented Apr 2, 2017 •

edited

Loading

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

madpilot commented Apr 3, 2017 •

edited

Loading

madpilot commented Apr 5, 2017

davisonja commented Apr 5, 2017 via email

madpilot commented Apr 5, 2017

igrr commented May 8, 2017

igrr May 8, 2017

igrr May 8, 2017

igrr May 8, 2017

igrr May 8, 2017

igrr May 8, 2017

madpilot May 9, 2017

igrr May 8, 2017

madpilot May 9, 2017

igrr May 8, 2017

igrr May 8, 2017

madpilot commented May 9, 2017

tuxedo0801 commented Jul 11, 2017

madpilot commented Jul 11, 2017

corbolais commented Nov 11, 2017

targence commented Dec 18, 2017

arihantdaga commented Jul 16, 2018

dumarjo commented Oct 4, 2018

earlephilhower commented Nov 9, 2018

Signed updates #3105

Signed updates #3105

Conversation

madpilot commented Apr 2, 2017 • edited Loading

Quick and Dirty documentation for those that know what they are doing

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

davisonja commented Apr 2, 2017 via email

madpilot commented Apr 2, 2017

madpilot commented Apr 3, 2017 • edited Loading

madpilot commented Apr 5, 2017

davisonja commented Apr 5, 2017 via email

madpilot commented Apr 5, 2017

igrr commented May 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madpilot commented May 9, 2017

tuxedo0801 commented Jul 11, 2017

madpilot commented Jul 11, 2017

corbolais commented Nov 11, 2017

targence commented Dec 18, 2017

arihantdaga commented Jul 16, 2018

dumarjo commented Oct 4, 2018

earlephilhower commented Nov 9, 2018

madpilot commented Apr 2, 2017 •

edited

Loading

madpilot commented Apr 3, 2017 •

edited

Loading