New PSK format

[countcobolt]

Hi all

I am trying to flash a BSD34 smart socket. I have downloaded the latest tuya master built using git. Working on an RPI 3B. I can connect the socket using the normal smart life app. I did this after I absolutely could not manage to flash it at first (still cant). Decided to dive a bit deeper into it today and found this:

In smarthack-psk.log I have a ton of entries as following

new client on port 443 from 10.42.42.20:53000
ID: 0242416f68626d64366147393149465231509241f729c9f0af3aa41e355b7cbeb1ece63da6ff54b74f271af0ef044466e6
PSK: d9488a1b4524ae3e31acd0342e6d0b2eedbb5d55e957a9a51073b95e36ab1c5c
('could not establish sslpsk socket:', SSLError(1, u'[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:727)'))
new client on port 443 from 10.42.42.20:5371
ID: 0242416f68626d64366147393149465231509241f729c9f0af3aa41e355b7cbeb1ece63da6ff54b74f271af0ef044466e6
PSK: d9488a1b4524ae3e31acd0342e6d0b2eedbb5d55e957a9a51073b95e36ab1c5c
('could not establish sslpsk socket:', SSLError(1, u'[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:727)'))

My smartphone and device are actually connecting to the acces point. In smarthack-wifi.log you see

wlan0: AP-STA-DISCONNECTED ac:57:75:99:f2:9f
wlan0: AP-STA-CONNECTED ac:57:75:99:f2:9f
wlan0: AP-STA-CONNECTED c4:4f:33:bc:10:c6
wlan0: AP-STA-DISCONNECTED ac:57:75:99:f2:9f
wlan0: AP-STA-CONNECTED ac:57:75:99:f2:9f
wlan0: AP-STA-DISCONNECTED ac:57:75:99:f2:9f
wlan0: AP-STA-CONNECTED ac:57:75:99:f2:9f

The AC mac is my smartphone, while the C4 is the actual device I am trying to flash.

The mqtt log remains quite virgin

1578041479: mosquitto version 1.5.7 starting
1578041479: Using default config.
1578041479: Opening ipv4 listen socket on port 1883.
1578041479: Opening ipv6 listen socket on port 1883.

as does the UDP log

Listening for Tuya broadcast on UDP 6666
Listening for encrypted Tuya broadcast on UDP 6667

Smarthack-web.log does not give me more either
Listening on 10.42.42.1:80

I have browsed and google for hours now and mostly I find that this might be due to not using the ESP8x chip. Yet funnily enough, when I simply configure them using the smart life app, I see them in my network as ESP_some_ref_number in it. And it does get an IP address.

The socket looks like this

Any clues on why the SSL error shows up?
Using openssl 1.1.1d / Python 2.7.16 and 3.7.3

Kind regards
Steve

[kueblc]

Hi @countcobolt and @rsbob, it looks like you may have discovered a new version of the Tuya firmware. The PSK identity in your psk log does not have the expected prefix, which explains why the calculation is failing.

If you are flashing by wire, could you please make a backup of the stock firmware? We can use this to further debug and possibly develop a new workaround.

[rsbob]

https://github.com/espressif/esptool
https://github.com/espressif/esptool/blob/master/README.md

[kueblc]

Once you have it installed you can use something like:

esptool.py read_flash 0 0x100000 firmware-backup.bin

[rsbob]

Here the orig.FW
BSD34--image1M.zip

[Farfar]

If anyone is interested, here's the firmware from my Deltaco SH-P01E plug.
Deltaco_SH-P01E_20200110_image1M.zip
The firmware is the latest up until Jan 10th 2020.

[kueblc]

Took some time on this today. Unfortunately these latest firmware builds seem to have changed a number of things making this a non-trivial fix.

• firmware OS SDK version did not change, but the hash and build time did
  • OS SDK ver: 2.0.0(e8c5810) compiled @ Jan 25 2019 14:26:04
  • OS SDK ver: 2.0.0(29f7e05) compiled @ Sep 30 2019 11:19:12
    • interesting that this build date corresponds with the release of tuya-convert 2.0, coincidence?
• psk begins with 02 rather than 01
  • guessing this will tell us the psk algorithm version
• disabled most of the serial debugging output
  • unfortunate as this was a useful reverse engineering resource
• did not respond to our smartconfig procedure
  • it may use a new mechanism or additional restrictions have been imposed
• MAC address in flash appears to be compared with actual device MAC
  • likely to foil reverse engineering attempts by running firmware on a dev board
• new factory written key found in flash, "pskKey"
  • this may mean the encryption key will no longer be derived deterministically, big bummer
  • only found in @rsbob's backup, looks like @Farfar's was updated to this firmware and thus lacks this factory written key
    • this may mean there is still hope for devices that came with a lower firmware, since updating to this firmware would require fetching the key
    • this is a very interesting new endpoint: tuya.device.uuid.pskkey.get

Thus the cat and mouse game with Tuya continues.

[kueblc]

If anyone is able to capture network traffic from one of these devices pairing with the cloud, along with the corresponding firmware backup, this would be instrumental in reverse engineering efforts

[Tollbringer]

If anyone finds a US variant of this I would be happy to get a couple sacrificial units and give it a go.

[kueblc]

Personally, I'm now going to go old school and disassemble the new dumped firmware to firstly see if I can determine how the psk is calculated or determined.

Earlier in this thread we determined that the PSK is stored plaintext in flash. It is not calculated.

[digiblur]

I wasn't aware of my comment about making sure people did proper 2M or 1M flash dumps was off topic? ¯_(ツ)_/¯

[kueblc]

Nothing personal @digiblur, just trying to keep the conversation focused and make discoveries and data easier to find. Github makes me choose a reason for hiding a comment, and a lot of comments are between two people for which the conversation has run it's course. I've hid these comments including my own as an attempt to tidy up.

I'm thinking this is not working since we are having a lot of repeat conversation and we're at nearly 300 comments.

Perhaps a better solution would be to create a wiki page for collaboration on findings and data. This could help keep this stuff easier to find while having a separate channel for discussion. It might take some time to sift through this thread and copy things over, please feel free to help out with editing.

Edit: I've created Collaboration document for PSK Identity 02

[digiblur]

If 1M backups of 2M flash is fine then rock and roll. I am cool with it .

[finnzz]

@kueblc Thanks for creating the collaboration doc, that's a good idea. This thread is probably only going to grow faster as more people get the PSK error.

@chrisp42 Is this right:
You bought a Powertech SL225X bulb, and it was TC compatible
"prod_idx":"71685817"
Version 3.3.16

You bought more Powertech SL225X bulbs but they were patched against TC
"prod_idx":"55260257"
Version 3.3.30

When you connected the old bulb (version 3.3.16) to the Tuya app, there was no option to upgrade the firmware to 3.3.30, or above 3.3.16.

Did you try to manually update the firmware in the Tuya app? I believe there is an option in the settings to force a firmware upgrade.

You said when you registered the old bulb, it took some time (like maybe something was downloading). I'm curious was the old bulb still TC compatible after the registration with the Tuya app?

So perhaps Tuya isn't issuing a patch to devices that are in consumer hands. Or it's just not available yet though the app. We might need to try a different device or brand.

[scholzmichael]

The stock firmware you flashed must not have been the one from that device. Indeed each device has a unique auzKey stored in flash that is used to authenticate a specific gwId (based on the MAC address) with the cloud. This is why the process described by @finnzz needs to be completed entirely with an original device.

Understood, started the procedure with a new device again, but so far no luck getting an update in the Cloud. I will leave it on there, can't actually take too long in order for them to prepare a new fw for these devices as they arre very common here in Europe

Thanx a lot for the wiki page. As soon as I have News I will try to post there.

[chrisp42]

@finnzz Yes your summary is correct.

I did try to force an update by going way down into the menus, and selecting Check for Firmware Update. The popup says No updates available, Main Module 3.3.16, MCU Module 3.3.16, Confirm. Admittedly this is with a skinned app (Brilliant Smart), but I doubt the Tuya app or Smart Life app would be much different. I have left the bulb on my test AP for a couple of days in case an update came through, but nothing much is happening.

I have just had to shutdown my system, so I may try Smart Life this weekend. And try to re-TC it.

I downloaded the flash again (on both bulbs) after the pairings, but nothing much had changed except some blocks that appear to be configuration.

[scholzmichael]

Admittedly this is with a skinned app (Brilliant Smart), but I doubt the Tuya app or Smart Life app would be much different.

Tried both apps (Gosund and smart life, no difference so far with my device, waiting for a reaction.

[jpeletier]

I don't understand why Tuya does not provide an easy way for us to flash our stuff. After all, the bulb is already sold, the manufacturer paid for the Tuya license and the fewer devices on their cloud after their firmware is replaced, the lower running costs they would have.

I'd argue they'd actually sell more this way.

[claymen]

I don't understand why Tuya does not provide an easy way for us to flash our stuff. After all, the bulb is already sold, the manufacturer paid for the Tuya license and the fewer devices on their cloud after their firmware is replaced, the lower running costs they would have.

I'd argue they'd actually sell more this way.

I'd say it's usually liability, money, and support. Liability being fairly straight forward, think of countries where there are approvals for devices, and the approval is based on that device working as it was submitted. Any deviation of the device from what was originally submitted could mean it is no longer approved for use. Now we could argue that it's not their problem, that the end user modified it, and that we can still mod it manually. Which is probably true, but at least this way it locks out the less skilled person from messing with it, which given what I have seen in some of the automation groups around mains AC equipment... it's probably a good thing.

On the money side, I would imagine that they potentially have customers who have paid solutions in place, and this would allow users to buy a device and then never pay for the subscription. Now I know a lot of it is free, but not all of it is, and at the same time, as a manufacturer of a device you'd have put in some R&D which you'd recoup out of that subscription. So potentially there is a money aspect for some of their clients, so again, easier to cut out the average user from messing with it.

And finally support. People modding it in software, then breaking it, and causing a headache on returns/warranty, or just generally support. Most of us would agree that once you software mod it, all warranty is off. But then what happens if the hardware fails? What if the software flashed on there did something unexpected, switching a relay thousands of times a second till it failed, or constantly wrote to the onboard flash wearing it out?

So there are plenty of reasons to lock out the DIY modders. I know it's annoying, as it was great to easily convert all these IoT devices to something you control. But I can understand why they would lock it down, just be glad we can still hardware flash them.

[Elkropac]

If 1M backups of 2M flash is fine then rock and roll. I am cool with it .

It seems to be really 1M . I have created 2MB dump of flash, then split the file in half and run md5sum on both parts and they are indentical

[MrLemur]

I don't understand why Tuya does not provide an easy way for us to flash our stuff. After all, the bulb is already sold, the manufacturer paid for the Tuya license and the fewer devices on their cloud after their firmware is replaced, the lower running costs they would have.
I'd argue they'd actually sell more this way.

I'd say it's usually liability, money, and support. Liability being fairly straight forward, think of countries where there are approvals for devices, and the approval is based on that device working as it was submitted. Any deviation of the device from what was originally submitted could mean it is no longer approved for use. Now we could argue that it's not their problem, that the end user modified it, and that we can still mod it manually. Which is probably true, but at least this way it locks out the less skilled person from messing with it, which given what I have seen in some of the automation groups around mains AC equipment... it's probably a good thing.

On the money side, I would imagine that they potentially have customers who have paid solutions in place, and this would allow users to buy a device and then never pay for the subscription. Now I know a lot of it is free, but not all of it is, and at the same time, as a manufacturer of a device you'd have put in some R&D which you'd recoup out of that subscription. So potentially there is a money aspect for some of their clients, so again, easier to cut out the average user from messing with it.

And finally support. People modding it in software, then breaking it, and causing a headache on returns/warranty, or just generally support. Most of us would agree that once you software mod it, all warranty is off. But then what happens if the hardware fails? What if the software flashed on there did something unexpected, switching a relay thousands of times a second till it failed, or constantly wrote to the onboard flash wearing it out?

So there are plenty of reasons to lock out the DIY modders. I know it's annoying, as it was great to easily convert all these IoT devices to something you control. But I can understand why they would lock it down, just be glad we can still hardware flash them.

Is there a market for manufacturers selling pre-flashed stuff? https://www.amazon.co.uk/Tasmota-MQTT-WiFi-Bulb-White/dp/B07Y2214VJ

[jschwalbe]

I have a number of unopened light switches that I purchased a few months ago. Since it's not trivial to open them up for flashing, can anyone recommend the best way to know if there is/might be a Tuya update available before I bother with it?

Edited (to reduce noise): Gosund SW1 & SW2, Treatlife 3-way,

[finnzz]

@chrisp42 I was going to do this myself, but I can't find any contact information for Powertech. Is there an email for them?

Could you email them, tell them you have several bulbs, one with old firmware, and ask them if a firmware upgrade will be made available through the Tuya-like app. Maybe they will give you some useful information or a timeline :)

@jschwalbe what's the brand/model, possibly someone else knows if there is a firmware upgrade? Again you could contact the tech support and ask. Maybe ask them the latest firmware version and release date available in the Tuya app, or when they plan to push one out. Not sure if they care why you are asking, but you can just say your device is acting buggy

[chrisp42]

@finnzz Powertech is a brand that seems to be distributed by https://www.techbrands.com/ and exclusively retailed by Jaycar https://www.jaycar.com.au/. The app that is documented with Powertech is the Smart Life app so they don't appear care about their own branding and having their own website etc.
Jaycar is an electronics and hobbyist parts supplier, they should be at least a bit sympathetic to DIY modders. I'll ask around next time I'm in there.

[finnzz]

Ok I'll see if Techbrands can provide a contact

[Fexiven]

If 1M backups of 2M flash is fine then rock and roll. I am cool with it .

It seems to be really 1M . I have created 2MB dump of flash, then split the file in half and run md5sum on both parts and they are indentical

are you sure? even if it would be a 4MB dump, if you split in in 4 parts it would also be the same md5sum on the first split because its maybe just a little part of the firmware, can you see wat's inside the dump with binwalk?

[Elkropac]

If 1M backups of 2M flash is fine then rock and roll. I am cool with it .

It seems to be really 1M . I have created 2MB dump of flash, then split the file in half and run md5sum on both parts and they are indentical

are you sure? even if it would be a 4MB dump, if you split in in 4 parts it would also be the same md5sum on the first split because its maybe just a little part of the firmware, can you see wat's inside the dump with binwalk?

Yeah, i did that too and all 4 parts have the same md5sum. This command says, it's 1MB.
Each 1MB ends with some strings, for example PSK

[Ryan-J]

Wouldn't this be expected for a fresh device that has never been updated? I was under the impression this is how the OTA is implemented. Two equal size partitions with one actively running so the other can be updated and then swapped at boot. Or do I have that wrong?

[Fexiven]

Wouldn't this be expected for a fresh device that has never been updated? I was under the impression this is how the OTA is implemented. Two equal size partitions with one actively running so the other can be updated and then swapped at boot. Or do I have that wrong?

i actually don't know, could be right

[4g3nth4aus3r]

managed to get a firmware dump of my light, looks like they've updated the SDK for mine "OS SDK ver: 2.1.1(317e50f) compiled @ Dec 10 2019 11:05:04"

only thing done with it since unbox was to attempt TC in case i got (un)lucky with old stock with compatible firmware and then use esptool to dump the firmware.

when looking at the dumped strings i'm not sure if the lines i'm looking at relate to 3rd party vendor firmware/config version?
3.3.31
Apr 9 2020
02:33:43
5.57

also not sure if its relivant but there is a string near device.c that starts with key "keyj979nf3q3theh" not sure if its related to a seed for generating the PSK or just something to identify the hardware ID?

firmware-backup.zip

[kueblc]

Please make sure you read the thread or the collaboration document before posting here. The PSK is not generated.

[kueblc]

Future collaboration on solving this issue will take place on the wiki.

Please read this document to understand the issue and what we know so far. It is publicly editable so feel free to add your data and findings after getting caught up.

I am locking this thread now to make sure everyone is on the same page. Once (if) we have a solution, we will post here to make sure all subscribers are notified.