Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question - around serial port output #6

Open
chrish987 opened this issue May 26, 2021 · 23 comments
Open

Question - around serial port output #6

chrish987 opened this issue May 26, 2021 · 23 comments

Comments

@chrish987
Copy link

Does the serial port output anything useful (i.e. can you determine cat going in and out) in normal operations?

I am thinking a possible solution to avoid MITM and hang an ESP32 (or similar) running some custom code off the serial port output.

I am not really bothered about controlling the flap outside the supplied app - just the ability to get cat movements locally without polling unsupported API or delay issues.
@plambrechtsen
Copy link
Owner

There is nothing useful on the serial port at all. Just the boot message and attached device be it feeder or door etc.
The only options are to use the docker stack to receive the MQTT messages from the hub which IMHO is the simplest option or if you want to replace the hub with your own stack then further developing the wemos code to do that.
But if you already have the hub then there is no point as I already have decoded all MQTT messages and if you're building your own hub you still need to decode the messages from the doors or feeder which is the same message as the MQTT messages but with a extra layer of complexity on top.
I personally can't see any easier solution than the docker compose stack I have built.

@chrish987
Copy link
Author

Is it possible to subscribe to a topic in AWS and get the messages from there (once you have the cert) and avoid the MITM? Are the messages on the MQTT topic in AWS still encrypted, or are they decrypted prior to MQTT transmission? Would this mean that the cert might change whenever the hub did a f/w upgrade?

@plambrechtsen
Copy link
Owner

plambrechtsen commented May 26, 2021 •
edited
Loading

The only config you need for MQTT is:
https://github.com/plambrechtsen/pethublocal/blob/main/docker/mqtt/default/mosquitto.conf#L22-L28
So the traffic put onto the MQTT topic isn't encrypted it is just that the hub connects to MQTT over TLS so you need a listener on the right port for the hub to connect to.

https://github.com/plambrechtsen/pethublocal/tree/main/docker#pet-hub-local-hub-altered-boot-process

That documents the process the hub takes to connect to the local instance.

The two main components you need are the Python flask web server to query the official surepet service to get the credentials, alter them then return the credentials file to the hub, and then the MQTT endpoint. Then you have a straight feed of unencrypted MQTT messages on the mqtt topic of your choice. If you have the certificate password by doing the firmware update and getting the password then you can generate your own so you don't even need to connect to the surepet backend to get the credentials file for the first time. But doing the firmware update requires downloading the xor hashed firmware specific for your hub from official surepet service and then caching it locally which the python flask web server also does.
https://github.com/plambrechtsen/pethublocal#aws-mtls-certificate

@chrish987
Copy link
Author

Sorry been a while since I looked at this.

Just trying to understand some of your code. This bit..
#Updating payload dns entry so that you don't need the aws api endpoint and everything points to hub.api.surepet.io which is the locally hosted mqtt instance and use a consistent topic of surepetlocal
payloadsplit=payload.decode('utf-8').split(':')
payloadsplit[6]='pethublocal' #Update topic name
payloadsplit[7]='hub.api.surehub.io' #Update dns entry
Are you are updating the AWS MQTT endpoint to be the same as the HTTP endpoint (I am assuming it is different), and does this mean that if I create a DNS entry for the AWS MQTT endpoint locally then I don't need to run the webserver bit to proxy the credentials web call (assuming I already have the certs etc)? The default topic is a UUID?? - which I assume is fixed and unique to the hub?
Do you think it would also be possible to bridge the MQTT messages from the local MQTT to the AWS instance and then still have the app working but have local visibility of the messages? I assume the local MQTT could subscribe to the remote topic with the same certs?
Last question :). From your diagram it looks like the initial post to /api/credentials has the hub MAC and Serial. If I have these can I craft a POST request in some PC tool and get the cert details back? Do you have the format of the POST request?

I already have MQTT, Node-red, Home assistant, and all my 'logic' in node-red. So I am thinking I might be able to do something with existing components - I am only interested in the in/out messages from the flap (and only have 1 cat) so building some message decode logic in node-red seems like it might be pretty simple.

Thanks for you help.

@plambrechtsen
Copy link
Owner

plambrechtsen commented Jul 26, 2021 •
edited
Loading

The http endpoint is hard coded in the hub to be hub.api.surehub.io so you need to poison DNS as part of the setup anyway to get the hub to hit the webserver. So since I already know that DNS name is local I update the AWS MQTT to be the same endpoint.
If you look through the PolarProxy folder you see how I figured out how to fully MITM all traffic but for that you need to hook up a serial console and capture the firmware update process as it prints the certificate password during the process.

If you look through pethubpacket.py it has all the heavy lifting converting the MQTT messages into something human readable and you will need to look at the 126/127 in the parseframe and parsemultiframe functions and rewrite them.

https://github.com/plambrechtsen/pethublocal/blob/main/docker/source/pethubpacket.py#L263

@plambrechtsen
Copy link
Owner

I also updated the hub documentation with the debug console commands... but there isn't really much useful to get the certificate password out in there just a bit of debug info's.

https://github.com/plambrechtsen/pethublocal/tree/main/docs/Hub

@chrish987
Copy link
Author

I have edited some of the personal values here - not sure how sensitive some of this is

I think I have made some progress on my goal - but not there.

I can't get your web docker image to work for some reason (I am just starting the web container - not everything). It seems to be doing a GET for firmware which seems not to be what is expected from your doc. Get this output:

XXX
root@ubuntu:/opt/pethublocal# docker-compose up
WARNING: Some networks were defined but are not used by any service: petnet
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Starting web ... done
Attaching to web
web | SUREHUBIO hard-coded in app.py
web | * Serving Flask app 'app' (lazy loading)
web | * Environment: production
web | WARNING: This is a development server. Do not use it in a production deployment.
web | * Serving Flask app 'app' (lazy loading)
web | Use a production WSGI server instead.
web | * Environment: production
web | * Debug mode: off
web | WARNING: This is a development server. Do not use it in a production deployment.
web | Use a production WSGI server instead.
web | * Debug mode: off
web | * Running on all addresses.
web | WARNING: This is a development server. Do not use it in a production deployment.
web | * Running on http://172.20.0.2:80/ (Press CTRL+C to quit)
web | * Running on all addresses.
web | WARNING: This is a development server. Do not use it in a production deployment.
web | * Running on https://172.20.0.2:443/ (Press CTRL+C to quit)
web | Post payload : {"serial_number": "H002-XXXXXX5", "mac_address": "000XXXXXXXXXDFF", "product_id": "1", "firmware_version": "2.43"}
web | /usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'hub.api.surehub.io'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
web | warnings.warn(
web | 172.20.0.1 - - [28/Jul/2021 05:50:32] "GET /api/firmware HTTP/1.1" 405 -
web | [2021-07-28 05:50:32,392] ERROR in app: Exception on /api/credentials [POST]
web | Traceback (most recent call last):
web | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2070, in wsgi_app
web | response = self.full_dispatch_request()
web | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1515, in full_dispatch_request
web | rv = self.handle_user_exception(e)
web | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1513, in full_dispatch_request
web | rv = self.dispatch_request()
web | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in dispatch_request
web | return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
web | File "app.py", line 73, in credentials
web | dlfirmware(request.form['serial_number'],0)
web | File "app.py", line 52, in dlfirmware
web | response.raise_for_status() # ensure we notice bad responses
web | File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
web | raise HTTPError(http_error_msg, response=self)
web | requests.exceptions.HTTPError: 405 Client Error: METHOD NOT ALLOWED for url: https://hub.api.surehub.io:443/api/firmware
web | 172.21.96.106 - - [28/Jul/2021 05:50:32] "POST /api/credentials HTTP/1.1" 500 -
XXX

The above however did give me the JSON for the /api/credentials POST, which I have done in a post tool (actually node-red with some javascript) and I get the below output.

Can I get the various certs and keys I need from this? The block of data in position (: delimited) 8 seems to be cert files, but I am not sure of the format or what is here and how many??

v02:63295:99XXX906-21a5-XXXX-XXXX-fa4e031d48de:::1:v2/production/99XXX06-21a5-XXX-XXXX-fa4e03XXXXXXX0226-ats.iot.us-east-1.amazonaws.com:MIIKLwIBAzCCCeUGCSqGSIb3DQEHAaCCCdYEggnSMIIJzjCCBEIGCSqGSIb3DQEHBqCCBDMwggQvAgEAMIIEKAYJKoZIhvcNAQcBMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAjePW87frkcLAICCAAwDAYIKoZIhvcNAgkFADAdBglghkgBZQMEASoEEL6wy3zj6KBp6Lh0Ll9NaqWAggPACeNvQZ1SMXm068HqOnZeFweojAQCCDpfvWHL1sCa+HDbEKCfx+WmrInuCobD/odmUuGcxW2YVQL7rMyk4d8itlp8AvV4Qhh5GvRl4bkDaiWufzUiX3oMzKwB7mKCeHaacOKdd/aapq7EKMIHJLBpacwWwT3bW8jDWxak0TVMTIaGuZ9R4qLVU79Dg23GWTx3Eilat3WV6KaACGTFDj4lc6uZFMgW2eqXiyP369HvTbRR4ER7AjfgfGSoLSMEb0KqRwhqVVBc/rG+ZLyI0gH0dhtj4drsIMunolY4zMyUYLlNpLq/XsSEyjTXWci1xTOuj1K1NfxFpZMIZEiuTq2Eoyr9fR6+9RVufHl3J0nMHDZzalo6FMaLyTd7xvBy3A+bUlLxeadGwgB0fG80WuNuxqP3xaoY8DPZqRs3+7stNkVP8yA3Om13FyXMSN7tWJcVF/Fj53MT6ImaYABGaK9pskZ4Lzarslwx7y4m6VjULljBsC97VOjmOgl7BYpyunucBQrYQWNVQhxwG6cDOi7T8MrUWrtRI+skV6EqL5lLv80JENQNTmDwmuCeknxMbQnCYbgkTFrNWDCerznXJ3sAC9qRaVuvWYvkGC6Vi57kZUuN1WSy/OTobZc/5mG4CnhTNcquzmxhePhe150nvpSnNkItc0Jif+rZ16ZSt2H0FqnDeUK7GQEU78U1nkZh7mk++uY+eEcTB+Fvo0sWWK1kIv4CtswTEqQTG+Ej0/1E4+/dcbhH4358PnulIETZA+DJaJxtS5SL97OEgWG6+F9PFcTB1Y2uztjqW2aUUR7we56AJ7A+ehIrTzhHcXGDch93uIlwnLBRk6maVq+uKac65CHFr2mFniAbLQiKXeZWoC4YojMe27aOSIkB2Fzl/rFumroa/zhufDlUpFb8dnRUb8S2N6D5QZf9OMOhAWe6O2FVRQxa0E4H6k01QxwwwVx6Eo3iKNbcywd5AtgDf9MxCyeErqH4UHukZgChFqWzwp7OP7WJII/GWNqE20b6QsJA5+VTnLXViDjYAMFVNrkkIvwTpaOOUSV7TTuEJgjw2Oeec5j3V9GwrHzfyMS34TeZ5kbcSTm4xZAewQaeCmCp9gvY47Oghy72YocSlV5Y7HOYBlDxUgyIXaPlO3BJgY8NgFuzrjyyAc9kUlrFmCIk9FL0/tMpkPxvxYgn0+LtsMopoosvhx8QqCH4VThhaaAvJyztajmFBNpMV5S4ebI/FzYVneO8OYLxyMgqRtUTuHbAq4nLp2nEthvfoAGAsAnn3ZgrGfl0Tvp7iN7oSRLI4ctXNW1OO0C+5QG1UJ/cYDATNZw6YBdJQNPIsCG8H6sqdofv12L8i9LBtX7xjzN4RzZwD3g/6aG5GSpxtLA9mUJNDVix0G2pgtUi9cxBHQiUuJZRbLFyZuyCfDiKQtw0wFVDu3sfFomALGnrdld1gnlHv+FJXBS/RO2il9JlF9JB6Vk+Q1qRXLjyOyb+3vq4vXnXf8Tcl3LgFv2NRzGmSN3mpGwF8r1IIe1ZtrqgyFtWF2JV6j72kWUDDYtWVAXoiOF9FO3zWD9mLyr9ZAIjHoVqtrbARC4+LTPSPjgnnbiwOynAuzGKUuhN47TaL+YV6cr1CzR8tCVs2RxLEIMSUwIwYJKoZIhvcNAQkVMRYEFBWRX7tNLb27BwlhRyQsjbyEpGN6MEEwMTANBglghkgBZQMEAgEFAAQgSa1n/uZCkYWj8ktOd7Ar+gaxuKIRB47pvWq0IVBEnywECN3XTOpvbJEvAgIIAA==

I have chopped some chunks out of the cert bits above - but it seems to have 6 different blocks that start MII. Do you know how to extract usable certificates from this?

@plambrechtsen
Copy link
Owner

plambrechtsen commented Jul 28, 2021 •
edited
Loading

So it seems my firmware download method has been broken by surepet backend changes and I will try and fix that in the next few days. But the certificate is a PKCS12, it's not multiple different files it is a single payload. You need the password to open the certificate up as it is encrypted with a 32 character / 16 byte string which is the "long_serial" that is stored in the hubs protected flash and is unique per hub. The only way to get the password is to hook up the serial console and do the firmware update process as documented here:

https://github.com/plambrechtsen/pethublocal/blob/main/docker/source/fwlogtopw.py

You could probably try and brute force it, but it would take years with JTR or similar tools.

I have also documented the fields for the credentials file here under credentials, the second field in your above post is your hubs serial number:
https://github.com/plambrechtsen/pethublocal/blob/main/docs/Hub/README.md#debug-menu

        Host:           **8th Field in Creds file**
        Client ID:      **3rd Field in Creds file**
        Base Topic:     **7th Field in Creds file**
        Version:        **1st Field in Creds file**
        ID:             xxxxxx (Serial Number which is also 2nd field ) and ClientID as a single string
        Username:       **4th Field in Creds file, should be empty**
        Password:       **5th Field in Creds file, also empty**
        Network Type:   1 **6th Field in Creds file**
        Certificate:    **9th Field in Creds file**

But I do highly recommend spending the time to get the certificate password out, all it takes is a bit of soldering and a TTL serial adapter but once it is done you are set to MITM or do whatever you need and who knows if surepet change their backend in the future preventing it.

@plambrechtsen
Copy link
Owner

Added a stand alone firmware downloader, just takes a command line argument of your serial number:
https://github.com/plambrechtsen/pethublocal/blob/main/docker/source/downloadfirmware.py

@chrish987
Copy link
Author

more progress. I now have the password from the hub.

Still trying to extract the certificate. I have the raw data from the web response, and the password (from the fw upgrade). How do I get the certs. I am trying openssl:
openssl base64 -d -in cert_raw.pfx -out conv.pfx
openssl pkcs12 -in cert.pem -nocerts -out test.key
My 'cert' data is just one long string starting MII and ending ===, no cert headers or anything.

I will keep trying different options - but thought i would ask in parallel.

Thanks for the help again.

@plambrechtsen
Copy link
Owner

plambrechtsen commented Jul 29, 2021 •
edited
Loading

This should help with the PolarProxy stack.

https://github.com/plambrechtsen/pethublocal/blob/main/PolarProxy/gomqtt.sh

awk -F":" '{print $9}' ../docker/output/web/$serialnumber-$macaddress-2.43.original.bin | base64 -d > $serialnumber.p12

openssl pkcs12 -nodes -passin pass:$certpassword -in $serialnumber.p12

@plambrechtsen
Copy link
Owner

You can always email me directly if you have some messages you don't want to post publicly as they contain personal information. As I think this case has been covered let me know if there is more help you want otherwise I will just close this ticket.

@chrish987
Copy link
Author

I have this nearly working :) It did work briefly. I naively thought that the hub getting the cert might be a one time thing - not a new cert every time it boots :(

I am trying to setup mitmproxy (which I use elsewhere) to grab the cert data when the hub boots. I am having issues with the certs for this - I assume that the hub validates the hub.api.sureflap.io cert when it boots?? how did you workaround this in your flask webserver?

@plambrechtsen
Copy link
Owner

plambrechtsen commented Aug 2, 2021 •
edited
Loading

The hub.api returns the same cert each time the hub boots... or it seems to from my experience and it sometimes gets rolled but the AWS IOT Client Cert is good until 2040 but surepet seem to re-issue it periodically.
Check out the script I created for PolarProxy as that should be easy to alter to download the creds first, but re-using the same cert works fine.
I have created a self signed certificate for hub.api which the hub happily accepts (thankfully, otherwise none of this project would work) and then cache the creds file locally so each subsequent boot from the hub to flask I return the locally cached version as that version is fine to work with the hub as the hub is happy to decrypt it.
Flask doesn't make the backend call back to the real hub.api each time as we don't need to do it more than once, but just returned the cached and modified version copy of the full creds file if it exists locally.

@chrish987
Copy link
Author

chrish987 commented Aug 2, 2021 via email

@plambrechtsen
Copy link
Owner

I would stick with PolarProxy rather than mitmproxy, as I had all sorts of problems getting mitmproxy to work correctly as the http headers needed to be returned in the correct order for the hub to accept it. The content-length header needed to be returned last.
Are you trying to MITM the hub.api https traffic or the AWS MQTT traffic? As the AWS traffic is really what you care about and if you look in the gomqtt.sh script.
https://github.com/plambrechtsen/pethublocal/blob/main/PolarProxy/gomqtt.sh
And then alter it as needed, and check the last line in the script where it calls PolarProxy:

./PolarProxy -v -p 8883,1883 --autoflush 10 -o mqtt --insecure --clientcert a5kzy4c0c0226-ats.iot.us-east-1.amazonaws.com:$serialnumber.p12:$certpassword --servercert a5kzy4c0c0226-ats.iot.us-east-1.amazonaws.com:iot.p12:password --nosni a5kzy4c0c0226-ats.iot.us-east-1.amazonaws.com

That will nicely spit out decrypted pcap files you can easily load into wireshark, or you can even pipe the output somewhere else.

@chrish987
Copy link
Author

chrish987 commented Aug 2, 2021 via email

@plambrechtsen
Copy link
Owner

To get mosquitto running as a MQTT Broker that the hub can connect to you can either use my docker stack or just this snip from the config plus the associated certs in that folder is what you need.
https://github.com/plambrechtsen/pethublocal/blob/main/docker/mqtt/default/mosquitto.conf#L22-L28
Needs to be listening on port 8883, and the aws.pem is the public key of the AWS IOT CA, then the hub.pem/key are just self signed certs.

@chrish987
Copy link
Author

chrish987 commented Aug 2, 2021 via email

@plambrechtsen
Copy link
Owner

The aws.pem is the public key of the AWS IOT MQTT CA since the hub uses a client certificate which is the PKCS12 certificate passed in credentials to connect to AWS. I assume you're using mosquito MQTT broker?
The broker needs to present a MTLS Client Certificate challenge as per

https://github.com/plambrechtsen/pethublocal/blob/main/docker/mqtt/default/mosquitto.conf#L26

Which points to the aws.pem which you can't change.

There is the iot.pem/key in the same folder which should work if you're using the AWS endpoint rather than the hub.pem/key.

Feel free to email me directly

@chrish987
Copy link
Author

chrish987 commented Aug 3, 2021 via email

@chrish987
Copy link
Author

were you able to have any thoughts on my issue above. It seems pretty simple that the hub could connect to an internal MQTT broker with just the generic server certs that you have on this site. Not sure where I am going wrong.

@plambrechtsen
Copy link
Owner

Not sure... it seems very odd since other folks have managed to get it working. I would suspect it is something network related, what is the main host OS you are running, and I assume that you have sorted out the DNS spoofing? Send me an email and I might be able to help diagnose what is going on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@plambrechtsen @chrish987