NIP-104: Double Ratchet (End-to-End Encrypted) DMs #1206
Conversation
|
|
||
| ### KDF Chains | ||
|
|
||
| The double ratchet algorithm used here depends on KDF (key derivation function) chains. If you're not familiar with what those are, the Signal docs have a great [description of KDF chains](https://www.signal.org/docs/specifications/doubleratchet/#kdf-chains), you should read it before moving on. |
|
|
||
| #### Root Chain Key (Diffie-Hellman Ratchet) | ||
|
|
||
| The initial key for the root chain is calculated based on the X3DH key exchange described above. From that point onwards, each time the DH Ratchet is turned, a new RK and a new CKs or CKr is output. This ratchet provides post-compromise security. In other words, if an attacker gets steals one participant's CKs and CKr, they can calculate all future message keys and encrypt/decrypt future messages. Ratching the root chain breaks this by creating a new RK and a new key for each sending participant's sending and receiving chains. This ratchet is turned twice each time the active participant changes. For example, if Alice sends 1 message, Bob then sends 3, and Alice sends 1 more, each participant will have performed the DH ratchet 6 times. Twice for each change in sender so that we can calculate a new CKs and CKr. |
|
Nice! I think this idea of using a double ratchet inside GiftWraps is neat. It basically creates a second encryption layer, with different keys that are not stored inside relays and thus cannot be decrypted even if the main key leaks, while keeping the GiftWrap's routing mechanism intact (poor man's explainer for Backward and Forward Secrecy) I'd call this PR Double Ratchet DMs just to differentiate from all the other Private, end-to-end DM PRs around here. :) I would love to get @paulmillr's take on this. |
|
Any effort to improve DMs is good. I didn't do the ratcheting/hierarchy of keys in NIP-44, because it's too complicated and has many non-obvious caveats. I have been discussing this with @erskingardner directly. Have been waiting for a working demo that allows outside researchers to audit how it would work in real-world scenario. @erskingardner does the demo contain full functionality now? |
|
Hello, I am new and pardon me to ask stupid question, how about Pre-Shared Key? so Alice and Bob could just use the specify agreement key without something Diffie-Hellman exchange handshake. That could maybe more secure or keep secret? |
|
this is great, just want to add, look at finding a way to do MLS as well and the misery of nostr DM will at last be over |
|
@mleku just take a look at RFC 9420 and try to comprehend it. It's complicated beyond any reasonable means. Which means it won't be implemented simply by volunteers. Which means it's much worse. It has also been focused on group chats - a different beast. |
| "sig": "" // No signature because these are only sent inside gift wraps. | ||
| } | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Clients MUST check if the pubkey of kind 443 and 444 is the same as the Seal's, kind 13, pubkey. Otherwise, impersonators can impersonate. :)
| } | ||
| ``` | ||
|
|
||
| The `"prekey_sig"` tag value is a Schnorr signature of the sha256 hashed value of the prekey's pubkey, signed with the prekey's private key. |
There was a problem hiding this comment.
Clients MUST verify the prekey_sig before accepting this event.
This allows anyone out there to copy the prekey_sig string and resign it under their own main nostr keys. Shouldn't the signature be a hash of (nostr.pubkey || prekey.pubkey) to avoid the shenanigans of just reusing somebody else's key?
There was a problem hiding this comment.
In my demo I'm verifying the signature, but I'll make it clear that it's a MUST.
Since the prekey has the signature of the user's nostr identity key in the sig field, and the signed value in the prekey_sig tag value, I think you're covered there from imposters? But maybe not? Maybe you can elaborate a little on what you're thinking here?
There was a problem hiding this comment.
I don't know. As it is today, one can just copy any other user's prekey tag and it will be valid. They won't have the private key so maybe they can't do anything with it, but there might be a social attack somewhere. The event doesn't prove the prekey is linked with the main key.
There was a problem hiding this comment.
True. that's a good point. I was thinking validation of the event + validation of the prekey signature would be enough but you're right, they aren't technically linked and it's easy to link them.
I don't think it's strictly necessary given how we do the multiple pairs of DH calculations in the initial key exchange (which does require private key signatures from both the prekey and the identity key) to derive the initial root key but it certainly wouldn't hurt.
| ["p", <pubkey of the recipient>], | ||
| ["dh_sending", <current DH sending pubkey>], | ||
| ["current_index", <number of current message in the current message chain>], | ||
| ["previous_length", <length of previous message chain>] |
There was a problem hiding this comment.
When sharing the key between clients/devices, do we also need to share the current rachet state (current_index, previous_length)? Or should a new client start from 0?
There was a problem hiding this comment.
This DOES NOT work out of the box on multiple clients or devices. Each client/device combo should be thought of as a different "inbox" that will have it's own initial setup and will maintain it's own set of chains/keys for each conversation with another person. At a very high level, say I have Primal on iOS and Amethyst on Android and you have just Amethyst on Android. You're client would have to know that it's having the same conversation with two device-clients but using different sets of keys/chains. It would encrypt each message you send for each recipient and then publish those GW events separately.
There was a problem hiding this comment.
This is similar to how this sort of scheme works for group conversations as well.
There was a problem hiding this comment.
@staab @paulmillr high level description of multiple devices question here... ☝️
you can also read more about how signal does it here: https://www.signal.org/docs/specifications/sesame/
I want to read up a bit more before committing to any path here. In an ideal world we have multi-device conversations, it's significantly more complex for clients. From the beginning I viewed this more as a solution that a dedicated secure messaging client would be using, and less as the normal standard for DMs in social clients. That said, nothing stopping those clients from implementing it, just a question of making the UI good enough.
There was a problem hiding this comment.
If that is the case, then I would add a:
Clients MUST not re-use/re-import keys from other clients or devices. The Key is not supposed to leave the app/device it was created from.
There was a problem hiding this comment.
I agree. I need to make this whole device/client limitation more clear in the NIP.
| "content": <ciphertext of initial message symmetrically encyrpted using first sending chain/message key>, | ||
| "tags": [ | ||
| ["p", <pubkey of the recipient>], | ||
| ["prekey", <pubkey of the recipient prekey used for DH calculation>], |
There was a problem hiding this comment.
Change the tag to recipient_prekey to make it more clear?
|
Initial thoughts:
|
No, I think synchronization needs to be solved for this to become the de-facto messaging standard. But I'm ok with deferring that discussion until later. Nice work btw @erskingardner, very excited for the prospect of NIP 17 eventually being improved upon. |
|
Signal doesn't allow multiple devices. Their web version proxies stuff from phone, which acts as a main device. It doesn't seem like syncing could be solved in a simple way. Whatsapp allows multiple devices, may be worth looking into. |
|
Keep in mind that, in Nostr, DMs should work not only multiple devices, but also multiple Clients in the same device. Signer Apps can be used to share pre-key private keys among many Clients, however, the big question on my mind is that if all clients and all devices get a copy of all these pre-keys, doesn't it break the Backward and Forward Secrecy just because the prekeys are being inserted into all the places that the main private key also is? Meaning... wouldn't all keys also leak if the private key leaks? And if so, what's the advantage of this scheme over the basic NIP-17? It feels like we need a scheme where each app/device has a pre-key and can talk to all other pre-keys in other devices. |
|
Glad to see this is finally happening. We attempted to solve this problem a while back but there was no engagement from anyone in the community so it fizzled out: #410 closing in favor of this. |
This must be done. The only alternative to having this is centralization among a few big relays. |
copy change Co-authored-by: Max Hillebrand <30683012+MaxHillebrand@users.noreply.github.com>
erskingardner
changed the title
@paulmillr Yes, it has the basic full functionality. I need to build more in the demo so I can test out of order and lost messages but in theory they should already work. |
@vitorpamplona the prekeys are basically per "device-client". I'm imagining that the clients themselves manage the prekey but I hadn't actually thought through the management of the private key of the prekey. You're right, that a remote signer like nsecbunker or otherwise could handle that but it's not ideal. An easy way to solve this would be to make these PRE instead of simple replaceable events. That way each device-client could manage it's own prekey. There is a can of worms there though on which prekey a user trying to connect would use for the initial conversation request event... I'll think about this a bit more. My initial thought here was that these sorts of DMs would be single purpose client instead of a another feature in every client out there. But maybe that is too narrow a view. |
|
|
||
| ## Known trade-offs & open questions | ||
|
|
||
| 1. Messages are decrypted and stored on each device/client pairing. This means that there is no way to load and decrypt historic messages directly from relays on a different client or device. However, you can create backups of your decrypted conversations that could be loaded into another device (this is beyond the scope of this NIP). |
There was a problem hiding this comment.
How do the other protocols (Signal and OMEMO) address this issue? Do they rely on at least one client to be online with the full history to relay it to the new client?
There was a problem hiding this comment.
No, in most cases that I've seen there is no historic conversation syncing via the protocol. With some of them there is a way to download/backup chat history and then load that into a new client (Signal does this) but it's not syncing conversations between devices ever.
One thought I had about this was that if Blossom becomes something then it would be a way that we could sync this chat data but I think that's still too early to tell...
There was a problem hiding this comment.
https://conversations.im/ suggests "If you are installing Conversations on a new device or catching up after being offline for a while, Conversations will use XEP-0313: Message Archive Management to fetch the message history from your server.". Do they mean that it is a new device with the same private key loaded from an old device?
There was a problem hiding this comment.
Signal does not address the issue. Signal can't download historical messages and doesn't have sync (besides a very trivial one, phone-to-desktop-web that relies on phone-to-web pairing and LAN webservers)
There was a problem hiding this comment.
Signal does not address the issue. Signal can't download historical messages and doesn't have sync (besides a very trivial one, phone-to-desktop-web that relies on phone-to-web pairing and LAN webservers)
That's partially true. They don't handle syncing but they do have a mechanism to back up the history and then load that file into a new device (done via bluetooth) https://support.signal.org/hc/en-us/articles/360007059752-Backup-and-Restore-Messages
@AndySchroder No idea. I think we have to approach the syncing question from first principles based on how Nostr works instead of just trying to copy whatever XMPP is doing.
With signal, there are two types of prekeys, basic prekeys and one-time use prekeys. The basic prekeys are rotated on a regular basis but aren't onetime use so they can't be exhausted. The addition of a one-time prekey does give you better replay attack protection but I chose to leave that out to keep things more manageable for now.
I'm not sure what this means, it doesn't matter how many or which relays you publish to, as long as the other user shares some relays (or they use outbox) then they shuold be able to find you. |
@paulmillr The clients aren't syncing keys with each other they're always deriving them all from the initial shared root key (secret key). And bob can't "send" message while offline. If the client can't publish the event to a relay it should fail immediately so bob knows that those messages weren't ever sent. retrying at a later date with those messages would have to create a new 444 event encrypted off the latest keys in the chain instead of trying to just republish. For generally missed messages (either they arrive out of order or anything else) the incoming messages give the pubkey (root key), the message number, and the previous chain's length of messages. The receiving client can use that to ratchet through the right chains to create and store the message keys for use later when (hopefully) the missing messages arrive. If they never arrive, the client could show a message to the user "missing message" if it wanted to. |
@fiatjaf what about the fact that it's more obvious to people watching that you're receiving DMs? I guess it's still speculation but it is a degree more information about what's going on than a bystander would otherwise have... |
A possible solution could use time-window keys and sharing those keys with all of the other clients: Build another ratchet that creates a new pre-key every month. You can share that prekey with other clients, but never the main ratchet key. Meaning, if the inner key leaks, it leaks only a window of time. Within that time, all clients can encrypt and decrypt. Users would need to periodically reload all clients with new keys. |
DH is the most secure way to agree on a shared secret value and it also authenticates the user's to each other in the process. |
People can select relays that will hide events targeting you from non-AUTHed users, same as in NIP-17. For that reason it is better to allow people to select specialized DM relays other than their default NIP-65 relays. |
|
Great work but unfortunatelly as other people said, there is the lack of multi-device support problem, which is a noticeable downgrade from NIP-04. |
|
I think the threat of total device compromise is far greater than the threat of a cryptographic key being derived via cryptanalysis. In that much more common scenario, whatever your device can do the attacker can do. If you can read historic messages, so can eve. If you can't, well that's quite a usability shortcoming. I'm not against this double ratchet NIP. I think this is very cool. I just want to (if I'm right) make people aware of how limited of an improvement all this complexity is providing. It ticks a marketing box and limits the damage caused by certain kinds of remote cryptographic breaks. But that threat was never as great IMHO as device compromise is. Your devices have |
Good point. I do make a note about using only relays that support AUTH but I can make that more clear here. Will update. |
First, I think we should do the best we can, even with all the complexity it adds. My point is just that we should be careful in our communications. If we ever say "this is the maximum security possible" in our pride, we should also be very clear that device security matters more and that we can't do anything about that. I don't want non-techies thinking this is super secure way to do DMs and then communicating about really dangerous topics, and then getting executed by their state because their devices weren't secure. |
|
I agree @mikedilger - it's really important that people don't think this is some sort of panacea solution that makes their comms perfect. |
the model of relays introduces a trust problem, in herently, and at minimum, a signals intelligence metadata leak problem any discussion about "what we can do" on this protocol should omit any pointless handwringing about the fact that it is inherently not trustless if you want a trustless relay network, you need to create a trustless way to pay for relay service, and you need to enable clients to sync state with each other... these are the two problems that a trustless p2p network faces, and any discussion about what nostr can do that pecks at the problem of what it inherently can't do is a total waste of time |
|
@erskingardner what is the threat model you’re trying to work in, with this improvement? Is it different than the previous one? |
The user I have had in mind since starting this work is activists in hostile jurisdictions. People that are working under authoritarian rule and need to have secure communications that can't just be blocked by the regime in power. The goal of this NIP is to create a scheme for secure direct messages between two individuals on Nostr that does three main things:
The tradeoffs inherent here are: (if anyone has others, please speak up)
I do think that @mikedilger's point is fair. The chances of device compromise is likely the largest risk that any of us face. It's also the one that is most understandable to a nontechnical user). Having as unimpeachable as possible encryption and metadata protection coupled with an unstoppable network goes a long way towards giving people piece on mind on the things that they can't control and leaves them to worry about the things they can control (their own security and device practices). |
|
Thanks, that’s thoughtful. What do you think about incorporating ML-KEM aka Kyber to gain additional pq security? I wanted to do it as v3 of nip44 at some point. Signal already added it. |
Not to discourage this PR, but I fail to see how this PR improves the situation for that specific user. I initially thought this was for other types of users. As described before, this PR only provides benefits IF Ratchet keys are kept separate from the private key. If they are kept together, in the same client (both leak at the same time), this PR provides exactly the same privacy and security as NIP-17. I think we all agree that NIP-44 encryption is nearly unbreakable, so there are two main threats for activists:
The use of ratchets by themselves doesn't solve either 1 or 2. But, if this PR could force clients to use security chips to store the ratchet information in such a way that no one can take them out of the phone/browser, then it would solve it. But that is a big IF that is barely mentioned in the PR. However, if that way of securing secrets can be coded, the same privacy gains could be achieved with burner Nostr accounts that use similar inability to export protections. And it would be a simpler implementation.
This is also very unclear to me. If the previous point is not solved, there is no additional deniability provided. If private keys and rachet info can be accessed by attackers 1 or 2, then this is exactly like NIP-17, but more complex to code. After all, GiftWraps with the user's main pubkey are still flying back and forth. NIP-17 key aliases could solve that, but we don't really explore that option neither here, not in NIP-17 (because it also needs its own little protocol to let friends know which keys to message). But again, even Key aliases are just the same thing as a rotating Nostr burner account. Their gains are small considering the alternatives. So, unless I am missing something, the use of rachets alone does not address the problem you are trying to solve. It's how you manage those rachet states, or other crypto primitives, that might solve it. But if that is the problem, then we might not need ratchets at all. -- Now to the tradeoffs, I actually think they are the biggest problem we face today.
This can be solved with either Tor or our own GiftWrap routing network using DVMs to forward GiftWraps to the next relay/DVM until it gets to the user. I like building our own GiftWrap routing network better just because it solves privacy (of transferring a GiftWrap with the user's main key listed as p-tag) and IP tracking at the same time.
This is a problem that we tried to solve with key aliases on the GiftWrap. It works, but we need a protocol to create a priority inbox where users can send giftwraps to and other users would be listening. The protocol can rotate keys constantly, with a ratchet, or just randomly. It's not difficult, it's just more work.
If by "compromised" we mean confiscated, I do think the ratchet here is a good solution, provided that we can keep the Ratchet's key away from people who have physical access to the phone. If by "compromised" we mean some code is running in the phone or desktop to read these messages, there is literally nothing we can do at the NIP level. Clients can make things harder for non-root agents, but it is up to each client to code these things correctly. |
|
I think these are the differences:
If NIP-17 or 104 could tick the remaining boxes it would be perfect, even if increasing complexity. |
|
Ok, if we had to build something from scratch for "activists in hostile jurisdictions". How would you do it?
In that way:
Anything else? |
|
Whoa a regular nostr user doesn't need all of that for sure =o. If an activist is kidnapped and forced to use biometrics to unlock the phone it is game over. Malware on the phone is mostly game over too. |
Sure, but we could make a client just for high-risk individuals to truly protect them, even from their own user mistakes. That's the goal of this PR, I believe. Build something that activists can use without thinking too much about their own custom security.
Not on this model... because the biometrics wouldn't open anything up since the key is not in the phone and discarding an NFC tag, especially if its paper-based, is imperceptible. |
|
I appreciate the extra level of scrutiny @vitorpamplona. While that was who I had in mind, I think that we probably want to work towards the fully paranoid version, and also not wait to get this out there. Getting forward and post-compromise security + having this built on an nearly unstoppable network is already a huge benefit over tools like Signal. IMO, the threat model for having your phone confiscated with Signal on it would be the same as this PR would provide, which is a good starting place. And clients do have a little bit of leeway in how they implement this spec (along with others that would be required for a truly secure messenger). It'll be up to those clients to be clear about how secure they are and what steps they are taking to protect users. In other words, this spec is only here to provide clear direction on how to set up double ratchet e2ee DMs that are tolerant of skipped/out of order/never arrived messages. It's not meant to detail every single thing that a good client design would require to be truly secure.
@paulmillr - yes, I did look a bit into Kyber and PQ security but I haven't really gotten to the point where I understand it. If you're keen, I'd love some help on that. BUT, it probably belongs in a v3 of NIP-44 instead of this PR, wouldn't you agree? @arthurfranca thanks for the chart. I'm not sure where the column about alerting for new device usage came from since that isn't something we've talked about yet, but it's definitely doable. I'm still writing up details on how we'd do multi-device sync with NIP-104 but I'm confident that it's going to work and we can also let users know/see a list of the active devices that are watching/listening and revoke those at any time. |
BTW @vitorpamplona I really like this idea. It's super cypherpunk. 🤘 It might be a fun thing to mess around with and involve @arcbtc |
Isn't that true for the base NIP-17 as well? If a user is using this PR with private key + rachet info loaded on the phone or NIP-17 with private keys on the phone, the confiscator will see the same amount of information. This PR doesn't offer anything extra in that case. This PR would only help if the confiscator gets the device that has the private key, but doesn't have the ratchets.
A NIP-49-like ncryptsec model can be where people store their ratchet information. The private key could be in the device at all times, but the user can enter the cypherpunk mode when loading the rachet info into it. Then more chats appear and now the user can reply with the ratchet information. |
|
@vitorpamplona the ratchet keys are destroyed as they are used. We don't keep around the keys forever. Instead, once the messages are decrypted, it's up to the client to store those in a way that is safe on the device. This will vary significantly for each device type and client but I think there are many ways to make this safe, even for confiscation situations. |
|
Hey folks, apologies for the silence on this PR for so long. I've been privately talking with lots of people about this Double Ratchet spec. The main feedback from folks was that if we are going to the trouble of doing something complicated we should make sure that it handles group messages and multi-client/device (which is basically group messages) very well. I agree. With that in mind, I started doing more research in to the MLS protocol. For those not familiar, MLS is basically a modern reimagining of the signal protocol that makes group messaging way more efficient (log vs linear) and was built to be used in centralized or federated environments. I've heard nostr devs talk about it in the past but it was always overlooked as being too complicated or too early. However, I now believe that it's actually the best solution for getting secure direct and group messaging in Nostr. It also has the added benefit that we can upgrade the underlying crypto primitives over time in a sane way. The MLS protocol specifies "a key establishment protocol that provides efficient asynchronous group key establishment with forward secrecy (FS) and post-compromise security (PCS) for groups in size ranging from two to thousands." The spec I (and anyone who wants to help me) will create will detail the ways that we implement this protocol into the Nostr environment (namely, how do we use our crypto primitives, use events as control mechanisms, and use relays for storage, while obfuscating metadata). Now, the catch. MLS really is much more complicated than the spec I've detailed here. I'm still researching the details and understanding what needs to be done in order to bring this to Nostr but I've done enough at this point to be quite sure that it's possible. Here's my plan (and I'd love feedback on this):
cc/ @pablof7z @fiatjaf @vitorpamplona @paulmillr @MaxHillebrand @jb55 @mleku @staab @mikedilger |
|
@erskingardner check out xmtp's implementation, I'm sure it will be informative: https://github.com/xmtp/libxmtp/tree/main/xmtp_mls |
i think it's worth noting that forward secrecy can be valuable in the event of partial compromise. The double-ratchet scheme does not offer any protection better than NIP-44 in the event of total device compromise, but ratchet keys for example are more likely to be in-memory than root keys are at any given moment... so any sort of attack vector dealing in side-channel malware, temporary device access, etc or anything else short of total device compromise, there may be a scenario where forward secrecy definitely can save your ass and should be considered when comparing the security profiles of NIP44 vs NIP104 |
This is not a good mental model. The root keys are almost always in memory because every action needs to be signed in Nostr. For instance: in some cases even seeing a post is a signed event to mark as read. |
Ah ok that's a good point. I guess was speaking generally in the context of an end-to-end encryption scheme where compromise of per-message keys vs the keys that derive per-message keys might actually matter. My point was just that forward secrecy can be valuable in certain situations. That sucks that the root key can't be protected better... |
|
Yeah, this was one of my concerns when thinking about these ratcheting schemes: they are all conceptualized with the idea that the root key or seed words are closer to a cold storage situation. But Nostr is as hot as it gets and the Nostr key can be much hotter than the ratchet keys. So, if the ratchet keys are derived or stored by the root key, it's easier for an attacker to simply dismiss the entire encryption scheme and just attack the root key itself, breaking all the claims of the encryption procedure. |
This NIP adds several events and details a protocol to add E2EE DMs to Nostr that use a double ratchet (based on the Signal protocol, but without the centralized servers).
Easy to read version