Determinstic secrets / ecash restore

[callebtc]

This PR introduces the ability to restore the user's ecash balance from a single secret with the help of the mint.

Teaser: https://twitter.com/callebtc/status/1620186555993456641

User testing

To test this feature, please install cashu from the GitHub repository. Then add the testnut mint that already runs this branch. This is the process:

cd cashu/
git pull && git checkout deterministic_secrets
poetry install && poetry shell

# add the testnut mint to this wallet
echo "MINT_URL=https://testnut.cashu.space" > .env 

Your wallet should be ok now. You can now mint tokens, send them to yourself, delete your wallet and restore it:

# running cashu once will now generate a seed phrase for you
# you can check the seed phrase with and copy it somewhere
cashu info --mnemonic

# now for the fun part
cashu invoice 12345
cashu send 10
cashu receive <token>

# let's delete your wallet
rm -r data/wallet

# let's restore it - enter your seed phrase
cashu restore

# your balance should be restored now
cashu balance

Deterministic secret generation

The wallet generates a private key and has a counter (called counter below) that starts at 0. There are two (previously random) inputs for token generation: secret and blinding_factor to generate BlindedMessages.

Previously, we used random numbers for these. Here, we generate them deterministically from the private_key and a counter that increments for every token we create.

Deriving secrets

We produce a standardized BIP32-compatible derivation path for generating secret and blinding factor r for each token. The path encodes:

• purpose' = 129372' (UTF-8 for 🥜)
• coin type' = Bitcoin 0'
• keyset id' = Keyset ID as integer keyset_id_int'
• coin counter' = counter' <-- this one goes up
• secret or r = 0 or 1

Thus:

secret_derivation_path = f"m/129372'/0'/{keyset_id_int}'/{counter}'/0"
r_derivation_path = f"m/129372'/0'/{keyset_id_int}'/{counter}'/1"

The wallet increments counter after every derivation. When the wallet restores its funds, it restarts counter at 0 and increments it as long as necessary to restore its balance.

Quirks

When a wallet sends a POST /mint request, it attaches BlindMessages to it that will necessarily increment the counter. However, if the invoice has not been paid yet, they won't be converted into Proofs. Therefore, say, if the wallet would attach 10 BlindMessages and tried to request the mint 5 times without success, the counter would be incremented by 50 without having any corresponding Proofs for the appropriate values. The number of missed counter values a wallet might have missed is not easy to predict either. Therefore, a wallet MUST not increment counter for failed POST /mint requests but only for successful ones.

Backup restore

After a database wipe, we can use the deterministic derivation to re-generate the same BlindedMessages that we had generated before. Generally, the user won't remember the state of their last counter before the database wipe, so we generate many of these, say 100 per request.

We can send these 100 BlindedMessages to the mint and ask it politely to re-issue the BlindedSignatures for them. The mint was so kind to keep these in their database. The mint matches the provided BlindedMessages against its database and returns all BlindedSignatures it issued for these matches.

The wallet uses the endpoint POST /restore and sends an ordinary PostMintRequest (which includes the BlindedMessages (this is the same object we use for POST /mint).

The wallet then receives the object PostRestoreResponse:

class PostRestoreResponse(BaseModel):
    outputs: List[BlindedMessage] = []
    promises: List[BlindedSignature] = []

The fields outputs and promisesonly contain positive matches. If we had 10 tokens during the wallet's life time, we will receive 10 matches (and not 100 objects as we sent to the mint). The wallet now needs to take theseoutputsand filter the the correctsecretandblinding_factorfrom the100` it generated in the beginning of the restore process.

Things to decide:

• How to generate private keys and let user enter private keys? -> User enters BIP39 seed phrase
• How to derive secrets? -> BIP32
• Should I derive secret from r or generate them independently? -> independently

Todo:

• Verify that failed POST /mint requests do not increment the counter
• Restore tokens from multiple mints
• Store derivation path of each Proof in the db
• Store private key in database
• Restore tokens for multiple wallets

[codecov]

Codecov Report

Patch coverage: 71.63% and project coverage change: +0.44 🎉

Comparison is base (3374563) 56.18% compared to head (262c9ea) 56.62%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #131      +/-   ##
==========================================
+ Coverage   56.18%   56.62%   +0.44%     
==========================================
  Files          43       43              
  Lines        3713     3961     +248     
==========================================
+ Hits         2086     2243     +157     
- Misses       1627     1718      +91     

Impacted Files	Coverage Δ
cashu/mint/router.py	0.00% <0.00%> (ø)
setup.py	0.00% <ø> (ø)
cashu/mint/ledger.py	28.60% <7.69%> (-0.71%)	⬇️
cashu/wallet/api/router.py	69.33% <41.66%> (-2.80%)	⬇️
cashu/mint/crud.py	70.42% <50.00%> (-2.31%)	⬇️
cashu/wallet/cli/cli.py	48.76% <56.86%> (-0.63%)	⬇️
cashu/wallet/wallet.py	80.48% <79.42%> (-1.46%)	⬇️
cashu/wallet/crud.py	73.98% <88.23%> (+2.02%)	⬆️
cashu/wallet/helpers.py	81.65% <88.88%> (-0.60%)	⬇️
cashu/core/base.py	91.30% <100.00%> (+0.11%)	⬆️
... and 7 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[xphade]

Hey @callebtc, very cool new feature, I really like it.

I had a high-level look through the code and it looks good to me, just some minor comments. I probably don't yet understand all implications of this change though. 🙂

One additional question, from your description:

Should I derive secret from r or generate them independently?

Looks like you decided to generate them separately. Was there any specific reason for this or just a random choice?

Also, this will probably become a NUT, right?

[callebtc]

Hey @callebtc, very cool new feature, I really like it.

I had a high-level look through the code and it looks good to me, just some minor comments. I probably don't yet understand all implications of this change though. 🙂

One additional question, from your description:

Should I derive secret from r or generate them independently?

Looks like you decided to generate them separately. Was there any specific reason for this or just a random choice?

Also, this will probably become a NUT, right?

Hey there, yes at least for now r is independently derived, and not derived from x.

There are intricacies that arise if you derive one from the other. In order to keep those out, I've decided on this approach.

I will write a NUT for this for sure after it has proven to work well!

[RydalWater]

Great work, I tried to break it and found it difficult to do so. Will try put some eyes on the code, though with the codebase being quite mature as a whole it is difficult to fully create a mental model without substantial time reviewing/dabbling.

Main test that found inconsistent behavior was around the initiation of restoring wallets while there were pending ecash sends at play. Sometimes able to receive previous sends even after wallet is restored (simulating alternative device). This leads to confusion and inability to accurately get pending information. It also can result in it being difficult to initiate new sends potentially pinning the funds (temporarily?).

Also performed test for 1000 cycles of invoicing (1sat per cycle) from test mint. Got to around 700ish and then started to receive errors from the mint about the invoice secrets already being used.

Test with around 100 iterations of invoices and some sends restored and balance reflected correctly.

Similar test with 700+ iterations showed balance of funds as expected. Removed DB and restored by balance showed 0 on restore. This item could be related to the test mint. Does it have some time limit/restrictions on the tokens minted? (it did take some time to cycle through the 700+ iterations).

Some restorations showed 0 balance but then other times after some time restoration showed funds. Again this could be related to the test mint if there are some odd interactions with mint time limits or if the funds get swept after some time.

Happy to do some more testing/repeat actions as needed.

[callebtc]

Hello @Mynima thank you for that awesome detailed response. Could you share the exact steps and set of commands that lead to some of the inconsistencies you have described?

I'm not sure if there is an easy fix for restoring when pending tokens are in the db. The tokens don't have a special state from the perspective of the mint so a restore should make them available in the new wallet as if they are normal non-pending tokens again. If that was the case for you, everything should be fine.

As I said, I would love to be able to reproduce the other behavior you described. If you can help me with those, I should be able to write tests that reveal these inconsistencies. Cheers!

[RydalWater]

Hello @Mynima thank you for that awesome detailed response. Could you share the exact steps and set of commands that lead to some of the inconsistencies you have described?

I'm not sure if there is an easy fix for restoring when pending tokens are in the db. The tokens don't have a special state from the perspective of the mint so a restore should make them available in the new wallet as if they are normal non-pending tokens again. If that was the case for you, everything should be fine.

As I said, I would love to be able to reproduce the other behavior you described. If you can help me with those, I should be able to write tests that reveal these inconsistencies. Cheers!

No worries at all, it was definitely doing something funny where it appeared that funds (after restore) were pinned (perhaps temporarily) from the perspective of performing a 'SEND ecash' action.

Here is the sequence of events to re-test:

1. Create two new wallets (call wallet_a and wallet_b)
2. Fund wallet_a with an invoice of 200 sats
3. Create ecash send from wallet_a of 100sat (save token temporarily and do not commit to a receive)
4. Restore wallet_a into a new wallet_c
5. Check balance wallet_c (should be 200 sats)
6. Attempt to perform a send 100 sats from wallet_c (result should be error message)
7. Receive wallet_a send to wallet_b and check balances (wallet_b should have 100 sats)
8. Attempt to perform a send 100 sats from wallet_c (result should be error message, repeat for smaller amounts until successful)

You may want to repeat this test a few times as sometimes it looked like it could be OK and the funds were available in the restored wallet. The situation this simulates is if someone has tried to send funds and then restored or dual restored with a second device.

By offering recovery we inherently open the door to people thinking about having the same seed on multiple devices, as it is how we're conditioned to think about bitcoin. However, the state of the ecash is governed by the private DBs and not a shared ledger, which means that restoring in multiple places does give you access to the initial pool of funds, however each devices/restore will have a separate database on which the state of the wallet will be tracked. This means we need to provide clear guidance that users should consider restoration as a means of moving devices (or retrieval of lost funds), not means of having multiple installations. It would be more advisable that for multiple devices users maintain separate wallets and send/receive between these wallets as needed.

For the other tests I wrote a simple loops in the shell.

1. Firstly to 1000 sat invoices requested for a new wallet. That should be an easy test to do.
2. Once this loop is finished (if no errors about the secret already being used were found) a few send/receives should be sent to another wallet, the balance then checked.
3. Then forget/remove the DB
4. Restore from seed (making sure as many restore iterations are completed as necessary)
5. Check balance (this should match step 2)

[callebtc]

• Attempt to perform a send 100 sats from wallet_c (result should be error message)
• Receive wallet_a send to wallet_b and check balances (wallet_b should have 100 sats)

Gotcha, I think here lies the crux: When you do a cashu --wallet wallet_a send 100, what happens is that 100 sats are burned from the wallet (now spent!) and replaced with a new set of 100 sats that are then stored in the wallet with the state "pending". These are "normal" funds that can be received by anyone, including yourself. They are not "locked" from the perspective of the mint.

To your first point: Now, when you restore wallet_a to wallet_c and cashu send --wallet=wallet_c 100, the expected behavior is that it should not error because you are spending from the funds that are marked "pending" in wallet_a. I'm trying to say that wallet_c has no way of knowing that these 100 sats are "pending", neither does the mint.

To your second point: When you try to send the "pending" 100 sats from wallet_a to wallet_c something weird happens: you are trying to receive funds in wallet_c that you already have in your database. That's why your balance remains at 200 (200 your initial balance and 100 sats that are duplicate, being a total of 200 unique ecash).

Hope it makes sense. Very interesting case you've looked at. It will cause weird problems when you use the same seed across two wallets and I can't think of a way to prevent that.

[callebtc]

By offering recovery we inherently open the door to people thinking about having the same seed on multiple devices, as it is how we're conditioned to think about bitcoin. However, the state of the ecash is governed by the private DBs and not a shared ledger, which means that restoring in multiple places does give you access to the initial pool of funds, however each devices/restore will have a separate database on which the state of the wallet will be tracked. This means we need to provide clear guidance that users should consider restoration as a means of moving devices (or retrieval of lost funds), not means of having multiple installations. It would be more advisable that for multiple devices users maintain separate wallets and send/receive between these wallets as needed.

Perfect summary 🙏

[callebtc]

Thank you to all reviewers! LFG!