web3.eth.sign have different signatures generated in web3js and web3py

[sivachaitanya]

• Version: 3.11.1
• Python: 3.4
• OS: osx

What was wrong?

web3.eth.sign(account,web3.sha3(plaintextdata)) is generating different signatures in web3js and web3py
Please include any of the following that are applicable:

• The code which produced the error

def registration_onboarding(unique_key):
    hash_op = web3.sha3('password')
    signature = web3.eth.sign(account,hash_op)
    return signature

So the above code is generating different signatures in web3js and web3py. I doing ecrecover in Solidity and the appending the '\x19Ethereum Signed Message\n'+lenght(message)+message in the contract itself and was able to recover the address for a web3js to Solidity call. However the signed message from Python is giving a totally different signature for same steps and same message and the ecrecover is giving incorrect account address. I observed that SHA3 for the both web3js and web3py are matching but the signature is different ?

• What type of node you were connecting to.
  I have testrpc client running and signing the hash with its account

How can it be fixed?

No idea
Fill this section in if you know how this could or should be fixed.

[djrtwo]

@sivachaitanya What version of web3.js are you using? 0.x.x or 1.x.x?

EDIT: nevermind. confirmed bug

[djrtwo]

@sivachaitanya Looks like web3.py is expecting a string not in hex and encoding it as hex no matter what. So your sha3 is something of the form '0xb68fe43f0d1a0d7aef123722670be50268e15365401c442f8806ef83b612976b' and being hex encoded to something of the form '0x307862363866653433663064316130643761656631323337323236373062653530323638653135333635343031633434326638383036656638336236313239373662'. I tested and if we pass in the double-encoded hex into the web3.js implementation, we get the same signature.

The following in web3.py:

web3.eth.sign(account, 'password')

Is the same as this is web3.js

// 'password' encoded as hex
var encoded = '0x70617373776f7264';
web3.eth.sign(account, encoded);

I'll need to discuss with @carver and @pipermerriam on how we want this to behave, given the web3.js implementation. Ours should definitely be able to handle the output of web3.eth.sha3 correctly.

[sivachaitanya]

Thanks for the quick reply Danny, From your suggestion sending the generated web3js signature from hex of password to Ecrecover function of solidity gives me totally different address. The code that is working at web3js end which gives me correct address when I sign the sign the message is -

web3js code -

//client side code
 var message = document.getElementById("message").value;
    var givenuseraddr = document.getElementById("acaddress").value;
    var msg = web3.sha3(message) ;
     console.log('SHA3 - '+msg);
     var sig = web3.eth.sign(account,msg ); 
     console.log('SHA3 signed msg op - '+sig);
     var r = sig.substr(0,66) ;
     var s = "0x" + sig.substr(66,64) ;
     var v = 28 ;
      console.log('V,R,S at client -'+v,r,s);


      //  recover address at server side by calling verify function on the smart contract
     var res = util.fromRpcSig(sig);
     console.log('V at server'+res.v);
     console.log('R at server'+util.bufferToHex( res.r));
     console.log('S at server'+util.bufferToHex(res.s));
      deployedecrecTest = ecrecTestContract.at(contractAddress);
    
    deployedecrecTest.verify.call(msg,res.v,util.bufferToHex(res.r),util.bufferToHex(res.s),function (error, data){
      console.log('Address from Quorum - '+data);
       if(data == givenuseraddr){
        // do something
      }
      if(data != givenuseraddr){
        // do something else
      }
    })

Ecrectest.sol -

function verify( bytes32 hash, uint8 v, bytes32 r, bytes32 s) 
        constant returns(address) {
           bytes memory prefix = "\x19Ethereum Signed Message:\n32";
            bytes32 prefixedHash = sha3(prefix, hash);
            return ecrecover(prefixedHash, v, r, s);
    }

So this combination of web3js and doing Ecrecover of web3js generated signature gives me correct address which signed it.

However, on the flip side, in the web3py side when I' signing using the following function-

def registration_onboarding(unique_key):
    hash_op = web3.sha3('password')
    signature = web3.eth.sign(account,hash_op)
    return signature

when inputting the signature generated from web3py inside the util.fromRPCSig(), I'm getting V,R,S values. However the address that is computed from the same verify function Ecrecover on the smart contract gives me incorrect address. Here basically here python and Javascript will be a clients sending the signature to nodejs api which calls the verify function on deployed contract and checks if the signature matches. There are no issues from web3js end but having mismatch with signature of web3py. I confirm that both the clients have different testrpc instances running locally and to verify the signature generatedby python client, I'm hardcoding sig variable in javascript to signature generated from python which should give me account address that generated that signature in python. NOTE - SHA3 for both the clients are matching.

[carver]

sha3() takes an encoding parameter, which defaults to hex. If you want to pass it bytes instead, you can use:

web3.sha3('password'.encode('utf-8'), encoding='bytes')

If the value is all ascii characters, this is equivalent to:

web3.sha3(b'password', encoding='bytes')

The sha3 implementation could use some love, though:

• The encoding dispatch accepts any value besides hex and treats it as bytes.
• Personally, I think the default encoding should be bytes, and the alternate should be hex, but that's not based on a wide community survey, and also would break backwards-compatibility. So I'm not sure what the right answer there is.

[sivachaitanya]

@carver Thank you for your response, But here the problem is the signature generated by web3py when inputted inside the Ecrecover of solidity with v,r,s variables extracted from web3js gives me incorrect address with or without the prefix. SHA3 in both the web3js and web3py seems to matching but the eth.sign seems to produce different results. I have correct address generated from web3js end but can you please help me how can I get signature validation done correctly for the signatures from web3py ?

[djrtwo]

@sivachaitanya The following will work until web3.eth.sign can handle hex_encoded strings:

from eth_utils import decode_hex
web3.eth.sign(account, decode_hex(web3.sha3("password")))

Because sign expects a non-hex_encoded string ("non '0x' prefixed"), you can decode_hex on the output of sha3 then pass that to sign. I have verified that this method does replicate the output in web3.js of the following:

web3.eth.sign(account, web3.sha3("password"));

@carver I think this warrants a change to web3.eth.sign. The method cannot accept output of web3.sha3 (a hex_prefixed string) correctly. Or I suppose web3.sha3 should maybe return a byte string instead of a hex_prefixed string.

[djrtwo]

@sivachaitanya Note: do not run decode_hex on a regular string or byte string to sign. If you wanted to sign msg = "some super cool message" you would just do the following:

msg = "some super cool message"
web3.eth.sign(account, msg)

[sivachaitanya]

@djrtwo Danny, seems like the signature generated using web3.eth.sign in web3 py is not matching with web3.eth.sign in web3js,

Above is the output from web3.js in the browser console for below code -

var message = document.getElementById("message").value;
   var givenuseraddr = document.getElementById("acaddress").value;
   var msg = web3.sha3(message) ;
    console.log('SHA3 - '+msg);
   var sig = web3.eth.sign(account, msg); 
    console.log('SHA3 signed msg op - '+sig);
    var r = sig.substr(0,66) ;
    var s = "0x" + sig.substr(66,64) ;
    var v = 28 ;
     console.log('V,R,S at client -'+v,r,s);
   

Above is the output from web3py for the below code, [Note- I have decoded the hex as per your suggestion before signing it and Both clients are signing with same account at the moment]

def registration_onboarding(unique_key):
    hash_op = web3.sha3('password')
    print('SHA3 -'+hash_op)
    print('Decoded SHA3 -'+str(decode_hex(hash_op)))
    signature = web3.eth.sign(account,str(decode_hex(hash_op)))
    return signature

Clearly from the screenshots the sha3 and signature of sha3 are also not matching. Javascript side I am able to extract address by calling ecrecover on the solidity contract. I'm using web3.js v 0.18.2 and web3.py 3.13.3 which I got this from line in init.py

__version__ = pkg_resources.get_distribution("web3").version

[djrtwo]

@sivachaitanya Your sha3 hashes are definitely not matching so any message signing following that will not match. It appears to me that document.getElementById("message").value is not returning "password". Can you try running the js example with var msg = web3.sha3('password') ; and checking against the python code with the decode_hex still in place?

[djrtwo]

@pipermerriam Thoughts on this?

sign cannot take an 0x-prefixed hexstring because it encodes it as hex no matter what. This makes signing a sha3 hash fail: web3.eth.sign(account, web3.sha3(s)). The incorrect signature is unclear to the user, and it is also different functionality than web3.js.

The following is a current hack to make sure it signs correctly

from eth_utils import decode_hex
web3.eth.sign(account, decode_hex(web3.sha3("password")))

We could either

• add an optional parameter to sign specifying the format of the argument coming in
• detect "0x" prefix (@carver has shown that this is unsafe)
• at least document somewhere that sign expects an unencoded string of data

[pipermerriam]

I didn't have time to fully catch up on this issue, but I wanted to toss this into the mix now so you guys can discuss.

I think it is ok for us to introduce a breaking change to web3.eth.sign because it's currently pretty much unusable and basically entirely broken. So, we can depart fully from the previous API in whatever we do.

[carver]

Please take a look at #301

If that gets merged, the new way to use eth.sign would be:

web3.eth.sign(text="my message here")
web3.eth.sign(hexstr="0x6d79206d6573736167652068657265")
web3.eth.sign(0x6d79206d6573736167652068657265)
web3.eth.sign(b'my message here')

^ Those are all equivalent.

---

BTW, if you are using 'password' from the earlier example for anything else (like, say, a password), please don't use eth.sign in that way. It won't be long before there are rainbow tables of b'\x19Ethereum Signed Message:\n{len}'+common_password. People still won't be able to fake signing the password, but they will be able to back out your password from the published, signed hash.

Instead, the server can send a random challenge phrase for a user to sign.

[carver]

Please reopen if you are still experiencing issues.

[jfdelgad]

I have an issue with this. having a message = 'mymessage' I use w3.eth,account.sign(message_text=message, public_key=publickKey) and the messageHash obtained in this signature differs from doing w3.sha3(text=message)

Any comment will be appreciated

[pipermerriam]

@jfdelgad that's because messages have a prefix prepended to them as part of the signing process to prevent usage of this API for transaction signing.