Skip to content

Commit

Permalink
Improve decrypt key deduction
Browse files Browse the repository at this point in the history
  • Loading branch information
ValueRaider committed Jan 14, 2023
1 parent 4ca9642 commit cd2c1ad
Showing 1 changed file with 20 additions and 4 deletions.
24 changes: 20 additions & 4 deletions yfinance/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,23 @@ def decrypt_cryptojs_aes_stores(data):
_cr = b"".join(int.to_bytes(i, length=4, byteorder="big", signed=True) for i in json.loads(_cr)["words"])
password = hashlib.pbkdf2_hmac("sha1", _cs.encode("utf8"), _cr, 1, dklen=32).hex()
else:
try:
password_key = next(key for key in data.keys() if key not in ["context", "plugins"])
except:
# Currently assume one extra key in dict, which is password. Print error if
# more extra keys detected.
new_keys = [k for k in data.keys() if k not in ["context", "plugins"]]
l = len(new_keys)
if l == 0:
return None
elif l == 1 and isinstance(data[new_keys[0]], str):
password_key = new_keys[0]
else:
msg = "Yahoo has again changed data format, yfinance now unsure which key(s) is for decryption:"
k = new_keys[0]
k_str = k if len(k) < 32 else k[:32-3]+"..."
msg += f" '{k_str}'->{type(data[k])}"

This comment has been minimized.

Copy link
@ifel

ifel Jan 24, 2023

That's not quite how their "protection" works. They basically load https://s.yimg.com/uc/finance/dd-site/js/main.__hash__.modern.js which does decryption.

  • 3376687f03f744ed0c2b version uses a value from the very particular key - 2f28c82aa38ad3e4dc1a
  • c3e5da0b83decfa64eba version combines values of the several keys.

Here is the diff:

-                        return (t.context.dispatcher.stores = JSON.parse(c().decrypt(n, t["2f28c82aa38ad3e4dc1a"]).toString(d()))), t;
+                        return (t.context.dispatcher.stores = JSON.parse(c().decrypt(n, `${t["5e971f17f196"]}${t["04b4af022e08"]}${t.f550b3f47950}${t["1e0c85f35401"]}`).toString(d()))), t;

Have no idea what they come up with next. It's not quite dynamic (yet). I see 2 ways of how to approach current situation:

  1. Use a static set of keys and try to decrypt with them, periodically adding new ones (until they come up with really dynamic approach). The current set is:
{"ad4d90b3c9f2e1d156ef98eadfa0ff93e4042f6960e54aa2a13f06f528e6b50ba4265a26a1fd5b9cd3db0d268a9c34e1d080592424309429a58bce4adc893c87",
"e9a8ab8e5620b712ebc2fb4f33d5c8b9c80c0d07e8c371911c785cf674789f1747d76a909510158a7b7419e86857f2d7abbd777813ff64840e4cbc514d12bcae"}
  1. Make a map js_file_hash -> key/function. Again, in this case the map needs to be updated periodically catching new changes.

At the moment, I see that they use the both methods (rollout shards maybe).

Maybe I make a patch on the weekend if someone else does not implement this by that.

P.S. Yahoo guys, if you read this, please stop wasting your time on this, this is simple stupid - it's a JS, source is available, it can be reverse engineered anyways. Yes, you break stuff for ppl for a few days, but you put a lot of effort into this. It's better to return the API you had, I bet there are many ppl who would like to keep paying a small fee instead of dancing with tambourine.

This comment has been minimized.

Copy link
@ValueRaider

ValueRaider Jan 24, 2023

Author Collaborator

I've hardcoded those keys into branch hotfix/decryption.

for i in range(1, len(new_keys)):
msg += f" , '{k_str}'->{type(data[k])}"
raise Exception(msg)
password_key = new_keys[0]
password = data[password_key]

encrypted_stores = b64decode(encrypted_stores)
Expand Down Expand Up @@ -105,7 +118,10 @@ def EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="m
key, iv = key_iv[:keySize], key_iv[keySize:final_length]
return key, iv

key, iv = EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5")
try:
key, iv = EVPKDF(password, salt, keySize=32, ivSize=16, iterations=1, hashAlgorithm="md5")
except:
raise Exception("yfinance failed to decrypt Yahoo data response")

if usePycryptodome:
cipher = AES.new(key, AES.MODE_CBC, iv=iv)
Expand Down

0 comments on commit cd2c1ad

Please sign in to comment.