Nvidia Snatcher Not Working for...Nvidia

[Cunningcory]

Nvidia changed their page so that "Out of Stock" is always displayed even when a buy button is available. As a result, I don't think the snatcher knows when there is stock. I missed two drops today (one in the afternoon and one about an hour ago) because no notifications were sent and no alarm went off. The latest apparently was up for 15 minutes, so there's some bug as to why it's missing it. I've been getting notifications for other sites (though mostly false alarms).

[brybry77104]

Yeah, i had my own bot going only on Nvidia and it didn’t alert either. I was able to actually get one out of pure dumb luck timing tonight.

Their site seems to only show stock on the 3080’s page (so no checking for the digital river ID on the search page) and the add to cart button doesn’t load onto the page for a few seconds after page load so you would have to wait and then check for that element. Also at high load it takes a few refreshes to see the add to cart button.

It’s a pretty clever system. I highly doubt any bot was notifying anyone today since the update to their website.

[Cunningcory]

Yup, I had...three different systems that were supposed to notify me, and none of them did. I've been at my computer most of the day and idly checking, but happened to be afk during both of those times. I just know I'm gonna have to wait at least another week to even get a chance. Really annoying.

[andrewmackrodt]

I managed to snag a card from the UK store after seeing alerts from snatcher; I'm querying the same API that the browser uses after page load: main...andrewmackrodt:develop#diff-7bb42febfcb7e3eec6d452a107dcbe8dR29

[matjam]

I think they implemented rate limiting. I backed off to 5 seconds per page load and set it to a 60 second timeout and that seems to have fixed it.

edit: it still times out from time to time.

[DarkeleEXE]

for me nvidia germany still doesnt work. I've seen that we have "nvidia" and "nvidia-api" as store now. Do I need to use nvidia-api?

[AdriaanDeVos]

Could this API be used for checking NVIDIA stock?
https://api-prod.nvidia.com/direct-sales-shop/DR/products/nl_nl/EUR/5438796700

This fork seems to have a working NVIDIA version combined with the API call instead of the webpage load. Thanks Andrew! https://github.com/andrewmackrodt/nvidia-snatcher/

[geman220]

We have nvidia and nvidia-api now, is this issue resolved? Aside from the timeouts, that is on Nvidia's side, not the bot (pages are taking too long to load).

[ghost]

When the stock had dropped last night I was using "nvidia" (not nvidia-api) and it didn't alert to any cards. It may just be that the site was not loading correctly due to everyone hitting it at once. "nvidia-api" doesn't work anymore from what I understand. It seems that the API key is no longer valid, and when I disable headless mode (checked again just now) I see it constantly opening pages that return 403 though the log says "out of stock". I haven't seen any commits since last night that change how nvidia and nvidia-api work so I don't think it's resolved.

As far as the timeouts, in my observation they only start happening once the script has been going for a couple minutes. There may be some rate limiting happening from nvidia. The script is also loading 2 links per card, and the time between those checks does not seem to respect any min/max timers set. The min/max timer only seems to come in to play once all checks for the nvidia store are completed. For example, I set a min of 20000 and max of 30000 (20-30 seconds). For the 3080 FE, it still hits the nvidia site twice in a 3 second period. Then it waits 20-30 seconds and does the same thing. It may help if there was a way to also specify minimum time between querying the same store. I tried IN_STOCK_WAIT_TIME but that didn't seem to be what I'm looking for, I think that delay only comes in to play once something is found to be "in stock".

[Cunningcory]

We have nvidia and nvidia-api now, is this issue resolved? Aside from the timeouts, that is on Nvidia's side, not the bot (pages are taking too long to load).

As far as I know, no, but it's hard to tell. Nvidia didn't have a drop today so no way to test, and I think it just shows "out of stock" regardless.

[brybry77104]

@tsumeone the API key isn't valid anymore, it stopped working sometime Wednesday afternoon. see #254

At the same time they updated their site to slightly alter how the add to cart works so now the out of stock element is still on the page (but hidden) when there is stock. It also takes a few moments for the add to cart to show up (seems like a separate API call but I wasn't able to find it in the networking call list. It seems like @andrewmackrodt seemed to have an idea of the API to query)

The API @AdriaanDeVos provided above seems to return stock correctly with no need for an API key (https://api-prod.nvidia.com/direct-sales-shop/DR/products/en_us/USD/{productId}) but I'm not sure if they are updating this information when the stock is available as there were no drops today to test against.

[geman220]

nvidia-api searches for stock correctly using the API see here. Auto-add to cart does not function. nvidia scrapes the page the same as every other store (although currently the site is very slow so it's tossing up timeouts unless you fiddle withPAGE_TIMEOUT). So I am still not sure Nvidia Snatcher Not Working for...Nvidia is a valid issue anymore. Is that correct? If the desire is for auto-add to cart to be re-implemented, we have #4 already.

Example in .env

STORES="nvidia-api"

[AdriaanDeVos]

@brybry77104 The API call I have provided above is the one that the nvidia frontend queries with javascript. This has been implemented by andrew in his fork.

@geman220 I think this is still a valid issue as the nvidia-api does not seem to work on my end (see screenshot)

[FireRx]

I still getting the perpetual timeouts on the Snatcher after like 20 or 30 pings. Can I get a little help here?

[zerocarbthirty]

Yeah nvidia doesn't work, there was a large drop at 9/27 12:30AM EST that it completely missed

[rodalpho]

That's good to know, as 9/27 is 2 days in the future! I'll stay up late. 👍

[joshuahiggins]

Nvidia's site is just horrendously slow. Here's a screenshot I just took from my browser of how long it took to load the 3090 page. I'm on 1GB fiber, so it's definitely not my connection.

The problem here is that the element that contains the pricing and link to add to cart is loaded after the page itself because it comes from a separate API request. The API request is also overly cached and horrendously inconsistent.

So since the page often isn't getting the button until 30+ seconds later and the button itself may be incorrect due to their API caching, Nvidia's site is a clusterfuck to automate scraping of. The default PAGE_TIMEOUT of 30s is solid for most sites, but unfortunately for the Nvidia site it's too short. It's basically luck of the draw whether the page will render before Puppeteer times out. That said, you can reduce the frequency of timeouts for this site by increasing PAGE_TIMEOUT to something like 60000.

They said they were making things harder for bots. Unfortunately it seems that they made it harder for bots and humans alike.

[rodalpho]

I hadn't noticed the button loading later, probably because I've never actually seen anything in-stock at the Nvidia store in my life. But the changes proposed in #281 may help out here?

[ghost]

nvidia-api searches for stock correctly using the API see here. Auto-add to cart does not function. nvidia scrapes the page the same as every other store (although currently the site is very slow so it's tossing up timeouts unless you fiddle withPAGE_TIMEOUT). So I am still not sure Nvidia Snatcher Not Working for...Nvidia is a valid issue anymore. Is that correct? If the desire is for auto-add to cart to be re-implemented, we have #4 already.

Example in .env

STORES="nvidia-api"

The console says "OUT OF STOCK" but that doesn't mean it's actually searching correctly. If I disable headless mode I can see it repeatedly opening windows that go to 403. It's not actually receiving any stock information that I can see if it's getting 403 every time. I think the "out of stock" message in console is in error, really it should be saying it failed to check. I think nvidia-api is not functioning. Please disable headless mode and see if you have the same result.

It's accessing this URL (using an invalid API key): https://api.digitalriver.com/v1/shoppers/me/products/5438481700/inventory-status?&apiKey=9485fa7b159e42edb08a83bde0d83dia&locale=en_us&_=1601051901846

Getting this message: https://i.imgur.com/xAOlhRN.png

Console shows this: https://i.imgur.com/sjtHEqY.png

[joshuahiggins]

Yea, the nvidia-api is currently broken. Nvidia blocked it off. I've been trying to figure out how to get a valid key for DR now to get that URL working again... But I think that was a secret key they were never supposed to have on their site in the first place.

I think what they did is set up a proxy at api-prod.nvidia.com where they add their new secret apiKey now and proxy API requests through that now. So things like actual inventory counts are no longer available to the public. I noticed there's some new endpoints available though that might be helpful. For instance, this one will tell you if a product is in stock or not:

https://api-prod.nvidia.com/direct-sales-shop/DR/products/en_us/USD/5438481700

Notice the DR, presumably for Digital River. There's also an exposed API on their public site where you can generate DR session tokens, but those don't work with the prior API this script was using. There's some add to cart and checkout functionality via that API, but I haven't quite sorted it all out yet.

[rodalpho]

Should we disable nvidia-api in the .env file, or does it not matter either way?