HTTP Error 401 when called on large amount of tickers

[rpfreitasxyz]

Hi, Thank you very much for the fix and comments here. getSymbols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download:

> getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE)
Warning: AAPL download failed; trying again.
Warning: Unable to import "AAPL".
AAPL download failed after two attempts. Error message:
HTTP error 401.
[1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n"
attr(,"class")
[1] "try-error"
attr(,"condition")
<simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

Originally posted by @edwinhung in #358 (comment)

Thank you all for the quick fix in regards to tq_get()! However, I believe that this new issue has arisen.

[Knxd3]

Is a fix planned for tq_get? getSymbols works with many tickers because of the 1 second pause. Thanks.

[msperlin]

The limits do not seem to be too restrictive. After reaching the 404 error, I was able to get successfull api calls in a few minutes.

After that, I downloaded all sp500 stocks (2010-today) in a single call:
df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

If anyone can confirm this in your own R session, please do.

It looks as the restrictions are based on time between api calls for the same ip. This could invalidate any parallel computation, which is what I'm testing now.

[msperlin]

As expected, any parallel use of quantmod::getSymbol() reaches the limit very easilly. As such, I'm removing the parallel option from yfR and BatchGetSymbols.

When using a single session (non-parallel), yfR runs fine for any large sample of stocks.

[msperlin]

If you can, please confirm if the code below runs fine:

remotes::install_github("msperlin/yfR")

df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

[rpfreitasxyz]

If you can, please confirm if the code below runs fine:

remotes::install_github("msperlin/yfR")

df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

The code runs fine, with a caveat:

The issue really seems to be the amount of API calls. Even though they're not in parallel, they are still (500+) sequential calls to the Yahoo! Finance API, and as such, it is quite inconsistent whether or not the whole dataframe will be downloaded. I believe this has already been worked around with your implementation of a cache system.

Thus, if it hasn't been downloaded completely at once, I suggest users wait for a few minutes, then run the code again, until completion.

Either way, thank you, @msperlin!

[msperlin]

Yes, lack of consistency in equivalent calls to BatchGetSymbols/yf_get can be troublesome. I'll see how I can control for this, at least letting the user know about the 404 error.

[joshuaulrich]

I just added an option to use the JSON endpoint instead of the CSV endpoint. Can you try that and see if you still get the 401 responses? You can install the patch via: remotes::install_github("joshuaulrich/quantmod@362-yahoo-json-endpoint"). Then call quantmod::getSymbols("SPY", use.json.api = TRUE)

[msperlin]

Sure, let me try..

[msperlin]

I just added an option to use the JSON endpoint instead of the CSV endpoint. Can you try that and see if you still get the 401 responses? You can install the patch via: remotes::install_github("joshuaulrich/quantmod@362-yahoo-json-endpoint"). Then call quantmod::getSymbols("SPY", use.json.api = TRUE)

Changed the call to getSymbols and I tried my best to reach the 401 response, with not success. I ranned yfR with parallel execution (14 cores) and it worked as expected.

[msperlin]

looks good..

[msperlin]

anyone can test it here:

remotes::install_github("msperlin/yfR@testing-json-entry")

library(yfR)

n_workers <- parallel::detectCores() - 1
future::plan(future::multisession, workers = n_workers)
available_collections <- yf_get_available_collections()

df <- yf_collection_get(collection = "SP500",
                          first_date = Sys.Date() - 10*365,
                          last_date = Sys.Date(),
                          do_parallel = TRUE)

dplyr::n_distinct(df$ticker)

@joshuaulrich please let me know if and when you're incorporating these changes.. I'll wait for your update in CRAN.

thanks.

[joshuaulrich]

I'm considering whether or not to make the JSON endpoint the default for getSymbols.yahoo(). It seems like it's a better endpoint, but I don't have any experience with it. @msperlin, what do you think about switching from the CSV endpoint to the JSON endpoint for the default?

[msperlin]

Being honest, I'm not sure. I'll have to think about that. Quality wise, I suspect the YF data comes from the same source and, wheter it is json or csv, the output should be the same.

But, the csv entrypoint is restricted by IP, which forces user to behave better, which is good. The restriction is also not that bad (I can still download everything I need for my classes, for example). While I would prefer to allow parallel computing with yfR, I also know that we should be thankful to YF for still keeping the API open...

what do you think?

[ethanbsmith]

some thoughts:

since you've done the bulk of the work of moving to the V8 api, may as well loosen the validation on period to support intra-day and kill #351

also, why not just remove the v7 code path entirely. in principle, i think supporting code to work around throttling (if thats what yahoo is doing) is not really a worth while battle. moving to higher rev makes sense to me, but if yahoo is really throttling and serious about it, its going to comer up again sooner or later. i'd just see this as another notch in the growing list of issues w/ yahoo data in general

[joshuaulrich]

I suspect the YF data comes from the same source and, whether it is json or csv, the output should be the same.

I would hope so, but I wouldn't be surprised if there are some differences... because data is awful. ;)

Also, why not just remove the v7 code path entirely.

I'm thinking the same thing. Thanks for mentioning the intra-day issue. That's a great point.