0.2.29 bug: OperationalError('unable to open database file')

[githubbla]

Describe bug

fail when downloading array, work ok in 0.2.28

Simple code that reproduces your problem

from pandas_datareader import data as pdr
import yfinance as yf
from numpy import isnan

##yf.pdr_override()

def loadtickers(tickers,start="2005-1-1",end=None,renamefield={},prev=None):
    for i in range(0,2):
        try:
            df = yf.download(tickers, start, end, auto_adjust=True,ignore_tz = True)
            df.columns = ['_'.join(((str.lower(col[1]) if col[1] not in renamefield.keys() else str.lower(renamefield[col[1]]), str.lower(col[0])))) for col in df.columns]
            for t in tickers:
                if t in renamefield.keys():
                    t=renamefield[t]
                print("{}_return".format(t).lower(),"{}_close".format(t).lower() )
                df["{}_return".format(t).lower()] = df["{}_close".format(t).lower()].pct_change().fillna(0)
            if prev is not None:
                df=df.join(prev,how='left')
            return df
        except Exception as e:
            print(e)
            continue
        break
    return

p1=loadtickers(["SPY","TLT"],start="2004-12-31",renamefield={"^VXTLT":"VXTLT","^TNX":"TNX",})

Debug log

DEBUG Entering download()
DEBUG:yfinance:Entering download()
DEBUG Disabling multithreading because DEBUG logging enabled
DEBUG:yfinance: Disabling multithreading because DEBUG logging enabled
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
DEBUG Entering history()
DEBUG:yfinance: Entering history()
ERROR
24 Failed downloads:
ERROR:yfinance:
24 Failed downloads:
ERROR ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: OperationalError('unable to open database file')
ERROR:yfinance: ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: OperationalError('unable to open database file')
DEBUG ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect
self._state.set_connection(self._connect())
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect
conn = sqlite3.connect(self.database, timeout=self._timeout,
sqlite3.OperationalError: unable to open database file

      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/dist-packages/yfinance/multi.py", line 275, in _download_one
          data = Ticker(ticker, session=session).history(
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 114, in wrapper
          result = func(*args, **kwargs)
        File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 148, in history
          tz = self._get_ticker_tz(proxy, timeout)
        File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 1644, in _get_ticker_tz
          cache = utils.get_tz_cache()
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 985, in get_tz_cache
          _tz_cache = _TzCache()
        File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 926, in __init__
          db.connect()
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3191, in connect
          with __exception_wrapper__:
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3019, in __exit__
          reraise(new_type, new_type(exc_value, *exc_args), traceback)
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 192, in reraise
          raise value.with_traceback(tb)
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect
          self._state.set_connection(self._connect())
        File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect
          conn = sqlite3.connect(self.database, timeout=self._timeout,
      peewee.OperationalError: unable to open database file

DEBUG:yfinance: ['TMF', 'TMV', 'VBINX', 'SHV', '^GVZ', 'VXZ', '^VIX', 'SGOV', 'TLT', 'QQQ', '^VXN', 'GLL', 'SPHD', 'SPY', 'TQQQ', 'QLD', 'SPLV', 'TBF', 'IAU', '^VXTLT', 'GLD', 'XLU', 'VXX', 'UGL']: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect
self._state.set_connection(self._connect())
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect
conn = sqlite3.connect(self.database, timeout=self._timeout,
sqlite3.OperationalError: unable to open database file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/yfinance/multi.py", line 275, in _download_one
data = Ticker(ticker, session=session).history(
File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 114, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 148, in history
tz = self._get_ticker_tz(proxy, timeout)
File "/usr/local/lib/python3.10/dist-packages/yfinance/base.py", line 1644, in _get_ticker_tz
cache = utils.get_tz_cache()
File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 985, in get_tz_cache
_tz_cache = _TzCache()
File "/usr/local/lib/python3.10/dist-packages/yfinance/utils.py", line 926, in init
db.connect()
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3191, in connect
with exception_wrapper:
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3019, in exit
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 192, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3192, in connect
self._state.set_connection(self._connect())
File "/usr/local/lib/python3.10/dist-packages/peewee.py", line 3536, in _connect
conn = sqlite3.connect(self.database, timeout=self._timeout,
peewee.OperationalError: unable to open database file

DEBUG Exiting download()
DEBUG:yfinance:Exiting download()
2023-09-22-15-59 tickers.pkl
loading fresh 2023-09-22-15-59 tickers.pkl
spy_return spy_close
tlt_return tlt_close
iau_return iau_close
gld_return gld_close
qqq_return qqq_close
vix_return vix_close
vxtlt_return vxtlt_close
gvz_return gvz_close
qld_return qld_close
vxz_return vxz_close
tbf_return tbf_close
vxn_return vxn_close
vxx_return vxx_close
tmf_return tmf_close
tmv_return tmv_close
splv_return splv_close
sphd_return sphd_close
vbinx_return vbinx_close
xlu_return xlu_close
tqqq_return tqqq_close
ugl_return ugl_close
gll_return gll_close
sgov_return sgov_close
shv_return shv_close

Bad data proof

No response

yfinance version

0.2.29

Python version

No response

Operating system

No response

[ValueRaider]

You plan on editing with error message and working code?

[githubbla]

i added, tell me of this is enough, it's fail with 0.2.29, but ok with 0.2.28

[ValueRaider]

Run again debug logging enabled. Like the bug report template instructed.

[githubbla]

done!

[arieller]

same

[ValueRaider]

I can't debug until tomorrow, but here's summary in case someone else can.

Previous version was 99% thread-safe. In rare cases the thread-safety failed.

Latest version switches to peewee module. Supposed to be 100% thread-safe, and I tested on few threads. Maybe it's configured wrong?

[enzbus]

Confirmed. Fails on my CI pipeline in python <=3.10, works on my mac with 3.11.

[ValueRaider]

There's clearly a bug in 0.2.29. Keep replies constructive and concise.

[ValueRaider]

@enzbus how many CPUs on your mac, vs how many on system running pipeline?

[githubbla]

My sample fail on Google colab

[enzbus]

@enzbus how many CPUs on your mac, vs how many on system running pipeline?

So far I've only observed failure on github CI, which is single processor (I believe). My machine has multiple processors. The only way I call yfinance is via yf.download, from a single thread. The errors that I get on github testing are like this

ERROR:yfinance:
1 Failed download:
ERROR:yfinance:['AAPL']: OperationalError('unable to open database file')

(And then yf.download returns an empty dataframe, I believe.)

[MrDvs]

I am encountering the same problem, both on python 3.9 and 3.11. FWIW, the machine I'm running uses Ryzen 5 5600X CPU.

[Himel55]

Not sure if this is related, but I am having a similar issue when using yf.download, I am getting a sqlite3.OperationalError: unable to open database file error. I am running on Windows and I believe it may have something to do with the cache path folder not existing.

If I manually create the py-yfinance folder in C:\Users\Himel\AppData\Local\, I no longer get the exception.

[j-nx]

In my case (windows using Git Bash), I found that this is due to the fact that peewee.py is trying to create a db at %LocalAppData%/py-yfinance/tkr-tz.db and since sqlite3.connect cannot force create the py-yfinance folder this fails.

I manually created the %LocalAppData%/py-yfinance/ folder and was able to bypass this issue, in my instance.

Edit: I see it's been identified already here

[ValueRaider]

@Himel55 That's the problem. Can people try fix in #1698 (instructions #1080)

[federico999999]

but I don't know how to solve it

[The-Milad-A]

The problem is beyond just this. I tried a few other yahoo finance packages & they also couldn't return data. There might be an api issue...

[ValueRaider]

@The-Milad-A That's a different problem. But working for me.

[joshuacuellar1]

I can confirm the error on my Linux environment. The peewee library does not create the py-yfinance folder. Tried your fix @ValueRaider and it worked.

[ValueRaider]

Can I get more confirmation that my refactor hasn't introduced bugs? Then can get a release out before markets open.

[joshuacuellar1]

@ValueRaider Sorry for the late reply. Tested the latest version of yfinance (with your fixes) and it works perfectly in my simple use case: data = ticker.history(period="max", interval="1d", actions=False, auto_adjust=True).

[federico999999]

@ValueRaider
please can you explain to me better what code should I use to solve the problem?
Because I'm an inexperienced user... and I don't understand what I have to do.

[janfrederik]

@federico999999 The fix is included in the new release 0.2.30. You can just fetch the new version from pypi: pip install --upgrade yfinance if you installed with pip. Or git pull if you cloned this git repo.

[janfrederik]

@ValueRaider thanks for the fast fix!

[federico999999]

@federico999999 The fix is included in the new release 0.2.30. You can just fetch the new version from pypi: pip install --upgrade yfinance if you installed with pip. Or git pull if you cloned this git repo.

Thank you