Feature Proposal: Support Sector and Industry data from Markets Tab

[ericpien]

The following data is available on: https://finance.yahoo.com/sectors/:

• sector / industry overview
• sector / industry top companies
• sector / industry research reports
• sector top ETFs
• sector top Mutual Funds
• industries within sector
• industry's sector
• industry's top performing companies
• industry's top growth companies

Resolves:

• get top companies by sector/industry: Get top 5 tickers by sector #2047
• find peers in the same industry: [Feature Request] Include Stock Peers Dictionary #2054

Sample code:

import yfinance as yf
tech = yf.Sector('technology')
software = yf.Industry('software-infrastructure')

# Common information
tech.key
tech.name
tech.symbol
tech.ticker
tech.overview
tech.top_companies
tech.research_reports

# Sector information
tech.top_etfs
tech.top_mutual_funds
tech.industries

# Industry information
software.sector_key
software.sector_name
software.top_performing_companies
software.top_growth_companies

# Ticker to Sector and Industry
msft = yf.Ticker('MSFT')
tech = yf.Sector(msft.info.get('sectorKey'))
software = yf.Industry(msft.info.get('industryKey'))

# Sector and Industry to Ticker
tech_ticker = tech.ticker
tech_ticker.info
software_ticker = software.ticker
software_ticker.history()

[sshcli]

Thank you @ericpien 🙌

[ValueRaider]

I ran tech.symbol a few times, not many, and triggered a messy error:

ERROR    Failed to get sector data for 'technology' reason: 401 Client Error: Unauthorized for url: https://query1.finance.yahoo.com/v1/finance/sectors/technology?formatted=true&withReturns=true&lang=en-US&region=US&crumb=4WOcuFKaIqF
DEBUG    Got response: 
DEBUG    -------------
Traceback (most recent call last):
  File "/home/test/yfinance-ericpien/yfinance/domain/sector.py", line 53, in _fetch_and_parse
    result = self._fetch(self._query_url, self.proxy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/test/yfinance-ericpien/yfinance/domain/domain.py", line 59, in _fetch
    result = self._data.get_raw_json(query_url, user_agent_headers=self._data.user_agent_headers, params=params_dict, proxy=proxy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/test/yfinance-ericpien/yfinance/data.py", line 399, in get_raw_json
    response.raise_for_status()
  File "/home/test/.local/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://query1.finance.yahoo.com/v1/finance/sectors/technology?formatted=true&withReturns=true&lang=en-US&region=US&crumb=4WOcuFKaIqF

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/test/yfinance-ericpien/./test.py", line 12, in <module>
    print(tech.symbol)
          ^^^^^^^^^^^
  File "/home/test/yfinance-ericpien/yfinance/domain/domain.py", line 34, in symbol
    self._ensure_fetched(self._symbol)
  File "/home/test/yfinance-ericpien/yfinance/domain/domain.py", line 97, in _ensure_fetched
    self._fetch_and_parse()
  File "/home/test/yfinance-ericpien/yfinance/domain/sector.py", line 66, in _fetch_and_parse
    logger.debug(f" {data}")
                     ^^^^
UnboundLocalError: cannot access local variable 'data' where it is not associated with a value

Also, you can put hardcoded stuff like SECTOR_INDUSTY_MAPPING in const.py, keep the logic clean.

[ericpien]

@ValueRaider thanks for reviewing. I made the changes for const.py and the UnboundedLocalError portion of the log you shared.

I can't reproduce the HTTPError portion of the log. I ran the code below successfully which rules out the error being related to "running it a few times". Can you try again?

for s in const.SECTOR_INDUSTY_MAPPING:
    sector = yf.Sector(s)
    print(f'sector symbol: {sector.symbol}')
    for i in const.SECTOR_INDUSTY_MAPPING[s]:
        industry = yf.Industry(i)
        print(f'industry symbol: {industry.symbol}')

[ValueRaider]

Your change has improved the error - it still happens randomly but is cleaner. Probably related to my spam, I'm doing other fetches.

One more error:

tech = yf.Sector('technology')
tech.research_reports

  File "yfinance/domain/domain.py", line 54, in research_reports
    self._ensure_fetched(self._research_reports)
                         ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Sector' object has no attribute '_research_reports'. Did you mean: '_research_report'?

[ericpien]

Thanks. Corrected the param name and checked the other params are correct as well.

[ValueRaider]

I'm hoping to release in a week, I'm just finishing testing improvements to dividend repair.