Historical Inputs BIDPEROFFER_D post February 2021

[MattAmos]

Firstly, I would like to sincerely compliment all of the work completed on NEMpy thus far, the functionality and thorough documentation has been very helpful!

Following the 30 minute to 5 minutely bid changes around February/March 2021, the NEMweb monthly archives post February 2021 no longer seem to contain the PUBLIC_DVD_BIDDAYOFFER_D tables:
February 2021 monthly archive vs March 2021 monthly archive

As a result, the mms_db.DBManager.populate call fails when attempting to gather data for March 2021 onwards:

Downloading MMS table for year=2021 month=2
Downloading MMS table for year=2021 month=3
---------------------------------------------------------------------------
_MissingData                              Traceback (most recent call last)
~\AppData\Local\Temp\15/ipykernel_37364/3480331206.py in <module>
     34 download_inputs = True
     35 if download_inputs:
---> 36     mms_db_manager.populate(
     37         start_year=start.year,
     38         start_month=start.month,

...\.venv\lib\site-packages\nempy\historical_inputs\mms_db.py in populate(self, start_year, start_month, end_year, end_month, verbose)
    284                 self.DISPATCHREGIONSUM.add_data(year=year, month=month)
    285                 self.DISPATCHLOAD.add_data(year=year, month=month)
--> 286                 self.BIDPEROFFER_D.add_data(year=year, month=month)
    287                 self.BIDDAYOFFER_D.add_data(year=year, month=month)
    288                 self.DISPATCHCONSTRAINT.add_data(year=year, month=month)

...\.venv\lib\site-packages\nempy\historical_inputs\mms_db.py in add_data(self, year, month)
    683         None
    684         """
--> 685         data = _download_to_df(self.url, self.table_name, year, month)
    686         if 'INTERVENTION' in data.columns:
    687             data = data[data['INTERVENTION'] == 0]

...\.venv\lib\site-packages\nempy\historical_inputs\mms_db.py in _download_to_df(url, table_name, year, month)
    373     r = requests.get(url)
    374     if r.status_code != 200:
--> 375         raise _MissingData(("""Requested data for table: {}, year: {}, month: {} 
    376                               not downloaded. Please check your internet connection. Also check
    377                               http://nemweb.com.au/#mms-data-model, to see if your requested

_MissingData: Requested data for table: BIDPEROFFER_D, year: 2021, month: 3 
                              not downloaded. Please check your internet connection. Also check
                              http://nemweb.com.au/#mms-data-model, to see if your requested
                              data is uploaded.

Are there any plans to patch this and use an alternative table that is still supplied in the monthly archives, and/or do you have any advice for temporarily patching this in the meantime?

Thanks in advance.

[nick-gorman]

Hi @MattAmos

Thanks for your feedback!

I had seen this data missing a little while back, I was hoping it was temporary, and that the missing PUBLIC_DVD_BIDDAYOFFER_D data would be uploaded. Doesn't look like this is going to happen, but I might try and contact AEMO just to check.

If PUBLIC_DVD_BIDDAYOFFER_D has been retired, then we will need to rely on PUBLIC_DVD_BIDDAYOFFER, and use some additional filters on the query to get the right bids for each dispatch interval, as PUBLIC_DVD_BIDDAYOFFER contains rebids. This will probably apply to PUBLIC_DVD_BIDPERFOFFER_D as well.

I'll get back to you when I have more info.

Cheers,
Nick

[nick-gorman]

@MattAmos this should be fixed in the latest release, update with pip, see notes here: https://github.com/UNSW-CEEM/nempy/releases/tag/v1.1.3

[MattAmos]

Thanks @nick-gorman, appreciate the speedy response!