Help with refresh credential

[JiHyeongSeo]

Hi.

I am fetching data from dynamodb using django and aioboto3.
Usually there were no bugs, but after a few days intermittently, the following bugs occur.

task: <Task pending coro=<get_gentimes() running at /opt/apps/api/monitor/nearby_plant_helper.py:56> wait_for=<Future pending cb=[BaseSelectorEventLoop._sock_connect_done(18)(), <TaskWakeupMethWrapper object at 0x7f91ad537a68>()]> cb=[gather.<locals>._done_callback(0)() at /usr/local/lib/python3.6/asyncio/tasks.py:622]>
Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/credentials.py", line 265, in _protected_refresh
    metadata = await self._refresh_using()
  File "/usr/local/lib/python3.6/site-packages/aiobotocore/utils.py", line 94, in retrieve_iam_role_credentials
    token = await self._fetch_metadata_token()
RuntimeError: coroutine ignored GeneratorExit

The python package version is shown below.

aioboto3==8.0.5
aiobotocore==1.0.4
aiohttp==3.7.3
aioitertools==0.7.1
appdirs==1.4.4
arrow==0.15.5
async-timeout==3.0.1
attrs==18.2.0
autopep8==1.4
# awscli==1.17.0
backcall==0.1.0
beautifulsoup4==4.5.1
black==20.8b1
boto3==1.12.32
certifi==2018.8.24
chardet==3.0.4
click==7.1.2
colorama==0.4.1
coreapi==2.3.3
coreschema==0.0.4
dataclasses==0.8
decorator==4.3.0
defusedxml==0.6.0
Django==2.1.1
django-cors-headers==3.0.2
django-environ==0.4.5
django-extensions==2.1.2
django-filter==2.2.0
django-redis==4.9.0
django-rest-swagger==2.2.0
djangorestframework==3.8.2
djangorestframework-filters==0.10.1
djangorestframework-jsonp==1.0.2
djangorestframework-jwt==1.11.0
djangorestframework-xml==1.3.0
djangorestframework-yaml==1.0.3
docutils==0.15.2
drf-yasg==1.17.1
et-xmlfile==1.0.1
flake8==3.5.0
flake8-commas==2.0.0
flake8-mypy==17.8.0
gevent==1.3.6
greenlet==0.4.14
gunicorn==19.9.0
idna==2.7
idna-ssl==1.1.0
importlib-metadata==3.1.0
inflection==0.5.1
iniconfig==1.1.1
ipython==6.5.0
ipython-genutils==0.2.0
itypes==1.2.0
jdcal==1.4.1
jedi==0.12.1
Jinja2==2.11.2
jmespath==0.9.5
lxml==3.7.2
MarkupSafe==1.1.1
mccabe==0.6.1
multidict==5.0.2
mypy-extensions==0.4.3
numpy==1.12.1
openapi-codec==1.3.2
openpyxl==3.0.3
packaging==20.4
pandas==0.20.3
parso==0.3.1
pathspec==0.8.1
pexpect==4.6.0
pickleshare==0.7.4
Pillow==5.0.0
pluggy==0.13.1
prompt-toolkit==1.0.15
psycopg2-binary==2.7.5
ptyprocess==0.6.0
py==1.9.0
pyasn1==0.4.8
pycodestyle==2.3.1
pyflakes==1.6.0
Pygments==2.2.0
PyJWT==1.7.1
pyparsing==2.4.7
pytest==6.1.2
pytest-django==4.1.0
python-dateutil==2.8.1
pytz==2018.5
PyYAML==5.2.0
redis==2.10.6
regex==2020.11.13
requests==2.19.1
rope==0.11.0
rsa==3.4.2
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
s3transfer==0.3.3
scipy==0.19.0
simplegeneric==0.8.1
simplejson==3.17.0
six==1.11.0
toml==0.10.2
traitlets==4.3.2
typed-ast==1.4.1
typing-extensions==3.7.4.3
uritemplate==3.0.1
urllib3==1.23
wcwidth==0.1.7
Werkzeug==0.14.1
wrapt==1.12.1
yarl==1.6.3
zipp==3.4.0

The django code is below.

async def async_call_dynamodb(table, key_col, key, start_date, end_date):
    async with aioboto3.resource("dynamodb", region_name="ap-northeast-2") as dynamodb:
        table = await dynamodb.Table(table)
        each_item = await table.query(
            KeyConditionExpression=Key(key_col).eq(key)
            & Key("date").between(start_date, end_date),
        )
        return each_item

async def get_gentimes(plant: Plant, query_when: str, query_type: str):
      blah...
      each_item = await async_call_dynamodb(
            ins.yearly_table_name,
            ins.yearly_pk_column_name,
            ins.yearly_pk,
            start_date,
            end_date,
        )
    blah...
    return dic

def loop_get_gentimes(plants, query_when, query_type):
    # dynamodb async call
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop = asyncio.get_event_loop()
    gen_times = loop.run_until_complete(
        async_gather_gen_times(
            plants=plants,
            query_when=query_when,
            query_type=query_type,
        )
    )
    loop.close()
    return gen_times

Currently, the django server is running in aws' elasticbeanstalk environment. And elasticbeanstalk ec2 is given iam role. I don't know why I keep getting credential issues intermittently.

Thanks

[thehesiod]

confirmed your issue to the very first discussion so we can figure out what's going on first

[thehesiod]

strange, I don't see where a yield can happen in that flow to generate that issue, perhaps related to this: https://bugs.python.org/issue39116

[JiHyeongSeo]

hmm. So what does the following log mean?
Refreshing temporary credentials failed during mandatory refresh period.

[thehesiod]

I don't see where that error message is coming from, perhaps AWS itself? I see this related issue on botocore: boto/botocore#1866