How can I retry a failed request from the aiohttp client ?

[iasukas]

I find that my crawler request speed is fast, and it is easy to fail the request. In this case, how can I retry the request and ensure that the JSON result is returned in order in the gather of asyncio

import  asyncio

import aiohttp
import pandas as pd


async def fetch(session, url):
    async with session.get(url) as response:
        json_result = await response.json()
        
    return json_result


async def fetch_all(task_url_list):
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(
            *[fetch(session, url) for url in task_url_list],
            return_exceptions=True
        )

    return results


def main():
    df = pd.read_csv("./result.csv")
    url_list = df["origin_url"].to_list()
    asyncio.run(fetch_all(url_list))


if __name__ == "__main__":
    main()

[Dreamsorcerer]

Just wrap it an another function?
Something like:

async def retry(...):
    tries = 3
    while tries > 0:
        try:
            return await fetch(...)
        except aiohttp.ClientResponseError:
            if tries <= 0:
                raise
        tries -= 1

[Dreamsorcerer]

You could then easily turn that into a decorator function so you can just write the code like:

@retry_client
def fetch(...):

Or similar, and leave fetch_all() unchanged.

[webknjaz]

Or use backoff/tenacity from PyPI...

[iasukas]

Thank you very very very much Dreamsorcerer and webknjaz