Update the docker image to use the latest odc-stats version 1.0.33

[vikineema]

The docker image tagged latest uses the odc-stats version 1.0.32 instead of the latest odc-stats version which is 1.0.33.
docker container run -it opendatacube/datacube-statistician:latest pip list| grep odc-stats

odc-stats 1.0.32

[fangfy]

Hi @emmaai @SpacemanPaul @omad, it looks like dockerise test failed (https://github.com/opendatacube/odc-stats/actions/runs/4191291551/jobs/7265577644). Any idea why?

[emmaai]

The connection is reset by s3. Rerun will fix it.

[vikineema]

Hi @emmaai @spac @omad . I've tested the latest published image from the rerun workflow and the odc-stats version is still 1.0.32. What could be causing this?

[emmaai]

had a look, it used the cache available in ecr, since nothing has changed with the files involved in the docker image building.

[fangfy]

Hi @emmaai, do you have an estimate of when this can be fixed? we urgently need this new image to generate S2 gm products.

[jmettes]

I've been noticing a lot of intermittent failures in tests using external S3 data in odc-tools - especially those using s3://sentinel-s2-l2a-cogs. I wonder if that's what's happening here too. I wonder if that bucket recently changed to a less highly-available storage class. Ideally these tests would be replaced with mocks, or locally stored data.

Maybe in the meantime, it's worth trying to set GDAL_HTTP_MAX_RETRY somewhere?
rasterio/rasterio#2119 (comment)

Apparently max_retry default is 0:
https://gdal.org/user/virtual_file_systems.html

[emmaai]

Hi @emmaai, do you have an estimate of when this can be fixed? we urgently need this new image to generate S2 gm products.

Within this week.

[emmaai]

I've been noticing a lot of intermittent failures in tests using external S3 data in odc-tools - especially those using s3://sentinel-s2-l2a-cogs. I wonder if that's what's happening here too. I wonder if that bucket recently changed to a less highly-available storage class. Ideally these tests would be replaced with mocks, or locally stored data.

Maybe in the meantime, it's worth trying to set GDAL_HTTP_MAX_RETRY somewhere? rasterio/rasterio#2119 (comment)

Apparently max_retry default is 0: https://gdal.org/user/virtual_file_systems.html

For the test, GDAL_HTTP_MAX_RETRY is set here

odc-stats/docker/docker-compose.yml

Line 21 in 0826115

- GDAL_HTTP_MAX_RETRY=5

and here

odc-stats/.github/workflows/main.yml

Line 137 in 0826115

GDAL_HTTP_MAX_RETRY: 5

Still, the connection can be reset by s3 server, which can not be caught or retried.

[emmaai]

Could you have a look at PR #88, it should resolve the issue.