openEO random_forest_classifier validation/inference fails in CDSE

[Pratichhya]

When testing https://open-eo.github.io/openeo-python-client/machine_learning.html, we can train the model and download the model weights too. But, when trying to validate/infer from the trained model, it fails with the following error:

"OpenEO batch job failed: OpenEOApiException(status_code=400, code='Internal', message='No random forest model found for job j-231029b6164049ba92c3eada7603f037', id='no-request')"

However, this is only the case when executing openeo.dataspace.copernicus.eu but works well when tested in openeo.cloud or openeo.vito.be

[jdries]

FYI we now have mounted the object storage bucket as a posix directory in the containers.
Hence models can be saved and read from there, just like on YARN, so this should start working again with limited remaining effort.
Do note that the new bucket functionality may still be feature flagged, to be checked with @tcassaert .

[tcassaert]

There's a PR (#684) for this. It's not yet in use.

[EmileSonneveld]

Hey, I re-launched the job of j-231029b6164049ba92c3eada7603f037 on https://openeo.dataspace.copernicus.eu/ and got a result. (My job ID was j-240308cce2b542f691fdab4b0c088f1b)
As I understand @tcassaert his fix did indeed work.

[tcassaert]

This is not on prod yet, so something else made it work then.

[EmileSonneveld]

CDSE staging and prod assign a different href to the model asset. The s3:// version works best.

Both backends have all files:
dev: /OpenEO-data/batch_jobs/j-240327428e7f48269cbf04840296cb9f
staging: /OpenEO-data/batch_jobs/j-240322a516184973a25ae51c55005416

[JeroenVerstraelen]

Currently waiting on https://github.com/eu-cdse/openeo-cdse-infra/issues/115 to deploy the required mounts on CDSE prod. Then this issue should be finished.

[JeroenVerstraelen]

@Pratichhya You can use the following code for training and inference.

Training:
https://gist.github.com/JeroenVerstraelen/ddaf818b5ae71f0fca2dfc68271cec1b

Inference:
https://gist.github.com/JeroenVerstraelen/b26d89cf9c84e2dc4435fb47802ebaed

The training extent + polygons need to be adjusted a bit because it currently outputs an empty randomforest model.