Initial support for PV and PVC creation #684
Conversation
|
Things to add:
|
|
|
||
| rendered = jinja_template.render(**kwargs) | ||
|
|
||
| return yaml.safe_load(rendered) |
There was a problem hiding this comment.
this looks quite convoluted: jinja-rendering a YAML file and directly YAML-parsing it to produce a python dictionary.
Isn't it easier to just produce a dictionary directly?
There was a problem hiding this comment.
I was repeating what we're already doing with the sparkapplication.yaml file. It's true that we don't really need the jinja2 templating here at this point. But if we want to make it more dynamically configurable (for example the csi part in the PersistentVolume) as we do with the SparkApplication, I think this jinja templating is easier?
There was a problem hiding this comment.
I'm fine with leaving it as-is for now in this PR.
But I think we should consider making the configuration aspect more explicit and robust. Because if I understand correctly at the moment, the configuration is done by overwriting certain (brittle) deep paths in the container at container run time
For sure. I was not thinking about fixing those in #283, I was just thinking about maybe feature-flagging this and I didn't want to introduce more technical debt. |
|
The new recommended way to define config options or feature flags is GpsBackendConfig, where you can define a new field, it's type and assign it a default value. In particular deployments this field can then be overriden from python based config files. (e.g. backendconfig_mep.py in openeo-deploy or backendconfig_prod.py in os_creodias_openeo_k8s) |
…ent of the batch job
|
@soxofaan should the cleanup of the PV and PVCs be handled by openEO and if so, where exactly would this have to happen? |
I don't think that clean up (how and triggered from where) is in scope of this ticket, right @jdries ? |
…ing process graphs.
…1000 process graphs from CDSE-prod.
…ted' crs doesn't really matter and we return a correct object. The receiving end can then reproject as necessary.
|
@soxofaan I've added a feature flag. Is this how it should be done? |
yes that |
To FUSE mount an S3 bucket in the batch jobs, a PV and PVC should be precreated. These resources use the CSI-S3 driver, which requires us to do static provisioning if we want to mount a subdirectory of a bucket.