add job database implementation that uses stac

[jdries]

To solve still:

• how to detect authentication settings
• use aggregate api -> won't do, that api is still on dev and not clear if it works
• filtering by status doesn't work yet
• avoid dependency on external stac builder library

[soxofaan]

this is quite a large PR and I didn't go through all of it already, just some initial comments

[soxofaan]

I don't think we can already remove this as a required method to implement:
it's still being called from start_job_thread:

openeo-python-client/openeo/extra/job_management.py

Line 367 in 47172cd

df = job_db.read()

however, It seems to be unused there, so we could actually remove this.

[soxofaan]

with 29d9b16 (in master branch) I now moved read from JobDatabaseInterface to FullDataFrameJobDatabase

[soxofaan]

I'm not sure this method must be a required part of the JobDatabaseInterface interface. There are still some conceptual issues with initialize_from_df at the moment (e.g. see #667), so we might want to avoid painting ourselves in the corner here.

initialize_from_df is not something that should be called automatically from within the job manager. It is explicitly intended for end users to call explicitly on their job database. So in that sense, it should not be part of the JobDatabaseInterface interface, which is a contract between the developer of the database and job manager, not a contract between user and job manager.

[soxofaan]

I think we should start making job_manager a package instead of module, so that files are organised like

- openeo
    - extra
        - job_management
            - __init__.py
            - stac_job_db.py

[soxofaan]

With a811bff I made openeo.extra.job_management a package now, so you can move this new module to openeo.extra.job_management.stac_job_db now

[soxofaan]

I also merge master now in this feature branch to resolve all conflicts.

so make sure to pull first before continuing on this feature branch

[soxofaan]

FYI: this is a bit simpler and cheaper:

Suggested change

	return len([c.id for c in self.client.get_collections() if c.id == self.collection_id]) > 0
	return any(c.id == self.collection_id for c in self.client.get_collections())

[soxofaan]

you only use item_dict["properties"] here so the line above that sets item_dict["datetime"] has no use?

[soxofaan]

as discussed, pystac_client is an optional dependency at the moment.

you should document that at

openeo-python-client/docs/installation.rst

Lines 82 to 94 in 47172cd

	Optional dependencies
	======================
	Depending on your use case, you might also want to install some additional libraries.
	For example:
	- ``netCDF4`` or ``h5netcdf`` for loading and writing NetCDF files (e.g. integrated in ``xarray.load_dataset()``)
	- ``matplotlib`` for visualisation (e.g. integrated plot functionality in ``xarray`` )
	- ``pyarrow`` for (read/write) support of Parquet files
	(e.g. with :py:class:`~openeo.extra.job_management.MultiBackendJobManager`)
	- ``rioxarray`` for GeoTIFF support in the assert helpers from ``openeo.testing.results``
	- ``geopandas`` for working with dataframes with geospatial support,
	(e.g. with :py:class:`~openeo.extra.job_management.MultiBackendJobManager`)

[soxofaan]

for dummy/fake URLs, use the .test TLD, which is designed for especially for test situations

[soxofaan]

some more notes

[soxofaan]

Suggested change

	import concurrent
	import concurrent.futures

[soxofaan]

statuses: Iterable[str] only guarantees that you can iterate through it once.
You are using statuses twice in this function, so the second time it might be an exhausted/empty collection.
It's best, to first make a copy of the statuses that you can use twice, e.g.

statuses = set(statuses)

[soxofaan]

because of items: Iterable[Item], it's not really valid to do items[0] (Iterable only allows to do a for loop, once, not random access with e.g. [0]).
In this case, I think you should just use type annotation List[Item]

[soxofaan]

Using private MultiBackendJobManager._normalize_df from get_by_status looks a bit problematic.

It's also used from initialize_from_df, which is ok at the moment, but subject to change (see #667)

Using it in another context than initialize_from_df, like here, seems to indicate that we have to rethink all this initialization business. Not sure yet what to do instead

[soxofaan]

I guess it's a matter of taste, but I would avoid importing generic names like Collection and Item in the global namespace like this.
I would just use pystac.Collection() and the like in the code, which is a bit more typing work, but a lot clearer reading-wise

(same with Client from pystac_client)

[soxofaan]

Extra advantage of import foo over from foo import bar: it reduces risk on circular import issues