Ships-detection indicator on-the-fly integration

[dmoglioni]

@aapopescu @lubojr
Here we can track the integration steps of the ships-detection indicator on the dashboard.

We agreed that the most straightforward way to integrate this on-the-fly indicator will be via POST request to a WPS end-point.

Leah, Juraj, Selim will work on:

• identifying and tagging the cells on the JN to be parametrised
• enhancing the code for taking into account failure conditions
• logging valuable information for maintenance and future post-processing operations
• producing a mock-up of the visuals they'd like to be displayed on the dashboard
• generating a requirements file to set-up the workspace

I discussed this solution with Lubo and we're in line with this way forward. Lubo will verify in parallel with Stephan the setting of a dedicated end-point for this task. As soon as we have access to it we'll start with the testing and integration activity.

[lubojr]

@dmoglioni Sorry for a bit of confusion on our side.
We discussed internally again and for the integration into eodash as another indicator, we shall try to proceed with a different route instead.

As the algorithm is not producing any output imagery to be "persistent" but just returning the detections geometries and it should be used on the dashboard in a synchronous manner (user of eodash draws area, selects time interval and clicks the "fetch detections"), it will be best (performance-wise) if we remove as much overhead of Headless execution or BYOA of EOxHub as possible, as they are both aimed on async handling.

We shall use an existing service (endpoint) that we have developed in other framework, which is capable of returning a value (here GeoJSON geometries) from the process directly in a synchronous manner.
For this, we would need to have the all above points still done (except that we do not need the JN cells to be parameterized).

Please once they are finished, send us the resulting code + export of the environment used to run the inference (or the requirements.txt) via email. Thanks

[Patrick1G]

@lubojr CC @aapopescu
could you share a geoJson example of the RACE ship detection, as we had it running on EDC/eodash over the last years?
It would be helpful for our interns to see how e.g. attributes are included and to ensure that their outputs can be directly integrated...

[lubojr]

@patrick-griffiths
This would be an example from a single detection extracted from the GeoDB for E1b indicator - Vessel density and detections.

[
    {
        "id": 39242,
        "created_at": "2023-03-13T18:12:30.260978+00:00",
        "modified_at": null,
        "geometry": "0104000020E6100000230000000101000000B9533A58FF272440E50AEF7211E74C400101000000132C0E677E452440C217265305E74C400101000000D4D4B2B5BE182540AE0D15E3FCE54C4001010000003A3B191C251F2540D8F50B76C3B64C400101000000545227A0892025409EEFA7C64BB74C4001010000003A4030478F2F25408E06F01648D84C400101000000302FC03E3A3525400938842A35B74C400101000000069E7B0F973C2540137EA99F37ED4C400101000000CD069964E43C2540CFF753E3A5D74C400101000000732EC555653F254004E275FD82D54C400101000000AB048BC3993F2540B58993FB1DD24C400101000000BE6A65C22F4525406E8B321B64FE4C400101000000650113B875472540890CAB7823D74C40010100000098FA7953914A2540742497FF90DA4C4001010000007B6649809A4A25405ABBED4273D94C400101000000B3B5BE48684B2540DC114E0B5ED84C400101000000715AF0A2AF5025400C1F115322D54C4001010000006CB2463D44532540809A5AB6D6D34C400101000000DDEA39E97D53254012DA722EC5B54C400101000000BA2C26361F5725405778978BF8D64C400101000000D5CA845FEA5725400C3CF71E2ED94C400101000000E3A59BC420602540FF959526A5D44C4001010000001A69A9BC1D612540AA2B9FE579D44C4001010000001AA88C7F9F6125401557957D57D84C400101000000A83AE466B861254086032159C0D84C400101000000637FD93D796825407B832F4CA6D64C40010100000008C90226706B254086200725CCD84C4001010000006FF085C954912540BBB88D06F0B64C400101000000A83AE466B891254065A54929E8BE4C4001010000001EFE9AAC519F254066A032FE7DBE4C400101000000C5E6E3DA50A12540554D10751FC04C400101000000573ECBF3E0CE2540BD6F7CED99D94C400101000000DFC325C79DF22540B459F5B9DAEA4C400101000000E6CB0BB08F8E2640A987687407C54C400101000000D8F0F44A599626402EE7525C55CA4C40",
        "indicator_code": "E1b",
        "aoi_id": "DK2",
        "time": "2021-02-23T05:32:08"
    }
]

We do not have GeoJSON directly returned from API, we convert to it after parsing the response.

Whichever attributes of the GeoJSON are required to be shown by the client, we shall display on hover...

[lubojr]

Just realized that the above posted is not really helpful, as it involves the structure in which the data are in GeoDB.

This is an example of a GeoJSON with features directly used https://github.com/eurodatacube/eodash/blob/staging/app/public/eodash-data/features/E12b/E12b_UK9_20200829.geojson
So this can be replicated instead

[Patrick1G]

that's perfect @lubojr many thanks.
Should we not track the S2 product identifier (on which the inference was run) there somewhere?
This would simplify identification of the S2 scenes for visualization and overlay later...

[lubojr]

@patrick-griffiths The properties object can contain any additional metadata to the detections (for example "probability" of valid detection to be shown on hover) or also the scene identifier(s), but I would prefer to instead use the SCENE time of the scene(s) used.
Regarding the addition of the S2 scenes - we are using the Sentinel-Hub layers, we are not directly requesting WMS of scenes, but rather display a layer along a time axis (start/end), so we can use the time start and time end input being passed into the detection algorithm as well.

If there is a need to subset this input start/end provided by the user on a per detection level (each detection can be extracted from another scene), we can use the same logic like in displaying the wind turbines together with an S2 layer where the TIME information coming from the GeoJSON properties of each detection that the user clicks on. https://gtif.esa.int/explore?x=1823968.87838&y=6094954.34461&z=14.76947&poi=Austria-REP6

[lubojr]

Mockup on the planned integration.

[dmoglioni]

@aapopescu @lubojr
Here's the logging file to be produced (write in append mode) by the application as agreed.
Could you check its integration status? Thank you.

Ships-detection_log.csv

[lubojr]

@Selimgit @leahsturm @jzvolensky As we do not have other ways of contacting you, could you please let us know if there has been any developments on adding the planned logging of requests of the Vessel inference - https://github.com/ShipDetectionExperts/draft_inference/tree/RACE_DASHBOARD and returning the detected features in EPSG:4326 coordinates?

Thanks!
Lubo

[jzvolensky]

Hi @lubojr, me and Selim had a chat today. For now, there have been no new developments. We are going to start again and hopefully provide an update on Monday on how things are going. Thank you.

[jzvolensky]

Hi @lubojr, as promised here is the update on the situation. After our discussion with @Selimgit, we spent a couple of days working on the code. There was no significant progress made.

Since you mentioned that you have done this kind of integration before with other scientists, we think it might be easier for you to take over our algorithm and add the required features yourselves. Unfortunately, we are not able to dedicate the time right now to finish anytime soon. Thank you for your understanding.

Juraj

[lubojr]

Server side integration was finished.

Example request: https://gtif-backend.hub.eox.at/ship_detection?lon_min=3.051167&lat_min=36.741406&lon_max=3.146095&lat_max=36.797510&start_date=2023-08-15&end_date=2023-08-18&threshold=0.5

Client integration within RACE dashboard still to finish

[lubojr]

Client side integration is merged to staging and pending a markdown description, to be added to https://github.com/eurodatacube/eodash-assets/blob/main/collections/E13c_ship_detections_algorithm/E13c_ship_detections_algorithm.md

[lubojr]

Client side integration is finished. For possible enhancements, please create a followup issue.