Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EPIC: As MaaS4Italy I would like to receive from the Open Data Hub a standard interface for planned and real-time mobility data (NeTEx and SIRI) #1

Closed
3 tasks done
rcavaliere opened this issue Feb 22, 2024 · 86 comments
Assignees

Comments

@rcavaliere
Copy link
Member

rcavaliere commented Feb 22, 2024

In the scope of the new MaaS4Italy project the Open Data Hub is request to provide a standard interface for its mobility data, so that it can be integrated in a new MaaS architecture to be developed at national level. The following specification provides details between available Open Data Hub data and the requested fields.

240701_PianificazioneSviluppi.docx

The services to be considered are:

  • static parking data, incl. e-charging stations (section 2.1.3)
  • static sharing mobility data (section 2.3)
  • real-time data (section 3.2)

Below the national specifications to be considered:

230711_Linee-guida-compilazione-NeTEx-IT-v.3.0.pdf

240502_SpecificaSIRI_v.1.0.3.pdf

DSRM-Architettura Target_Dati dinamici e sharing v1.2_signed.pdf
(relevant: paragraph 2.4)

NeTEx exports can be compared with an XSD before being published, available there https://github.com/5Tsrl/netex-italian-profile

The API is online at

The URL schemes are :

  • /netex
  • /netex/parking
  • /netex/sharing
  • /siri-lite/fm
  • /siri-lite/fm/parking
  • /siri-lite/fm/sharing

Update 14.05.2024: STA has also requested to include in the exports bike parking services, since they are bookable and interesting for MaaS providers. I have amended in track change mode the internal specification document with indication on how to include this additional data. Please also note that there has been a new version of the SIRI national standard, some fields for parking have changed (removed).

@rcavaliere rcavaliere changed the title EPIC: As MaaS4Italy I would like to receive from the Open Data Hub a standard interface for planned and real-time mobility data EPIC: As MaaS4Italy I would like to receive from the Open Data Hub a standard interface for planned and real-time mobility data (NeTEx and SIRI) Feb 23, 2024
@clezag
Copy link
Member

clezag commented Mar 6, 2024

After discussing this in person, some more details

  • NeTEx and SIRI-FM services will be implemented as one or more custom REST endpoints, that produce the XML documents specified in the document on request
  • The intended audience are STA backend services, the endpoints will not be advertised to the public (though accessible)
  • No authentication will be necessary as everything is Open Data
  • For NeTEx, we return the complete dataset as one document
  • For SIRI-FM, we return only the document to one specific station/facility in the request.
  • Where possible, we will validate against the XSD to guarantee formal correctness
  • There is still lots of data missing in the Open Data Hub. The endpoints will follow the mapping in the specification as if the data existed, as it will be added later
  • The provision that "all exports / end-points are made publicly available for all interested 3rd parties on the Open Data Hub" refers to a different, proxy-like service that will be implemented at a later time

@ohnewein
Copy link
Member

ohnewein commented Mar 8, 2024

The deadline for the first PoC is set to end of March 2024

@clezag clezag transferred this issue from noi-techpark/it.bz.opendatahub.api.mobility-ninja Mar 18, 2024
@clezag
Copy link
Member

clezag commented Mar 18, 2024

@rcavaliere First PoC is up for static parking data:

https://sta-netex.opendatahub.testingmachine.eu/netex/parking

I've changed the ID a bit and added IT:OpenDataHub (it has to be globally unique), aside from that it should be faithful to the document.

Booleans currently default to false if no value in OpenDataHub is present, maybe we should implement ternary logic (e.g. leave it empty in netex where we don't have data).

XSD validation is also not implemented yet.

@rcavaliere
Copy link
Member Author

@clezag that's look good, great work.
For the ID: this is not OK, since we need to put as prefix also the ISTAT code of the region. We could use OpenDataHub (in my view, not the proper solution, since the IDs should be related to the data, not to the platform), so something like it:ITH10:xxxx
I send you a technical specification for the IDs in the public transport, which is under implementation by STA.
230116_NeTExProfileSouthTyrol_GlobalID.pdf

@clezag
Copy link
Member

clezag commented Mar 19, 2024

I've updated the ID to the format

IT:ITH10:Parking:<sanitized scode>

where sanitized scode is the scode, with all characters that are not characters, numbers, hypens or underscores replaced with underscores.

e.g. station with scode=me: Karl-Wolf02 becomes IT:ITH10:Parking:me__Karl-Wolf02

This is how I understand the specification of NeTEx (4.2.1), and also matches with their examples.

The document from STA is not consistent with the NeTEx spec:

  • country code must be capitalized, but in the examples is not
  • the parens of the (edp) prefix are not actually part of the prefix, e.g. in the NeTEx spec it's just edp:IT:xxx
  • the edp prefix is only required for frame identifiers, not for entities themselves

Origin is not necessary, since our scode is unique for station type. Each parking station is guaranteed to have a unique scode, and this way we avoid chaos when we deprecate/change origin in the future

@rcavaliere
Copy link
Member Author

@clezag this is fine for me! Thanks for the input for the STA IDs

@clezag
Copy link
Member

clezag commented Mar 25, 2024

A PoC for sharing services (currently bike sharing BZ only) is now available:
https://sta-netex.opendatahub.testingmachine.eu/netex/sharing

A PoC for SIRI-FM is available here:
https://sta-netex.opendatahub.testingmachine.eu/siri/fm/115
the last part of the URL is the Open Data Hub scode, currently for Parking stations only

I think now would be a good time to talk both internally and with STA to define the next steps.

On our side, we mainly need to figure out how to integrate all the missing data (e.g. bike/car models, Operator address, zone polygons etc.).

With STA I'd like to finalize the workflow, format and interfaces of the API, and possible validation, as soon as possible

@rcavaliere
Copy link
Member Author

@clezag wonderful work! I will analyze your work more in deep starting next week. I agree with you, we can start sharing this work with STA and define with them the next steps. I will contact them and put you in CC!

@rcavaliere
Copy link
Member Author

rcavaliere commented May 2, 2024

@rcavaliere
Copy link
Member Author

@clezag I have started to insert the metadata. Parking areas and operators are completely filled.

@rcavaliere
Copy link
Member Author

@clezag a remark: I would suggest to have at the end three different CompositeFrames:

  • parking
  • bike sharing (with all operators / services included, like for parking)
  • car sharing

@rcavaliere
Copy link
Member Author

@clezag an update on the SIRI interfaces. I have got an informal information that we would need to provide these interfaces using the "Lite" approach foreseen in the standard, which is much simpler. At the end it is exactly to provide end-points that a 3rd party can access with simple HTTP requests. Check as examples https://developer.entur.org/pages-real-time-api
So, perfectly in line with what you implemented. Let's wait for the formal specification to complete this part. It's probable that they will ask to implement in the API call some filtering options

@rcavaliere
Copy link
Member Author

@clezag I have modified the user story description. The request is to add bike parking data (static -> NeTEx and dynamic -> SIRI) in the interfaces. Please note that we have also a new SIRI profile, some fields have been removed.
In case of questions let me know!

240514_PianificazioneSviluppi.docx
240502_SpecificaSIRI_v.1.0.3.pdf

@rcavaliere
Copy link
Member Author

@clezag I have completed the insertion of metadata, including for bike parking. Let me know if there are some doubts, I would suggest to include this data as metadata for the reference station types, and then include in the mapping with the NeTEx export. Let me know when you are done so that I can check if in the NeTEx exports we have all fields available with the proper values.

@rcavaliere
Copy link
Member Author

rcavaliere commented May 14, 2024

@clezag I have had also a look at the current exports. In general, it looks very good! There is something I would change.

  • In the parking data of Skidata, in which we have the complexity of the ParkingFacility, we have the issue that names are not unique, so we have three parkings named as "Parcheggio Stazione Bressanone". I would change the Data Collector so to append here the information contained in the metadata field FacilityDescription (seems to be new, was not present when the Data Collector was implemented...)
  • you already told me, please ensure to add all bike sharing services and not just the one of the Municipality of Bolzano
  • in the mobility constraint zone, you just put a "fake" polygon, isn't it? We should create it here, probably on OSM we could easily extract such geographical data.

@clezag
Copy link
Member

clezag commented May 14, 2024

Just writing down what we discussed in person in addition to the points above:

Metadata

  • Parking will be added by data collectors . Where already implemented, supplied via spreadsheet
    • parking-offstreet-meranobolzano
    • parking-tn (trento + rovereto)
    • (blocked due to endpoint not working) parking-offstreet-sta
    • parking-offstreet-a22
    • bike-boxes
  • Operators will be added by the netex endpoint on the fly
  • Bike models will be added by data collector (hardcoded)
  • Car models will be added by data collector (configurable via csv or similar)

Bike Parking

  • Bike parking station type & mapping to be added to parking export

@rcavaliere
Copy link
Member Author

@clezag I have modified the user story description. The request is to add bike parking data (static -> NeTEx and dynamic -> SIRI) in the interfaces. Please note that we have also a new SIRI profile, some fields have been removed. In case of questions let me know!

240514_PianificazioneSviluppi.docx 240502_SpecificaSIRI_v.1.0.3.pdf

@clezag I have seen that in the document the integration modalities for car sharing data in the SIRI interface was missing. I have updated the document.

240521_PianificazioneSviluppi.docx

@rcavaliere
Copy link
Member Author

@clezag an important update in relation to the implementation of the SIRI interfaces. We have received the attached documentation, available also in the description of this epic.
Relevant here is at the beginning of paragraph 2.4: we have to provide two end-points, one for parking and one for sharing services. You can see the names of the end-points we have to implement and what kind of input parameters we have to support.
Let me know in case of issues!

DSRM-Architettura Target_Dati dinamici e sharing v1.2_signed.pdf

@clezag
Copy link
Member

clezag commented Jun 6, 2024

@rcavaliere Just some progress updates:

static parking is now fully implemented:

  • bikeparking implemented
  • missing metadata added to data collectors and mapped in the export
  • operator details configurable via config file in repo
  • operator FBK is split into FBK-Trento and FBK-Rovereto on basis of smetadata.municipality
  • skidata is excluded due to data collector not working

In general I think it's best to limit the dataset to what we know works, and only add new providers/origins explicitly.
To that end I have added a static config file in the repo where you have to explicitly add new origins in the future.
I think this will be better than having the export fail every few months due to new data and someone having to go in there and fix it.

@rcavaliere
Copy link
Member Author

@clezag agree. I have just checked, I inserted all metadata here: https://noibz-my.sharepoint.com/:x:/r/personal/c_zagler_noi_bz_it/_layouts/15/Doc.aspx?sourcedoc=%7B0E6FBB54-B477-4D50-B9A4-1E133893F3B5%7D&file=NeTEx%20missing%20data.xlsx&fromShare=true&action=default&mobileredirect=true

Are you checking this file, isn't it? Let me know if there is something missing.

Do you have some new NeTEx exports that I can check?

Thanks for your great work here

@clezag
Copy link
Member

clezag commented Jun 7, 2024

@rcavaliere
Yes I've used this file.

  • Bike sharing bz information about the locks is missing. I've set it to false for now
  • For bike sharing merano I have assumed they are all muscular (judging from the images) with a basket, lamp and lock

On the static parking export, I've implemented all the missing points, so from my side I consider it complete. Only skidata is missing, which IMO is a "wontfix" until the data collector works again or better is in production.
You can take a look here:
https://sta-netex.opendatahub.testingmachine.eu/netex/parking

Sharing is here:
https://sta-netex.opendatahub.testingmachine.eu/netex/sharing
Bike sharing bolzano, which is the only one implemented, should now be complete.
I've implemented the missing metadata (bicycles, bike models, operators, service constraint polygon).

Still missing are the other bike sharing providers, the car sharing, and then the new SIRI developments

clezag added a commit that referenced this issue Jun 10, 2024
@rcavaliere
Copy link
Member Author

@clezag let's try to close this first implementation iteration.

I have looked at the NeTEx parking export, in my opinion it's nearly done. I would just one thing: I would consider as names for the operators not their origin, but a "standard name". I have made a proposal in the shared Excel file, see here in red:

Screenshot from 2024-06-10 15-04-33

Can you match the correspondent fields with these values?

@clezag
Copy link
Member

clezag commented Jun 10, 2024

@rcavaliere I've added the mapping, but I'm not completely sure I understood:
Should I also use the operator name as ID/key? e.g. should "skidata" and "bicincitta" refer to the same single operator with ID = STA?. Or should I maintain two operators, who happen to have the same name and address?

@rcavaliere
Copy link
Member Author

@clezag thanks! I would also change the IDs accordingly, so that we have consistency in the data.Yes, skidata and bicincittà have to refer to the same operator STA

@leonardehrenfried
Copy link

leonardehrenfried commented Jul 3, 2024

Have you looked into the content itself? Is it conforming the the "SIRI standard" in OTP? From the little experience I've had with this SIRI/Netex stuff, the word "standard" is to be understood loosely, to say the least.

This is also my impression. NeTEx/Siri are really (very big) XML schemas and it's up to the implementers to fill it with life and agree on a subset of it. Most of them call it a profile, so we have EPIP, Nordic profile, UK profile. There appears to be huge room for interpretation, which is a shame.

What I miss most is some form of community where you can ask "Here is my XML, am I using it how it is intended?".

To answer your question: OTP doesn't use the JSON version of Siri so it's hard to say if it conforms to a schema but if I translate it back to XML in my head it looks correct on first glance. However, I would prefer to try it out with a proper parser and then give you feedback.

@clezag
Copy link
Member

clezag commented Jul 3, 2024

@leonardehrenfried
All good, I'll include the XML output in the next round of implementations and ping you here once it's up

@rcavaliere
Copy link
Member Author

Do they know it's open data and we're not doing any authorization anyway? This is a lot of effort for something that doesn't do anything. I suggest we create them some keycloak credentials, they can then request and supply the authentication token, but we don't implement anything that looks at the token and validates it. If further down the line we need authentication/authorization, we can add the authentication code.

@clezag thanks for your feedback. OK, let's push our message that since all this data is licensed CC0 we won't implement any security mechanism in front of the end-points

@rcavaliere
Copy link
Member Author

To be clear, only the filters outlined in the document?

@clezag, yes only these filters are really necessary!

@leonardehrenfried
Copy link

@clezag I tried out the Netex parking feed today and it conformed flawlessly to the schema (that OTP uses). If the Siri XML is of the same quality, I expect it no problems there as well.

@clezag
Copy link
Member

clezag commented Jul 8, 2024

@rcavaliere
Fixed the parking space metadata issue

@leonardehrenfried
I've implemented the SIRI XML feed again.

URL scheme is:

siri-lite/fm
siri-lite/fm/sharing
siri-lite/fm/parking

It defaults to JSON, you can force XML via Accept: application/xml or by setting (and I would leave this an undocumented feature for testing convenience) query string ?format=xml

Let me know if you run into any issues with the XML

@rcavaliere
Copy link
Member Author

@clezag both are siri-lite, the difference with SIRI "standard" is that there is this simple URL, without a publish / subscribe mechanism at the beginning between data producer and data consumer

@clezag
Copy link
Member

clezag commented Jul 8, 2024

@rcavaliere I've edited my comment above, the Accept header and URL scheme are now implemented according to the spec document

@rcavaliere
Copy link
Member Author

@clezag can you please put in the main description of this user story the consolidated end-points for the NeTEx and SIRI exports? I have now a bit confusion about what are the correct URLs :-) For the SIRI end-points, I would suggest, if you already didn't fix this, to change "sta-netex" at the beginning and just put "siri.opendatahub...." as mentioned above. Thanks a lot!

@clezag
Copy link
Member

clezag commented Jul 10, 2024

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

@rcavaliere
Copy link
Member Author

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

@clezag can you please tell me exactly where this kind of filter is mentioned? I can not find it...

@clezag
Copy link
Member

clezag commented Jul 10, 2024

@rcavaliere I'm implementing the filters, and I'm not sure what "datasetId" is supposed to be. To which fields does it map (either in OpenDataHub or netex)

@clezag can you please tell me exactly where this kind of filter is mentioned? I can not find it...

https://github.com/user-attachments/files/16069269/240620_RAP.-.Specifiche.interfacce.dati.dinamici.pdf
2.5

@rcavaliere
Copy link
Member Author

@clezag I would implement just the facilityRef filter. The datasetId would be useful if we have in the SIRI feed a field specifying the data provider for certain records, which would correspond to our "sorigin" field. But it's not clear to me how we could implement this...

@clezag
Copy link
Member

clezag commented Jul 11, 2024

@rcavaliere

I've implemented the filters (aside from datasetId).
Due to implementation details, and because we also have a combined endpoint for parking and sharing, all filters work on all SIRI endpoints (e.g. you can filter realtime parking data with gps coordinates and by operator, and have maxSize everywhere).

Regarding the endpoint URL, since the service is running both netex and siri at the same time, what do you think of making the endpoint transmodel.api.opendatahub.com?
You would then do transmodel.api.opendatahub.com/netex/... and transmodel.api.opendatahub.com/siri-lite/...

I can also set it up like you suggested, and have two URLs,

  • netex.api.opendatahub.com/parking/...
  • siri.api.opendatahub.com/lite/fm/parking

but it's a more complicated setup with custom proxy configuration, separate swagger definitions and so on, because we will fake a single service being two separate ones

I would prefer one single URL, and then have the /netex/ and /siri-lite/ paths to differentiate between services, but the decision is yours

@rcavaliere
Copy link
Member Author

@clezag thanks for your work. I agree, let's have one single end-point called transmodel.api.opendatahub.com
Can you set it up today, both in a testing and productive mode? Testing end-point should therefore be transmodel.api.opendatahub.testingmachine.eu

@clezag
Copy link
Member

clezag commented Jul 12, 2024

@clezag can you please put in the main description of this user story the consolidated end-points for the NeTEx and SIRI exports? I have now a bit confusion about what are the correct URLs :-) For the SIRI end-points, I would suggest, if you already didn't fix this, to change "sta-netex" at the beginning and just put "siri.opendatahub...." as mentioned above. Thanks a lot!

@rcavaliere
I've added the new endpoints that are now online to the main story.
I've also changed this repo's name accordingly

I'm still working on providing swagger etc. which I will also link in the main story (should just be the / base URL as in our other APIs)

One minor thing: please note that in production, using the SIRI maxDistance filter currently results in an error. This is because it relies on a new ninja API feature that has not gone in to production yet (should do so early next week)

In the sprint meeting today, we also emphasized to not forget to link and document this API on out main channels and websites once it has reached some level of stability (when it goes into testing with the NAP).

@clezag
Copy link
Member

clezag commented Jul 12, 2024

@leonardehrenfried
the API has been migrated to

https://transmodel.api.opendatahub.com (production)
https://transmodel.api.opendatahub.testingmachine.eu (testing)

and the old URL will be dismissed shortly

@rcavaliere
Copy link
Member Author

@clezag can you please put here some examples on how you can make some API calls with active filters? I was not able alone to reproduce this. Thanks!

@clezag
Copy link
Member

clezag commented Jul 15, 2024

@rcavaliere
There was a bug so that latitude and longitude were reversed, now it's fixed.
The change to the ninja API are now in production as well, so all filters should work:

@leonardehrenfried
Copy link

Note to myself, the Netex parking feed is now at: https://transmodel.api.opendatahub.testingmachine.eu/netex/parking

@clezag
Copy link
Member

clezag commented Jul 15, 2024

Note to myself, the Netex parking feed is now at: https://transmodel.api.opendatahub.testingmachine.eu/netex/parking

Just FYI, this is the testing link, production is https://transmodel.api.opendatahub.com/netex

@clezag
Copy link
Member

clezag commented Jul 15, 2024

Openapi spec is now up:

https://transmodel.api.opendatahub.com/

Feel free to suggest additional documentation we could include.

@leonardehrenfried
Copy link

@clezag I'm currently working on consuming the Siri feed. What does the following mean?

		<FacilityCondition>
				<FacilityRef>IT:ITH10:Parking:IT_RBK_EBZ002_3</FacilityRef>
				<FacilityStatus>
					<Status>notAvailable</Status>
				</FacilityStatus>
				<MonitoredCounting>
					<CountingType>presentCount</CountingType>
					<CountedFeatureUnit>devices</CountedFeatureUnit>
					<Count>1</Count>
				</MonitoredCounting>
			</FacilityCondition>

This is a parking lot with RechargingAvalable=true, so presumably that's a charger for an electric car. But does <Status>notAvailable</Status> mean that the status is unknown or that the charger is currently not available, so cars cannot park there?

@clezag
Copy link
Member

clezag commented Jul 17, 2024

@clezag I'm currently working on consuming the Siri feed. What does the following mean?

		<FacilityCondition>
				<FacilityRef>IT:ITH10:Parking:IT_RBK_EBZ002_3</FacilityRef>
				<FacilityStatus>
					<Status>notAvailable</Status>
				</FacilityStatus>
				<MonitoredCounting>
					<CountingType>presentCount</CountingType>
					<CountedFeatureUnit>devices</CountedFeatureUnit>
					<Count>1</Count>
				</MonitoredCounting>
			</FacilityCondition>

This is a parking lot with RechargingAvalable=true, so presumably that's a charger for an electric car. But does <Status>notAvailable</Status> mean that the status is unknown or that the charger is currently not available, so cars cannot park there?

@leonardehrenfried

The specs of the mapping are [here, section 3.2.x] (https://github.com/noi-techpark/transmodel-api/blob/main/documentation/240603_PianificazioneSviluppi.docx)

TL;DR: In general, the status is to be interpreted as the state of parking, not charging availability. In your case the one parking space is occupied (presentCount=1, capacity=1), so parking is 'notAvailable'

For the specific parking station you are looking at, we happen to deduce it from the charging state, because that's all we have, but a SIRI consumer wouldn't know about that.
Having recharging can be a feature of a parking area, but it's not the thing we are reporting about via SIRI.

@rcavaliere correct me if I'm wrong here, you wrote the spec 😉

@rcavaliere
Copy link
Member Author

@clezag the feedback is correct. At the end the e-charging is location is not available in the sense that a person can not make a charging operation, or is not available because somebody else is charging the car. In both cases the parking is to be considered not available, and that's the reason why, at the moment, we do not differentiate these situations. Also for routing purposes, such parking shouldn't be considered in such situations.

@rcavaliere
Copy link
Member Author

@clezag we have received an approval for our SIRI FM end-point by the reference national staff! The only thing they have asked to change is the naming of our end-points, as follow:

Current situation:

Final situation:

Can you please fix this? Please do it also on our testing environment, thanks!

@clezag
Copy link
Member

clezag commented Jul 22, 2024

@clezag we have received an approval for our SIRI FM end-point by the reference national staff! The only thing they have asked to change is the naming of our end-points, as follow:

Current situation:

* https://transmodel.api.opendatahub.com/siri-lite/fm/parking

* https://transmodel.api.opendatahub.com/siri-lite/fm/sharing

Final situation:

* https://transmodel.api.opendatahub.com/siri-lite/facility-monitoring/parking

* https://transmodel.api.opendatahub.com/siri-lite/facility-monitoring/sharing

Can you please fix this? Please do it also on our testing environment, thanks!

Done

@rcavaliere
Copy link
Member Author

@clezag I think we can close the EPIC. In case of further enhancements, let's open new specific tickets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants