diff --git a/documentation/adr/0004-trip-modifications-mqtt-topic-structure.md b/documentation/adr/0004-trip-modifications-mqtt-topic-structure.md new file mode 100644 index 0000000000..9234078d6b --- /dev/null +++ b/documentation/adr/0004-trip-modifications-mqtt-topic-structure.md @@ -0,0 +1,179 @@ +# 4. Trip Modifications MQTT Topic Structure + +Date: 2024-07-04 + +## Status + +Draft + +## Context + +[With the creation of #ADR0003](./0003-use-mqtt-to-publish-trip-modifications.md), +we are trying to decide on the structure for _how_ we send and persist these +[Trip Modifications](https://gtfs.org/realtime/reference/#message-tripmodifications) between Skate and Transit Data. + +We know that we'll need these messages to be available if the Transit Data +service is restarted. + +## Decision + +We'll use a +[topic hierarchy](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) +which allows us to manage individual GTFS-RT messages and update them, and does +not require that the Transit Data service has some way to persist state between +restarts. + +### Topic Hierarchy +We'll nest all of our Trip Modifications related data under the +[MQTT "topic level"](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/) +`trip_modifications`, which will be under our "topic prefix" +configured in the DevOps repo for each environment. + +For Example: + +Our ``s would be derived from the environment and are described via IaC +in the devops repo +``` +skate/prod +skate/dev +skate/dev-blue +skate/dev-green +``` +And the final combined topic hierarchy would be formed by prepending _one_ of +the ``s +``` +/trip_modifications/ +``` +For instance, +``` +skate/dev-blue/trip_modifications/ +``` + + +For each new and unique trip modification, we'll generate a unique ID and use +that as our Trip Modification identifier, named `id`. + +> [!NOTE] +> The GTFS-RT Trip Modifications spec does not currently have a ID for each modification, +> so this `ID` _will not_ appear with the Trip Modification Message. + +The topic hierarchy is then formed using this ID as the topic level +``` +/trip_modifications/#{id}/ +``` + +Then, to create or update an associated `trip_modification`, we'll use the topic +level `trip_modification` +Creating the topic +``` +/trip_modifications/#{id}/trip_modification +``` + +Any associated [`Shape`](https://gtfs.org/realtime/reference/#message-shape) +Messages will be published to the topic level `shape` in the hierarchy. +``` +/trip_modifications/#{id}/shape +``` + +On `Shape` Messages, the +[`Shape.shape_id`](https://gtfs.org/realtime/reference/#message-shape) +field will also have a unique ID generated by Skate, which will be +referenced by the corresponding Trip Modification's +[`SelectedTrips.shape_id`](https://gtfs.org/realtime/reference/#message-selectedtrips) +field. + +#### Active Trip Modifications State +Messages pushed to these topics will have the +[`retain` property](https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/) +set so that new connections get active Trip Modifications and Shapes published +to them on connect. + +Deactivating a Trip Modification will be done by +[publishing a zero-byte payload](https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/#heading-how-to-delete-retained-messages-in-mqtt) +each topic within the `/trip_modifications/#{id}/` +topic hierarchy, which will clear any retained messages and ensure that new +subscribers are not notified of these now deleted messages and existing +subscribers are made aware of the removal. + +### Schema +We'll publish using JSON encoded versions of the GTFS-RT Messages to these topics. +The `payload` of the MQTT message will contain a JSONAPI compatible string +where the `data` key contains the GTFS-RT Message encoded as JSON. + +`/trip_modifications/#{id}/trip_modification` +```json5 +{ + "data": { + # + } + "meta": { + // this field will be removed once we give users the ability to activate and + // deactivate detours themselves. + // + // Currently, this indicates that this message should not be published to + // applications outside of the MBTA, and is for internal testing and + // development only. + "is_draft?": boolean + } +} +``` +`/trip_modifications/#{id}/shape` +```json5 +{ + "data": { + # + } +} +``` + +### Subscriber Outcomes +With this Structure, _theoretically_, the only subscription that Transit Data +should need to make is to the following topics. +``` +/trip_modifications/+/trip_modification +/trip_modifications/+/shape +``` +OR, Using only [wildcard topics](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/#heading-avoid-subscribing-to-wildcards): +``` +/trip_modifications/+/+ +/trip_modifications/# +``` + +## Consequences + +This topic hierarchy requires that Skate is ensuring that the open retained +messages on MQTT are synced with our own internal SOT of which +Trip Modifications are active. + + +## Alternatives Considered + +1. A single topic in which typed events are published, with or without + differential feeds in addition to the typed events. + (Detour A exists now, Detour B now applies to trip xyz, Detour C is + deactivated, etc). + + It seemed like the complexity of managing retained messages and a bunch of + message "types" would be a bit more of a hassle instead of leaning into MQTT + topics, in addition to figuring out retention/"rehydration". + +2. Topics split out by detour ID with retained messages, but instead of + deactivating with a 0-byte message, deactivating with an explicit "deactivated" + message. + + Before we 100% knew how retained messages were removed, we considered + publishing "deactivation" messages to the topics which contained + Trip Modifications previously. Given MQTT has a deletion method already, we + are opting to use that. + +3. No deactivating at all - we just stop adding trip ID's to a particular detour + (or remove them if they've already been added) once a detour doesn't apply to + future or current trips anymore. + + Because we need to keep Trip Modifications up to date with the trips that + the modification applies to, we'll be publishing frequent updates depending + on how many trips out we're configured to say will be affected. We + theoretically could publish a Trip Modification with an empty + `selected_trips` field, but this would go against the + [GTFS spec of Required: Many, Cardinality: One](https://gtfs.org/realtime/reference/#message-selectedtrips) +