doc: create ADR for Skate MQTT Hierarchy

Co-authored-by: Hannah Purcell <69368883+hannahpurcell@users.noreply.github.com>
Co-authored-by: Josh Larson <jlarson@mbta.com>

documentation/adr/0004-trip-modifications-mqtt-topic-structure.md

@@ -0,0 +1,183 @@
+# 4. Trip Modifications MQTT Topic Structure
+
+Date: 2024-07-04
+
+## Status
+
+Accepted
+
+## Context
+
+[With the creation of #ADR0003](./0003-use-mqtt-to-publish-trip-modifications.md),
+we are trying to decide on the structure for _how_ we send and persist these 
+[Trip Modifications](https://gtfs.org/realtime/reference/#message-tripmodifications) between Skate and Transit Data.
+
+We know that we'll need these messages to be available if the Transit Data
+service is restarted.
+
+## Decision
+
+We'll use a
+[topic hierarchy](https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/)
+which allows us to manage individual GTFS-RT messages and update them, and does
+not require that the Transit Data service has some way to persist state between
+restarts.
+
+### Topic Hierarchy
+We'll nest all of our Trip Modifications related data under the
+[MQTT "topic level"](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718107)
+`trip_modifications`, which will be under our "topic prefix"
+configured in the DevOps repo for each environment.
+
+For Example:
+
+Our `<prefix>`s would be derived from the environment and are described via IaC
+in the devops repo
+```
+skate/prod
+skate/dev
+skate/dev-blue
+skate/dev-green
+```
+And the final combined topic hierarchy would be formed by prepending _one_ of 
+the `<prefix>`s
+```
+<prefix>/trip_modifications/
+```
+For instance,
+```
+skate/dev-blue/trip_modifications/
+```
+
+
+For each new and unique trip modification, we'll generate a unique ID and use
+that as our Trip Modification identifier, named `id`.
+
+> [!NOTE]  
+> The GTFS-RT Trip Modifications spec does not currently have a ID for each modification,
+> so this `ID` _will not_ appear with the Trip Modification Message.
+
+The topic hierarchy is then formed using this ID as the topic level
+```
+<prefix>/trip_modifications/#{id}/
+```
+
+Then, to create or update an associated `trip_modification`, we'll use the topic
+level `trip_modification`
+Creating the topic
+```
+<prefix>/trip_modifications/#{id}/trip_modification
+```
+
+Any associated [`Shape`](https://gtfs.org/realtime/reference/#message-shape)
+Messages will be published to the topic level `shape` in the hierarchy.
+```
+<prefix>/trip_modifications/#{id}/shape
+```
+
+On `Shape` Messages, the
+[`Shape.shape_id`](https://gtfs.org/realtime/reference/#message-shape)
+field will also have a unique ID generated by Skate, which will be
+referenced by the corresponding Trip Modification's
+[`SelectedTrips.shape_id`](https://gtfs.org/realtime/reference/#message-selectedtrips)
+field.
+
+#### Active Trip Modifications State
+Messages pushed to these topics will have the
+[`retain` property](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349265)
+set so that new connections get active Trip Modifications and Shapes published
+to them on connect.
+
+Deactivating a Trip Modification will be done by
+[publishing a zero-byte payload](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349265)
+each topic within the `<prefix>/trip_modifications/#{id}/`
+topic hierarchy, which will clear any retained messages and ensure that new
+subscribers are not notified of these now deleted messages and existing
+subscribers are made aware of the removal.
+
+### Schema
+We'll publish using JSON encoded versions of the GTFS-RT Messages to these topics.
+The `payload` of the MQTT message will contain a JSONAPI compatible string
+where the `data` key contains the GTFS-RT Message encoded as JSON.
+
+`<prefix>/trip_modifications/#{id}/trip_modification`
+```json5
+{
+  "data": {
+    // <GTFS-RT FeedEntity Message as JSON>
+    id: "", // <Trip Modification ID>
+    trip_modifications: {}, // <GTFS-RT Trip Modification Message as JSON>
+  }
+  "meta": {
+    // this field will be removed once we give users the ability to activate and
+    // deactivate detours themselves.
+    //
+    // Currently, this indicates that this message should not be published to
+    // applications outside of the MBTA, and is for internal testing and
+    // development only.
+    "is_draft?": boolean
+  }
+}
+```
+`<prefix>/trip_modifications/#{id}/shape`
+```json5
+{
+  "data": {
+    // <GTFS-RT FeedEntity Message as JSON>
+    id: "", // <Shape ID>
+    shape: {}, // <GTFS-RT Shape Message as JSON>
+  }
+}
+```
+
+### Subscriber Outcomes
+With this Structure, _theoretically_, the only subscription that Transit Data
+should need to make is to the following topics.
+```
+<env_prefix>/trip_modifications/+/trip_modification
+<env_prefix>/trip_modifications/+/shape
+```
+OR, Using only [wildcard topics](https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718107):
+```
+<env_prefix>/trip_modifications/+/+
+<env_prefix>/trip_modifications/#
+```
+
+## Consequences
+
+This topic hierarchy requires that Skate is ensuring that the open retained
+messages on MQTT are synced with our own internal SOT of which
+Trip Modifications are active.
+
+
+## Alternatives Considered
+
+1.  A single topic in which typed events are published, with or without
+    differential feeds in addition to the typed events.
+    (Detour A exists now, Detour B now applies to trip xyz, Detour C is
+    deactivated, etc).
+
+    It seemed like the complexity of managing retained messages and a bunch of
+    message "types" would be a bit more of a hassle instead of leaning into MQTT
+    topics, in addition to figuring out retention/"rehydration".
+
+2.  Topics split out by detour ID with retained messages, but instead of
+    deactivating with a 0-byte message, deactivating with an explicit "deactivated"
+    message.
+
+    Before we 100% knew how retained messages were removed, we considered
+    publishing "deactivation" messages to the topics which contained
+    Trip Modifications previously. Given MQTT has a deletion method already, we
+    are opting to use that.
+
+3.  No deactivating at all - we just stop adding trip ID's to a particular detour
+    (or remove them if they've already been added) once a detour doesn't apply to
+    future or current trips anymore.
+
+    Because we need to keep Trip Modifications up to date with the trips that
+    the modification applies to, we'll be publishing frequent updates depending
+    on how many trips out we're configured to say will be affected. We
+    theoretically could publish a Trip Modification with an empty
+    `selected_trips` field, but this would go against the
+    [GTFS spec of Required: Many, Cardinality: One](https://gtfs.org/realtime/reference/#message-selectedtrips)
+