Define how to expose version through the collection metadata API

[Ndpnt]

This PR is an experiment to create RFC that will lead to a decision record delivery.

The idea is to add comments right onto the document, much like you would with code. I will update the document for each consensus reached.

The deadline for this RFC is 27/11 end of day AoE (Anywhere on Earth).

[MattiSG]

Several stylistic changes and complements.

[MattiSG]

This multi-line example is invalid. We would need to transform newlines to \n. The example should also provide an example with JSON data, to demonstrate how it would be encoded.

[Ndpnt]

Solutions A and C are interesting, but I find that while it's still possible to pass metadata via HTTP headers, this approach is less common and less discoverable than solution B.
Furthermore, in the case of solution C, the need to make a new request due to a redirect for each call does not seem desirable to me. Especially as the caching and transmission of metadata can be achieved in another way and are also doable in solution B.

So, I vote for the solution B as it aligns with the common and anticipated conventions of API responses. It provides a straightforward and highly extensible method for passing metadata in a standardized manner.

@MattiSG @clementbiron @madoleary @michielbdejong What do you think ?

Note: The deadline elapsed a week ago, and I am starting the implementation this afternoon. The decision on the chosen solution will be confirmed tomorrow morning on the basis of the votes taken between now and then.

[MattiSG]

Thanks @Ndpnt!

Reading through solutions A, B and C, I came up with solution D that aims at keeping the best parts of each:

• A for ease of content access
• B for metadata extensibility
• C for performance capabilities

It has the added benefit of being possibly split in implementation in two parts, which are made clear in the proposal. Indeed, I feel that we have two orthogonal choices to make rather than 3 full-fledged solutions:

1. Do we redirect to leverage caching?
2. Do we serve JSON or Markdown?

…and that choice 2 can be delayed: for sure, there is relevance in JSON extensibility. Let's start with implementing that one and, while we're at it, defining in this RFC how to implement the Markdown version, without committing to implementing it at this stage as we first collect feedback from the JSOn version 🙂

I thus vote for solution D, with only implementation of the JSON version of the endpoint!

[Ndpnt]

Thanks @MattiSG for this relevant contribution!

I really like the idea of mixing solutions to keep the best parts of each 👍. At this stage I still find that using redirections to leverage caching is a premature optimization. And I prefer to wait until the need arises before choosing how to manage the cache. In addition, this caching optimisation also has the cost of introducing an extra round trip between client and server, which I still don't find desirable.

So for now I'm in favor of the solution D without the redirection mechanism. It means where solution D returns HTTP 302, I suggest to returns HTTP 200 with the version (either in markdown format or in JSON format according to the Content-Type). I added it in the document in solution E.

I thus vote for solution E (and still with only implementation of the JSON version).

[clementbiron]

I'm in favour of solution E, which I feel is sufficient for the uses identified.

Well done Nico for your work!

[madoleary]

I, too, am in favor of solution E. I think the implementation is solid, particularly returning HTTP 200 with the version, and I agree that optimization can come later.

For posterity, I'd like to recap a conversation that happened outside of GitHub regarding this RFC and certain questions of mine.

Generally, I was confused by the term "version" and the function of the date. I also didn't understand the difference between a version and a dataset.

@MattiSG explained to me that a “version” is indeed “the content" of the term text, in Markdown format, and that a dataset is the consolidation of all those versions.

Here is an example of a version: https://github.com/OpenTermsArchive/contrib-versions/blob/46d10a6c2ef50a11eaa39d4fa5ae3913e9258831/Atlassian/Privacy%20Policy.md

Here is an example of a dataset: https://github.com/OpenTermsArchive/contrib-versions/releases/tag/dataset-contrib-2023-12-04

Regarding the date, I wondered whether it simply represents the day the terms were added or updated, and whether there could be multiple dates for the same terms type. I also questioned how handling the dates on the ToS;DR end would work, i.e., how we'd know which date to use for the API request.

Matti answered these questions in kind. There is an infinity of valid dates between their first declaration in Open Terms Archive and the current time. This enables clients such as ToS;DR to request any date and let the API retrieve the version (terms text) that was valid at that date. In our use case, it's likely that we'll always use the current date and time to get the latest version, but this is something we should verify.

Thank you!

[Ndpnt]

Solution E received the most votes and will thus be selected.

Thanks for your participation :).

[michielbdejong]

Maybe pin the timezone to UTC? So "A full UTC data and time"?

[Ndpnt]

If the timezone is defined as an offset from UTC (ex: 2023-12-06T12:34:56+02:00) is still ISO 8601 valid.
So, why forcing the timezone to be UTC?

[michielbdejong]

And require that the timezone be UTC?

[michielbdejong]

Ah, looks like I came to this discussion just a day after it finished :)
OK, cool! You can take my suggestion to require UTC as the only allowed timezone or not, we can also simplify our code on our side to always use UTC even if your API supports more.

I think one way we could use this API is by setting the date far into the future, to get the latest known version, right? For instance:

GET /version/Open%20Terms%20Archive/Privacy%20Policy/9999-01-01T00%3A00%3A00.000Z`

And that would be cacheable! :)

[Ndpnt]

Ah, looks like I came to this discussion just a day after it finished :) OK, cool! You can take my suggestion to require UTC as the only allowed timezone or not, we can also simplify our code on our code to always use UTC even if your API supports more.

👍

I think one way we could use this API is by setting the date far into the future, to get the latest known version, right? For instance:

GET /version/Open%20Terms%20Archive/Privacy%20Policy/9999-01-201T00%3A00%3A00.000Z`

And that would be cacheable! :)

We could also use a more elegant alias (latest for example) which would also allow caching:

GET /version/Open%20Terms%20Archive/Privacy%20Policy/latest

[MattiSG]

I think one way we could use this API is by setting the date far into the future, to get the latest known version, right?

In the current proposal, this would not work and would reply with a 416 Range Not Satisfiable: the idea is that you're asking for the legal text that was applicable at the given date, and we can't guess what text will be applicable in the future.

I believe that this spontaneous usage, instead of using Date.now(), justifies further the relevance of a /latest endpoint.

[MattiSG]

There is an inconsistency with the Markdown / JSON content in this proposal here.

Suggested change

	- HTTP 400 Bad Request with JSON content `# Error\n\n_Requested date <requested-date> is in the future, no version can exist there_` if the date is in the future.
	- HTTP 400 Bad Request with Markdown content `# Error\n\n_Requested date <requested-date> is in the future, no version can exist there_` if the date is in the future.

[MattiSG]

Thanks everyone! That works for me 🙂 👍

My only concern is the lack of a /latest route for implementers, but since we introduced a 416 for far-future dates, the /9999 trick suggested will not work and will thus enforce the use of Date.now. This might not be comfortable but is at least forward-compatible, and we could thus introduce the /latest route as an improvement after shipping the first version.