SPIKE: What's involved in serving multiple versions of the Hub API?

[edgarrmondragon]

See #1418 (comment)

[tayloramurphy]

FYI @edgarrmondragon I've added to the engineering board cc @seth-meltano

[edgarrmondragon]

Because the https://github.com/meltano/hub/tree/99731c055bc1a1513228745db673aab789d4047c/_data/meltano directory is the source of truth for the API, with the current process any changes to the source files will be reflected in the API responses.

To support multiple versions of the API and have updates to plugin data reflect in all of them (meaning v1 can be used to retrieve new plugins), the following steps are required:

1. Refactor the script that builds the API files to output one directory for each version. The script will have to convert newer properties to be compatible with older versions of Meltano. For example, changing instances of sensitive: true and hidden: true to kind: password and kind: hidden, respectively.

2. Start hosting plugin files in S3 under s3://<BUCKET>/hub-api/v1/plugins and s3://<BUCKET>/hub-api/v2/plugins instead of s3://<BUCKET>/hub-api/plugins.

3. Adjust our Netlify redirects from /meltano/api/v1/*" to /meltano/api/*"

With this approach, no changes are required to our infra and changes can be tested in the staging environment.

Step 1 above involves the bulk of the effort and about ~1 week of work.

[tayloramurphy]

To support multiple versions of the API and have updates to plugin data reflect in all of them (meaning v1 can be used to retrieve new plugins),

@edgarrmondragon is this something we actually need? I'd say after we switch to using sensitive we wouldn't need to update v1 anymore. But maybe I'm wrong there. cc @pnadolny13 if you have a thought

[edgarrmondragon]

@edgarrmondragon is this something we actually need?

@tayloramurphy the alternative is to freeze v1. New plugins would not be available there, so users would have to update Meltano to be able to meltano add them to their project. We'd still need to restructure the S3 bucket: move the current /hub-api/plugins to /hub-api/v1/plugins and update the upload script to push files to /hub-api/v2.

[tayloramurphy]

@edgarrmondragon how would we determine if the metadata was appropriate for v1? Would we just introspect the settings and not update it if we see sensitive? It's not clear to me how we freeze some settings but not the others.

It seems like it'd be easier to do a freeze for v1 but maybe I'm over thinking it.

[pnadolny13]

@tayloramurphy I think in an ideal world both API versions would still continue to update as plugin metadata changes. If its too complex then freezing it would be the backup plan but it feels pretty harsh. To me the benefits of this sensitive/hidden feature dont outweigh the negative affects of freezing the API.

It might be a little more overhead but I was thinking along the same lines as Edgar described above:

Refactor the script that builds the API files to output one directory for each version. The script will have to convert newer properties to be compatible with older versions of Meltano. For example, changing instances of sensitive: true and hidden: true to kind: password and kind: hidden, respectively.

This doesnt sound like a lot of work to me especially because in the API generation script we're already reading every plugin definition and deleting properties that are only for rendering the hub pages like usage, settings_preamble, etc.

One other thing to consider is that we might want another schema change like this in the near future for settings group validation improvements, like if we support json schema format instead of our custom format.

[edgarrmondragon]

@edgarrmondragon how would we determine if the metadata was appropriate for v1? Would we just introspect the settings and not update it if we see sensitive? It's not clear to me how we freeze some settings but not the others.

It seems like it'd be easier to do a freeze for v1 but maybe I'm over thinking it.

@tayloramurphy what I imagine is that we build the Pydantic models for v1 and v2 plugins. For each plugin from the source data, we instantiate a v2 plugin and also create a v1 instance from that. We save both in their respective directory and push to S3.

[tayloramurphy]

@edgarrmondragon thanks for the explanation. I think that makes sense.

I think we'll need to pick a level of long term support for earlier versions of the API at some point. Perhaps v1 and v2 are deprecated at a Meltano v4 launch (or something like that). At some point we won't want to maintain them.

But I like the proposed approach for supporting both for now!