Allow IIA APIs to be distributed

[wrygiel]

@georgschermann proposed that we should have an alternate set of APIs for exchanging data in a decentralized way.

[wrygiel]

Conclusion from our current meeting is that we should prepare and alternate set of APIs for editing IIAs in a distributed environment.

The way I see it, we should split the work in the same fashion as we did with other APIs so far: @erasmus-without-paper/wp4 will propose the API signatures, while @erasmus-without-paper/wp3 will later fill them with detailed schemas.

[wrygiel]

We have described a distributed version of IIA APIs during the meeting in Warsaw. Now we need to document them.

[wrygiel]

I believe that due to the nature of this problem, EWP Host X may end up hosting some IIAs which are not related to any of the HEIs covered by X. Is this a problem?

This may happen for example when the list of partners changes after the IIA has been first published. This scenario is unlikely, but possible.

[wrygiel]

Also:

• if we still want a central "fallback" web implementation, and
• if we still want to make implementing servers and clients for this API optional,

then all EWP Hosts which choose to implement the server part of the API will be REQUIRED to allow our "fallback" web client to access all their IIAs which are related to HEIs which had not implemented their own clients.

In other words, our central "fallback" web client will still need to have write-access to the majority of IIAs, regardless of where they are stored.

When joined with all the other complexities of distributed systems, this becomes really crazy...

[georgschermann]

This may happen for example when the list of partners changes after the IIA has been first published. This scenario is unlikely, but possible.

we could forsee the possibility for an IIA to go into a finalized state after it was published and received by every partner, which means that at this state every partner has a copy (PDF or XML, etc.) of this agreement and it (possibly) can't be further edited and is safe for removal until it probably gets reopened by a partner (pobably with states as discussed, proposed, agreed on, finalized)
of course the one hosting the IIA is obligated to keep it as long as it is valid

In other words, our central "fallback" web client will still need to have write-access to the majority of IIAs, regardless of where they are stored.

this client would just call the regular API of the other EWP Host in behalf of the HEI using the client, no special write permissions required

for those HEIs without the server part of this API in place, this part could also be generated and hosted by the EWP server as a fallback (probably as an opt in possibility by just specifying the EWP server url in the manifest for the IIA API)
this would only require the owner of the IIA to also manipulate it through the API, which would be a good design anyways I think

so the scenario when no participating HEI has the APIs in place would be:
one HEI using the generated IIA client to communicate via the regular API calls with the EWP hosted IIA server component (acting for the target HEI) and the owner HEI of the IIA would also have to communicate to its own IIA server API (located on the EWP server) as everybody else

for convenience and approval the (web) client could be bundled with a tomcat and optional lightweigth database and provided to the HEIs instead of beeing hosted by EWP, so they can host it themselve
this client if generated could probably benefit from using certificates to authorize the sending HEI since it could remove the necessarity to login

but as also stated in Warsaw that is exaclty what SOP is gonna do anyways, provide a ready to use client (for all or most APIs, not only IIA) either hosted by the HEI themselve as a standalone EWP host or by our SAAS cluster, which will align perfectly with the distributed nature of EWP

[wrygiel]

we could forsee the possibility for an IIA to go into a finalized state after it was published (...)

What I meant by "after the IIA has been first published" was "after a first draft of the IIA has been published". At this point, other partners need to be able to edit this draft, so the master of this IIA has to be already established. And, if this master is selected incorrectly at this point, then it will need to stay this way.

client (...) could probably benefit from using certificates to authorize the sending HEI since it could remove the necessarity to login

That's an interesting idea. However, if I understood it correctly, it would require the sending HEI to provide some kind of OAuth-like workflow for the IRO staff member (and then sign the token with HEI's certificate...?). So this could be a bit cumbersome.

Our idea was much simpler: Set up automatic user accounts for all proper IRO staff members which the HEI has indicated in its Institutions API. This way, the only work HEI needs to do is to provide the current list of their IRO staff member emails. (And the central client needs to query these lists periodically.)

I am currently updating the specs and will post the proposal for the new version soon.

[wrygiel]

BTW, the "opt-in" for the central client would also be done via setting a flag in Institutions API. Manifests - currently - serve a bit different purpose (they describe EWP Hosts more than individual HEIs).

[georgschermann]

I think if the master of the IIA leaves the IIA and is no longer related to it during the negotiations it would most likely be safe to discard the whole IIA. Keeping them could make other things ambiguous like how they should be handled by other APIs e.g. the list-agreements

However, if I understood it correctly, it would require the sending HEI to provide some kind of OAuth-like workflow for the IRO staff member (and then sign the token with HEI's certificate...?). So this could be a bit cumbersome.

you are right, the authentication of the different staff members would still be needed to be handled by the client, but this could be done relatively simple by using e.g. the app servers container managed security which would be compatible with a fast range of authentication systems
so the HEi would have to configure the client accordingly to their needs

[wrygiel]

I think if the master of the IIA leaves the IIA and is no longer related to it during the negotiations it would most likely be safe to discard the whole IIA. Keeping them could make other things ambiguous like how they should be handled by other APIs e.g. the list-agreements

You mean IIA Search API? This API provides the list of all IIAs hosted by the particular provider. However, in order for a HEI X to discover all IIAs which X is related to, X would still need to query all providers periodically. I don't see any way to simplify this, regardless of whether we discard such IIAs or not.

[wrygiel]

so the HEi would have to configure the client accordingly to their needs

But we were considering using your client in the Central IIA Repository too. This wouldn't work then, I think?

[georgschermann]

our software stores the database parameters in a config file per host and all other configurations inside the database per customer, they would all be free to configure their SSO systems or use our internal authentication system and (probably) may opt out of specific EWP APIs

with central IIA repository you mean the fallback server part of the IIA API? In that case I don't think that much configuration or user authentication would be necessary, since it would only be used by the IIA client parts

[wrygiel]

I suspect that my questions will make more sense after you read the newer versions of the specs (the ones I haven't yet posted). So, we'll talk then :)

[wrygiel]

I have posted the new specifications here:

https://github.com/erasmus-without-paper/ewp-specs-api-iias-host

Note, that this is a separate APIs with 4 endpoints. Once we decide to go this way, it will replace the IIAs API and IIA Search API.

[wrygiel]

We have one more - and much easier - proposal to consider. Because there are so many of them, let's assign some numbers to each.

IIA1 - Central IIA Repository

This is the solution proposed initially. We have a single host for all agreements. Only this host implements the APIs. All other partners store their IIAs on this host, by signing in onto the web GUI hosted there. Only the approved IIAs are available via the APIs.

IIA1B - Central IIA Repository with Remote Update APIs

This is similar to IIA1, but:

• Also draft IIAs will be visible via the APIs.
• <timeline/> element will be added to the hosted IIAs.
• Partners will be allowed to implement a client and edit IIAs remotely. (Master copies of these IIAs will still be stored on the central server.)

IIA2 - Distributed, single-master-per-IIA

This solution is described here: Interinstitutional Agreements Host API. In short:

• Each IIA has a single master.
• Every implementer of the Host API will be a IIA hosting provider - it will be able to become the master for any IIA.
• We will still have a fallback central server which will be able to be the master for all the IIAs, but HEIs will be able to pick a master on their own.
• We will still have a fallback web client somewhere on EWP domains which users will be able to use for accessing all providers.

IIA3 - Distributed, no-master, "read-only"

This is the new proposal to think about.

As we have pointed out here, there are many disadvantages of having a possibly-unreliable IIA master in the network (solution IIA2). But perhaps we don't really need it.

IIA3 solution is as follows:

• There will be no central server at all.
• There will be only two IIA-related APIs in this solution: IIAs API and IIA CNR API. Both will be serving similar features as they did in IIA1.
• Implementing IIAs API will NOT be required anymore (but still recommended). That is, server will be able to implement other mobility APIs without implementing IIAs API.
• Partners still SHOULD exchange IIAs with each other before the mobility process begins, but EWP will not strictly enforce it. However, partners will still be required to exchange at least IIA IDs.
• IIAs API will serve only a single copy of the agreement (with no history of previous versions). This copy should be the copy which is currently used by the HEI which is serving the API.
• Each partner will be allowed to have a different copy of a single IIA.
• There will be no APIs which would allow other partners to modify IIAs stored on other servers. However, the clients will still be allowed to access all of these different copies, compare them with each other, and display appropriate notices to the user who is viewing them. The clients will also be allowed to fetch versions from other servers and store them to their own server.
• In other words, EWP Network will allow all the partners to peek at each other's copies of their IIAs, but it is for the partners themselves to design the data flow between the partners.

[wrygiel]

@erasmus-without-paper/all-members - Could you please tell us what you think about IIA3 proposal?

• IIA1 - seems to be unacceptable due to its being too central.
• IIA2 - we judge it equally unacceptable due to its being much too complex. And please, don't deny it, until you have read (and fully understood) the proposed specs (note, that it has a few endpoint sub-pages). It is doable, but there's a huge chance that it will simply scare all the partners off...
• IIA3 - it is much easier, distributed, and it seems to provide the minimum subset of what we actually need. It seems the only acceptable solution. If you agree, then we would like to the draft an alternate set of specifications for IIA3 (while deprecating both IIA1 and IIA2).

[georgschermann]

It would remove a bit of functionality but would also remove the most complexity from the network.
I think the network would be more accessible.
IIA3 should be sufficient and is probably the most feasible solution.

A distributed way of creating the agreements could be added at a later stage.

[wrygiel]

New proposal is now published. Older APIs were marked as obsolete and removed from the document index at http://developers.erasmuswithoutpaper.eu/