How to tell if caller covers the receiving Hei of this mobility?

[GeniusJonathan]

Hi,

In https://github.com/erasmus-without-paper/ewp-specs-api-omobility-las/blob/stable-v1/endpoints/get.md in the permission section states:

• If the caller covers the receiving HEI of this mobility, then he MUST be allowed access to its learning agreement.

As the server how do we know if caller covers the receiving Hei of the mobility? When calling the /get endpoint the caller only passes the parameters sending_hei_id and the mobility_id's. So how do you know who the caller/requester is?

Can somebody please enlighten me? Thanks in advance

[sascoms]

Hi,

Unfortunately, many APIs do not have sending or receiving (both in terms of mobility and/or in terms of HTTP request) HEI IDs in API parameters.

You need to check the caller's signature or header data and find which HEI is the caller.
or
You can fetch the LA and check the partners section inside and find the recipient HEI. (applicable to iias-get and las-get)

Yet, this cause many troubles and also many work to be done internally on software.

We have opened issues related to this problem in the past for many APIs. Explained the problems in details.
And proposed adding sending/receiving HEI IDs to the params.

Unfortunately we have not seen any improvements on these issues. They are pending.

I do not know how the proposals are accepted or implemented by API maintainers.

Maybe, there must be (i have not seen any yet) a procedure document that describes how proposals and also new api versions are accepted and by whom and how they are implemented/updated in github.

Hope this issue becomes a motivation for all of us and such problems are soon fixed and EWP APIs become more efficient.

[GeniusJonathan]

Thank you for your answer @sascoms .

It's still unclear how to achieve the above requirement.

Scenario: When Host A covers institutions A and institution B and Institution A tries to get a LA from institution B, which he has no permission to get.

You need to check the caller's signature or header data and find which HEI is the caller.
Both the signature of institution A and institution B will have the same signature because they are within the same host. So you still cannot tell which HeiID the call is initiated from. And i don't see any header data to extract the heiID from.

You can fetch the LA and check the partners section inside and find the recipient HEI. (applicable to iias-get and las-get)
How can you tell if recipient Hei is the one who initiated the get call?

@janinamincer-daszkiewicz Can you maybe elaborate on on this?

[janinamincer-daszkiewicz]

See also:

• Adding receiving_hei_id parameter to request parameters and making it required ewp-specs-api-omobility-cnr#2
• Missing sending_hei_id parameters ewp-specs-api-imobilities#2
• Adding CNR receiver hei_id to the request parameters and making it required ewp-specs-api-iia-cnr#1
  We plan to add missing sending_hei_id/receiving_hei_id to the approriate APIs after this proposal is approved by the community (service providers) and we find a fitting time window for backward incompatible changes to the specifications. June 2021 has not been a good time for such changes.

I hope to delivered more concrete information soon.

[sascoms]

@janinamincer-daszkiewicz
As it seems nearly all providers and HEIs are still in testing phase, we need to make this decision in a very short time.

In my opinion, it would be more difficult to make changes after the testing period ends.

PS: I will create a new issue related to the decision and approval procedures for proposals.

[sascoms]

You can fetch the LA and check the partners section inside and find the recipient HEI. (applicable to iias-get and las-get)
How can you tell if recipient Hei is the one who initiated the get call?

@GeniusJonathan Can you please add a full scenario from the beginning to this step? It will easier to understand the problematic points in a full scenario.

[mkurzydlowski]

@GeniusJonathan, in EWP client credentials are tied to an EWP host, that may represent multiple HEIs. So in general you need to check if any of the HEIs represented by the client are "linked" to the data being asked for.

Isn't this enough in this case? Do you need to know which particular HEI is linked to the data?

[CACIdev]

@mkurzydlowski So if we understand it correctly, the permission for omobillity:

If the caller covers the receiving HEI of this mobility, then he MUST be allowed access to its learning agreement.

should be interpreted as:

If the calling host covers the receiving HEI of this mobility, then he MUST be allowed access to its learning agreement

So when host H1 covers institution A and B and institution A requests information from institution C (covered by host H2) then is must be allowed acces to a learning agreement "linked" to institution B?

[mkurzydlowski]

You are right that from EWP point of view it's hosts that are calling methods, so when authorizing a request you have to take into account that a host can represent multiple HEIs and can ask for data linked to any of the hosts it covers.

[janinamincer-daszkiewicz]

From version v5.1.0 of the https://github.com/erasmus-without-paper/ewp-specs-api-discovery no extra parameters to identify sender/receiver are needed as "A unique key is enough to identify the sender/caller, and a unique service URL (API) to identify the recipient/callee".

[umesh-qs]

I have raised this question in email communication earlier as well, but wasn't answered. Let me try this with larger forum.

The solution proposed will require the service provider to purchase and maintain CA signed certificate for each of their clients. This involves significant cost for service provide with lot of clients. Why is this need hasn't been clarified yet.

I am wondering how will Dashboard accommodate this cost and future maintenance.

[janinamincer-daszkiewicz]

1. If HEI uses an HTTP signature, a key is needed but it can be generated automatically.
2. If HEI uses TLS, self-signed certificates can be generated automatically. The problem is that they are not accepted by everyone. Anyway, TLS should be deprecated, and eventually will. Everybody is adviced to use HTTP signature.

[sascoms]

... and a unique service URL (API) to identify the recipient/callee".

If not misunderstood, does that mean each HEI covered by the provider will be obliged to have a different and individual API endpoint?

[janinamincer-daszkiewicz]

Yes

[umesh-qs]

1. If HEI uses an HTTP signature, a key is needed but it can be generated automatically.
2. If HEI uses TLS, self-signed certificates can be generated automatically. The problem is that they are not accepted by everyone. Anyway, TLS should be deprecated, and eventually will. Everybody is adviced to use HTTP signature.

Does it mean that the solution proposed will only be implemented when TLS auth is deprecated? When is it going to be deprecated?

[janinamincer-daszkiewicz]

How many nodes in the network use only TLS?

[umesh-qs]

Not sure if this question is for the entire forum or MoveOn. But what is the context?

[janinamincer-daszkiewicz]

Entire network. The context is if TLS is a real problem.

[umesh-qs]

So you mean if only a few service providers use TLS, it is not a real problem?

[janinamincer-daszkiewicz]

We will ask them to switch to HTTP signature which is a better solution anyway.

[umesh-qs]

Well, then why not set a date to make TLS obsolete. What is the point of asking?
And why asking now. I have raised this in October 2021. This could have been closed by now.

[sascoms]

We still have doubts about the solutions proposed.

We do not think these are feasible and good solutions to solve the detection of caller or the callee.

Don't the providers still have to generate certificates for each HEI?

And why generating a different URL for each API endpoint?

A screenshot from the catalogue is below.

@georg, @umesh-qs, dashboard-developers:

Are you willing to take this solution and create countless number of different API endpoints for each HEI?

Is not this putting more work and making it more complex instead of adding what parameters are needed to the API specs?

Am I missing some point?

1. If HEI uses an HTTP signature, a key is needed but it can be generated automatically.
2. If HEI uses TLS, self-signed certificates can be generated automatically. The problem is that they are not accepted by everyone. Anyway, TLS should be deprecated, and eventually will. Everybody is adviced to use HTTP signature.

[umesh-qs]

I completely agree with @sascoms. A simple solution has been unnecessarily complicated. We are in support of new parameter in API call and not identify based on certificates.
But only if EWP Consortium is willing to listen.

I am not sure about others, we already have raised some objections to the new proposed registry changes in Oct-2021 itself via email, but in vain.

[umesh-qs]

@janinamincer-daszkiewicz can you share how, when and who made this decision on this issue?

[kkaraogl]

These changes, inculding the "a unique service URL (API) to identify the recipient/callee", have been presented in providers' meetings, I can't recall, though, which specific meeting this was mentioned the very first time. We' ve started preparations to accommodate them, some time ago, Dashboard-side.

[umesh-qs]

@kkaraogl .. looks like you have budget now to start on unapproved changes also :). Question is not about when you started. It is about who approved it and when. We raised the objection in Oct-2021. Was it discussed and approved explicitly after that? MoveOn did not approve it till now.

[janinamincer-daszkiewicz]

@kkaraogl , for the first time it has been discussed during the tech workshop back in October 2021, a long time ago.

@umesh-qs , these changes have been approved and the upgraded specifications are already publicly available. See new tags in:
https://github.com/erasmus-without-paper/ewp-specs-architecture
https://github.com/erasmus-without-paper/ewp-specs-api-discovery

[janinamincer-daszkiewicz]

@sascoms , you generate these endpoints automatically anyway. You can add the SCHAC at the end and it will be unique.
I do agree that developers who support many HEIs in one node will have some work to do.
Changing parameters of the APIs will mean extra work for everybody.

[umesh-qs]

@kkaraogl , for the first time it has been discussed during the tech workshop back in October 2021, a long time ago.

@umesh-qs , these changes have been approved and the upgraded specifications are already publicly available. See new tags in: https://github.com/erasmus-without-paper/ewp-specs-architecture https://github.com/erasmus-without-paper/ewp-specs-api-discovery

@janinamincer-daszkiewicz who approved the changes?

[umesh-qs]

@sascoms , you generate these endpoints automatically anyway. You can add the SCHAC at the end and it will be unique. I do agree that developers who support many HEIs in one node will have some work to do. Changing parameters of the APIs will mean extra work for everybody.

I don't agree. Changing the API parameters is a cleaner and simplest solution. It will take way less time compare to the current solution. Everybody will anyway have to make changes now also.

[janinamincer-daszkiewicz]

At the meeting the change proposals were shown, nobody objected, nobody sent comments after the meeting, this is regarded as approval by the community.

[umesh-qs]

@janinamincer-daszkiewicz you are misleading. Please check your emails in Oct-2021. We have raised objections.

Also please see the rules for API changes
https://user-images.githubusercontent.com/53461949/159250793-045cc459-6d35-49ab-9c56-b0bc3393e981.png

[sascoms]

At the meeting the change proposals were shown, nobody objected, nobody sent comments after the meeting, this is regared as approval by the community.

And unfortunately, this shows there is no procedure or principles or a written document on how the decisions are taken and new versions/changes are decided/accepted.

See: erasmus-without-paper/general-issues#36

And also this also means approval by the community to all proposals made here and was not given any comments/rejection?

At the meeting the change proposals were shown, nobody objected, nobody sent comments after the meeting, this is regared as approval by the community.

And it is not true that no objections made to the proposal document you presented in the meeting:
See Umesh's comments: https://docs.google.com/document/d/1-4kkn_dTqWnOpt-rAKjPf0GH9c5SYZCF/edit

[umesh-qs]

@sascoms Thanks for putting this here. I wasn't aware that we have this document discussion going on. But this is only part of conversation I have in the email. After this there was a further discussion on email and they had no answers and stopped responding. In-fact the very thought of new registry is an overhead for everyone. It hardly resolved any issues as claimed in Oct 2021 call. Not sure why EWP Consortium is bulldozing it without proper discussion. May be they have to show some development to EC.

[umesh-qs]

Access to "https://docs.google.com/document/d/1-4kkn_dTqWnOpt-rAKjPf0GH9c5SYZCF/edit" has been revoked all of a sudden.

[umesh-qs]

At the meeting the change proposals were shown, nobody objected, nobody sent comments after the meeting, this is regared as approval by the community.

Some months back I was given answer as being "disingenuous" and "interest lies elsewhere" on raising question about the API changes approval process. How about this answer?

[muratyuceer]

It will cause big changes in our infrastructure :(

[jpbacelar]

hi umesh-qs et all: happy to share a primer on EDSSI and EWP governance at our next Infrastructure Forum meeting - it's well within your right to ask about it.

conversely this is not the first time QS pushes back on proposals that were well received by the majority of the participants in the technical stakeholders meetings - the document Eser linked to shows clearly that only 1 party raised concerns, so this begs the question of whether you are ready to accept and respect decisions that you don't personally agree with.

I think we set a positive example of how this must work in practice when we dropped the XML signature suggestion further to our joint discussion, so really quite keen to clarify these matters on the 6th with you all.

[umesh-qs]

Hi Joao,
I would love to hear this from majority of participations as you mentioned. I don't see yet anyone apart from Dashboard who has responded in agreement to this. And I am happy that you at least accepted that concerns were indeed raised by QS and only QS. But your answer gives me a feeling that those were deliberately ignored.

I would like to correct you on some of your statements. We (QS and not my personal opinion) have supported whatever makes sense (can give positive examples if you wish to) and we (QS and not my personal opinion) have opposed and will oppose whatever doesn't. Looking forward to the call on 6th. I hope I get the invitation.

For your information, mentioning again the rules for API changes that are not defined by me personally.
https://github.com/erasmus-without-paper/ewp-specs-management/tree/v1.2.1

[jpbacelar]

Thanks Umesh. It stands to reason that you/QS will support what makes the most sense, but that was not my question: what I asked was whether you/QS are ready to accept and respect decisions made by the majority of the stakeholders? Consensus is ideal but not always possible, so it's important to know what can be expected in such cases.

As for the upcoming meetings QS has been invited to them since the end of the EWP project(s) several years ago, so I assure you there's no reason why the invitation would not arrive. As explained just a few weeks ago, further to the start of EWP+ we are looking to make these exchanges more rather than less regular. Experience shows more joint discussions bode well for building shared understandings, so QS's participation in the Infrastructure Forum is valuable to us - along with all other 3PP providers and HEIs with own systems, evidently.

[umesh-qs]

Great. Majority rule for major changes will make sense. But that should only be after concerns/suggestions raised by anyone, reaches everyone, for them to understand advantages/disadvantages. Discussion on the concerns/suggestions should not be restricted to EWP Consortia. A link to take a vote will ease the process. I would suggest lets start voting process with this issue itself.

Invitation should clarify the agenda. Often the invitation does not come to our tech team directly because it is assumed that some meeting are not for technical discussion. When I say agenda, it should say if there will be any API related discussion or not.

[jpbacelar]

Good to hear, thanks Umesh. Can't comment on who receives what internally at QS but other points will help feed upcoming discussions.

[georgschermann]

@umesh-qs the document seems accessible again
I see the points for the proposed changes and the ones against it.
We will most likely be quite affected by the changes ourself, less because of the manifest changes but more because of our caching machanism beeing rather sensitive to URL changes, which will happen for nearly all URLs then.

I don't see much problems with the points raised in the document:

1. Most service providers already have measures in place that technically prevent duplicates from beeing published (registry check prior to update).
2. Will cause a some extra effort, but again if a provider already handles dozens to hundreds of HEIs, there is probably some automation to certificate / key-pair creation and manifest generation in place.
3. This can be automatically checked by the registry as it is also done now (if the user doesn't point to a different correct manifest).
4. Definitely a proper transition period is needed, but since this only affects the registry I think this can be easily accomplished, as soon as a provider is ready, he changes the manifests and the registry-imported result should stay the same and after time X all manifests which haven't been changed are discarded.

The changes are far from beeing "well received" by us, but we see potential for them reducing long term organizational overhead in exchange for some (hopefully) short term technical issues/effort.
I am neither a friend of changeing a running system nor manually writing emails to add / change / verify / resolve registry entries. And we are already 3-4 major reworks of our implementation/infrastructure into this project so there is that :)

From a technical point of view - generating/persisting the key-pairs/certificates, changing the manifest generation algorithm, adding the hei as a path-parameter to the endpint-urls (and probably ignoring them since we currently don't have any problem identifying sending/receiving heis) is in our case little effort compared to some previous and upcoming changes.

@sascoms

We do not think these are feasible and good solutions to solve the detection of caller or the callee.
Don't the providers still have to generate certificates for each HEI?
And why generating a different URL for each API endpoint?
@georg, @umesh-qs, dashboard-developers:
Are you willing to take this solution and create countless number of different API endpoints for each HEI?
Is not this putting more work and making it more complex instead of adding what parameters are needed to the API specs?

We currently don't have any issues identifying sending/receiving HEIs I am aware of and we check both on all APIs, I doubt that this is the main reason for these changes, but rather the organizational overhead of managing the registry entries/permissions.

As said generating the additional (currently around 6000-7000) URLs on our manifest is minimal effort and path parameters are the same effort as query paramters in our case so we don't have a strong opinion against it, probably a timout adjustment on the registry side will be needed since the manifest size will increase significantly with the added urls and key pairs (we will probably remove the certificates).

looking forward to the next meetings / discussions

[umesh-qs]

I think we need to identify the objective of the new registry solution. Sadly, it is still not clear to us.

1. If the purpose of this new solution is to handle duplicates, then it is not serving the purpose. What is the point of having so many parties make investment in a solution that is not full proof? Why not stick to the current solution where there are hardly any duplicates. As of now I don’t see any duplicates.
2. If the purpose of the solution is the find the caller HEI ID, having an additional parameter serves the purpose. For us and many other providers, this is a cleaner, faster and maintenance free solution. Why invent a complicated and costly solution of unique url (QS is fine but other service provider might have issues) and unique CA certificates.
3. If the purpose of the solution is to reduce maintenance of the registry by the EWP registry team, then lets only focus on that.

Please add to it if there is any other purpose that was in mind when building this solution.

[sascoms]

Just to note again clearly:
We are not in favor of these proposed changes and have objections as stated above.

It is not only the QS. And it seems more providers have concerns about these changes/proposal.

And also we fully agree on @umesh-qs's concerns in the document mentioned and in his comments above.

All this thread and the comments and the discussion shows clearly that there should be a procedure and written document accepted by all that defines the acceptance rules for APIs or similar changes and such proposals. What is a consensus? If voting, how? etc. Even mini open source projects have such procedures and documents. We mentioned this in one of the issues we have opened. This is important because such discussions cause loss of trust in the project management.

And, we hope we will be invited to the March meeting.

[janinamincer-daszkiewicz]

The next meeting will take place 2022-04-06 at 10 am, as usual on the first Wednesday of the month.

[umesh-qs]

Just to note again clearly: We are not in favor of these proposed changes and have objections as stated above.

It is not only the QS. And it seems more providers have concerns about these changes/proposal.

And also we fully agree on @umesh-qs's concerns in the document mentioned and in his comments above.

All this thread and the comments and the discussion shows clearly that there should be a procedure and written document accepted by all that defines the acceptance rules for APIs or similar changes and such proposals. What is a consensus? If voting, how? etc. Even mini open source projects have such procedures and documents. We mentioned this in one of the issues we have opened. This is important because such discussions cause loss of trust in the project management.

And, we hope we will be invited to the March meeting.

Joao, I hope that the proposed registry changes will be put on hold until there is a larger discussion and majority approval

[umesh-qs]

Hi @georgschermann thanks for your inputs.
As you said there are both plus and minus points for the newly proposed registry service. But I am really not able come up with any 100% working advantage, other then giving control to individual HEIs (that may also change once we get more details on how the solution will be executed). It will help us and other providers help make a decisions if you can list down the advantages of the new solution vs current registry.

[janinamincer-daszkiewicz]

The old discussion worth reminding: erasmus-without-paper/ewp-specs-api-echo#3

[umesh-qs]

@janinamincer-daszkiewicz what is the worth in reminding something that is discussed in 2016 when initial architecture was being discussed. Also I can see from the discussion that idea of unique certificate for each host was dropped and also the idea for single host per institution.

Instead of this, since you proposed the changes pretty recently, I believe it will be on top of your mind, to quickly list down the advantages of the new solution vs existing registry even though the request originally was for @georgschermann

[umesh-qs]

@jpbacelar since there is no objection to what I have mentioned, should we not automatically assume that it is correct and hold these changes before people start implementing and then they would have to revert it?

[jpbacelar]

No - we use GitHub for reporting issues, not for voting.

Meanwhile we have read your statements with interest - including the ones you removed, urging me to rejoin the discussion (so here I am) - and there are some obvious problems. Let’s start here:

1. You asked how decision making functions in EWP
2. I offered to take up the issue at our next meeting
3. Then you proceed to declare (and then delete) that we fail to follow due process, meaning you make your own assumptions about what those processes are.

Decision making process aside, you repeated several times that not enough discussion has taken place. Allow me to challenge that claim:

1. On the 19th of October we invited all providers (with an EWP MoU) to a meeting where Janina presented the specification changes in detail and where all participants were invited to put forward their feedback.
2. Further to that QS raised questions by email which we did follow up on (contrary to your claims). I believe you were the only party to do so.
3. On the 10th of February we again invited all providers to a meeting where participants were reminded about the planned changes and an account of the progress was shared.
4. On the 2nd of March we had another meeting where the final spec adjustments were presented. Colleagues were invited to flag possible critical issues within the week - all of the presented information was also available on our repositories, as is usually the case, and it’s important to note that at this point we are no longer discussing in which direction to take the changes but ascertaining they are implemented in a manner that everyone endorses. We did receive feedback from other providers, which we also engaged with (obviously).
5. At the March meeting mentioned above we also discussed the calendar for deploying the new API versions. The feedback we received entailed some delays but we proceeded as planned and started upgrading version numbers on March 11th.

It’s not serious to argue that after more than five months, three meetings, hundreds of emails and several rounds of feedback not enough room for discussion was afforded. Instead if feels as though the real issue is that you/MoveOn feel EWP cannot move forward unless MoveOn is in agreement – hence my question of whether MoveOn is ready to respect the opinion of the majority of the community, which objectively takes no issue with the proposed and discussed changes.

Some colleagues have already started implementing the changes we discussed together, and I would urge everyone in the community to do the same, to make sure deadlines are met and the implementations meet the needs and expectations of your users. This is it from our side on non-technical topics until we meet again.

[umesh-qs]

@jpbacelar
You are acting so naive here. There are many API changes discussed and suggestions sought in GitHub. It is not for only reporting issues.

I have not deleted any comments. I might have edited some of them for more clarity. But that was done immediately. You might have felt that it was deleted and out of excitement jumped the gun.
Please be assured, I always stick by what I write or say.

I am not sure if you are not able to get the reasoning or you are being led to not reason by people close to you. In both the case I believe your limited technical expertise is to blame. And hence I believe you or your team do not want to debate on technical aspects.
So here I challenge you again for a technical debate on this.

As per MoveOn, process is not followed. Had there been any process, there would have been at least some people coming out in support if not "majority”. Not sure if there is any process. Please see erasmus-without-paper/general-issues#36
And see the rules of accepting new API changes here https://github.com/erasmus-without-paper/ewp-specs-management/tree/v1.2.1 which is not followed.

Now coming to your challenge (I like challenges), on "not enough discussion has taken place", let me correct your narrative.

1. In meeting on 19th, QS asked what the benefits of new solution are. Only answer given by Janina, was that it will solve all the issues that are there in existing registry. No further details on how.
2. After the meeting QS raised objections via email. There were email exchanges on the objections. After Oct-29th you (your team) stopped responding without being able to justify the proposed changes. Can share the email here or directly to you if you wish.
3. Then the same change was proposed on Feb-10. I saw the recording. No discussion on this or on the points raised by QS. Only mentioned that Janina will implement it. Full stop.
   Here again Janina mentioned that not a single duplicate will be there after the new registry portal. So, what, she said this so many times before. But at least once, technically explain how?

What were you and your team were doing for 4 months? Why no one responded or discussed with QS, the issues raised? Why these issues pointed by QS, not presented before larger audience?

On this Feb-10 meeting, Girish from QS attended it. He is not a technical person. And there was no mention of any technical discussion in the agenda for the meeting. The call was for giving status updates. I believed you assumed that it was for technical discussion but forgot to mention in the agenda.

4. Meeting on March-02. QS was not invited. Janina is already aware. So, your 2nd March narrative flops, at least from QS perspective. Couldn’t find any recording to see if any points mentioned by QS came up for discussion in this meeting.

There have been cases in past where changes have been reverted even after implementation by almost everyone. And there will be cases in future as well when there is a realization that what was decided at one point doesn't make sense now. So, your argument that, some colleagues have started doesn't cut the ice, if the changes are not worth. And hence I mentioned to put it on hold until it is discussed.

The problem is that there are people in your team who don’t like to be proved wrong. Sooner you fix this better it is for you and this project.

[janinamincer-daszkiewicz]

list down the advantages of the new solution vs existing registry

It will be one of the topics for the incoming tech meeting on 2020-04-06.