Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IIA version change impact on cooperation conditions hash #109

Closed
mkurzydlowski opened this issue Mar 20, 2023 · 372 comments
Closed

IIA version change impact on cooperation conditions hash #109

mkurzydlowski opened this issue Mar 20, 2023 · 372 comments

Comments

@mkurzydlowski
Copy link
Contributor

As we already know IIA major version change impacts the cooperation conditions hash, as it changes the XML namespace that is part of the cooperation conditions XML element being hashed. So even if a IIA version change doesn't impact the fields in this element, the hash changes.

Let's discuss how this case should be handled and what impact this has.

Should the cooperation conditions hash change after version change or should it be stored and stable until IIA changes?
If we choose to recalculate hashes then should we send CNRs?
If the partner chooses to recalculate hashes, should we recalculate IIA approvals?

@demilatof
Copy link

I think that your question may involve three issues:

  1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
  2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?
  3. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

But I think that we should realize that the hash code is a useless computation.
I have to compute the hash code for the same piece of code you have considered too and check that they are the same.
Bravo! Me and you are able to compute a hash code! And now?
It's useful only as version identifier; let's keep it as it arrives from the partner and all the problems go away.

@umesh-qs
Copy link

I think that your question may involve three issues:

  1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
  2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?

This is not true. Hash will change if the major version changes. Timing of implementation has no impact.

  1. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

Last approved version can be retrieved from local backup as well, if backups are kept. Problem here is, how would such agreements be show to IROs, so that they don't have to spend time on them.

But I think that we should realize that the hash code is a useless computation. I have to compute the hash code for the same piece of code you have considered too and check that they are the same. Bravo! Me and you are able to compute a hash code! And now? It's useful only as version identifier; let's keep it as it arrives from the partner and all the problems go away.

Agree. Hash should be just considered as version id. IIA approval should be reset based on the change in the content of the cooperation condition.

@demilatof
Copy link

I think that your question may involve three issues:

  1. When there is a major version, should we all conform to it and in what amount of time? Or should we be able to use both of them? Until when? Because they can become three versions, four an so on...
  2. If we all switch together to the new versions, there could not be any problem for the IIA even mutually approved; but if we don't?

This is not true. Hash will change if the major version changes. Timing of implementation has no impact.

Yes, hash changes if the major version changes; but if I'm with 7.0 and you still with 6.0, how do we exchange data?
The hash code is not the only problem, obviously, but one of the problems.
Depending on how I have designed my internal system, I may use the saved hash code (6.0) to answer to your request about an old approval or compute it again but with the new version of my API (7.0). Same data, different hash.

  1. As concern the mutually approved versions, I think that we could save the snapshot as a pure XML and return it if we'll have the possibility to ask for the last approved version and not only for the current one. Who receive the XML should read the version that is written in the XML (I think it should be possible...) to elaborate it correctly.

Last approved version can be retrieved from local backup as well, if backups are kept. Problem here is, how would such agreements be show to IROs, so that they don't have to spend time on them.

This is true if we save both of the backups; our version and partner's version.
But if you look at ewp-specs-api-omobilities you can see that there are sending and receiving IIA-IDs.
One of them says (the other is similar): "Clients can fetch more information on this agreement from IIAs API served on the sending HEI's servers"
So, if I develop a omobilities client, I need to be able to fetch data even from the partner's system, not only for local backup

@mkurzydlowski
Copy link
Contributor Author

We would like to propose serving a snapshot of partner's IIA in the IIA Approval API. That way version changes won't affect the approval. IIA version update would change the hash in the IIA get response but this would not affect what has already been approved.

@demilatof
Copy link

I repeat here some considerations I've already made in the other issue because this should be a more appropriate thread to discuss about the consequences of the namespace upgrade.

We would like to propose serving a snapshot of partner's IIA in the IIA Approval API. That way version changes won't affect the approval. IIA version update would change the hash in the IIA get response but this would not affect what has already been approved.

Could you provide an example?
I think that it should be mandatory that the snapshot served with approval API should be saved when Approval CNR was sent.
Nevertheless, this is not the point now.

The question to understand your proposal is: why I have to re-approve the IIA if the hash code changes due to namespace version upgrade?
To be more clear, what are the hash codes you compare?
Because I have to evaluate if your IIA requires a new approval or not, not my IIA!
And with Approval API you provide me my IIA; I already know if my own IIA requires a new approval or not, I don't need your copy of my IIA for this.
As a minimum, you should provide the snapshots of both IIAs.

I think, as you if I remember well, that what could rule the need of an approval is the partner will: if he sends us an IIA CNR, we can suppose that he needs a new approval (may even be he doesn't need it, of course).
The main problem is that we can miss his IIA CNR. To overcome the problem it could be enough adding a new element to the general section of the IIA: <last-cnr-sent-on-date>
Doing so we may implement our logic relying on partner's will, even if we have missed his IIA CNR: if it is after the approval, we have to give it again, otherwise not.

This is not the perfect solution, I know: the partner may send us an IIA CNR even if he doesn't need a new approval.
We may implement some further check, even comparing the new IIA with our snapshot. But we should introduce the mobility-ID to facilitate this.

We should catch the opportunity of the next namespace upgrade to add all the elements may help us for a better management.

@mkurzydlowski
Copy link
Contributor Author

The main problem is that we can miss his IIA CNR. To overcome the problem it could be enough adding a new element to the general section of the IIA:

I think that the modified_since parameter of IIA index should be enough.

@demilatof
Copy link

I think that the modified_since parameter of IIA index should be enough.

Do you mean the modified_since in the request of IIA Index?

@mkurzydlowski
Copy link
Contributor Author

Exactly.

@demilatof
Copy link

Exactly.

I'm trying to make EWP Network more stable asking for something that could fix the problem for missing IIA CNRs, and you tell me that you "think that the modified_since parameter of IIA index should be enough"?
That is, another element that is not sure neither mandatory?
From the specifications:

If given, then the server SHOULD filter the returned IIA IDs to the ones which have been either created or modified after the given point in time

This means that the server SHOULD and not MUST filter; "modified after" in our implementation (and not only in our implementation, I think) means every modification, even those we don't need to communicate immediately to the partner because we want to give to the IRO a small amount of time to avoid mistakes.

Servers MAY include IIAs which were not modified.

So it is not much useful...

Servers MAY ignore the modified_since parameter completely, and always respond with the full list of IIA IDs

Again, if this may happen, the parameter it is useless

As we previously explained here, clients MAY use the index and get endpoints as a pull-based method of synchronization, complementary to CNRs

INDEX and GET are useful for the synchronization, but we cannot substitute the IIA-CNR thanks to this parameter.
Only inserting the information about the IIA-CNR in the IIA-GET response can give us a robust synchronization by means of INDEX- GET, because we receive the same piece of information we would have received with IIA-CNR, that is the exact moment when the partner decided to inform us about an update.

@janinamincer-daszkiewicz
Copy link
Member

We are looking for the simplest possible solution which would allow us to avoid any automatic triggering of re-approval after the change of the name space in IIA (version change from 6.2.0 to 7.0.0) due to changes we plan for the current window. We have some flexibility in choosing the final solution for IIA negotiate/approve/terminate/modify/delete, but the general guidelines are the following:

  • it should be simple, clear, easy to understand and implement;
  • limited number of options on the network level (local system can offer flexibility, but not complicate the interoperability scenarios);
  • CNR is a main tool for triggering exchange of IIA data, not index;
  • CNR can get lost but this should be treated like an exceptional situation, not a general rule;
  • no mercy for errors in the network (this phrase has often been repeated during the technical workshop in Thessaloniki).

I will post here the more elaborated proposal, trying to take into account recent discussion in GitHub and Thessaloniki.

@demilatof
Copy link

which would allow us to avoid any automatic triggering of re-approval after the change of the name space in IIA (version change from 6.2.0 to 7.0.0) due to changes we plan for the current window.

I noticed discussing with @mkurzydlowski that we all agree that presently there is the risk of an automatic triggering of re-approval after the change of the name space in IIA.
But I even notice that due to the different implementations we may have different views of what is the element that triggers the re-approval.
I think that we should specify what is that triggers a new approval and when before looking for a solution good for all.
Can someone share his opinion?

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

If the content is same and only change is in the namespace then it is not complicated to handle it.
Just compare the content. If there is no change, calculate the hash from the content and match it with partner calculated hash.
If it matches then update the hash in you local system without resetting any approval.

@demilatof
Copy link

Just compare the content.

But how? Field by field? And in which order?
It can work, without warranties anyway.
A mobility ID may be useful...

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

Just compare the content.

But how? Field by field? And in which order? It can work, without warranties anyway. A mobility ID may be useful...

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

@demilatof
Copy link

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order).

But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner?
The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)?
Adding extra logic for every change?

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order).

But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner? The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)? Adding extra logic for every change?

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

@janinamincer-daszkiewicz
Copy link
Member

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is out of the question. We have to find a solution where re-approval is not needed.
I think about a timestamp which would allow us to tell if the change in the hash is caused by the business logic change or a technical change (like change in a name space). Only in the first case new action should be triggered.

I will describe this proposal it in more detail later today or tomorrow.

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is out of the question. We have to find a solution where re-approval is not needed. I think about a timestamp which would allow us to tell if the change in the hash is caused by the business logic change or a technical change (like change in a name space). Only in the first case new action should be triggered.

I will describe this proposal it in more detail later today or tomorrow.

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

@janinamincer-daszkiewicz
Copy link
Member

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

It depends on a specific case. The timestamp would allow us to decide when we have real changes and when only 'technical' changes. Name space is such a technical change. No need to approve again IIA which differs from the approved one only in a name space.
Another example: we change the list of academic years to the range <start-academic-year, end-academic-year>. Nothing changes on a business level. Just representation is different.

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

timestamp

Ok. Will wait for the timestamp proposal before further comments

@demilatof
Copy link

I think about a timestamp

so you are saying that even if the hash logic has changed to include new fields, old approvals will remain intact even if the old IIA has new data in it?

A timestamp of what?
Three days ago I already suggested a solution by means of a timestamp <last-cnr-sent-on-date>.

it could be enough adding a new element to the general section of the IIA: <last-cnr-sent-on-date>
Doing so we may implement our logic relying on partner's will, even if we have missed his IIA CNR: if it is after the approval, we have to give it again, otherwise not.

Doing so, if the partner for whatever reason (valid or not) ask us for a new approval, we can give him a new approval, switching to the new namespace version. If there is no request (IIA CNR), we have nothing to do.

@janinamincer-daszkiewicz
Copy link
Member

Timestamp of a last business change in IIA.

@demilatof
Copy link

last business change in IIA.

This may require a shared definition of "last business change in IIA". May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

@janinamincer-daszkiewicz
Copy link
Member

janinamincer-daszkiewicz commented Apr 2, 2023

This may require a shared definition of "last business change in IIA".

Yes, it may be intuitive, but difficult to define in such a way that everybody will understand this the same way. Anyway, we need a method of saying that there are some cases when the change of hash does not mean a (business) change in IIA (only change in its format) and need not trigger an action on the partner side (like re-approval or even comparing the new object with the old object).

May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

We will have (at least) these changes in this round:

  • format of academic years
  • mobilities-per-year required
  • name space

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

This may require a shared definition of "last business change in IIA".

Yes, it may be intuitive, but difficult to define in such a way that everybody will understand this the same way. Anyway, we need a method of saying that there are some cases when the change of hash does not mean a (business) change in IIA (only change in its format) and need not trigger an action on the partner side (like re-approval or even comparing the new object with the old object).

May be that the solution is the set of fields computed in the hash code, but I think we have to investigate on what providers have to modify.

We will have (at least) these changes in this round:

  • format of academic years
  • mobilities-per-year required
  • name space

How is it different when it comes to implementation, when compared with to having different logic for each version (v6 vs v7)?

@janinamincer-daszkiewicz
Copy link
Member

janinamincer-daszkiewicz commented Apr 2, 2023

How is it different when it comes to implementation, when compared with to having different logic for each version (v6 vs v7)?

I am not sure I understand your question. Moving from version 6.2.0 to version 7.0.0 will not change logic of already approved IIAs. These is what we care about. We need a solution that will allow us to automatically recognize between two situations:

  • this mutually approved IIA has been modified, needs negotiations and then approval;
  • this mutually approved IIA has not been modified, only its format changed from 6.2.0 to 7.0.0, no negotiations nor re-approval are needed.

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

Yes I get the problem. My question was, how is the solution you are proposing different from what I have proposed in terms of implementation?

@janinamincer-daszkiewicz
Copy link
Member

Yes I get the problem. My question was, how is the solution you are proposing different from what I have proposed in terms of implementation?

Can you link to this proposal?
I have not yet described my proposal in full. Need some time to polish it.

@umesh-qs
Copy link

umesh-qs commented Apr 2, 2023

You define a object and populate that with the data. Then compare the object. It will work with full warranty :) as long as the sequence is not changed

Maybe it works, even if I had a bad experienced in the past using that system in Java (I've had to implement an algorithm that collects all fields in a given order).
But what are the two versions you try to compare? Your snapshot and the one fetched again from the partner? The new IIA should be downloaded with IIA-Get v. 7; with your solution you can resolve the changes in hash code, but how you do manage different fields due to the new version (even if for the same information)? Adding extra logic for every change?

If the new fields are part of hash calculation then there is a problem. In that case there either we have separate logic for each version or reset the approval.

This is what I have proposed which you just rejected. In any solution you will have to add a special logic in the implementation.

@demilatof
Copy link

P.S. This will work under the condition that the local system keeps the whole set of values in the database, not only the values which are relevant in this particular institution.

There are more problem than the one you consider:

* the v6 ISCED must be a valid ISCED; 071 brings you nowhere and we can have wrong two digit ISCEDs in v6; here your algorithm could fail

071 Engineering and engineering trades - it is OK 0710 Engineering and engineering trades not further defined - is a good 4-digit equivalent

v6 ISCED should be compliant with v6 specification, ie. belong to the dictionary (but may have less digits than 4).

My typo; 077 is a valid value for v6, but doesn't exist.
The fact that the v6 ISCED should be compliant with v6 specification doesn't mean that it is.

Yes, but it should belong to the dictionary, i.e. should be a correct ISCED value. It makes sense to me to give the advice to providers how to pick up a proper value (better then by padding with zeroes which may produce incorrect value)

You could give the advice, but it's outside of the XSLT scope.
And there is no warranty that the users follow your advice, as they didn't follow the advice for v6.

This a useless complication, we are interested only in exchanging the hash code; it's up to the internal system how implementing the substitution strategy. The only requirement is that as soon as they publish v7 they will have to share a four digit code (with a valid value or padding the XML, and not their DB, with leading zeros and adding v6_value attribute).

According to v7 spec they have to send a correct ISCED code, padding with zeroes does not guarantee that.

There is no warranty in any way without a constraint list in the XSD.
The user can use a four digit code without zeros and being invalid anyway.
We should not mix the filling of XML to satisfy v7 spec with the values in the database to exchange IIA.
Here we are discussing about populating v7 to be a valid document for an XSLT transformation, not valid for a new approval.
It's different from populating it to exchange data.

And, in any case, have you decided what we have to do for the approval?
My proposal was to use the XSLT to compare the hash code version 6 and version 7.
Then @mkurzydlowski said, if I'm not wrong, that to check the v6 IIA approval we should have to use the approval v1, and in this way we have less need for a cross XSLT.
But after that you said that we have to decide if having only one version at the time or not.

If we don't know what you choose, we cannot propose any valid strategy.

@janinamincer-daszkiewicz
Copy link
Member

You could give the advice, but it's outside of the XSLT scope.

I agree. Although saying in the specifiation that the value should be a proper dictionary value is not just an advice, the partner should reject the value which is not compliant.

Yes, it may happen that due to an error v6 value may not be from dictionary, also v7 value with 4-digit may not be from dictionary but these are errors which should be rejected by the partners.

Here we are discussing about populating v7 to be a valid document for an XSLT transformation, not valid for a new approval.

We do both, at least this is my goal. Providers may ask, we should not tell them to pad with zeroes.

And, in any case, have you decided what we have to do for the approval?

Michał will answer.

@mkurzydlowski
Copy link
Contributor Author

And, in any case, have you decided what we have to do for the approval? My proposal was to use the XSLT to compare the hash code version 6 and version 7.

That is still needed when both partner's upgrade to version 7 and they want to algorithmically check that the other side responds with a previously approved version.

Then @mkurzydlowski said, if I'm not wrong, that to check the v6 IIA approval we should have to use the approval v1, and in this way we have less need for a cross XSLT.

This is only valid during the upgrade window and enables the side that is not yet upgraded to still read the old-style hash approved.

But we should ask if this is something that is needed. Maybe this compatibility issue that occurs during transition period isn't something we should be tackling? This seems to be a business decision.

@demilatof
Copy link

You could give the advice, but it's outside of the XSLT scope.

I agree. Although saying in the specifiation that the value should be a proper dictionary value is not just an advice, the partner should reject the value which is not compliant.

Yes, it may happen that due to an error v6 value may not be from dictionary, also v7 value with 4-digit may not be from dictionary but these are errors which should be rejected by the partners.

The fact that we are discussing about invalid v6 ISCED codes means that what you think that should happen, in the real world doesn't happen.

Here we are discussing about populating v7 to be a valid document for an XSLT transformation, not valid for a new approval.

We do both, at least this is my goal. Providers may ask, we should not tell them to pad with zeroes.

Of course, but the only answer is that we need a four digit code. They have few options:

  • patching all the codes before switching to V7 (this triggers a new approval request); it's up to them and their IROs choose the right code, because the choice impact an approval
  • Keeping the old value, transmitting it with the v6_value attribute, and padding the old value (only inside the tag) with zeros, just to satisfy the 4 digit code requirement, nothing else. We have no new approval in this case.

Michał will answer.

ok

@demilatof
Copy link

And, in any case, have you decided what we have to do for the approval? My proposal was to use the XSLT to compare the hash code version 6 and version 7.

That is still needed when both partner's upgrade to version 7 and they want to algorithmically check that the other side responds with a previously approved version.

So it's still useless suggesting any algorithm here to patch the real ISCED code, because both of the partners should have to implement the same strategy and anyway we need to compare only the old ISCED value, not the new one.
In any case, to match the new value, they should communicate to the partner the new ISCED value, because there is no warranty that their algorithms bring to the same result.

This is only valid during the upgrade window and enables the side that is not yet upgraded to still read the old-style hash approved.

But we haven't an upgrade window, because it was said that there will be places for only one API version per HEI.
Therefore, if I switch to v7 in January, from that moment I can have only approval v2 and I can exchange IIA only with partners that have implemented v7.

We need a clear address about the road map to v7

@janinamincer-daszkiewicz
Copy link
Member

But we haven't an upgrade window, because it was said that there will be places for only one API version per HEI.

There will be one week for the upgrade.

Therefore, if I switch to v7 in January, from that moment I can have only approval v2 and I can exchange IIA only with partners that have implemented v7.

We need a clear address about the road map to v7

It is ready and will be send to providers by DG EAC (probably on the 1st of August).

@Piargentieri
Copy link

One week to test a new version!! really?
I'm sorry but if you leave a week for the upgrade pheraps you can leave more time.

@demilatof
Copy link

But we haven't an upgrade window, because it was said that there will be places for only one API version per HEI.

There will be one week for the upgrade.

For a week of overlapping, doesn't worth discussing about managing v6 IIA approval.
But I think that it will be not realistic to switch all in a week, without testing anything in the development environment.

We need a clear address about the road map to v7

It is ready and will be send to providers by DG EAC (probably on the 1st of August).

So you already know if we will have one or two versions at the same time in the registry?

@Piargentieri
Copy link

Could someone explain why it's not possible have one or two versions at the same time in the registry? What is the problem?

@janinamincer-daszkiewicz
Copy link
Member

One week to test a new version!! really?

Testing will be done in DEV network and there will be more time for that.
One week window will be for deploying changes in PROD.

@Piargentieri
Copy link

I'm sorry but if I remember correctly You told that the DEV network would have been only for the 6 version .

@demilatof
Copy link

One week to test a new version!! really?

Testing will be done in DEV network and there will be more time for that. One week window will be for deploying changes in PROD.

During the infrastructure forum it was said something different, v6 in DEV, v7 in PROD.
How can we have two versions in DEV, if the choice is to have only one version per registry?
How can we dismiss the support to v6 (that require testing before releasing any patch) if we need to have only v7 for testing?
Anyway, DEV is almost empty

@janinamincer-daszkiewicz
Copy link
Member

Anyway, DEV is almost empty

As to my knowledge each providers has at least one installation in DEV. Most 3rd party providers have more the one. Anybody who wants to be accepted to PROD should first test in DEV, according to 'How to join' procedure.

@demilatof
Copy link

Anyway, DEV is almost empty

As to my knowledge each providers has at least one installation in DEV. Most 3rd party providers have more the one. Anybody who wants to be accepted to PROD should first test in DEV, according to 'How to join' procedure.

For IIAs
565 Institutions in DEV
2820 Institutions in PROD

We make test and exchange data with institutions, not providers.
And what is missing is data.

Do you say I need to create a new fake HEI, e.g. unige2.it and a fork of my my program, to be in the same DEV environment with two different versions?

@janinamincer-daszkiewicz
Copy link
Member

565 Institutions in DEV
2820 Institutions in PROD

The Dashboard has only 3 accounts in DEV, as I remember.

We make test and exchange data with institutions, not providers.
And what is missing is data.

This is why we always encourage to use real but anonymised data, as we (MUCI) do in our installations in DEV.

PROD is not for testing.

@mkurzydlowski
Copy link
Contributor Author

Please review changes to ISCED field description:
not-yet-defined...isced

@mkurzydlowski
Copy link
Contributor Author

I also attach changes to XSLT:
b1ec26b

@skishk
Copy link

skishk commented Jul 28, 2023

565 Institutions in DEV

most of them are not working...

@demilatof
Copy link

PROD is not for testing.

I know, but please explain how do you think to have DEV with only one version that is both 1.6.3 and 1.7.0 at the same time.
Because you said that we have to implement 1.6.3 only in test (and I don't understand why we need it at this point), but in the mean time we need to test 1.7.
And we have anything yet.

@fioravanti-unibo
Copy link

most of them are not working...

many of those that work do not display data, or display 3 or 4 elements, often not formally valid

@janinamincer-daszkiewicz
Copy link
Member

DG EAC should think about uniforming the three lists.

From DG EAC: We have checked and confirm that the field of education list in the BM Data dictionary is aligned with the UNESCO list. It seems to us that providing the link to UNESCO is easier to access and more stable for developers as a reference to enforce the mandatory use of 4 digit ISCED codes.

@demilatof
Copy link

From DG EAC: We have checked and confirm that the field of education list in the BM Data dictionary is aligned with the UNESCO list. It seems to us that providing the link to UNESCO is easier to access and more stable for developers as a reference to enforce the mandatory use of 4 digit ISCED codes.

And I tell you that they should check better, as I did months ago (one by one).
Find the differences, a game for the week-end.
BM Data Dictionary, sheet named EDUCATION_FIELDS
UNESCO list

isced dictionary
Where is ISCED 0200 in the BM dictionary?
And ISCED 0299? And 0300?

I leave you and DG EAC the fun of finding the other eight ISCED codes that are in the UNESCO list and not in the BM Dictionary.

@mkurzydlowski
Copy link
Contributor Author

There is one more thing we need to change in the XSLT. The not-yet-defined attribute must also be used for the recommended-language-skill element as it will be required in StudentMobilitySpecification and StaffMobilitySpecification. But what makes things more complicated is that this change:
stable-v7...rls-restructure

Changes the order of elements and XSLT would not work without further changes...

@mkurzydlowski
Copy link
Contributor Author

I must also add an important disclaimer. After we introduce the XSLT template we run into an important issue. If for some reason we decide to improve the template then we are required to change the API major version. Otherwise people would use different templates. Before the XSLT we would only change the major number if there are incompatibilities in the XML structure.

@mkurzydlowski
Copy link
Contributor Author

This is the updated branch with the not-yet-defined attribute in the recommended-language-skill element:
https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/rls-restructure

@demilatof
Copy link

There is one more thing we need to change in the XSLT. The not-yet-defined attribute must also be used for the recommended-language-skill element

As you have probably seen, the XSLT can work with not-yet-attribute wherever you want without the need to be changed.

Changes the order of elements and XSLT would not work without further changes...

This is a choice you made and that complicate the structure uselessly, therefore is up to you finding the proper solution for the XSLT.
XSD/XML don't require that if an element becomes compulsory, it must be moved at the top of the sequence.

If for some reason we decide to improve the template then we are required to change the API major version. Otherwise people would use different templates. Before the XSLT we would only change the major number if there are incompatibilities in the XML structure.

What do you mean with this disclaimer?
Already with the current hash code algorithm if you introduce a new optional element, the previous hash code changes, as you need a version upgrade if you introduce a new compulsory element.
I don't see any difference from this point of view; beside that, the XSLT is more version proof: you can add a new compulsory element without the need for a version upgrade, because you can use not-yet-defined to take it apart temporally from the hash calculation.

@demilatof
Copy link

This is the updated branch with the not-yet-defined attribute in the recommended-language-skill element: https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/rls-restructure

I thank you for your consideration, I'm happy to have suggested some possible solution to address the problem, but I think we all need to come back to our respective role.

We other developers have little if nothing knowledge of relevant information about the decisions and the future of the version 7, we cannot spend more time than this in finding a solution that the day after needs to be changed because the DG EAC decided something different that we didn't know even that it was discussing about.
The same is for the policy, all internal of your maintainer group, to conform to rules that are all internal, such as the order of the elements compulsory or not, that nothing has to do with XSD/XML specifications.

And even the problem of the versioning prevents us to give a useful contribution that is not a waste of time.
You haven't yet clearly declared:

  • if we have to support more versions at the same time in the same registry or we cannot or we must not
  • if not, how we can test the v7 in DEV and debug v6 in the same environment
  • how we test the hash code between v6 and v7 if we can/cannot have multiple version in the same registry
  • what role play the approval v1, approval v2 and XSLT
  • if approval v1 and approval v2 exist at the same time (they are two different versions) and how long

Therefore, personally, I'll return to examine what specifications we will have to comply in our further developments only when they will be officially released with the description of the solution to switch from a version to another with or without keeping both of them, at the same time, in the same registry.

I hope that the final conclusion obtained in Thessaloniki, the famous strict and simple rules, has not been already abandoned in the name of the flexibility aimed by the DG EAC and by the solutions of writing XSD with recommendation and not formal declaration.
Because this kind of flexibility goes in the opposite way of strict and simple rules, the flexibility represented to us in the last two infrastructure forums nothing has to see with a rule, it's a complete anarchy: do whatever you want.

Therefore, I'm ready to return in a world where the DG EAC keeps on doing the product owner and you implement their requirements, later the BPOs and us (technicians/providers) will examine what will be the final result of your efforts and if it will be compatible with our IROs work from an administrative and technical point of view.

I wish you a good job, see you later, after at least an alpha v7 version will be published, hoping not too late to be ready for the next month of march.

@mkurzydlowski
Copy link
Contributor Author

@demilatof, adding extensions in subtypes like StudentMobilitySpecification makes elements appear at the end of the sequence. We didn't change the order "on purpose".

@mkurzydlowski
Copy link
Contributor Author

I propose using XSD restriction and group (to minimize copying of element sequences) to avoid changing the order of elements:
https://github.com/erasmus-without-paper/ewp-specs-api-iias/compare/rls-v2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests