Make PDE builds standalone

[akurtakov]

This is an umbrella issue for work needed to be done in order to make PDE release standalone. That would allow the project to rely on LSP for editors (as requested in eclipse-platform/eclipse.platform.releng.aggregator#1476 ), release more often and even simplify the work on releasing Eclipse Platform project.

[akurtakov]

There has already been work done on the topic like #1060 , #898 , eclipse-jdt/eclipse.jdt.core#1918 , eclipse-platform/eclipse.platform#1152 , #826 . These are just few of the things that happened and are provided for overview purposes and doesn't try to be a complete list .
Future work should be linked here.

[akurtakov]

With eclipse-jdt/eclipse.jdt.core#1997 all runtime dependencies (even for tests) should be gone so we should be ready to think/act for the next step here.

[laeubi]

@akurtakov just wondering but is ECLIPSE_HOME provided by PDE? If yes looking at your implementation it seems it could be more generically provided by JDT in the first place?

[akurtakov]

@laeubi It's in PDE and the implementation actually points to target platform thus it can't be moved to JDT

eclipse.pde/ui/org.eclipse.pde.core/src/org/eclipse/pde/internal/core/EclipseHomeInitializer.java

Line 22 in 792018e

public class EclipseHomeInitializer extends ClasspathVariableInitializer {

[laeubi]

Ouch ... this most probabbly never works well and is completely misleading and only working if the eclipse is the running target, even the used method getLocation mentions that this is unreliable, is this actually used anywhere except JDT? If not I think we should just use your implementation in JDT and remove it from PDE alltogether...

It seems if no target is set this is even equal to your implementation...

[akurtakov]

If you want to continue on this one please move it to separate issue/PR to not hijack this one.

[HannesWell]

With eclipse-jdt/eclipse.jdt.core#1997 all runtime dependencies (even for tests) should be gone so we should be ready to think/act for the next step here.

Good work so for from everybody involved.

As already mentioned in eclipse-platform/eclipse.platform.releng.aggregator#1472 (comment), as one of the next steps we should discuss the PDE repository structure and when it is build and if the artifacts of pde are mirrored into the eclipse/updates repo or if the PDE repo is only referenced (to save storage space).

Furthermore we would have to think about the release procedure, when and how to do it. From my experience with M2E the release dates 'automatically' align with the SimRel eventually.

[akurtakov]

So here is a proposal for how to proceed here.

PDE builds and downloads content:

• Master build is done and published to https://download.eclipse.org/pde/builds/master/ (already done)
• Branches (maintenance ones like R_4_31_maintenance ) are build and published to https://download.eclipse.org/pde/builds/BRANCH_NAME
• At release time latest build from BRANCH_NAME (branching happens in RC phases) is copied to https://download.eclipse.org/pde/releases/SDK_VERSION/ (e.g. 4.31)
• Content of PDE p2 repo are only pde bundles and their dependencies that are not already dependencies of bundles in JDT and/or Platform/Equinox

Releng aggregator builds and downloads content:

• I-builds target platform includes https://download.eclipse.org/pde/builds/master/ p2 repo
• PDE bundles are not build by releng aggregator but referred to and reused from PDE p2 repo
• Branches use respective PDE branch p2 repo content
• Produced zip files and p2 repos mirror PDE content. At least initially this would be easiest as if they are not there tests run by nightly I-build will not execute.

Once the above is in good shape we can move on to next step:

• Define and use PDE's own target platform in order to make it possible for it to include LSP4E/TM4E in order to make writing and maintaining editors easier.

[merks]

A "moving target repository" that stomps its contents from day to day or even from minute to minute, breaking downstream builds and consumers, is the very last thing we need to throw into the build mix (that is already hard to keep consistent and afloat).

It's not so difficult to manage an update site properly:

https://eclipse.dev/justj/?page=tools

What's being proposed here looks like a formula for disaster. This is not now a mature project should be managing their update site for consumption downstream.

[akurtakov]

@merks Would you please ellaborate what do you mean to manage properly? PDE is moving target for the I-build right now and this proposal doesn't change that from my POV. Which tool and how will be used would not change the fact that we build latest master of PDE in aggregator I-build and I assume we want to continue that way.

[merks]

I'm going to leave this with @HannesWell because I'm sure he knows what I mean.

This is a moving target as well:

https://download.eclipse.org/eclipse/updates/4.32-I-builds/

But it is a composite and it is "moved" by virtue of updating it to compose a new simple repository as produced by the I-build.

For PDE you are proposing to use a moving target that is a simple repository:

https://download.eclipse.org/pde/builds/master/

The contents here can be (and literally are) deleted or modified at any moment, even in the middle of being used by a downstream build. This is simply asking for trouble.

If the long term plan is for PDE to be like a proper project, it will need to produce milestones and releases. I actually not sure they long term plan has been spelled out. The title here talk about release, but the proposal you outline doesn't even cover that topic.

[akurtakov]

I fully agree that using composite repo would help us provide better repositories.

[HannesWell]

If the long term plan is for PDE to be like a proper project, it will need to produce milestones and releases. I actually not sure they long term plan has been spelled out. The title here talk about release, but the proposal you outline doesn't even cover that topic.

In general PDE will and should stay part of the Eclipse top-level project and will also have the same release cycle as the other projects in the Eclipse-SDK, just like it is now.

Once the above is in good shape we can move on to next step:

* Define  and use PDE's own target platform in order to make it possible for it to include LSP4E/TM4E in order to make writing and maintaining editors easier.

That is to me the main motivation, plus to ensure that in the graph of dependencies PDE remains a leave of the tree, at least with respect to PDE. This was already done in preparation of this and splitting up the build will hopefully ensure in the future that there are no accidental dependencies introduce from Equinox, Platform or JDT to PDE.

So here is a proposal for how to proceed here.

PDE builds and downloads content:

* Master build is done and published to https://download.eclipse.org/pde/builds/master/ (already done)

* Branches (maintenance ones like R_4_31_maintenance ) are build and published to https://download.eclipse.org/pde/builds/BRANCH_NAME

* At release time latest build from BRANCH_NAME (branching happens in RC phases) is copied to https://download.eclipse.org/pde/releases/SDK_VERSION/ (e.g. 4.31)

As already mentioned by Ed with respect to the Eclipse-SDK integration build we should make the master a composite repo.
And I would like to propose the following refined repository structure:

Below the https://download.eclipse.org/pde/SDK_VERSION/ we should have the following sub-repositories

• master, main, latest, integration, nightly, snapshot or I-builds
  a composite that contains the p2-repo produced by the master-branch build and is only retained during the release-cycle (maybe limited to some arbitrary number, maybe 10?).
• milestones
  a composite that contains the M1-M3 and RC1/2 repo and is only retained during the release-cycle
• release
  contains the p2 repo of the corresponding release directly and is retained forever.
• release-patches
  a composite that contains the p2-repo produced by the corresponding maintenance branch build

Alternatively the position of the version-number in the URL could be reversed, so that we have ../pde/I-builds/SDK_VERSION, releases ../pde/releases/SDK_VERSION, ...
AFAICT this is the more common structure and would have the advantage that there could be a latest alias for those repo-types where it is suitable. E.g. ../pde/releases/latests. Alternativly the container could also be a composite that contains all version which makes its use equivalent to latest. But for the latter variant, the resolution is probably slower since all children are queried and not only the really latest.

* Content of PDE p2 repo are only pde bundles and their dependencies that are not already dependencies of bundles in JDT and/or Platform/Equinox

Sounds good.

Releng aggregator builds and downloads content:

* I-builds target platform includes https://download.eclipse.org/pde/builds/master/ p2 repo

* PDE bundles are not build by releng aggregator but referred to and reused from PDE p2 repo

* Branches use respective PDE branch p2 repo content

* Produced zip files and p2 repos mirror PDE content. At least initially this would be easiest as if they are not there tests run by nightly I-build will not execute.

In general this sounds good.
But in order to ensure binary compatibility respectively to get notified about comparator-errors a build of the PDE master against the just created equinox+platform+jdt p2-repo should be started as part of the I-build procedure. If that build fails due to comparator-errors (assuming that no changes/errors in the binaries means binary-compatibility of the parts used by PDE) the entire I-build should fail, similar to how it is now. Or at least the build should be marked as unstable.
But somehow PDE must then be re-build against the new repo (which is difficult if it is marked as unstable or not even published).
But this can be done in parallel, while running the I-build tests.

And speaking about test, I think the PDE tests should also run as part of the I-builds. But AFAICT the I-builds run based on the repo, so it isn't an issue that the PDE bundles are consumed through a repo.

[mickaelistria]

The contents here can be (and literally are) deleted or modified at any moment, even in the middle of being used by a downstream build. This is simply asking for trouble.

A site with movable content is the strategy that is used for snapshots by many other projects (m2e, LSP4E, Wild Web Developer, WindowBuilder...) and there is no a noticeable trouble induced in practice. I imagine that it's just that the probability of this issue happening are relatively low, and that people who use snapshot repos are aware of the risk and just relaunch there build in case they get a related error.
Also, in practice, there are very few project the do depend on PDE, and even less that will require this snapshot site.
So to me, the risk -while not null- seems very low.

[laeubi]

This is simply a technical issue, if one wants a "consucrent safe snapshot site" it can be done in this way (maven actually do it in a similar way):

1. make the snapshot a composite site
2. let each deployed snapshot life in an own folder (e.g. with timestamp)
3. add it to the composite

Then one can remove the eldest entries (e.g. once a day/hour) from the composite and after some delay (e.g after another hour) remove the data folders as well

[merks]

A site with movable content is the strategy that is used for snapshots by many other projects (m2e, LSP4E, Wild Web Developer, WindowBuilder...)

WindowBuilder is a counter example:

https://download.eclipse.org/windowbuilder/updates/

It is one of many:

https://eclipse.dev/justj/?page=tools#p2-manager-examples

Just because one hasn't noticed problems doesn't mean there are no problems that one hasn't noticed. It's also a question of who and what is downstream from such flushable, transient content. Do "we" really want the Platform to have occasional/potential hiccups consuming content that disappears while it is being consumed or shortly after it is consumed when that problem is entirely avoidable?

[laeubi]

In fact "platform" can consume PDE like every other "third party" dependency (e.g. m2e, Orbit, EMF, ECF, ...) if one wants really stable (released only) sites... for me its not a requirement to build PDE in the aggregator with the latest master...

[merks]

Did someone say that someone else wants only stable (released only sites)?

What @HannesWell concretely suggests sounds very good. But then @mickaelistria suggested that movable content is fine, even though that's not what @HannesWell is suggesting. Then you / @laeubi suggested that even maven snapshots don't simply stomp on existing content. Then I pointed out that stomping on content isn't ideal, even one hasn't noticed problems; it's simply not a best practice.

So I'm not sure the direction being taken in this previous comment. Of course PDE should (must) provide content, of different different qualities, that can be consumed by the Platform or by anything other project the same way that other projects (following best practices) do exactly that. That that's exactly what @HannesWell is suggesting. Or are you suggesting PDE doesn't want to produce its own release repository, like normal independent projects do?

In any case, it's hard to figure out how all the subsequent commentary is related to the concrete outline from @HannesWell.

[laeubi]

I just said that maven uses a similar strategy instead of "simply overwrite" existing files. I can't see that this is contrary to anything else you mentioned here. Just the fact that currently PDE only publishes a simple repository does not mean it could be made better, its just something to start with.

So basically what kind of tooling is missing for PDE to produce such "better" snapshot site?

Independent from that, platform might in the future even want to used only released repositories that never change at all like it is done for m2e already what simply contributes the latest release to the simrel, what for me would be the ultimate goal as it mean PDE can release any time with any frequency.

[merks]

It sounds like we pretty close to violent agreement. 😁

[HannesWell]

In fact "platform" can consume PDE like every other "third party" dependency (e.g. m2e, Orbit, EMF, ECF, ...)

Actually "platform" (with which I precisely mean equinox, eclipse-platform and JDT) must not or at least should not consume PDE, I think that's the main goal of this exercise.
Only the I-build must or at least should consume PDE in the sense that PDE should remain part of the eclupse/updates repo and part of the SDK products build as part of the I-builds.
And with that I think the I-build repos should contain the latest PDE I-build while the latest release and milestone should contain the latest PDE release respectively milestone.
If that's done via referencing the repo or via mirroring is something that can be decided when adjusting the I-build to that.

But as said if the I-build consume the latest PDE master we should have a stabilized PDE master-repo implemented with the strategies suggested by Ed and Christoph. The difference of M2E or other projects with ad-hoc updated snapshots to PDE is that M2E (AFAIK) is not consumed by other CI builds regularly. If an update of an IDE has to be restarted that's not so bad, but if an I-build fails just because somebody submitted something to the PDE master at an unfortunate point of time that's very annoying since it means one maybe has to wait or another hour or two until the next I-build completes.
And since, once this is implemented, nothing has to be done I don't see a drawback to implement that. Since I assume we can already rely on existing work to add this for PDE.

(released only) sites... for me its not a requirement to build PDE in the aggregator with the latest master...

We can do that once PDE only uses public API of all other 'platform' projects. But as long as we use internal elements, we should be notified about necessary changes rather sooner than later.
But I also don't see a big problem to trigger a PDE build as part of the I-build. That can run in parallel while the I-builds repo and the SDK product is build. We can maybe even trigger the Jenkins job in the PDE JIPP and the I-builds job just have to await its completion and check the result. Since it can run in parallel the I-build runtime will probably even be reduced compared to today.

Independent from that, platform might in the future even want to used only released repositories that never change at all like it is done for m2e already what simply contributes the latest release to the simrel, what for me would be the ultimate goal as it mean PDE can release any time with any frequency.

At least from my experience with PDE the release cycle more or less aligns to the SimRel anyways. To only advantage I see is that intermediate-releases are possible in order to fix serious bugs. But that could also be done via the maintenance branches and the release-patch repos proposed above.

But regardless of how it is done eventually, I think in any case the next step would be to prepare the stabilization of the PDE 'master' repo and to prepare the milestone/release repo creation process.
Using the https://eclipse.dev/justj/?page=tools#p2-manager-examples as suggested by Ed sounds very good to me.
@merks would you have the time to prepare this for PDE or could you provide some pointers to where it has been done already so we can 'copy' it?
Btw. is it possible to fine-tune the names? For example I wonder if we should use the plural or singular for release/milestone and if we should use an alternative name to nightly (we also produce at day time 😛)? If it is possible I think it's best to discuss the final names in the PR that sets this up.

I assume the only thing not covered by the justj-p2-manager is the handling of the proposed release-patches, is it? But that's probably just a matter of enhancing the Jenkins pipeline to target to another repo for the maintenance branch. That's all feasible without problems. We would just need an agreement about the retention policy in those repos. But I think that's better discussed in a separate issue/PR.

[merks]

I will preface what is to follow with some general general comments and considerations:

• I have a limited capacity for helping on this topic so I cannot allow this to become an ocean-boiling exercise.
• I have limited patience for "my project is special and therefore my projects needs x things that are different from the needs of the 20 other projects that are currently already reusing JustJ's p2 manager". I emphasize this because every configuration option needs to be documented and must maintained forever, so if you insist on trivial things like "the folder name must be releases instead of release", and "I don't like the name nightly", I will most likely lose my patience and will suggest that you build your own solution.
• I have no personal need to solve this problem, so please decide early because I have many other things on which I can spend my time.
• Please read and understand https://eclipse.dev/justj/?page=tools fully and decide early if you can live with what is outlined there.

---

Generally the projects using this infrastructure have builds that are parameterized like this:

I.e., when you do a build, you tell it what type of build so that it's promoted according. The document describes the build type.

Here is how it's defined:

https://github.com/eclipse-orbit/orbit-simrel/blob/main/maven-osgi/MavenOSGi.jenkinsfile

If you don't like this, we're probably quickly already at the stage of "I can't help you".

Here is how a POM to drive promotion is defined:

https://github.com/eclipse-orbit/orbit-simrel/blob/main/maven-osgi/promotion/pom.xml

---

Here is a first rough example of how a generated update site looks like specifically for PDE.

It's kind of an inconsistent mess with poor feature names and poor bundle names. Also, the version numbers are all over the map, so there is no feature that clearly identifies a PDE release version number.

[laeubi]

It's kind of an inconsistent mess with poor feature names and poor bundle names.

Why it became an issue now? I suspect they are not more consitent in current I build either, improving this sounds more like an orthogonal issue for me (at least for me it does not look too bad...)

Also, the version numbers are all over the map, so there is no feature that clearly identifies a PDE release version number.

I can read a big PDE 3.15 on top of the site (not that I think it is the version) but...