Configure CI/CD for the whole UI

[joshuawilson]

From @joshuawilson on June 23, 2017 13:43

Currently the UI repos build independently but they need to role up into the components that use them.

Additionally we need to have every push to master update prod automatically for CD.

Copied from original issue: fabric8-ui/fabric8-ui#1591

[joshuawilson]

@jstrachan were you ever able to build the thing that would do this?

[joshuawilson]

From @jstrachan on July 25, 2017 7:41

its on my TODO list once I've finished a few other things... hopefully I'll get to it soon

[jstrachan]

@joshuawilson is the main issue that we need to consolidate version change PRs together?

I guess there's

1. reusing PRs for subsequent releases. e.g. if component A releases 1.0.1 and then 1.0.2 then the latter should cancel the first PR and recreate another one - so that there's only one PR for release A of 1.0.2
2. combining version change PRs into a single PR (e.g. if component A and component B release and C uses both A and B then make sure there's only 1 PR for a change to A and B).

For 2 I wonder if we should always combine; or if sometimes we should keep them separate? e.g. a change to, say, planner could break the tests and a change to login might not - but if we combine them together how do we know what really broke things? I guess if a PR fails we could try split them up again?

Maybe for 2 we could define what dependencies to group together using dependency name includes/exclues? e.g. group all changes in fabric8-ui/* projects together; or list all the permutations of stuff you want to group together; otherwise leave them seperate. e.g. anything angular?

[rawlingsj]

reusing PRs for subsequent releases. e.g. if component A releases 1.0.1 and then 1.0.2 then the latter should cancel the first PR and recreate another one - so that there's only one PR for release A of 1.0.2

Yeah, this currently happens in the pipeline today https://github.com/fabric8io/fabric8-pipeline-library/blob/master/vars/npmUpdateOrganisationDependencies.groovy#L61 so if updatebot did the same thing it shouldn't change any current expected behaviour.

combining version change PRs into a single PR (e.g. if component A and component B release and C uses both A and B then make sure there's only 1 PR for a change to A and B).

This is the main issue we have with the auto version updates provided by our pipelines.

For 2 I wonder if we should always combine; or if sometimes we should keep them separate? e.g. a change to, say, planner could break the tests and a change to login might not - but if we combine them together how do we know what really broke things? I guess if a PR fails we could try split them up again?
Maybe for 2 we could define what dependencies to group together using dependency name includes/exclues? e.g. group all changes in fabric8-ui/* projects together; or list all the permutations of stuff you want to group together; otherwise leave them seperate. e.g. anything angular?

The initial example used was if a change to ngx-base is made we currently update all other repos in the fabric8-ui org with individual PRs containing just the ngx-base version update. But the PR that updates fabric8-ui will always fail because it must include the update to fabric8-planner which in turn has the new ngx-base else there are version mismatches and will cause failure.

This is a diagram from @joshuawilson, ok runtime console has gone but it shows the repo dependencies.

I wondered if we knew that a PR requires multiple package updates if we could open up a branch and commit the version update but only make a PR when the branch contains all required updates? Then when we go to update a downstream repo we check if there's a branch already open for the version update? Not sure this is a good approach or not yet.

[joshuawilson]

You have it mostly figured out. Here is updated diagram.

Here is what I have to do manually. Change ngx-base. Update the version of ngx-base in all the other components but since they are also added I do it in order. Update ngx-login-client and ngx-fabric8-wit. Once those pass, update fabric8-planner and farbric8-stack-analysis. The update to those will be an update of ngx-base and the version of ngx-login-client and ngx-fabric8-wit that now has ngx-base in it. Once all the children are updated I update fabric8-ui. That will include a new version everything that was just changed.

If you just change ngx-widgets then the updates aren't as wide spread but work in a similar fashion following the diagram. Update fabric8-planner and farbric8-stack-analysis. Then update all 3 versions in fabric8-ui.

[joshuawilson]

I just thought of another layer of complexity. What happens when I need to update a package that is used by all the components?

When I update Angular, I still need to start from the bottom and work my way up. But now, Angular has to be updated along with the ngx-base at the next layer up.

[jstrachan]

for most common stuff (e.g. angular) I'm hoping we can configure a single project to be the master so that any change to, say, @angular/core in ngx-base then gets propagated through to all the dependent projects.

I'm about to add a project specific set of includes/excludes patterns so that we can define owners of sets of dependencies at different points in the tree; the simplest possible thing is to let ngx-base take care of most things

[jstrachan]

am still pondering on how best to tackle the fabric8-ui tree problem in updatebot.

I guess the main issue is we want to delay updating, say, fabric8-planner until we have a new release of ngx-base plus all its dependencies which use that dependency too (the other ngx-* libraries) so that a consistent set of dependencies are all using the same ngx-base version. The worry I guess it that if we're waiting for all these releases to go out and one doesn't happen; we may not update fabric8-planner at all.

Are we sure its a huge issue to not be able to upgrade, say, fabric8-planner until all of the ngx-* libraries are updated, released and updated on fabric8-planner? i.e. if we update ngx-base and one of the ngx-* libraries is that gonna break fabric8-planner? I'd hope the CI tests would break though to help us delay the merge until things are working ;)

Am just balancing up complexity versus expedience really.

In terms of simplicity, the easiest thing to do would be something like:

• changes to ngx-base raises a PR on each fabric8-ui project to use the new ngx-base library
• we could maybe optimise by sorted by 'least dependent' library first - e.g. the ngx-* stuff before fabric8-* stuff
• when the ngx-* projects get released and those PRs get generated - reuse the same PR as the ngx-base PR.
  • e.g. for fabric8-planner we'd have 1 PR for updates to ngx-base, ngx-wigdets, ngx-login-client, ngx-fabric8-wit. The PR could be accepted/merged at any point if a user decides.
• we may get smart enough to deduce that a PR for fabric8-planner won't be really complete until, say, 4 commits to 4 versions are included; so rename the PR as [WIP] or something to indicate its not ready for merge yet

how's that sound as a first stab?

[jstrachan]

@joshuawilson btw do you really want to manually curate by hand dependencies in, say, ngx-base or just use the version ranges in ngx-base and have the versions updated automatically? We've the latter in updatebot we're just missing the ability to push the version of dependencies from ngx-base down into downstream projects; but thats fairly easy - adding it now

[joshuawilson]

Ideally, I only want to hand control the library versions in https://github.com/fabric8-ui/fabric8-npm-dependencies.

Ranges on dependencies have bit me before, since this is production I want to keep it locked down. So it only gets updated when I change it.

There are 2 issues with not having the same version in all the projects.

1. there can be conflicts using different libraries
2. even if there are no conflicts you now have at least 2 of the same library in your web app, making it even bigger and slower.

If you change ngx-base and add it to planner but there is a failure in ngx-login-client then it may be weeks before it gets fixed and then you have ngx-base updated in planner but not in sub-components. This could get messy very fast. I really doing that kind of thing for an emergency.

Doing all of manually, I will continue to follow the roll-up path of components to ensure there are no failures and no duplicates of the library.

[jstrachan]

here's an overview of how we're planning on using updatebot to solve this:
fabric8-updatebot/updatebot#4

[jstrachan]

@joshuawilson more detailed docs here: https://github.com/fabric8io/updatebot#pushing-other-dependency-versions

but here's an example of updatebot generating PRs from an 'exemplar' project in this case we're using ngx-base as the canonical set of base versions of stuff
jstrachan-testing/ngx-widgets#13

so that changing, say, the angular version in ngx-base then generates PRs on downstream projects:
jstrachan-testing/ngx-widgets@a3ade93

we then define the versions to push from each repo in the updatebot YAML like this (using includes/excludes):
https://github.com/fabric8io/updatebot/blob/master/updatebot-core/src/test/resources/npm/source/updatebot.yml#L5-L13

[joshuawilson]

I comment on the issue for updatebot, nice idea.

Why not use https://github.com/fabric8-ui/fabric8-npm-dependencies to store/track/update the dependencies?

I don't want to have to define every library I use in a project that is not meant for that. The ngx-base project has a specific task. Or did I misunderstand your intent.

[jstrachan]

use any git repos you like to own the exemplar versions of any sets of dependencies - you just associate the dependency sets that a repo ‘owns’. Feel free to do whatever works best.

I mostly picked ‘ngx-base’ as an example due to its name and it having quite a lot of dependencies ;)

[joshuawilson]

Nice. Thanks.

[sthaha]

@joshuawilson I am closing this issue as it seems to apply to the old GKE jobs. Please reopen if the issue is still valid.