[RfD] Using a testing/beta branch and releases

[ThomasKaiser]

I'm picking up some suggestions by Tido now and have to admit that I don't know whether discussions like that already happened in the past.

But since count of boards and amount of testing has increased a lot in 2016 I think we've to find a solution for some of the problems otherwise it gets too frustrating both for us and our users.

What about a testing/beta branch that feeds the http://beta.armbian.com repo and http://image.armbian.com/betaimages/

And also reducing public releases (maybe adopting a release scheme like Ubuntu's so current release would be 16.10 and next 17.01 or something like that)?

This way we could develop new features without the constant danger of breaking existing installations, could also do a call for testers prior to new releases or after important changes and if we don't get enough feedback for a specific board then postpone/skip one release or maybe even drop support for it.

Of course this will only work if we get an appropriate count of testers on board that feel somewhat responsible for their devices and also know the caveats somehow.

[ThomasKaiser]

BTW: I thought about discussing this in http://forum.armbian.com/index.php/forum/12-armbian-build-framework/ but the 'huge amount' of feedback eg. to Provide an 'Armbian security defaults' document in lib.docs were somewhat discouraging.

But I think we've here also some sort of a chicken-and-egg problem.

[zador-blood-stained]

but the 'huge amount' of feedback eg. to Provide an 'Armbian security defaults' document in lib.docs were somewhat discouraging.

Offtopic Well, end users mostly don't care about this stuff, and I prefer adding info to existing page instead of trying not to break anything by adding new. Just add a new page and I'll try to extend it if I have anything to add.

What about a testing/beta branch that feeds the http://beta.armbian.com repo and http://image.armbian.com/betaimages/

I agree with this, and in addition I would propose something like a review policy on incoming pull requests and a "freeze" period in main branch before the release (no adding new features, only bugfixes).

[golfromeo-fr]

yeap, the project needs to scale up (because Igor's baby has grown up with many boards). So adjustments are needed.
Yes, beta concept or release concepts are welcome. Team Tester as well.
For the best, be sure.

[igorpecovnik]

In last update we had an "urgent kernel fix", but at the same time, our rfc to the system was not done / tested yet. It would be better that we do more smaller upgrades when and if possible. On this one I took a shortcut, a known and calculated risk, and fail.

We usually do fixing before release so perhaps we only need to stick to this rule more strictly, involve some more people to do the testing and give them enough time to adopt.

A slow down, that we are not under this much pressure, would already solve half of problems of this kind. We don't do this first time, but the project has escalated so some adjustments are needed.

No matter how good we will be in this, there still will be problems left. Ubuntu's way of doing also produces problems and they need to push fixes short after release.

[igorpecovnik]

Another idea, since we can't do just the way Ubuntu does. Our case is a bit different. What about if we always do it this way: rebuilding images -> update -> bug fix update -> rebuilding images ... ? And if we define 2 months period between images rebuild and unspecified for the rest?

[ThomasKaiser]

I'm still thinking that a testing/beta branch providing stuff in http://beta.armbian.com repo and http://image.armbian.com/betaimages/ is a requirement.

We can develop in this branch without the risk to break anything on productive systems while users who always want the latest and greatest can also rely on beta images updated from http://beta.armbian.com later (and therefore giving valuable feedback also in case stuff breaks between releases).

And then every n months we merge commits from testing to master, provide new images at http://image.armbian.com/release-test/ (or something like that) do a call for testers and react on feedback while in the same time only fixing bugs in master branch any more.

Adding to these ideas we could add useful tests to the unattended-tests function in firstrun based on experiences/fails.

Some of these tests might require testers willing/able to provide needed prerequisits (like another SBC playing AP for Wi-Fi tests or a BT enabled smartphone that has to be detected by bluetooth test routines and stuff like that) but basic set of tests can and should run unattended on the device itself which makes testing efforts for users simple.

[igorpecovnik]

o.k.

Boot yes/no and boot yes/no after upgrade is top priority. Back in the old days I test all images at least for that, now this: http://forum.armbian.com/index.php/topic/2422-more-proper-testing-better-armbian-experience/?p=18566

The data could be actually be collected automatically withing firstrun / upgrade script. We need just people who would burn and run those test images on various devices. No boot no data :)

[ThomasKaiser]

Useless.

[ThomasKaiser]

Why implementing a stable branch with changes that receive testing before rolling them out if it's so much more fun to brick installations all the time after upgrades...

b680e9a#commitcomment-27203859%E2%80%8B