docs: new release process to include Status fleets #2825

fryorcraken · 2024-06-20T01:56:22Z

New detailed release process that involves more validation in Status app context.

Some decisions explanation (feel free to debate):

Release isn't finalised until validated with Status
- 2024 is focus on Status, it seems less costly to block release until checked in Status context than doing patch releases.
- Problems introduced in latest release blocked Status QA (nwaku blocked node), we must avoid this in the future.
Which means release process now takes 2 weeks:
- 1st week testing release on Waku fleet
- 2nd week testing release on Status fleet
Added task to create a summary of changes impact on Status users: needed to know what we check in the Status app beyond sanity check.

Also:

not sure how we deploy on the various fleets: do we run jenkins ourselves or open an issue to infra?
Once merged, I think we should delete https://www.notion.so/Release-Process-61234f335b904cd0943a5033ed8f42b4?pvs=4#4e249cfff5d24ea08f7b8087a286f5c6 and point to this template to avoid dupe/conflicting information.

Ivansete-status

Thanks for that ❤️ ! I just added a few comments that I hope you find interesting.
In general, I like the new approach and the details.

IMO, only the release branch should receive cherry-picks from master if, during the release period, we consider an important commit that should be delivered and was merged after the creation of the release branch.

With that, I suggest the following lifetime for any release branch:

create the release branch
update the CHANGELOG.md accordingly in the release branch
cherry-pick any interesting new commit from master to the release branch
once release is validated and deployed, then merge the release branch into master, but without removing the release branch

.github/ISSUE_TEMPLATE/prepare_release.md

gabrielmer

Thanks so much! It looks reaaally good 😍

.github/ISSUE_TEMPLATE/prepare_release.md

gabrielmer · 2024-06-20T16:59:33Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] [deploy release to `waku.sandbox` fleet](https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox)
+  - [ ] announce the release
+
+- [ ] **Promote release to fleets**.
+  - [ ] Deploy final release to `waku.prod`


Suggested change

- [ ] [deploy release to `waku.sandbox` fleet](https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox)

- [ ] announce the release

- [ ] **Promote release to fleets**.

- [ ] Deploy final release to `waku.prod`

- [ ] announce the release

- [ ] **Promote release to fleets**.

- [ ] Deploy final release to `waku.sandbox`

I think it should be like this instead, as waku.sandbox is our "prod" fleet

gabrielmer · 2024-06-20T17:03:40Z

.github/ISSUE_TEMPLATE/prepare_release.md

+
+  - [ ] **On Waku fleets**
+    - [ ] Lock `waku.test` fleet to release candidate version
+    - [ ] Continuously stress `waku.test` fleet for a week (e.g. from `waku.sandbox`)


I think here it should be waku-dev machine instead of waku.sandbox

We sometimes called the waku-dev machine waku-sandbox, so it might generate confusions. But IMO better to stick to the waku-dev naming

I think here it should be waku-dev machine instead of waku.sandbox

We sometimes called the waku-dev machine waku-sandbox, so it might generate confusions. But IMO better to stick to the waku-dev naming

oh yes good point! I agree that the "sandbox" term is confusing (I will stop using it as well to refer to that machine ;P)

We can mention wakudev or more specifically: metal-01.he-eu-hel1.wakudev.misc.

fryorcraken · 2024-06-21T04:36:20Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] [Deploy final release to `waku.sandbox` fleet](https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox)
+  - [ ] Deploy final release to `waku.prod`
+  - [ ] Deploy final release to `status.staging`
+  - [ ] Deploy final release to `status.test` (soon to be `status.prod`)


Should we add CI links for all these steps?
Or do the last steps means pinging infra?

Yes, good point!

Links:

waku.test - https://ci.infra.status.im/job/nim-waku/job/deploy-waku-test/

waku.sandbox - https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox/

shards.staging - https://ci.infra.status.im/job/nim-waku/job/deploy-shards-staging/

shards.test - https://ci.infra.status.im/job/nim-waku/job/deploy-shards-test/

won't nwaku master get deployed automatically to waku.test? isn't it how ti works?

won't nwaku master get deployed automatically to waku.test? isn't it how ti works?

Yes, you are absolutely right. Nevertheless, during the testing time, we disable the automatic deployments and leave the release candidate version running for some days (ideally one week.)

Yes, you are absolutely right. Nevertheless, during the testing time, we disable the automatic deployments and leave the release candidate version running for some days (ideally one week.)

Yes, we have an entry unlock waku.test already, so I assume the deployment will resume. Let me know if it's not clear enough in the new tempalte.

gabrielmer

Thanks so much!

Ivansete-status · 2024-06-21T13:35:29Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] Deploy final release to `waku.prod`
+  - [ ] Deploy final release to `status.staging`
+  - [ ] Deploy final release to `status.test` (soon to be `status.prod`)


We don't have waku.prod.

Suggested change

- [ ] Deploy final release to `waku.prod`

- [ ] Deploy final release to `status.staging`

- [ ] Deploy final release to `status.test` (soon to be `status.prod`)

- [ ] Deploy final release to `shards.staging`

- [ ] Deploy final release to `shards.test` (soon to be `status.prod`)

we don't? What's TWN then? where do you deploy final Waku releases for TWN?

we don't? What's TWN then? where do you deploy final Waku releases for TWN?

Yes, TWN is deployed in waku.sandbox.

Ah btw, I think we are missing to add a point where we update the version pointed by nwaku-compose

Ivansete-status · 2024-06-21T13:37:59Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] [Deploy final release to `waku.sandbox` fleet](https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox)
+  - [ ] Deploy final release to `waku.prod`
+  - [ ] Deploy final release to `status.staging`
+  - [ ] Deploy final release to `status.test` (soon to be `status.prod`)


Yes, good point!

Links:

waku.test - https://ci.infra.status.im/job/nim-waku/job/deploy-waku-test/

waku.sandbox - https://ci.infra.status.im/job/nim-waku/job/deploy-waku-sandbox/

shards.staging - https://ci.infra.status.im/job/nim-waku/job/deploy-shards-staging/

shards.test - https://ci.infra.status.im/job/nim-waku/job/deploy-shards-test/

fryorcraken · 2024-06-24T10:59:01Z

Need to add running status-cli tests against nwaku release candidate.

fryorcraken · 2024-06-26T00:02:56Z

what do we do for mobile? - light client
need to be sure to log results on an issue

fryorcraken · 2024-06-26T00:06:28Z

Also ensure that deploy date does not conflict with ongoing Status/QA tests
contact request might not be needed

fryorcraken · 2024-06-26T00:06:51Z

Clarify who are the actors/owners

fryorcraken · 2024-06-27T06:10:34Z

At which point do we ask Vac/QA to run their test suite against the release candidate?

Ivansete-status · 2024-06-27T06:55:45Z

At which point do we ask Vac/QA to run their test suite against the release candidate?

I think this is something that we can run right after we create the release candidate, and before we have waku.test and shards.staging running for one week.

On the other hand, we need to properly define what we understand by "running VAC/QA test suite against a RC".

Wrt VAC I first of all envision the process in which @AlbertoSoutullo helps us check that the mesh stability is reached reasonably fast.

And re QA I'm envisioning running a Jenkins job. I think is that. In that case, I think that's something we can do ourselves manually if the Jenkins job just runs against the shards.staging.

fryorcraken · 2024-06-27T23:14:24Z

At which point do we ask Vac/QA to run their test suite against the release candidate?

I think this is something that we can run right after we create the release candidate, and before we have waku.test and shards.staging running for one week.

On the other hand, we need to properly define what we understand by "running VAC/QA test suite against a RC".

Wrt VAC I first of all envision the process in which @AlbertoSoutullo helps us check that the mesh stability is reached reasonably fast.

And re QA I'm envisioning running a Jenkins job. I think is that. In that case, I think that's something we can do ourselves manually if the Jenkins job just runs against the shards.staging.

Hi @Ivansete-status, I mean the Vac-QA team. if Vac-DST is also able to run test for mesh stability, that is good.
Ultimately, I'd like to see Vac-DST running performance difference between release to ensure non regression.

fryorcraken · 2024-07-02T06:59:22Z

.github/ISSUE_TEMPLATE/prepare_release.md

+    - [ ] Perform checks based _end user impact_
+    - [ ] Ask other (Waku and Status) CCs to point their instance to `status.staging` for a week and use the app as usual.
+    - [ ] Ask Status-QA to perform sanity checks (as described above) + checks based on _end user impact_
+    - [ ] Ask Status-QA or infra to run the automated Status e2e tests against `status.staging`


We believe this is the test suite: https://ci.infra.status.im/job/status-desktop/job/e2e/job/manual/

unclear how to run it against a specific fleet (shards.staging).

fryorcraken · 2024-07-02T07:02:39Z

.github/ISSUE_TEMPLATE/prepare_release.md

+
+- [ ] Automated testing
+  - [ ] Ensures js-waku tests are green against release candidate
+  - [ ] Ask Vac-QA and Vac-DST to perform available tests against release candidate


cc @fbarbu15 @Zorlin

fryorcraken · 2024-07-02T07:04:36Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] Submit a PR from the release branch to master. Important to commit the PR with "create a merge commit" option.
+
+- [ ] **Promote release to fleets**.
+  - [ ] Update infra config with any deprecated arguments or changed options


I added this to avoid bad surprises.

fryorcraken · 2024-07-05T05:34:23Z

@waku-org/nwaku-developers I think this is ready to merge. Let me know.

NagyZoltanPeter

Looks massive!

Ivansete-status

LGTM! Thanks for it! 💯
I just added a minor comment but this is a great reference.
Testing in waku.test and status.staging could be made concurrently.

Ivansete-status · 2024-07-10T16:56:10Z

.github/ISSUE_TEMPLATE/prepare_release.md

+  - [ ] Create GitHub release
+  - [ ] Deploy the release to DockerHub
+  - [ ] Announce the release
+  - [ ] Submit a PR from the release branch to master. Important to commit the PR with "create a merge commit" option.


That line is mentioned as a post-release step :)

Suggested change

- [ ] Submit a PR from the release branch to master. Important to commit the PR with "create a merge commit" option.

fryorcraken requested a review from a team June 20, 2024 01:56

github-actions bot assigned fryorcraken Jun 20, 2024

fryorcraken changed the title ~~New release process to include Status fleets~~ docs: new release process to include Status fleets Jun 20, 2024

fryorcraken force-pushed the prod-promotion-status branch from 19a5b79 to 9e1d57c Compare June 20, 2024 01:59

Ivansete-status reviewed Jun 20, 2024

View reviewed changes

.github/ISSUE_TEMPLATE/prepare_release.md Outdated Show resolved Hide resolved

.github/ISSUE_TEMPLATE/prepare_release.md Outdated Show resolved Hide resolved

.github/ISSUE_TEMPLATE/prepare_release.md Show resolved Hide resolved

gabrielmer reviewed Jun 20, 2024

View reviewed changes

fryorcraken force-pushed the prod-promotion-status branch from 9e1d57c to 3580432 Compare June 21, 2024 04:35

fryorcraken commented Jun 21, 2024

View reviewed changes

gabrielmer self-requested a review June 21, 2024 07:40

gabrielmer approved these changes Jun 21, 2024

View reviewed changes

Ivansete-status reviewed Jun 21, 2024

View reviewed changes

fryorcraken force-pushed the prod-promotion-status branch from 3580432 to 6c7f994 Compare June 25, 2024 01:40

fryorcraken force-pushed the prod-promotion-status branch 2 times, most recently from 53c568a to ddee9ad Compare July 2, 2024 06:57

fryorcraken commented Jul 2, 2024

View reviewed changes

fryorcraken requested a review from Ivansete-status July 3, 2024 01:36

NagyZoltanPeter approved these changes Jul 5, 2024

View reviewed changes

Ivansete-status approved these changes Jul 10, 2024

View reviewed changes

fryorcraken added 2 commits July 12, 2024 16:44

docs: new release process to include Status fleets

5eec2d7

docs: update release process

0a77c2d

fryorcraken added 6 commits July 12, 2024 16:44

doc: specify release owner, clarify some steps

f55a1e5

some updates on feedback

195818d

Mention automated tests

03b864b

clarify fleet name

971a3bf

clarify what "unlock" means

94c0825

specify that QA wants to know versions to be sure

f70ca06

fryorcraken force-pushed the prod-promotion-status branch from 9198ce1 to f70ca06 Compare July 12, 2024 06:44

remove dupe instruction

dc35baa

fryorcraken merged commit 4264666 into master Jul 12, 2024
8 checks passed

fryorcraken deleted the prod-promotion-status branch July 12, 2024 06:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: new release process to include Status fleets #2825

docs: new release process to include Status fleets #2825

fryorcraken commented Jun 20, 2024 •

edited

Loading

Ivansete-status left a comment

gabrielmer left a comment

gabrielmer Jun 20, 2024

gabrielmer Jun 20, 2024 •

edited

Loading

Ivansete-status Jun 21, 2024

fryorcraken Jun 21, 2024

Ivansete-status Jun 21, 2024

fryorcraken Jun 27, 2024

Ivansete-status Jun 27, 2024

fryorcraken Jul 2, 2024

gabrielmer left a comment

Ivansete-status Jun 21, 2024

fryorcraken Jun 27, 2024

Ivansete-status Jun 27, 2024

Ivansete-status Jun 21, 2024

fryorcraken commented Jun 24, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 27, 2024

Ivansete-status commented Jun 27, 2024

fryorcraken commented Jun 27, 2024

fryorcraken Jul 2, 2024

fryorcraken Jul 2, 2024

fryorcraken Jul 2, 2024

fryorcraken commented Jul 5, 2024

NagyZoltanPeter left a comment

Ivansete-status left a comment

Ivansete-status Jul 10, 2024

docs: new release process to include Status fleets #2825

docs: new release process to include Status fleets #2825

Conversation

fryorcraken commented Jun 20, 2024 • edited Loading

Ivansete-status left a comment

Choose a reason for hiding this comment

gabrielmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrielmer Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrielmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fryorcraken commented Jun 24, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 26, 2024

fryorcraken commented Jun 27, 2024

Ivansete-status commented Jun 27, 2024

fryorcraken commented Jun 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fryorcraken commented Jul 5, 2024

NagyZoltanPeter left a comment

Choose a reason for hiding this comment

Ivansete-status left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fryorcraken commented Jun 20, 2024 •

edited

Loading

gabrielmer Jun 20, 2024 •

edited

Loading