release: v19.2.0-beta.20190903

[solongordon]

Candidate SHA: e8faca6
Deployment status:
http://solon-release-v1920-alpha20190805-0001.roachprod.crdb.io:26258
http://solon-release-v1920-beta20190903-0001.roachprod.crdb.io:26258
Nightly Suite:
https://teamcity.cockroachdb.com/viewLog.html?buildId=1460988&buildTypeId=Cockroach_Nightlies_NightlySuite

Release process checklist

Prep date: 2019-08-29

• Pick a SHA, fill in Candidate SHA above, notify #release-process of SHA.

• Tag the provisional SHA

• Publish provisional binaries

• Check binaries
  Update to v19.2.0-beta.20190903 bincheck#122
  AppVeyor tests are failing but they appear to just be hanging at the end of the test. All checks pass.

• Deploy to test clusters
  http://solon-release-v1920-alpha20190805-0001.roachprod.crdb.io:26258
  http://solon-release-v1920-beta20190903-0001.roachprod.crdb.io:26258

• Verify node crash reports

• Start release qualification suite
  https://teamcity.cockroachdb.com/viewLog.html?buildId=1460912&buildTypeId=Cockroach_ReleaseQualification

• Start nightly suite
  https://teamcity.cockroachdb.com/viewLog.html?buildId=1460988&buildTypeId=Cockroach_Nightlies_NightlySuite

• fill in Deployment status above with clusters and Nightly Suite with the link to Nightly TeamCity Job

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: WIP fill in date: usually Monday WIP

• Check cluster status

• Tag release

• Bless provisional binaries

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
  • Update the image tag in our Orchestrator configurations
  • Announce version to the registration cluster
  • Upgrade the registration cluster (reg runs latest stable version)

• Update docs

• External communications for release

[solongordon]

Failed roachtest nightlies:

• cancel/tpcc/distsql/w=10,nodes=3 roachtest: cancel/tpcc/distsql/w=10,nodes=3 failed #38417
• hotspotsplits/nodes=4 roachtest: hotspotsplits/nodes=4 failed #33660
• jepsen/monotonic/start-kill-2 Looks like a hardware flake to me. Passed in subsequent build
• kv50/rangelookups/relocate/nodes=8 Passed in subsequent build
• scaledata/jobcoordinator/nodes=6 roachtest: scaledata/jobcoordinator/nodes=6 failed #37595
• tpcc/mixed-headroom/n5cpu16 roachtest: tpcc/mixed-headroom/n5cpu16 failed #39460 Release blocker. See discussion below. Addressed by storage: fix two botched migrations #40415.
• tpccbench/nodes=9/cpu=4/chaos/partition roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #39005
• tpccbench/nodes=9/cpu=4/multi-region roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #40360
• backup2TB/n10cpu4 roachtest: backup2TB/n10cpu4 failed #38794
• clearrange/checks=false roachtest: clearrange/checks=false failed #38772
• clearrange/checks=true roachtest: clearrange/checks=true failed #38720
• hibernate roachtest: hibernate failed #38288
• import/tpch/nodes=4 Per @dt on last week's release: We'll look into it but it isn't blocking: this has been timing out for awhile so this isn't a regression compared to the alpha.
• import/tpch/nodes=8 See previous.
• psycopg roachtest: psycopg failed #39011
• scrub/all-checks/tpcc/w=100 roachtest: scrub/all-checks/tpcc/w=100 failed #39327 AOST-related error. Asked @lucy-zhang to take a look. Addressed in sql: fix SCRUB for indexes with AS OF SYSTEM TIME #40420.
• scrub/index-only/tpcc/w=100 roachtest: scrub/index-only/tpcc/w=100 failed #39328 Same as above.
• typeorm roachtest: typeorm failed #38180 Same failure as in a previous alpha release: release: v19.2.0-alpha.20190701 #38381 (comment)

[tbg]

I'm just starting to look into mixed-headroom for #40370 and feel strongly that we ought to get it to pass reliably before continuing with further releases. This test is our primary long-running mixed-version test (and there isn't a "secondary" - this is really the only one), and it seems to be badly broken in ways that would certainly be felt by anyone upgrading into a 19.2 beta release.
I will document my findings in #39460.

[tbg]

The TL;DR of the analysis over in #39460 is that there are three issues that I'm seeing so far, two in core and which I think I'll have a PR out tomorrow. The third looks like a SQL error, perhaps related to backwards-incompatible workload change? I'm not sure, I asked @jordanlewis to assist.
If we fix that last one, there might be more lurking.

[jordanlewis]

As an update, it looks like the SQL error was potentially user error - and also that Tobi managed to track down the remaining problems.

[thoszhang]

#40420 should fix the two SCRUB failures.