Merge #104752

104752: upgrades: fix txn retry bug in upgrade batching r=stevendanna a=adityamaru

In #104545 we broke up the txn that is responsible for backfill the `system.job_info` table as part of an upgrade. That diff had a bug where a txn retry inside the `db.Txn` closure could result in us skipping rows to backfill. The consequence of this is that some jobs will not have their payload and progress copied over from the `system.jobs` table to the `system.job_info` table. This is bad because once the cluster is fully upgraded, the job system will **only** consult the `system.job_info` table during execution. When it does so, the job is destined to fail as there will be no payload or progress entry corresponding to that job.

Fixes: #104653
Release note (bug fix): fixes a bug where a txn retry during the backfill of the jobs info table could result in job rows being missed

Co-authored-by: adityamaru <adityamaru@gmail.com>

pkg/upgrade/upgrades/backfill_job_info_table_migration.go

@@ -49,6 +49,7 @@ func backfillJobInfoTable(
 	for step, stmt := range []string{backfillJobInfoPayloadStmt, backfillJobInfoProgressStmt} {
 		var resumeAfter int
 		for batch, done := 0, false; !done; batch++ {
+			var lastAdded int
 			if err := d.DB.KV().Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
 				last, err := d.InternalExecutor.QueryBufferedEx(
 					ctx,
@@ -63,9 +64,8 @@ func backfillJobInfoTable(
 				if err != nil {
 					return errors.Wrap(err, "failed to backfill")
 				}
-				resumeAfter = 0
 				if len(last) == 1 && len(last[0]) == 1 && last[0][0] != tree.DNull {
-					resumeAfter = int(tree.MustBeDInt(last[0][0]))
+					lastAdded = int(tree.MustBeDInt(last[0][0]))
 				} else {
 					done = true
 				}
@@ -77,6 +77,7 @@ func backfillJobInfoTable(
 			if done {
 				break
 			}
+			resumeAfter = lastAdded
 		}
 	}
 	return nil