[YSQL] Inflight transactions getting aborted while adding a node to the cluster

[srinivasa-vasu]

Jira Link: DB-880

Description

When we add a new node to the cluster, some leaders get moved to this new node. During this time, some of the inflight transactions in the existing nodes get aborted because of this leader shuffle. This throws a 40001 transaction aborted/expired error message to the client layer. As this is a server-side controlled change that happens during the load balancer movement, failures are expected to be handled transparently.

Error message,

I0224 05:25:45.151476  2113 write_operation.cc:135] Apply operation failed: Operation failed. Try again. (yb/tablet/tablet.cc:1198): Transaction was recently aborted: f2dd5c4c-f3fc-42f8-a7c8-dcdb49492e6c: 40001 (pgsql error 40001) (transaction error 1)
I0224 05:25:45.151877  2113 write_operation.cc:135] Apply operation failed: Operation failed. Try again. (yb/tablet/tablet.cc:1198): Transaction was recently aborted: b38b4a8c-650b-4b1d-bcbe-1ad429584c7e: 40001 (pgsql error 40001) (transaction error 1)
I0224 05:25:45.184355 27227 transaction_coordinator.cc:1225] T f585199208ad4fd3ba96900269caad76 P 699c70f5eca840ddac6fc4bdc6329d5c: Request to unknown transaction 5b8d5a0b-6e8e-4da8-ac9b-38d594f6dba6: transaction_id: "[\215Z\013n\216M\250\254\2338\325\224\366\333\246" status: PENDING
I0224 05:25:45.211586 22189 transaction_coordinator.cc:1225] T b7b1d2d67ccf4541bbfa1d96ffd23e4d P 699c70f5eca840ddac6fc4bdc6329d5c: Request to unknown transaction f863444a-b2af-4e87-8de9-67a623cc77b0: transaction_id: "\370cDJ\262\257N\207\215\351g\246#\314w\260" status: PENDING

[bmatican]

@srinivasa-vasu Do you recall what version this was on? Is this reproducible consistently? This is definitely not expected and the upper layer should retry on leader changes.

[srinivasa-vasu]

@bmatican - We observed this behavior with 2.8/2.9. Need to verify/test again with the latest release.

[shishir2001-yb]

Tried it on 2.19.3.0-b67 version but I didn't hit this issue.
cc: @sushantrmishra