Delete - LWT failure retries

[maheshrajamani]

What this PR does:
Delete - Handling LWT failure retries
Which issue(s) this PR fixes:
Fixes #217

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CLA Signed: DataStax CLA

[maheshrajamani]

@ivansenic @tatu-at-datastax I have done IT for this PR. Not sure how to generate LWT failure for IT test case.
Note: The transaction fails with first failure and throws errors. Error handling to return both count and error message will be handled as part of #214

[maheshrajamani]

/gcbrun

[ivansenic]

Some comments, plus missing integration test.. Should be done with multiple update threads and one delete thread.. Or one update and one delete thread, but then the test must run many times to see both can succeed..

[maheshrajamani]

Should be done with multiple update threads and one delete thread.. Or one update and one delete thread, but then the test must run many times to see both can succeed..

@ivansenic None of these guarantee delete retry logic is executed. Can't assert for the use case with above testing.

[tatu-at-datastax]

Should be done with multiple update threads and one delete thread.. Or one update and one delete thread, but then the test must run many times to see both can succeed..

@ivansenic None of these guarantee delete retry logic is executed. Can't assert for the use case with above testing.

We discussed this with Aaron, Mahesh and myself and considered it very difficult if not impossible to have a reproducible reliable test to verify this, as an IT. But that later on it could be tested with load test. Unit test can verify other aspects.

[ivansenic]

I approve, but there are still some issue that should be fixed:

• simplify reactive sequence
• assert JWT failure in the JsonApiException
• etc

[ivansenic]

Should be done with multiple update threads and one delete thread.. Or one update and one delete thread, but then the test must run many times to see both can succeed..

@ivansenic None of these guarantee delete retry logic is executed. Can't assert for the use case with above testing.

We discussed this with Aaron, Mahesh and myself and considered it very difficult if not impossible to have a reproducible reliable test to verify this, as an IT. But that later on it could be tested with load test. Unit test can verify other aspects.

But verify what? I mean it's fairly easy to do an update and delete in parallel. They both have to succeed. Run this N amount of times in a @RepeatableTest, and you get the assurance that retries work. Yes I agree, it's not directly assuring that retry was executed, but if you write this test and apply to the current state, it would fail actually. So if you ask me that's pretty good, way better than not having it.

@maheshrajamani cc

[maheshrajamani]

But verify what? I mean it's fairly easy to do an update and delete in parallel. They both have to succeed. Run this N amount of times in a @RepeatableTest, and you get the assurance that retries work. Yes I agree, it's not directly assuring that retry was executed, but if you write this test and apply to the current state, it would fail actually. So if you ask me that's pretty good, way better than not having it.