PUT operation on unversioned objects not always atomic

[l-mb]

When an object already exists and a PUT operation to the same object name happens, a concurrent GET operation may see a truncated or missing object.

The expected correct behaviour is that the GET sees either the old state or the new state only.

[giubacc]

A thing we must decide before starting to address this is what model we want to implement.
When there is an on-going PUT a GET sees:

• old state ?
• new state ?

The following reasoning assumes that a GET is issued without specifying the object version in the query string.
When a GET has the version object specified, IMO the problem should not exist (but I could be wrong).

To me, make the GET to see the old state makes more sense.
The old state here means:

1. Non-versioned or version suspended buckets: the latest committed version (the null version for suspended-versioned buckets)

2. Versioned buckets: a GET must point a version so I don't think the problem exists here; It could exist when the null version exists because the bucket has firstly been version suspended and then version re-enabled.

[irq0]

We addressed this in the design review https://github.com/aquarist-labs/s3gw/pull/497

[irq0]

S3 Tests

dirty read tests - halt a second write before finish, check if first write is still there
test_atomic_write_8mb
test_atomic_write_4mb
test_atomic_write_1mb

with two writes, see either one or the other
test_atomic_dual_write_8mb
test_atomic_dual_conditional_write_1mb

test_atomic_conditional_write_1mb

test_versioned_concurrent_object_create_and_remove fail
test_versioned_concurrent_object_create_concurrent_remove fail