Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

head_object/get_object/ more: Missing VersionId in response #526

Closed
irq0 opened this issue May 16, 2023 · 1 comment · Fixed by aquarist-labs/ceph#153
Closed

head_object/get_object/ more: Missing VersionId in response #526

irq0 opened this issue May 16, 2023 · 1 comment · Fixed by aquarist-labs/ceph#153
Assignees
Labels
area/rgw-sfs RGW & SFS related kind/bug Something isn't working

Comments

@irq0
Copy link
Member

irq0 commented May 16, 2023

Many responses don't have the VersionID / x-amz-version-id set.

For head_object requests:

head object requests..
are RGWGetObj RGW operations with get_data == false..
create a ReadOp, call prepare, never call iterate()..
retrieve attrs by calling get_attrs on the SAL object
retrieve the version_id from object get_instance()

relevant op code:

  op_ret = read_op->prepare(s->yield, this);
  if (op_ret < 0)
    goto done_err;
  version_id = s->object->get_instance();
  s->obj_size = s->object->get_obj_size();
  attrs = s->object->get_attrs();

Theory: we don't set the instance in the object's cached state (RGWObjState)

S3 Tests failing because of this:

test_object_lock_delete_object_with_retention_and_marker fail
test_object_lock_get_obj_metadata fail
test_object_lock_uploading_obj fail
test_versioned_object_acl_no_version_specified fail
test_versioning_multi_object_delete_with_marker fail
test_versioning_obj_plain_null_version_overwrite fail

@irq0 irq0 added kind/bug Something isn't working area/rgw-sfs RGW & SFS related labels May 16, 2023
@github-project-automation github-project-automation bot moved this to Backlog in S3GW May 16, 2023
@github-actions github-actions bot added the triage/waiting Waiting for triage label May 16, 2023
@0xavi0
Copy link
Contributor

0xavi0 commented Jun 1, 2023

I fixed this one today, working on the versioning stuff... PR coming in the next days.
Some of the tests in the list still fail, but not because of the missing VersionId in the response.
For example:

S3TEST_CONF=s3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_object_lock_delete_object_with_retention_and_marker

        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock

When versioning work is done I'll re-tests those s3 tests and will open new issues for those still not passing.

@0xavi0 0xavi0 self-assigned this Jun 1, 2023
@jhmarina jhmarina removed the triage/waiting Waiting for triage label Jun 6, 2023
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 7, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Jun 8, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
@github-project-automation github-project-automation bot moved this from Backlog to Done in S3GW Jun 9, 2023
0xavi0 referenced this issue in aquarist-labs/ceph Oct 5, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in 0xavi0/ceph Oct 5, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
0xavi0 referenced this issue in aquarist-labs/ceph Oct 18, 2023
Implements new versioning design.

Includes `rgw::sal::Attrs` type binding in `sqliteorm` code, so
VersionedObject code for conversions is no longer needed.

Adds transactions for the actions that require more than 1 database
access.

Simplifies the way a new version is created and moves its logic to a
sqlite transaction to avoid race conditions when creating N objects with
the same name at once from several threads.

Fixes: https://github.com/aquarist-labs/s3gw/issues/378
Fixes: https://github.com/aquarist-labs/s3gw/issues/472
Fixes: https://github.com/aquarist-labs/s3gw/issues/547
Fixes: https://github.com/aquarist-labs/s3gw/issues/526
Fixes: https://github.com/aquarist-labs/s3gw/issues/524
Fixes: https://github.com/aquarist-labs/s3gw/issues/519

Signed-off-by: Xavi Garcia <xavi.garcia@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/rgw-sfs RGW & SFS related kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants