Develop (#364)

* Loglevel Debug is sufficient as it gets retried (#337)

* Bugfix/339 rados config timeout (#340)

* 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config

* Feature/342 object search (#343)

* Develop (#331)

* #328: fix copy move (#330)

* Develop (#334)

* 217: fix unused code.

* 217: unused code and warning

* disabled man pages for rmb

* Bugfix/283 virtual mailbox fetch metadata (#284)

* Develop (#282)

* 217: fix unused code.

* 217: unused code and warning

* disabled man pages for rmb

* #283: fetch metadata for mails in virtual mailbox

* #283: fix metadata date.saved. date.received virtual mailbox

* #283: version

* Feature/286 use guid from UUID string (#287)

* #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing

* version 0.0.25 preparations

* #286: build issue

* merge

* version

* #286: prefere cached mail guid.

* #286: determine if uuid has - hyphon, if true preseve it.

* fix non void return

* Feature/289 GitHub actions (#290)

* #289: build

* #289: build plugin.

* #289: submodules

* submodules

* upgrade git

* clean outdated repos

* Bugfix/UUID record format (#293)

* bugfix initialisaction rados_mail->deprecated_uid

* rados_mail creation and default value check

* #295: update index after full object has been moved (metadata) (#296)

* Bugfix/298 295 bugfixes (#300)

* #289: fix force-resync (always use INBOX)
#295: rbox_mail_set_expunge (play save)

* #298: move ceph objects the standard way.

* clean up log.

* #298: log warning if client connection times out.

* Feature/302 retry expunge connection timeout (#303)

* #302: retry in case of ceph connection timeout

* rpm

* #302: max_retry = 10

* max 10 retry, in case object not available, fail with error

* #304: use last know uid to restore email flags (#305)

* #306: re-assign unreferenced mail objects to inbox (#309)

* #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311)

* version: 0.0.31

* #313: fix imap append crash (#314)

* version 0.0.32

* Bugfix/316 mailbox save (#317)

* #316: imap append (remove index entry twice in case of error)

* #316: changed storage for big attachments. (sync write)

* #316: operate api

* #316:  fix operate call

* #316: release preparations

* #316: fixed some legacy unit tests

* 316: review results

* #316: fix

* Bugfix/test output stream check (#323)

* init output stream and free mailbuffer

* build

* extracted write chunks

* mailsize -1;

* fix

* debug messages

* init stream any case

* added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes

* missing mock class

* version

* disabled (temporarily storage tests)

* disabled storage tests( api change)

* Feature/alternative save methods (#326)

* #322: alternative save methods

* version

* Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321)

* #319: automatically add lost objects to inbox (if no mailbox guid exist)

* #319: version 0.0.34

* catch error in case mailbox does not have obox header: e.v. Virtual Mailbox

* check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin)

* version

* #328: fix copy move (#330)

* Bugfix/332 quota and move (#333)

* 291: failed save mock test

* 291: fixed most of the tests

* 332 fix notify messages number and type

* Develop (#336)

* 217: fix unused code.

* 217: unused code and warning

* disabled man pages for rmb

* Bugfix/283 virtual mailbox fetch metadata (#284)

* Develop (#282)

* 217: fix unused code.

* 217: unused code and warning

* disabled man pages for rmb

* #283: fetch metadata for mails in virtual mailbox

* #283: fix metadata date.saved. date.received virtual mailbox

* #283: version

* Feature/286 use guid from UUID string (#287)

* #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing

* version 0.0.25 preparations

* #286: build issue

* merge

* version

* #286: prefere cached mail guid.

* #286: determine if uuid has - hyphon, if true preseve it.

* fix non void return

* Feature/289 GitHub actions (#290)

* #289: build

* #289: build plugin.

* #289: submodules

* submodules

* upgrade git

* clean outdated repos

* Bugfix/UUID record format (#293)

* bugfix initialisaction rados_mail->deprecated_uid

* rados_mail creation and default value check

* #295: update index after full object has been moved (metadata) (#296)

* Bugfix/298 295 bugfixes (#300)

* #289: fix force-resync (always use INBOX)
#295: rbox_mail_set_expunge (play save)

* #298: move ceph objects the standard way.

* clean up log.

* #298: log warning if client connection times out.

* Feature/302 retry expunge connection timeout (#303)

* #302: retry in case of ceph connection timeout

* rpm

* #302: max_retry = 10

* max 10 retry, in case object not available, fail with error

* #304: use last know uid to restore email flags (#305)

* #306: re-assign unreferenced mail objects to inbox (#309)

* #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311)

* version: 0.0.31

* #313: fix imap append crash (#314)

* version 0.0.32

* Bugfix/316 mailbox save (#317)

* #316: imap append (remove index entry twice in case of error)

* #316: changed storage for big attachments. (sync write)

* #316: operate api

* #316:  fix operate call

* #316: release preparations

* #316: fixed some legacy unit tests

* 316: review results

* #316: fix

* Bugfix/test output stream check (#323)

* init output stream and free mailbuffer

* build

* extracted write chunks

* mailsize -1;

* fix

* debug messages

* init stream any case

* added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes

* missing mock class

* version

* disabled (temporarily storage tests)

* disabled storage tests( api change)

* Feature/alternative save methods (#326)

* #322: alternative save methods

* version

* Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321)

* #319: automatically add lost objects to inbox (if no mailbox guid exist)

* #319: version 0.0.34

* catch error in case mailbox does not have obox header: e.v. Virtual Mailbox

* check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin)

* version

* #328: fix copy move (#330)

* Bugfix/332 quota and move (#333)

* 291: failed save mock test

* 291: fixed most of the tests

* 332 fix notify messages number and type

* #332: fix append additional error mssage in case connection can't be opend (#335)

* Develop (#341)

* Loglevel Debug is sufficient as it gets retried (#337)

* Bugfix/339 rados config timeout (#340)

* 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config

* #342: simple multithreading object search

* #342: version 0.0.40

Co-authored-by: Ewald Dieterich <ewald.dieterich@tallence.com>

* Bugfix/342 logging multithreading (#344)

* #342 fix threading and additional logs

* version

* Bugfix/rbox copy context (#347)

* 346; fix rbox-copy seg fault always init rbox_mail

* #346 fix segmentation fault rbox_copy

* #364 and code cleanup

* Feature/349 ceph object index (#350)

* #349: initial storage api

* #349 doveadm rmb create ceph index force resync strategy

* allow refresh of ceph object index via -r param

* #349 save ceph index objects to separate pool

* #349 poc impl ceph index done

* #349 new config setting rbox_index_pool_name

* #349 force reload of all objects, config setting defines if append of oid to ceph index happens

* #349 add , to every entry or entry list

* #349 add , to every entry or entry list, and test

* repair invalid object not debug level log

* #349 changed the remove function, initial index creation: append after each mailbox

* #349 fix unit tests

* #349 version

* #349 bugfix return code doveadm rmb (#351)

* #349 validate object (#353)

* Bugfix/355 fix gzip trailer for empty stream (#357)

* #355 fix write gzip trailer even the stream is empty.

* #355 minor improvements for quota messaging

* #355 fix gzip trailer for empty stream

* #349 fix threshold calc for ceph index file

* #355 fix buffersize for write method 1 and 2 (#360)

* Bugfix 355 fix buffersize write method (#363)

* #355 fix buffersize for write method 1 and 2

* 355 only use append write, clear read buffer in case of read timeout

* #355 check for gzip and fix invalid trailer

* #355 fix unit tests.

* #355 fix call to operate

* 355 stream size for uncompressed mails

* #355 additional tests

Co-authored-by: Ewald Dieterich <ewald.dieterich@tallence.com>

CHANGELOG.md

@@ -1,7 +1,11 @@
 # Change Log
 ## [0.0.47](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.47) (2022-12-05)
 - #355 fix gzip trailer when stream is empty
+       fix save_method 1+2 buffersize (1 byte short) 
+       bugfix-355-fix-buffersize-write-method
 
+
+## [0.0.46](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.45) (2022-11-22)
 - #349 bugfix doveadm rmb create ceph index validate object metadata
 
 ## [0.0.45](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.45) (2022-11-22)

src/librmb/rados-storage-impl.cpp

@@ -57,7 +57,7 @@ int RadosStorageImpl::split_buffer_and_exec_op(RadosMail *current_object,
   }
 
   int ret_val = 0;
-  uint64_t write_buffer_size = current_object->get_mail_size() -1;
+  uint64_t write_buffer_size = current_object->get_mail_size();
 
   assert(max_write > 0);
 
@@ -147,6 +147,26 @@ int RadosStorageImpl::delete_mail(const std::string &oid) {
   return get_io_ctx().remove(oid);
 }
 
+bool RadosStorageImpl::execute_operation(std::string &oid, librados::ObjectWriteOperation *write_op_xattr) {
+  if (!cluster->is_connected() || !io_ctx_created) {
+    return false;
+  }
+  return get_io_ctx().operate(oid, write_op_xattr) >=0 ? true : false;
+}
+
+bool RadosStorageImpl::append_to_object(std::string &oid, librados::bufferlist &bufferlist, int length) {
+  if (!cluster->is_connected() || !io_ctx_created) {
+    return false;
+  }
+  return get_io_ctx().append(oid, bufferlist, length) >=0 ? true : false;
+}
+int RadosStorageImpl::read_operate(const std::string &oid, librados::ObjectReadOperation *read_operation, librados::bufferlist *bufferlist) {
+if (!cluster->is_connected() || !io_ctx_created) {
+    return -1;
+  }
+  return get_io_ctx().operate(oid, read_operation, bufferlist);
+}
+
 int RadosStorageImpl::aio_operate(librados::IoCtx *io_ctx_, const std::string &oid, librados::AioCompletion *c,
                                   librados::ObjectWriteOperation *op) {
   if (!cluster->is_connected() || !io_ctx_created) {

src/librmb/rados-storage-impl.h

@@ -92,6 +92,10 @@ class RadosStorageImpl : public RadosStorage {
   std::set<std::string> ceph_index_read() override;
   int ceph_index_delete() override;
 
+  bool execute_operation(std::string &oid, librados::ObjectWriteOperation *write_op_xattr) override;
+  bool append_to_object(std::string &oid, librados::bufferlist &bufferlist, int length) override;
+  int read_operate(const std::string &oid, librados::ObjectReadOperation *read_operation, librados::bufferlist *bufferlist) override;
+
  private:
   int create_connection(const std::string &poolname,const std::string &index_pool);
 

src/librmb/rados-storage.h

@@ -232,6 +232,16 @@ underTest.ceph_index_add("dkfkjdf")
    * @return linux errorcode or 0 if successful
    * */
   virtual int read_mail(const std::string &oid, librados::bufferlist *buffer) = 0;
+
+  /*! read the complete mail object into bufferlist
+   *
+   * @param[in] oid unique object identifier
+   * @param[in] read_operation read operation
+   * @param[out] buffer valid ptr to bufferlist.
+   * @return linux errorcode or 0 if successful
+   * */
+  virtual int read_operate(const std::string &oid, librados::ObjectReadOperation *read_operation, librados::bufferlist *bufferlist) = 0;
+
   /*! move a object from the given namespace to the other, updates the metadata given in to_update list
    *
    * @param[in] src_oid unique identifier of source object
@@ -268,6 +278,11 @@ underTest.ceph_index_add("dkfkjdf")
    *
    */
   virtual bool save_mail(librados::ObjectWriteOperation *write_op_xattr, RadosMail *mail) = 0;
+
+  virtual bool execute_operation(std::string &oid, librados::ObjectWriteOperation *write_op_xattr) = 0;
+
+  virtual bool append_to_object(std::string &oid, librados::bufferlist &bufferlist, int length) = 0;
+
   /*! create a new RadosMail
    * create new rados Mail Object.
    *  return pointer to mail object or nullptr

src/storage-rbox/rbox-mail.cpp

@@ -452,13 +452,9 @@ static int read_mail_from_storage(librmb::RadosStorage *rados_storage,
     read_mail->read(0, INT_MAX, rmail->rados_mail->get_mail_buffer(), &read_err);
     read_mail->stat(psize, save_date, &stat_err);
 
-    //TODO: refactore to use operate instead of aio_operate.
-    librados::AioCompletion *completion = librados::Rados::aio_create_completion();
-    int ret = rados_storage->get_io_ctx().aio_operate(*rmail->rados_mail->get_oid(), completion, read_mail,
+    int ret = rados_storage->read_operate(*rmail->rados_mail->get_oid(), read_mail,
                                                   rmail->rados_mail->get_mail_buffer());
-    completion->wait_for_complete_and_cb();
-    ret = completion->get_return_value();
-    completion->release();
+
     delete read_mail;
 
     return ret;
@@ -506,19 +502,6 @@ static int rbox_mail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, s
 
     ret = read_mail_from_storage(rados_storage, rmail,&psize,&save_date);
 
-    /* duplicate code: get_attribute 
-      librados::ObjectReadOperation *read_mail = new librados::ObjectReadOperation();
-      read_mail->read(0, INT_MAX, rmail->rados_mail->get_mail_buffer(), &read_err);
-      read_mail->stat(&psize, &save_date, &stat_err);
-
-      librados::AioCompletion *completion = librados::Rados::aio_create_completion();
-      ret = rados_storage->get_io_ctx().aio_operate(*rmail->rados_mail->get_oid(), completion, read_mail,
-                                                    rmail->rados_mail->get_mail_buffer());
-      completion->wait_for_complete_and_cb();
-      ret = completion->get_return_value();
-      completion->release();
-      delete read_mail;
-    */
     if (ret < 0) {
       if (ret == -ENOENT) {
         // This can happen, if we have more then 2 processes running at the same time.
@@ -533,7 +516,7 @@ static int rbox_mail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, s
         return -1;
       } 
       else if(ret == -ETIMEDOUT) {
-        int max_retry = 10;
+        int max_retry = 10; //TODO FIX 
         for(int i=0;i<max_retry;i++){
           ret = read_mail_from_storage(rados_storage, rmail,&psize,&save_date);
           if(ret >= 0){
@@ -543,6 +526,8 @@ static int rbox_mail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, s
           i_warning("READ TIMEOUT retry(%d) %d reading mail object %s ",i, ret,rmail->rados_mail != NULL ? rmail->rados_mail->to_string(" ").c_str() : " no rados_mail");
           // wait random time before try again!!
           usleep(((rand() % 5) + 1) * 10000);
+          // clear the read buffer in case of timeout
+          rmail->rados_mail->get_mail_buffer()->clear();
         }
 
         if(ret <0){          
@@ -582,11 +567,33 @@ static int rbox_mail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, s
       return -1;
     }
 
+    i_debug("reading stream for oid: %s, phy: %d, buffer: %d", rmail->rados_mail->get_oid()->c_str(),
+                                                               physical_size, 
+                                                               rmail->rados_mail->get_mail_buffer()->length());
+    // validates if object is in zlib format (first 2 byte)
+    bool isGzip = check_is_zlib(rmail->rados_mail->get_mail_buffer());
+    if(isGzip) {
+      uint32_t result = zlib_trailer_msg_length(rmail->rados_mail->get_mail_buffer(),physical_size);
+
+      // get mails real physical size and compare against trailer length
+      uoff_t real_physical_size;
+      rbox_mail_get_physical_size(_mail, &real_physical_size);
+      // in case we have corrupt trailer, 
+      if(result-real_physical_size > zlib_header_length(rmail->rados_mail->get_mail_buffer())) {
+          i_warning("zlib size check failed %d trailer not as expected, fixing by adding 0x00 to msb",(result-real_physical_size));
+          rmail->rados_mail->get_mail_buffer()->append(0x00);
+          physical_size+=1;                 
+      }
+    }
+
     if (get_mail_stream(rmail, rmail->rados_mail->get_mail_buffer(), physical_size, &input) < 0) {
+      i_debug("get mail failed");
       FUNC_END_RET("ret == -1");
       delete rmail->rados_mail->get_mail_buffer();
       return -1;
     }
+
+    i_debug("get mail failed retval of get_mail_stream %d",ret);
 
     data->stream = input;
     index_mail_set_read_buffer_size(_mail, input);
@@ -595,6 +602,82 @@ static int rbox_mail_get_stream(struct mail *_mail, bool get_body ATTR_UNUSED, s
   FUNC_END();
   return ret;
 }
+uint32_t zlib_trailer_msg_length(librados::bufferlist* mail_buffer, int physical_size) {
+    unsigned char gzip_size[] = {
+                        mail_buffer->c_str()[physical_size-1], 
+                        mail_buffer->c_str()[physical_size-2],
+                        mail_buffer->c_str()[physical_size-3],
+                        mail_buffer->c_str()[physical_size-4]
+                        };        
+    uint32_t result = (gzip_size[0] << 24 | gzip_size[1] << 16 | gzip_size[2] << 8 | gzip_size[3]);
+
+    i_debug("length of message(%d) last byte of trailer %d / %d / %d / %d sizeof(char %d), sizeof(unsingned int: %d) ",
+      result,
+      gzip_size[0], gzip_size[1], gzip_size[2], gzip_size[3],
+      sizeof(char), sizeof(unsigned int));
+    return result;
+
+}
+bool check_is_zlib(librados::bufferlist* mail_buffer) {
+
+    unsigned char magic1 = mail_buffer->c_str()[0];
+    unsigned char magic2 = mail_buffer->c_str()[1];
+
+    i_debug("checking for z_lib header magic bytes check %x : %x compared to %x : %x",
+        magic1,magic2,
+        0x1f,0x8b);   
+
+    if(magic1 == 0x1f && magic2 == 0x8b){
+      i_debug("magic bytes 0x1f 0x8b found");
+      return true;
+    }                        
+    i_debug("magic bytes 0x1f 0x8b not found");
+    return false;
+}
+
+int zlib_header_length(librados::bufferlist* mail_buffer) {
+
+    int header_length = 11;    
+    const unsigned char FLG=mail_buffer->c_str()[3];
+
+    switch (FLG){
+        //ETEXT
+        case (unsigned char) 0x01:
+            header_length=10;
+            break;
+        case (unsigned char) 0x02:
+            header_length=10 + 2;
+            break;
+        //FXTERA
+        case (unsigned char) 0x04:
+            header_length=header_extra_size(mail_buffer->c_str());
+            break;
+        case (unsigned char) 0x08:
+            header_length=header_dynamic_size(mail_buffer->c_str());
+            break;
+        case (unsigned char) 0x10:
+            header_length=header_dynamic_size(mail_buffer->c_str());
+            break;
+    default:
+        break;
+    }             
+  i_debug("found header Type %x header size is %d", FLG, header_length);
+
+  return header_length;
+}
+int header_dynamic_size(const unsigned char *data){
+
+    int i=10 - 1;
+    do{
+        i++;
+    }while(data[i]!=(unsigned char) 0 );
+    return i++;
+}
+
+int header_extra_size(const unsigned char *data){
+    int extera_part_size= int(data[10] + data[11]);
+    return 10 + 2 + extera_part_size;
+}
 
 // guid is saved in the obox header, and should be available when rbox_mail does exist. (rbox_get_index_record)
 int rbox_get_guid_metadata(struct rbox_mail *mail, const char **value_r) {

src/storage-rbox/rbox-mail.h

@@ -40,4 +40,13 @@ extern int read_mail_from_storage(librmb::RadosStorage *rados_storage,
                                   struct rbox_mail *rmail,
                                   uint64_t *psize,
                                   time_t *save_date);
+
+
+extern bool check_is_zlib(librados::bufferlist* mail_buffer);
+extern int zlib_header_length(librados::bufferlist* mail_buffer);
+extern uint32_t zlib_trailer_msg_length(librados::bufferlist* mail_buffer, int physical_size);
+
+extern int header_dynamic_size(const unsigned char *data);
+extern int header_extra_size(const unsigned char *data);
+
 #endif  // SRC_STORAGE_RBOX_RBOX_MAIL_H_