Develop (#354)

* Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * Feature/342 object search (#343) * Develop (#331) * #328: fix copy move (#330) * Develop (#334) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * Develop (#336) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * Bugfix/283 virtual mailbox fetch metadata (#284) * Develop (#282) * 217: fix unused code. * 217: unused code and warning * disabled man pages for rmb * #283: fetch metadata for mails in virtual mailbox * #283: fix metadata date.saved. date.received virtual mailbox * #283: version * Feature/286 use guid from UUID string (#287) * #256: use guid_128_from_uuid_string instead of guid_128_from_string, to support older uuid formats like RECORD or MICROSOFT, always use compact for printing * version 0.0.25 preparations * #286: build issue * merge * version * #286: prefere cached mail guid. * #286: determine if uuid has - hyphon, if true preseve it. * fix non void return * Feature/289 GitHub actions (#290) * #289: build * #289: build plugin. * #289: submodules * submodules * upgrade git * clean outdated repos * Bugfix/UUID record format (#293) * bugfix initialisaction rados_mail->deprecated_uid * rados_mail creation and default value check * #295: update index after full object has been moved (metadata) (#296) * Bugfix/298 295 bugfixes (#300) * #289: fix force-resync (always use INBOX) #295: rbox_mail_set_expunge (play save) * #298: move ceph objects the standard way. * clean up log. * #298: log warning if client connection times out. * Feature/302 retry expunge connection timeout (#303) * #302: retry in case of ceph connection timeout * rpm * #302: max_retry = 10 * max 10 retry, in case object not available, fail with error * #304: use last know uid to restore email flags (#305) * #306: re-assign unreferenced mail objects to inbox (#309) * #310: read osd_max_object_size and fail if mail.size > osd_max_object_size (#311) * version: 0.0.31 * #313: fix imap append crash (#314) * version 0.0.32 * Bugfix/316 mailbox save (#317) * #316: imap append (remove index entry twice in case of error) * #316: changed storage for big attachments. (sync write) * #316: operate api * #316: fix operate call * #316: release preparations * #316: fixed some legacy unit tests * 316: review results * #316: fix * Bugfix/test output stream check (#323) * init output stream and free mailbuffer * build * extracted write chunks * mailsize -1; * fix * debug messages * init stream any case * added deprecation comment, writing chunks now default. new config param rbox_chunk_size default 10240 bytes * missing mock class * version * disabled (temporarily storage tests) * disabled storage tests( api change) * Feature/alternative save methods (#326) * #322: alternative save methods * version * Feature/319 force resync immediatelly assign bugfix/328 segmentation fault copy from virtual mailbox (#321) * #319: automatically add lost objects to inbox (if no mailbox guid exist) * #319: version 0.0.34 * catch error in case mailbox does not have obox header: e.v. Virtual Mailbox * check for virtual mailbox when moving or copying, (virtual mailbox not allowed as origin) * version * #328: fix copy move (#330) * Bugfix/332 quota and move (#333) * 291: failed save mock test * 291: fixed most of the tests * 332 fix notify messages number and type * #332: fix append additional error mssage in case connection can't be opend (#335) * Develop (#341) * Loglevel Debug is sufficient as it gets retried (#337) * Bugfix/339 rados config timeout (#340) * 339 retry 10 times with random wait. retry read for simple read e.g. rados_config. fail with assert in case we can't find rados_config * #342: simple multithreading object search * #342: version 0.0.40 Co-authored-by: Ewald Dieterich <ewald.dieterich@tallence.com> * Bugfix/342 logging multithreading (#344) * #342 fix threading and additional logs * version * Bugfix/rbox copy context (#347) * 346; fix rbox-copy seg fault always init rbox_mail * #346 fix segmentation fault rbox_copy * #364 and code cleanup * Feature/349 ceph object index (#350) * #349: initial storage api * #349 doveadm rmb create ceph index force resync strategy * allow refresh of ceph object index via -r param * #349 save ceph index objects to separate pool * #349 poc impl ceph index done * #349 new config setting rbox_index_pool_name * #349 force reload of all objects, config setting defines if append of oid to ceph index happens * #349 add , to every entry or entry list * #349 add , to every entry or entry list, and test * repair invalid object not debug level log * #349 changed the remove function, initial index creation: append after each mailbox * #349 fix unit tests * #349 version * #349 bugfix return code doveadm rmb (#351) * #349 validate object (#353) Co-authored-by: Ewald Dieterich <ewald.dieterich@tallence.com>
ceph-dovecot · Nov 26, 2022 · 7133b64 · 7133b64
1 parent cfc6dba
commit 7133b64
Show file tree

Hide file tree

Showing 12 changed files with 46 additions and 15 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,8 @@
 # Change Log
+
+## [0.0.46](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.45) (2022-11-22)
+- #349 bugfix doveadm rmb create ceph index validate object metadata
+
 ## [0.0.45](https://github.com/ceph-dovecot/dovecot-ceph-plugin/tree/0.0.45) (2022-11-22)
 - #349 bugfix doveadm rmb return code not set
 

diff --git a/configure.ac b/configure.ac
@@ -9,7 +9,7 @@
 AC_PREREQ([2.59])
 
 
-AC_INIT([dovecot-ceph-plugin], [0.0.45], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
+AC_INIT([dovecot-ceph-plugin], [0.0.46], [https://github.com/ceph-dovecot/dovecot-ceph-plugin/issues/new], ,[https://github.com/ceph-dovecot/dovecot-ceph-plugin])
 
 
 AC_CONFIG_AUX_DIR([.])

diff --git a/rpm/dovecot-ceph-plugin.spec b/rpm/dovecot-ceph-plugin.spec
@@ -14,7 +14,7 @@
 Name:		dovecot-ceph-plugin
 Summary:	Dovecot Ceph RADOS plugins
 
-Version:	0.0.45
+Version:	0.0.46
 
 Release:	0%{?dist}
 URL:		https://github.com/ceph-dovecot/dovecot-ceph-plugin

diff --git a/src/librmb/rados-storage-impl.cpp b/src/librmb/rados-storage-impl.cpp
@@ -576,6 +576,13 @@ void RadosStorageImpl::free_rados_mail(librmb::RadosMail *mail) {
   }
 }
 
+uint64_t RadosStorageImpl::ceph_index_size(){
+  uint64_t psize;
+  time_t pmtime;
+  get_recovery_io_ctx().stat(get_namespace(), &psize, &pmtime);
+  return psize;
+}
+
 int RadosStorageImpl::ceph_index_append(const std::string &oid) {  
   librados::bufferlist bl;
   bl.append(RadosUtils::convert_to_ceph_index(oid));

diff --git a/src/librmb/rados-storage-impl.h b/src/librmb/rados-storage-impl.h
@@ -85,6 +85,7 @@ class RadosStorageImpl : public RadosStorage {
 
   void free_rados_mail(librmb::RadosMail *mail) override;
 
+  uint64_t ceph_index_size() override;
   int ceph_index_append(const std::string &oid)  override;
   int ceph_index_append(const std::set<std::string> &oids)  override;
   int ceph_index_overwrite(const std::set<std::string> &oids)  override;

diff --git a/src/librmb/rados-storage.h b/src/librmb/rados-storage.h
@@ -220,6 +220,10 @@ underTest.ceph_index_add("dkfkjdf")
   */
   virtual int ceph_index_delete() = 0;
 
+  /**
+   * returns the ceph index size
+  */
+  virtual uint64_t ceph_index_size() = 0;
 
   /*! read the complete mail object into bufferlist
    *

diff --git a/src/librmb/tools/rmb/rmb-commands.cpp b/src/librmb/tools/rmb/rmb-commands.cpp
@@ -427,12 +427,21 @@ static void aio_cb(rados_completion_t cb, void *arg) {
 int RmbCommands::overwrite_ceph_object_index(std::set<std::string> &mail_oids){
     return storage->ceph_index_overwrite(mail_oids);
 }
-std::set<std::string> RmbCommands::load_objects(){
+std::set<std::string> RmbCommands::load_objects(librmb::RadosStorageMetadataModule *ms){
   std::set<std::string> mail_list;
-  librados::NObjectIterator iter_guid  =storage->find_mails(nullptr);
+  librados::NObjectIterator iter_guid = storage->find_mails(nullptr);
   while (iter_guid != librados::NObjectIterator::__EndObjectIterator) {
-        mail_list.insert((*iter_guid).get_oid());       
-        iter_guid++;     
+      librmb::RadosMail mail;
+      mail.set_oid((*iter_guid).get_oid());
+
+      int load_metadata_ret = ms->load_metadata(&mail); 
+      if (load_metadata_ret < 0 || !librmb::RadosUtils::validate_metadata(mail.get_metadata())) {    
+         std::cerr << "metadata for object : " << mail.get_oid()->c_str() << " is not valid, skipping object " << std::endl;
+         iter_guid++;     
+         continue;
+      }
+      mail_list.insert((*iter_guid).get_oid());       
+      iter_guid++;     
   } 
   return mail_list;
 }

diff --git a/src/librmb/tools/rmb/rmb-commands.h b/src/librmb/tools/rmb/rmb-commands.h
@@ -66,7 +66,7 @@ class RmbCommands {
   void set_output_path(librmb::CmdLineParser *parser);
 
   int overwrite_ceph_object_index(std::set<std::string> &mail_oids);
-  std::set<std::string> load_objects();
+  std::set<std::string> load_objects(librmb::RadosStorageMetadataModule *ms);
   int remove_ceph_object_index();
   int append_ceph_object_index(const std::set<std::string> &mail_oids);
  private:

diff --git a/src/storage-rbox/doveadm-rbox-plugin.cpp b/src/storage-rbox/doveadm-rbox-plugin.cpp
@@ -805,10 +805,7 @@ static int cmd_rmb_create_ceph_index_run(struct doveadm_mail_cmd_context *_ctx,
     return -1;
   }
   if(rmb_cmds.remove_ceph_object_index() < 0){
-      i_error(" Error overwriting ceph object index");
-      delete ms;
-      _ctx->exit_code = -1;
-      return -1;
+    i_debug(" Error overwriting ceph object index");
   }
 
   if (user->namespaces != NULL) {
@@ -832,7 +829,7 @@ static int cmd_rmb_create_ceph_index_run(struct doveadm_mail_cmd_context *_ctx,
             }
 
             //append to index.
-            i_info("found %d mails in namespace",mail_objects.size());
+            i_info("found %s mails in namespace %d",info->vname, mail_objects.size());
             if(rmb_cmds.append_ceph_object_index(mail_objects) < 0){
                 i_error(" Error overwriting ceph object index");
                 delete ms;
@@ -842,9 +839,10 @@ static int cmd_rmb_create_ceph_index_run(struct doveadm_mail_cmd_context *_ctx,
             mail_objects.clear();
           }
         }
-      } // end of for
+      } // end of for     
+
     }else{
-        mail_objects = rmb_cmds.load_objects();
+        mail_objects = rmb_cmds.load_objects(ms);
         if(rmb_cmds.overwrite_ceph_object_index(mail_objects) < 0){
           i_error(" Error overwriting ceph object index");
           delete ms;
@@ -900,7 +898,6 @@ static int iterate_list_objects(struct mail_namespace* ns, const struct mailbox_
       continue;
     }    
     std::string oid = guid_128_to_string(obox_rec->oid);
-    //TODO: add the oid to the ceph index!!!!
     object_list.insert(oid);
   }
   if (mailbox_search_deinit(&search_ctx) < 0) {

diff --git a/src/storage-rbox/rbox-save.cpp b/src/storage-rbox/rbox-save.cpp
@@ -899,6 +899,11 @@ int rbox_save_finish(struct mail_save_context *_ctx) {
         if( r_storage->config->get_object_search_method() == 2){
           // ceph config schalter an oder aus!
           r_storage->s->ceph_index_append(*r_ctx->rados_mail->get_oid());
+          uint64_t index_size = r_storage->s->ceph_index_size();
+          // WARN if index reaches 80% of max object size
+          if((r_storage->s->get_max_object_size() / index_size) > 80) {
+            i_warning("ceph_index file(%d) close to exceed max_object size(%d), recalc index !",r_storage->s->get_max_object_size(), index_size );
+          }
         }
       }
       if (r_storage->save_log->is_open()) {

diff --git a/src/storage-rbox/rbox-sync.cpp b/src/storage-rbox/rbox-sync.cpp
@@ -367,6 +367,9 @@ int rbox_sync_begin(struct rbox_mailbox *rbox, struct rbox_sync_context **ctx_r,
   rebuild = (hdr->flags & MAIL_INDEX_HDR_FLAG_FSCKD) != 0;
 #endif
 
+  //TODO: check ceph index size, if threshold size has been reached, stat mails
+  //      and repair mailbox
+
   ctx = i_new(struct rbox_sync_context, 1);
   ctx->rbox = rbox;
   i_array_init(&ctx->expunged_items, 32);

diff --git a/src/tests/mocks/mock_test.h b/src/tests/mocks/mock_test.h
@@ -83,6 +83,7 @@ class RadosStorageMock : public RadosStorage {
 
   MOCK_METHOD0(create_anker, int());
 
+  MOCK_METHOD0(ceph_index_size,uint64_t());
 
   MOCK_METHOD1(ceph_index_append,int(const std::string &oid));
   MOCK_METHOD1(ceph_index_append,int(const std::set<std::string> &oids));