According to https://github.com/facebook/rocksdb/wiki/Statistics stats are dumped after compaction. This file shows samples of the stats before and after the beginning and end of the performance degradation. Throughout the logfile, the following 2 messages are printed. [WARN] [default] Stalling writes because we have 21 level-0 files [WARN] [default] Stopping writes because we have 24 level-0 files I did some visual (inexact) analysis. Before the performance issue, there are around 60 stalls for every 1 stop. During the performance issue that ratio decreases to about 11 to 1. ###################################################################################### ################## Config ######################### ###################################################################################### 2016/05/20-05:11:08.937380 7f9b3313f700 [WARN] RocksDB version: 3.13.1 2016/05/20-05:11:08.937528 7f9b3313f700 [WARN] Git sha rocksdb_build_git_sha: 2016/05/20-05:11:08.937533 7f9b3313f700 [WARN] Compile date Oct 14 2015 2016/05/20-05:11:08.937539 7f9b3313f700 [WARN] DB SUMMARY 2016/05/20-05:11:08.937603 7f9b3313f700 [WARN] SST files in //Partition_50 dir, Total Num: 0, files: 2016/05/20-05:11:08.937608 7f9b3313f700 [WARN] Write Ahead Log file in //Partition_50: 2016/05/20-05:11:08.937620 7f9b3313f700 [WARN] Options.error_if_exists: 0 2016/05/20-05:11:08.937622 7f9b3313f700 [WARN] Options.create_if_missing: 1 2016/05/20-05:11:08.937625 7f9b3313f700 [WARN] Options.paranoid_checks: 1 2016/05/20-05:11:08.937627 7f9b3313f700 [WARN] Options.env: 0x7f9afb74cba0 2016/05/20-05:11:08.937629 7f9b3313f700 [WARN] Options.info_log: 0x7f9b2dae93b0 2016/05/20-05:11:08.937631 7f9b3313f700 [WARN] Options.max_open_files: 5000 2016/05/20-05:11:08.937633 7f9b3313f700 [WARN] Options.max_total_wal_size: 0 2016/05/20-05:11:08.937635 7f9b3313f700 [WARN] Options.disableDataSync: 0 2016/05/20-05:11:08.937637 7f9b3313f700 [WARN] Options.use_fsync: 0 2016/05/20-05:11:08.937639 7f9b3313f700 [WARN] Options.max_log_file_size: 0 2016/05/20-05:11:08.937641 7f9b3313f700 [WARN] Options.max_manifest_file_size: 18446744073709551615 2016/05/20-05:11:08.937643 7f9b3313f700 [WARN] Options.log_file_time_to_roll: 0 2016/05/20-05:11:08.937645 7f9b3313f700 [WARN] Options.keep_log_file_num: 1000 2016/05/20-05:11:08.937648 7f9b3313f700 [WARN] Options.allow_os_buffer: 1 2016/05/20-05:11:08.937650 7f9b3313f700 [WARN] Options.allow_mmap_reads: 0 2016/05/20-05:11:08.937652 7f9b3313f700 [WARN] Options.allow_mmap_writes: 0 2016/05/20-05:11:08.937654 7f9b3313f700 [WARN] Options.create_missing_column_families: 0 2016/05/20-05:11:08.937656 7f9b3313f700 [WARN] Options.db_log_dir: 2016/05/20-05:11:08.937658 7f9b3313f700 [WARN] Options.wal_dir: //Partition_50 2016/05/20-05:11:08.937660 7f9b3313f700 [WARN] Options.table_cache_numshardbits: 4 2016/05/20-05:11:08.937662 7f9b3313f700 [WARN] Options.delete_obsolete_files_period_micros: 21600000000 2016/05/20-05:11:08.937664 7f9b3313f700 [WARN] Options.max_background_compactions: 1 2016/05/20-05:11:08.937666 7f9b3313f700 [WARN] Options.max_background_flushes: 1 2016/05/20-05:11:08.937668 7f9b3313f700 [WARN] Options.WAL_ttl_seconds: 0 2016/05/20-05:11:08.937670 7f9b3313f700 [WARN] Options.WAL_size_limit_MB: 0 2016/05/20-05:11:08.937672 7f9b3313f700 [WARN] Options.manifest_preallocation_size: 4194304 2016/05/20-05:11:08.937675 7f9b3313f700 [WARN] Options.allow_os_buffer: 1 2016/05/20-05:11:08.937676 7f9b3313f700 [WARN] Options.allow_mmap_reads: 0 2016/05/20-05:11:08.937678 7f9b3313f700 [WARN] Options.allow_mmap_writes: 0 2016/05/20-05:11:08.937680 7f9b3313f700 [WARN] Options.is_fd_close_on_exec: 1 2016/05/20-05:11:08.937682 7f9b3313f700 [WARN] Options.stats_dump_period_sec: 600 2016/05/20-05:11:08.937684 7f9b3313f700 [WARN] Options.advise_random_on_open: 1 2016/05/20-05:11:08.937686 7f9b3313f700 [WARN] Options.db_write_buffer_size: 0d 2016/05/20-05:11:08.937688 7f9b3313f700 [WARN] Options.access_hint_on_compaction_start: NORMAL 2016/05/20-05:11:08.937691 7f9b3313f700 [WARN] Options.use_adaptive_mutex: 0 2016/05/20-05:11:08.937692 7f9b3313f700 [WARN] Options.rate_limiter: (nil) 2016/05/20-05:11:08.937709 7f9b3313f700 [WARN] Options.delete_scheduler.rate_bytes_per_sec: 0 2016/05/20-05:11:08.937712 7f9b3313f700 [WARN] Options.bytes_per_sync: 0 2016/05/20-05:11:08.937714 7f9b3313f700 [WARN] Options.wal_bytes_per_sync: 0 2016/05/20-05:11:08.937716 7f9b3313f700 [WARN] Options.enable_thread_tracking: 0 2016/05/20-05:11:08.937718 7f9b3313f700 [WARN] Options.row_cache: None 2016/05/20-05:11:08.937722 7f9b3313f700 Compression algorithms supported: 2016/05/20-05:11:08.937726 7f9b3313f700 Snappy supported: 1 2016/05/20-05:11:08.937728 7f9b3313f700 Zlib supported: 1 2016/05/20-05:11:08.937730 7f9b3313f700 Bzip supported: 1 2016/05/20-05:11:08.937732 7f9b3313f700 LZ4 supported: 1 2016/05/20-05:11:08.937735 7f9b3313f700 Fast CRC32 supported: 0 2016/05/20-05:11:08.937854 7f9b3313f700 Creating manifest 1 2016/05/20-05:11:09.369600 7f9b3313f700 Recovering from manifest file: MANIFEST-000001 2016/05/20-05:11:09.369682 7f9b3313f700 --------------- Options for column family [default]: 2016/05/20-05:11:09.369689 7f9b3313f700 [WARN] Options.comparator: rocksdb.InternalKeyComparator:leveldb.BytewiseComparator 2016/05/20-05:11:09.369691 7f9b3313f700 [WARN] Options.merge_operator: None 2016/05/20-05:11:09.369692 7f9b3313f700 [WARN] Options.compaction_filter: None 2016/05/20-05:11:09.369695 7f9b3313f700 [WARN] Options.compaction_filter_factory: TtlCompactionFilterFactory 2016/05/20-05:11:09.369697 7f9b3313f700 [WARN] Options.memtable_factory: SkipListFactory 2016/05/20-05:11:09.369698 7f9b3313f700 [WARN] Options.table_factory: BlockBasedTable 2016/05/20-05:11:09.369707 7f9b3313f700 [WARN] table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f9b2cf5e350) cache_index_and_filter_blocks: 0 index_type: 0 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x7f9b2cf5e778 block_cache_size: 104857600 block_cache_compressed: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 filter_policy: nullptr format_version: 0 2016/05/20-05:11:09.369709 7f9b3313f700 [WARN] Options.write_buffer_size: 33554432 2016/05/20-05:11:09.369710 7f9b3313f700 [WARN] Options.max_write_buffer_number: 3 2016/05/20-05:11:09.369712 7f9b3313f700 [WARN] Options.compression: Snappy 2016/05/20-05:11:09.369713 7f9b3313f700 [WARN] Options.prefix_extractor: nullptr 2016/05/20-05:11:09.369714 7f9b3313f700 [WARN] Options.num_levels: 7 2016/05/20-05:11:09.369715 7f9b3313f700 [WARN] Options.min_write_buffer_number_to_merge: 1 2016/05/20-05:11:09.369716 7f9b3313f700 [WARN] Options.max_write_buffer_number_to_maintain: 0 2016/05/20-05:11:09.369717 7f9b3313f700 [WARN] Options.compression_opts.window_bits: -14 2016/05/20-05:11:09.369718 7f9b3313f700 [WARN] Options.compression_opts.level: -1 2016/05/20-05:11:09.369719 7f9b3313f700 [WARN] Options.compression_opts.strategy: 0 2016/05/20-05:11:09.369721 7f9b3313f700 [WARN] Options.level0_file_num_compaction_trigger: 4 2016/05/20-05:11:09.369722 7f9b3313f700 [WARN] Options.level0_slowdown_writes_trigger: 20 2016/05/20-05:11:09.369723 7f9b3313f700 [WARN] Options.level0_stop_writes_trigger: 24 2016/05/20-05:11:09.369724 7f9b3313f700 [WARN] Options.target_file_size_base: 2097152 2016/05/20-05:11:09.369725 7f9b3313f700 [WARN] Options.target_file_size_multiplier: 1 2016/05/20-05:11:09.369726 7f9b3313f700 [WARN] Options.max_bytes_for_level_base: 10485760 2016/05/20-05:11:09.369727 7f9b3313f700 [WARN] Options.level_compaction_dynamic_level_bytes: 0 2016/05/20-05:11:09.369728 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier: 10 2016/05/20-05:11:09.369729 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[0]: 1 2016/05/20-05:11:09.369731 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[1]: 1 2016/05/20-05:11:09.369732 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[2]: 1 2016/05/20-05:11:09.369733 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[3]: 1 2016/05/20-05:11:09.369734 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[4]: 1 2016/05/20-05:11:09.369735 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[5]: 1 2016/05/20-05:11:09.369736 7f9b3313f700 [WARN] Options.max_bytes_for_level_multiplier_addtl[6]: 1 2016/05/20-05:11:09.369737 7f9b3313f700 [WARN] Options.max_sequential_skip_in_iterations: 8 2016/05/20-05:11:09.369738 7f9b3313f700 [WARN] Options.expanded_compaction_factor: 25 2016/05/20-05:11:09.369739 7f9b3313f700 [WARN] Options.source_compaction_factor: 1 2016/05/20-05:11:09.369740 7f9b3313f700 [WARN] Options.max_grandparent_overlap_factor: 10 2016/05/20-05:11:09.369766 7f9b3313f700 [WARN] Options.arena_block_size: 3355443 2016/05/20-05:11:09.369768 7f9b3313f700 [WARN] Options.soft_rate_limit: 0.00 2016/05/20-05:11:09.369773 7f9b3313f700 [WARN] Options.hard_rate_limit: 0.00 2016/05/20-05:11:09.369774 7f9b3313f700 [WARN] Options.rate_limit_delay_max_milliseconds: 1000 2016/05/20-05:11:09.369775 7f9b3313f700 [WARN] Options.disable_auto_compactions: 0 2016/05/20-05:11:09.369776 7f9b3313f700 [WARN] Options.filter_deletes: 0 2016/05/20-05:11:09.369777 7f9b3313f700 [WARN] Options.verify_checksums_in_compaction: 1 2016/05/20-05:11:09.369778 7f9b3313f700 [WARN] Options.compaction_style: 1 2016/05/20-05:11:09.369779 7f9b3313f700 [WARN] Options.compaction_options_universal.size_ratio: 1 2016/05/20-05:11:09.369780 7f9b3313f700 [WARN] Options.compaction_options_universal.min_merge_width: 2 2016/05/20-05:11:09.369781 7f9b3313f700 [WARN] Options.compaction_options_universal.max_merge_width: 4294967295 2016/05/20-05:11:09.369783 7f9b3313f700 [WARN] Options.compaction_options_universal.max_size_amplification_percent: 200 2016/05/20-05:11:09.369784 7f9b3313f700 [WARN] Options.compaction_options_universal.compression_size_percent: -1 2016/05/20-05:11:09.369785 7f9b3313f700 [WARN] Options.compaction_options_fifo.max_table_files_size: 1073741824 2016/05/20-05:11:09.369786 7f9b3313f700 [WARN] Options.table_properties_collectors: 2016/05/20-05:11:09.369787 7f9b3313f700 [WARN] Options.inplace_update_support: 0 2016/05/20-05:11:09.369788 7f9b3313f700 [WARN] Options.inplace_update_num_locks: 10000 2016/05/20-05:11:09.369789 7f9b3313f700 [WARN] Options.min_partial_merge_operands: 2 2016/05/20-05:11:09.369790 7f9b3313f700 [WARN] Options.memtable_prefix_bloom_bits: 0 2016/05/20-05:11:09.369791 7f9b3313f700 [WARN] Options.memtable_prefix_bloom_probes: 6 2016/05/20-05:11:09.369793 7f9b3313f700 [WARN] Options.memtable_prefix_bloom_huge_page_tlb_size: 0 2016/05/20-05:11:09.369794 7f9b3313f700 [WARN] Options.bloom_locality: 0 2016/05/20-05:11:09.369795 7f9b3313f700 [WARN] Options.max_successive_merges: 0 2016/05/20-05:11:09.369796 7f9b3313f700 [WARN] Options.optimize_fllters_for_hits: 0 2016/05/20-05:11:09.369903 7f9b3313f700 Recovered from manifest file://Partition_50/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0 2016/05/20-05:11:09.369906 7f9b3313f700 Column family [default] (ID 0), log number is 0 2016/05/20-05:11:09.453048 7f9b3313f700 DB pointer 0x7f9b2dae8aa0 2016/05/20-05:11:10.388436 7f9af8e16700 (Original Log Time 2016/05/20-05:11:10.388391) Calling FlushMemTableToOutputFile with column family [default], flush slots available 0, compaction slots available 1 2016/05/20-05:11:10.388464 7f9af8e16700 [default] [JOB 2] Flushing memtable with next log file: 3 2016/05/20-05:11:10.388573 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721070388515, "job": 2, "event": "flush_started", "num_memtables": 1, "num_entries": 131228, "num_deletes": 0, "memory_usage": 32717846} 2016/05/20-05:11:10.388601 7f9af8e16700 [default] [JOB 2] Level-0 flush table #4: started 2016/05/20-05:11:11.067954 7f9af8e16700 [default] [JOB 2] Level-0 flush table #4: 14337270 bytes OK 2016/05/20-05:11:11.068056 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721071067998, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 4, "file_size": 14337270, "table_properties": {"data_size": 14041182, "index_size": 504423, "filter_size": 0, "raw_key_size": 9949192, "raw_average_key_size": 75.8161, "raw_value_size": 19990408, "raw_average_value_size": 152.333, "num_data_blocks": 6908, "num_entries": 131228, "filter_policy_name": "", "kDeletedKeys": "0"}} 2016/05/20-05:11:11.166953 7f9af8e16700 Creating manifest 5 2016/05/20-05:11:11.387379 7f9af8e16700 Deleting manifest 1 current manifest 5 2016/05/20-05:11:11.387882 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.166901) [default] Level-0 commit table #4 started 2016/05/20-05:11:11.387885 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.387787) [default] Level-0 commit table #4: memtable #1 done 2016/05/20-05:11:11.387887 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.387834) EVENT_LOG_v1 {"time_micros": 1463721071387821, "job": 2, "event": "flush_finished", "lsm_state": [1, 0, 0, 0, 0, 0, 0]} 2016/05/20-05:11:11.387888 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.387862) [default] Level summary: files[1 0 0 0 0 0 0] max score 0.25 2016/05/20-05:11:11.390515 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.390506) Calling FlushMemTableToOutputFile with column family [default], flush slots available 0, compaction slots available 1 2016/05/20-05:11:11.390520 7f9af8e16700 [default] [JOB 3] Flushing memtable with next log file: 3 2016/05/20-05:11:11.390535 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721071390528, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 130910, "num_deletes": 0, "memory_usage": 32717850} 2016/05/20-05:11:11.390538 7f9af8e16700 [default] [JOB 3] Level-0 flush table #6: started 2016/05/20-05:11:11.973723 7f9af8e16700 [default] [JOB 3] Level-0 flush table #6: 14291520 bytes OK 2016/05/20-05:11:11.973818 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721071973761, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 6, "file_size": 14291520, "table_properties": {"data_size": 13996262, "index_size": 503911, "filter_size": 0, "raw_key_size": 9929444, "raw_average_key_size": 75.8494, "raw_value_size": 20015842, "raw_average_value_size": 152.898, "num_data_blocks": 6913, "num_entries": 130910, "filter_policy_name": "", "kDeletedKeys": "0"}} 2016/05/20-05:11:12.028226 7f9af8e16700 (Original Log Time 2016/05/20-05:11:11.989370) [default] Level-0 commit table #6 started 2016/05/20-05:11:12.028231 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.028157) [default] Level-0 commit table #6: memtable #1 done 2016/05/20-05:11:12.028235 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.028187) EVENT_LOG_v1 {"time_micros": 1463721072028177, "job": 3, "event": "flush_finished", "lsm_state": [2, 0, 0, 0, 0, 0, 0]} 2016/05/20-05:11:12.028238 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.028210) [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2016/05/20-05:11:12.037451 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.037435) Calling FlushMemTableToOutputFile with column family [default], flush slots available 0, compaction slots available 1 2016/05/20-05:11:12.037457 7f9af8e16700 [default] [JOB 4] Flushing memtable with next log file: 3 2016/05/20-05:11:12.037524 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721072037489, "job": 4, "event": "flush_started", "num_memtables": 1, "num_entries": 130343, "num_deletes": 0, "memory_usage": 32717798} 2016/05/20-05:11:12.037533 7f9af8e16700 [default] [JOB 4] Level-0 flush table #7: started 2016/05/20-05:11:12.633089 7f9af8e16700 [default] [JOB 4] Level-0 flush table #7: 14238467 bytes OK 2016/05/20-05:11:12.633157 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721072633114, "cf_name": "default", "job": 4, "event": "table_file_creation", "file_number": 7, "file_size": 14238467, "table_properties": {"data_size": 13942935, "index_size": 504562, "filter_size": 0, "raw_key_size": 9884618, "raw_average_key_size": 75.8354, "raw_value_size": 20071552, "raw_average_value_size": 153.99, "num_data_blocks": 6919, "num_entries": 130343, "filter_policy_name": "", "kDeletedKeys": "0"}} 2016/05/20-05:11:12.704381 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.670875) [default] Level-0 commit table #7 started 2016/05/20-05:11:12.704386 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.704305) [default] Level-0 commit table #7: memtable #1 done 2016/05/20-05:11:12.704388 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.704338) EVENT_LOG_v1 {"time_micros": 1463721072704328, "job": 4, "event": "flush_finished", "lsm_state": [3, 0, 0, 0, 0, 0, 0]} 2016/05/20-05:11:12.704390 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.704364) [default] Level summary: files[3 0 0 0 0 0 0] max score 0.75 2016/05/20-05:11:12.891056 7f9af8e16700 (Original Log Time 2016/05/20-05:11:12.891031) Calling FlushMemTableToOutputFile with column family [default], flush slots available 0, compaction slots available 1 2016/05/20-05:11:12.891062 7f9af8e16700 [default] [JOB 5] Flushing memtable with next log file: 3 2016/05/20-05:11:12.891118 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721072891095, "job": 5, "event": "flush_started", "num_memtables": 1, "num_entries": 130178, "num_deletes": 0, "memory_usage": 32718006} 2016/05/20-05:11:12.891127 7f9af8e16700 [default] [JOB 5] Level-0 flush table #8: started 2016/05/20-05:11:13.670310 7f9af8e16700 [default] [JOB 5] Level-0 flush table #8: 14204116 bytes OK 2016/05/20-05:11:13.670420 7f9af8e16700 EVENT_LOG_v1 {"time_micros": 1463721073670356, "cf_name": "default", "job": 5, "event": "table_file_creation", "file_number": 8, "file_size": 14204116, "table_properties": {"data_size": 13909480, "index_size": 503742, "filter_size": 0, "raw_key_size": 9871375, "raw_average_key_size": 75.8298, "raw_value_size": 20088487, "raw_average_value_size": 154.316, "num_data_blocks": 6920, "num_entries": 130178, "filter_policy_name": "", "kDeletedKeys": "0"}} 2016/05/20-05:11:13.734296 7f9af8e16700 (Original Log Time 2016/05/20-05:11:13.697361) [default] Level-0 commit table #8 started 2016/05/20-05:11:13.734301 7f9af8e16700 (Original Log Time 2016/05/20-05:11:13.734156) [default] Level-0 commit table #8: memtable #1 done 2016/05/20-05:11:13.734303 7f9af8e16700 (Original Log Time 2016/05/20-05:11:13.734189) EVENT_LOG_v1 {"time_micros": 1463721073734180, "job": 5, "event": "flush_finished", "lsm_state": [4, 0, 0, 0, 0, 0, 0]} 2016/05/20-05:11:13.734306 7f9af8e16700 (Original Log Time 2016/05/20-05:11:13.734279) [default] Level summary: files[4 0 0 0 0 0 0] max score 1.00 2016/05/20-05:11:13.734391 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/20-05:11:13.734407 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 54 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 19.3 3 4 0.704 0 0 0 Sum 4/0 54 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 19.3 3 4 0.704 0 0 0 Int 0/0 0 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 19.3 3 4 0.704 0 0 0 Flush(GB): cumulative 0.053, interval 0.053 Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 4.4 total, 4.4 interval Cumulative writes: 654K writes, 654K keys, 654K batches, 1.0 writes per batch, ingest: 0.14 GB, 33.87 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 0.05 GB write, 12.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 2.8 seconds Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 654K writes, 654K keys, 654K batches, 1.0 writes per batch, ingest: 147.82 MB, 33.87 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 0.05 GB write, 12.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 2.8 seconds Interval stall: 00:00:0.000 H:M:S, 0.0 percent ###################################################################################### ################## Begin stats samples (before degrade) ######################### ###################################################################################### 2016/05/26-14:00:39.807704 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:00:39.807736 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 23.5 22.3 0.0 0.0 0.0 0.5 51368 176798 0.291 8746 11M 7476 L1 0/0 0 0.0 8.0 7.7 0.3 7.7 7.4 0.0 1.0 1.4 1.4 5727 24229 0.236 0 69M 49K L2 161/0 326 0.0 3824.1 13.6 3810.4 3819.6 9.2 0.0 280.3 9.5 9.4 414142 87081 4.756 0 25G 874K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6209/0 12632 0.0 3884.2 51.8 3832.3 3896.5 64.2 0.0 174.3 8.4 8.4 475965 288449 1.650 8746 25G 2209K Int 0/0 0 0.0 4.8 0.0 4.7 4.8 0.0 0.0 317.0 6.8 6.8 714 293 2.436 155 11M 1286 Flush(GB): cumulative 22.360, interval 0.015 Stalls(count): 8033 level0_slowdown, 617 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 550170.4 total, 625.8 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.15 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3896.50 GB write, 7.25 MB/s write, 3884.16 GB read, 7.23 MB/s read, 475964.7 seconds Cumulative stall: 03:43:11.105 H:M:S, 2.4 percent Interval writes: 129K writes, 129K keys, 129K batches, 1.0 writes per batch, ingest: 30.39 MB, 0.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.76 GB write, 7.79 MB/s write, 4.75 GB read, 7.78 MB/s read, 713.8 seconds Interval stall: 00:01:8.182 H:M:S, 10.9 percent 2016/05/26-14:11:18.958458 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:11:18.958481 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 23.6 22.3 0.0 0.0 0.0 0.5 51460 177072 0.291 8904 11M 7476 L1 0/0 0 0.0 8.0 7.8 0.3 7.7 7.5 0.0 1.0 1.4 1.4 5731 24245 0.236 0 69M 49K L2 164/0 332 0.0 3829.2 13.6 3815.6 3824.7 9.2 0.0 280.4 9.5 9.4 414769 87097 4.762 0 25G 875K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6212/0 12638 0.0 3889.3 51.9 3837.5 3901.7 64.2 0.0 174.4 8.4 8.4 476688 288755 1.651 8904 25G 2210K Int 0/0 0 0.0 5.2 0.0 5.1 5.2 0.0 0.0 332.3 7.3 7.3 723 306 2.364 158 12M 1193 Flush(GB): cumulative 22.375, interval 0.016 Stalls(count): 8177 level0_slowdown, 631 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 550809.6 total, 639.2 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.18 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3901.68 GB write, 7.25 MB/s write, 3889.34 GB read, 7.23 MB/s read, 476688.2 seconds Cumulative stall: 03:43:53.326 H:M:S, 2.4 percent Interval writes: 134K writes, 134K keys, 134K batches, 1.0 writes per batch, ingest: 31.52 MB, 0.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 5.18 GB write, 8.30 MB/s write, 5.17 GB read, 8.29 MB/s read, 723.5 seconds Interval stall: 00:00:42.222 H:M:S, 6.6 percent ###################################################################################### ################## Perf degrades somewhere between 2:10-2:20 ######################### ###################################################################################### 2016/05/26-14:22:07.485490 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:22:07.485522 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 23.6 22.3 0.0 0.0 0.0 0.5 51560 177321 0.291 9058 11M 7476 L1 0/0 0 0.0 8.0 7.8 0.3 7.7 7.5 0.0 1.0 1.4 1.4 5735 24259 0.236 0 70M 49K L2 167/0 338 0.0 3833.8 13.7 3820.1 3829.3 9.2 0.0 280.4 9.5 9.4 415407 87111 4.769 0 25G 876K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6215/0 12644 0.0 3893.9 51.9 3842.0 3906.3 64.2 0.0 174.5 8.4 8.4 477431 289032 1.652 9058 25G 2212K Int 0/0 0 0.0 4.6 0.0 4.6 4.6 0.0 0.0 320.7 6.4 6.4 742 277 2.680 154 11M 1253 Flush(GB): cumulative 22.390, interval 0.014 Stalls(count): 8317 level0_slowdown, 645 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 551458.1 total, 648.5 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.21 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3906.29 GB write, 7.25 MB/s write, 3893.94 GB read, 7.23 MB/s read, 477430.7 seconds Cumulative stall: 03:50:54.276 H:M:S, 2.5 percent Interval writes: 124K writes, 124K keys, 124K batches, 1.0 writes per batch, ingest: 29.08 MB, 0.04 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.61 GB write, 7.28 MB/s write, 4.61 GB read, 7.27 MB/s read, 742.5 seconds Interval stall: 00:07:0.949 H:M:S, 64.9 percent 2016/05/26-14:32:26.076299 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:32:26.076321 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 17/0 1 0.0 1.2 0.0 1.2 23.6 22.3 0.0 0.0 0.0 0.5 51650 177568 0.291 9217 11M 7476 L1 0/0 0 0.0 8.0 7.8 0.3 7.7 7.5 0.0 1.0 1.4 1.4 5740 24273 0.236 0 70M 49K L2 170/0 343 0.0 3838.5 13.7 3824.8 3834.0 9.2 0.0 280.5 9.4 9.4 416015 87125 4.775 0 25G 877K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6217/0 12649 0.0 3898.6 51.9 3846.7 3911.0 64.3 0.0 174.6 8.3 8.4 478132 289307 1.653 9217 25G 2213K Int 0/0 0 0.0 4.7 0.0 4.7 4.7 0.0 0.0 328.8 6.8 6.8 702 275 2.551 159 11M 1195 Flush(GB): cumulative 22.404, interval 0.014 Stalls(count): 8461 level0_slowdown, 660 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 552076.7 total, 618.6 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.24 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3910.98 GB write, 7.25 MB/s write, 3898.63 GB read, 7.23 MB/s read, 478132.2 seconds Cumulative stall: 03:58:51.976 H:M:S, 2.6 percent Interval writes: 123K writes, 123K keys, 123K batches, 1.0 writes per batch, ingest: 28.82 MB, 0.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.69 GB write, 7.76 MB/s write, 4.68 GB read, 7.75 MB/s read, 701.5 seconds Interval stall: 00:07:57.700 H:M:S, 77.2 percent 2016/05/26-14:42:49.742971 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:42:49.742993 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 23.6 22.4 0.0 0.0 0.0 0.5 51733 177814 0.291 9375 11M 7476 L1 0/0 0 0.0 8.1 7.8 0.3 7.8 7.5 0.0 1.0 1.4 1.4 5744 24287 0.237 0 70M 49K L2 173/0 349 0.0 3843.2 13.7 3829.5 3838.7 9.2 0.0 280.5 9.4 9.4 416625 87139 4.781 0 25G 878K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6221/0 12655 0.0 3903.4 52.0 3851.4 3915.7 64.3 0.0 174.7 8.3 8.4 478831 289581 1.654 9375 25G 2214K Int 0/0 0 0.0 4.8 0.0 4.7 4.8 0.0 0.0 336.8 7.0 7.0 698 274 2.549 158 11M 1244 Flush(GB): cumulative 22.418, interval 0.014 Stalls(count): 8605 level0_slowdown, 674 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 552700.4 total, 623.7 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.27 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3915.74 GB write, 7.25 MB/s write, 3903.38 GB read, 7.23 MB/s read, 478830.6 seconds Cumulative stall: 04:06:2.845 H:M:S, 2.7 percent Interval writes: 122K writes, 122K keys, 122K batches, 1.0 writes per batch, ingest: 28.58 MB, 0.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.76 GB write, 7.82 MB/s write, 4.76 GB read, 7.81 MB/s read, 698.3 seconds Interval stall: 00:07:10.869 H:M:S, 69.1 percent 2016/05/26-14:53:23.615351 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/26-14:53:23.615374 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 23.6 22.4 0.0 0.0 0.0 0.5 51817 178065 0.291 9534 11M 7476 L1 0/0 0 0.0 8.1 7.8 0.3 7.8 7.5 0.0 1.0 1.4 1.4 5749 24301 0.237 0 70M 49K L2 175/0 354 0.0 3848.0 13.7 3834.3 3843.5 9.2 0.0 280.6 9.4 9.4 417246 87153 4.788 0 25G 879K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6223/0 12660 0.0 3908.2 52.0 3856.2 3920.6 64.3 0.0 174.8 8.3 8.4 479540 289860 1.654 9534 25G 2215K Int 0/0 0 0.0 4.8 0.0 4.8 4.8 0.0 0.0 335.7 7.0 7.0 709 279 2.541 159 11M 1196 Flush(GB): cumulative 22.432, interval 0.014 Stalls(count): 8747 level0_slowdown, 691 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 553334.2 total, 633.9 interval Cumulative writes: 204M writes, 204M keys, 204M batches, 1.0 writes per batch, ingest: 47.30 GB, 0.09 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 3920.58 GB write, 7.26 MB/s write, 3908.21 GB read, 7.23 MB/s read, 479539.6 seconds Cumulative stall: 04:14:26.499 H:M:S, 2.8 percent Interval writes: 125K writes, 125K keys, 125K batches, 1.0 writes per batch, ingest: 29.14 MB, 0.05 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.84 GB write, 7.82 MB/s write, 4.83 GB read, 7.81 MB/s read, 709.1 seconds Interval stall: 00:08:23.654 H:M:S, 79.5 percent ###################################################################################### ################## Perf issue continues for almost 23hrs ######################### ###################################################################################### 2016/05/27-12:46:53.660221 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-12:46:53.660242 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 17/0 1 0.0 1.2 0.0 1.2 24.8 23.6 0.0 0.0 0.0 0.4 58241 199068 0.293 22775 11M 7476 L1 0/0 0 0.0 9.2 9.0 0.3 8.9 8.6 0.0 1.0 1.5 1.5 6124 25482 0.240 0 80M 50K L2 413/0 834 0.0 4534.2 14.8 4519.4 4529.0 9.7 0.0 305.0 9.4 9.4 494990 88334 5.604 0 26G 989K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6460/0 13139 0.0 4595.6 54.3 4541.3 4608.4 67.1 0.0 195.0 8.3 8.4 564083 313225 1.801 22775 26G 2325K Int 0/0 0 0.0 5.7 0.0 5.7 5.7 0.0 0.0 797.1 8.7 8.7 673 139 4.838 78 6103K 718 Flush(GB): cumulative 23.637, interval 0.007 Stalls(count): 20679 level0_slowdown, 2000 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 632144.3 total, 640.9 interval Cumulative writes: 215M writes, 215M keys, 215M batches, 1.0 writes per batch, ingest: 49.67 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4608.40 GB write, 7.47 MB/s write, 4595.56 GB read, 7.44 MB/s read, 564083.4 seconds Cumulative stall: 22:29:44.453 H:M:S, 12.8 percent Interval writes: 62K writes, 61K keys, 62K batches, 1.0 writes per batch, ingest: 14.45 MB, 0.02 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 5.70 GB write, 9.11 MB/s write, 5.70 GB read, 9.11 MB/s read, 672.5 seconds Interval stall: 00:09:19.807 H:M:S, 87.3 percent 2016/05/27-12:58:13.475846 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-12:58:13.476106 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 18/0 1 0.0 1.2 0.0 1.2 24.8 23.6 0.0 0.0 0.0 0.4 58281 199193 0.293 22855 11M 7476 L1 0/0 0 0.0 9.2 9.0 0.3 8.9 8.6 0.0 1.0 1.5 1.5 6127 25489 0.240 0 80M 50K L2 414/0 836 0.0 4539.9 14.9 4525.1 4534.7 9.7 0.0 305.3 9.4 9.4 495663 88341 5.611 0 26G 990K L3 482/0 974 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1237 262 4.722 0 73M 37K L4 644/0 1303 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1143 54 21.170 0 58M 30K L5 1001/0 2023 0.0 11.4 6.5 4.8 10.1 5.3 0.0 1.6 12.1 10.8 961 18 53.375 0 36M 31K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6462/0 13142 0.0 4601.3 54.3 4547.0 4614.1 67.2 0.0 195.1 8.3 8.4 564799 313364 1.802 22855 26G 2326K Int 0/0 0 0.0 5.7 0.0 5.7 5.7 0.0 0.0 794.3 8.2 8.2 716 139 5.148 80 6106K 733 Flush(GB): cumulative 23.645, interval 0.007 Stalls(count): 20751 level0_slowdown, 2008 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 632824.1 total, 679.8 interval Cumulative writes: 215M writes, 215M keys, 215M batches, 1.0 writes per batch, ingest: 49.69 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4614.12 GB write, 7.47 MB/s write, 4601.28 GB read, 7.45 MB/s read, 564798.9 seconds Cumulative stall: 22:39:41.497 H:M:S, 12.9 percent Interval writes: 62K writes, 62K keys, 62K batches, 1.0 writes per batch, ingest: 14.56 MB, 0.02 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 5.72 GB write, 8.62 MB/s write, 5.72 GB read, 8.62 MB/s read, 715.6 seconds Interval stall: 00:09:57.044 H:M:S, 87.8 percent 2016/05/27-13:08:14.500745 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-13:08:14.500766 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 0 0.0 1.2 0.0 1.2 24.8 23.6 0.0 0.0 0.0 0.4 58300 199257 0.293 22868 11M 7476 L1 0/0 0 0.0 9.2 9.0 0.3 8.9 8.6 0.0 1.0 1.5 1.5 6127 25490 0.240 0 80M 50K L2 1/0 1 0.0 4539.9 14.9 4525.1 4534.7 9.7 0.0 305.2 9.4 9.4 495674 88378 5.609 0 26G 990K L3 1/0 2 0.0 12.5 7.1 5.3 11.4 6.0 0.0 1.6 10.3 9.4 1239 266 4.658 0 73M 37K L4 1/0 2 0.0 12.0 7.4 4.7 10.8 6.1 0.0 1.5 10.8 9.7 1144 55 20.791 0 58M 30K L5 2446/0 4944 0.0 16.4 9.6 6.8 15.0 8.2 0.0 1.6 11.0 10.1 1524 19 80.222 0 40M 59K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6353/0 12954 0.0 4606.3 57.4 4549.0 4619.0 70.0 0.0 195.3 8.3 8.4 565395 313472 1.804 22868 26G 2354K Int 0/0 0 0.0 5.0 3.0 2.0 4.9 2.9 0.0 1349.9 8.7 8.3 596 108 5.515 13 3659K 27K Flush(GB): cumulative 23.648, interval 0.004 Stalls(count): 20763 level0_slowdown, 2009 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 633425.1 total, 601.0 interval Cumulative writes: 215M writes, 215M keys, 215M batches, 1.0 writes per batch, ingest: 49.70 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4618.97 GB write, 7.47 MB/s write, 4606.32 GB read, 7.45 MB/s read, 565394.6 seconds Cumulative stall: 22:48:57.980 H:M:S, 13.0 percent Interval writes: 31K writes, 32K keys, 31K batches, 1.0 writes per batch, ingest: 7.37 MB, 0.01 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 4.85 GB write, 8.27 MB/s write, 5.04 GB read, 8.58 MB/s read, 595.6 seconds Interval stall: 00:09:16.483 H:M:S, 92.6 percent ###################################################################################### ################## Perf issue subsides between 1:00-1:10 ######################### ###################################################################################### 2016/05/27-13:18:14.759540 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-13:18:14.759560 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 0 0.0 1.2 0.0 1.2 24.9 23.6 0.0 0.0 0.0 0.4 58599 200241 0.293 22868 11M 7476 L1 0/0 0 0.0 9.3 9.0 0.3 8.9 8.6 0.0 1.0 1.5 1.5 6167 25595 0.241 0 80M 50K L2 4/0 7 0.0 4541.5 14.9 4526.6 4536.3 9.7 0.0 304.2 9.4 9.4 496058 89070 5.569 0 26G 990K L3 9/0 17 0.0 12.5 7.2 5.4 11.4 6.0 0.0 1.6 10.3 9.4 1250 284 4.401 0 74M 37K L4 16/0 31 0.0 12.1 7.4 4.7 10.9 6.2 0.0 1.5 10.7 9.7 1150 58 19.827 0 58M 30K L5 2446/0 4944 0.0 16.4 9.6 6.8 15.0 8.2 0.0 1.6 11.0 10.1 1524 19 80.222 0 40M 59K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6380/0 13005 0.0 4608.0 57.5 4550.5 4620.7 70.2 0.0 194.9 8.3 8.4 566136 315274 1.796 22868 26G 2354K Int 0/0 0 0.0 1.7 0.1 1.6 1.8 0.2 0.0 30.9 2.4 2.4 741 1802 0.411 0 15M 4 Flush(GB): cumulative 23.705, interval 0.057 Stalls(count): 20763 level0_slowdown, 2009 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 634025.4 total, 600.3 interval Cumulative writes: 215M writes, 215M keys, 215M batches, 1.0 writes per batch, ingest: 49.81 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4620.73 GB write, 7.46 MB/s write, 4608.03 GB read, 7.44 MB/s read, 566135.7 seconds Cumulative stall: 22:48:57.980 H:M:S, 13.0 percent Interval writes: 490K writes, 490K keys, 490K batches, 1.0 writes per batch, ingest: 115.39 MB, 0.19 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 1.76 GB write, 3.01 MB/s write, 1.71 GB read, 2.92 MB/s read, 741.2 seconds Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2016/05/27-13:28:17.955704 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-13:28:17.955741 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 4/0 0 0.0 1.2 0.0 1.2 24.9 23.7 0.0 0.0 0.0 0.4 58895 201075 0.293 22874 11M 7476 L1 0/0 0 0.0 9.3 9.0 0.3 8.9 8.7 0.0 1.0 1.5 1.5 6231 25748 0.242 0 80M 50K L2 9/0 16 0.0 4543.3 15.0 4528.4 4538.1 9.8 0.0 303.4 9.4 9.4 496461 89459 5.550 0 26G 990K L3 9/0 17 0.0 12.6 7.2 5.4 11.4 6.1 0.0 1.6 10.2 9.3 1257 289 4.350 0 74M 37K L4 31/0 63 0.0 12.1 7.4 4.7 10.9 6.2 0.0 1.5 10.7 9.6 1161 59 19.678 0 59M 30K L5 2446/0 4944 0.0 16.4 9.6 6.8 15.0 8.2 0.0 1.6 11.0 10.1 1524 19 80.222 0 40M 59K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6402/0 13047 0.0 4610.0 57.6 4552.3 4622.7 70.4 0.0 194.6 8.3 8.3 566917 316656 1.790 22874 26G 2354K Int 0/0 0 0.0 1.9 0.1 1.8 2.0 0.2 0.0 40.7 2.5 2.6 781 1382 0.565 6 18M 5 Flush(GB): cumulative 23.753, interval 0.048 Stalls(count): 20769 level0_slowdown, 2009 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 634628.6 total, 603.2 interval Cumulative writes: 216M writes, 216M keys, 216M batches, 1.0 writes per batch, ingest: 49.90 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4622.70 GB write, 7.46 MB/s write, 4609.95 GB read, 7.44 MB/s read, 566917.2 seconds Cumulative stall: 22:48:58.207 H:M:S, 12.9 percent Interval writes: 413K writes, 413K keys, 413K batches, 1.0 writes per batch, ingest: 97.41 MB, 0.16 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 1.96 GB write, 3.33 MB/s write, 1.92 GB read, 3.26 MB/s read, 781.4 seconds Interval stall: 00:00:0.227 H:M:S, 0.0 percent 2016/05/27-13:38:18.882156 7f9afaaea700 [WARN] ------- DUMPING STATS ------- 2016/05/27-13:38:18.882178 7f9afaaea700 [WARN] ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 0 0.0 1.2 0.0 1.2 25.0 23.7 0.0 0.0 0.0 0.4 59192 201847 0.293 22885 11M 7476 L1 0/0 0 0.0 9.3 9.1 0.3 9.0 8.7 0.0 1.0 1.5 1.5 6297 25874 0.243 0 81M 50K L2 2/0 3 0.0 4545.4 15.0 4530.4 4540.2 9.8 0.0 302.6 9.4 9.4 496867 89698 5.539 0 26G 990K L3 2/0 3 0.0 12.6 7.2 5.4 11.5 6.1 0.0 1.6 10.2 9.3 1264 292 4.328 0 74M 37K L4 61/0 125 0.0 12.3 7.5 4.8 11.0 6.3 0.0 1.5 10.6 9.5 1190 60 19.826 0 60M 30K L5 2446/0 4944 0.0 16.4 9.6 6.8 15.0 8.2 0.0 1.6 11.0 10.1 1524 19 80.222 0 40M 59K L6 3903/0 8005 0.0 15.0 9.4 5.6 13.4 7.8 0.0 1.4 11.1 9.9 1387 7 198.150 0 15M 1178K Sum 6416/0 13080 0.0 4612.2 57.8 4554.4 4625.0 70.5 0.0 194.3 8.3 8.3 567720 317797 1.786 22885 26G 2354K Int 0/0 0 0.0 2.2 0.2 2.1 2.3 0.2 0.0 51.6 2.9 2.9 803 1141 0.704 11 21M 19 Flush(GB): cumulative 23.797, interval 0.044 Stalls(count): 20779 level0_slowdown, 2010 level0_numfiles, 96 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard ** DB Stats ** Uptime(secs): 635229.5 total, 600.9 interval Cumulative writes: 216M writes, 216M keys, 216M batches, 1.0 writes per batch, ingest: 49.99 GB, 0.08 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative compaction: 4624.97 GB write, 7.46 MB/s write, 4612.20 GB read, 7.43 MB/s read, 567719.9 seconds Cumulative stall: 22:49:6.906 H:M:S, 12.9 percent Interval writes: 378K writes, 378K keys, 378K batches, 1.0 writes per batch, ingest: 89.27 MB, 0.15 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval compaction: 2.27 GB write, 3.87 MB/s write, 2.24 GB read, 3.82 MB/s read, 802.7 seconds Interval stall: 00:00:8.699 H:M:S, 1.4 percent