Add stats about the total data read from source (client + BookKeeper)

qubole · Aug 21, 2020 · 28a141d · 28a141d
1 parent 4d220b6
commit 28a141d
Show file tree

Hide file tree

Showing 4 changed files with 19 additions and 5 deletions.
diff --git a/docs/metrics.rst b/docs/metrics.rst
@@ -119,9 +119,9 @@ Following sections cover the metrics available under both these types in detail.
 +------------------------------------------------+--------------------------------------------+
 | Metric                                         | Description                                |
 +================================================+============================================+
-| mb_read_from_cache                             | Data read from cache                       |
+| mb_read_from_cache                             | Data read from cache by the client jobs    |
 +------------------------------------------------+--------------------------------------------+
-| mb_read_from_source                            | Data read from Source                      |
+| mb_read_from_source                            | Data read from Source by the client jobs   |
 +------------------------------------------------+--------------------------------------------+
 | cache_hit                                      | Cache Hit ratio, between 0 and 1           |
 +------------------------------------------------+--------------------------------------------+
@@ -131,12 +131,17 @@ Following sections cover the metrics available under both these types in detail.
 +------------------------------------------------+--------------------------------------------+
 | Metric                                         | Description                                |
 +================================================+============================================+
-| mb_read_from_cache                             | Data read from cache                       |
+| mb_read_from_cache                             | Data read from cache by the client jobs    |
 +------------------------------------------------+--------------------------------------------+
-| mb_read_from_source                            | Data read from Source                      |
+| mb_read_from_source                            | Data read from Source by the client jobs   |
 +------------------------------------------------+--------------------------------------------+
 | cache_hit                                      | Cache Hit ratio, between 0 and 1           |
 +------------------------------------------------+--------------------------------------------+
+| total_system_source_mb_read                    | Total data read from source. This includes |
+|                                                | data read from source by BookKeeper (for   |
+|                                                | parallel warmup) in addition to data read   |
+|                                                | from source by the client jobs             |
++------------------------------------------------+--------------------------------------------+
 | cached_rrc_data_read                           | Data read from local cache                 |
 +------------------------------------------------+--------------------------------------------+
 | cached_rrc_requests                            | Number of requests served from local cache |

diff --git a/docs/release/release_notes/release-0.3.17-(next-release).rst b/docs/release/release_notes/release-0.3.17-(next-release).rst
@@ -4,3 +4,4 @@ Release 0.3.17 (next release)
 
 Fixes and Features
 ------------------
+* Added `total_system_source_mb_read` stat in detailed metrics to show total data read from source: during read + warmups
diff --git a/rubix-core/src/main/java/com/qubole/rubix/core/BasicCachingFileSystemStats.java b/rubix-core/src/main/java/com/qubole/rubix/core/BasicCachingFileSystemStats.java
@@ -34,7 +34,7 @@ public BasicCachingFileSystemStats(
         this.conf = conf;
     }
 
-    @Managed(description = "Gets the total size in MB that was read from Source")
+    @Managed(description = "Gets the total size in MB that was read from Source during the reads")
     public long getMB_read_from_source()
     {
         return BYTES.toMB(statsProvider.getStats().getDirectRRCDataRead() +

diff --git a/rubix-core/src/main/java/com/qubole/rubix/core/DetailedCachingFileSystemStats.java b/rubix-core/src/main/java/com/qubole/rubix/core/DetailedCachingFileSystemStats.java
@@ -130,4 +130,12 @@ public int getcorrupted_file_count()
     {
         return statsProvider.getStats().getCorruptedFileCount();
     }
+
+    @Managed(description = "Total data read from source by the system")
+    // getMB_read_from_source only includes the data read from source during the reads
+    // It ignores the data read by BookKeeper in async warmup
+    public long getTotal_system_source_MB_read()
+    {
+        return getMB_read_from_source() + statsProvider.getStats().getDownloadedFromSourceParallel(bookKeeperFactory, conf);
+    }
 }
Original file line number	Diff line number	Diff line change
Expand Up		@@ -4,3 +4,4 @@ Release 0.3.17 (next release)

		Fixes and Features
		------------------
		* Added `total_system_source_mb_read` stat in detailed metrics to show total data read from source: during read + warmups