keep import position in whisper reader #814

replay · 2018-01-09T13:54:21Z

Tested in my test env. I ran the importer-reader with the following command:

./mt-whisper-importer-reader \
-dst-schemas /home/mst/documents/code/go/src/github.com/grafana/metrictank/scripts/config/storage-schemas.conf \
-http-endpoint http://localhost:5432/chunks \
-name-prefix my.test. \
-orgid 1 \
-threads 3 \
-verbose \
-whisper-directory ziggurat  \
-position-file /tmp/mypos

After a few seconds I aborted with Ctrl+C. The file /tmp/mypos now contains the list of files that have already been processed:

mst@mst-nb1:~/documents/code/go/src/github.com/grafana/metrictank$ cat /tmp/mypos
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_count.wsp
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceFailures_count.wsp
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_sum.wsp

When I now restart the same command it skips those that have already been completed:

DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceFailures_count.wsp because it was listed as already done 
DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_count.wsp because it was listed as already done 
DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_sum.wsp because it was listed as already done

fixes issue #727

Dieterbe · 2018-01-10T20:57:21Z

cmd/mt-whisper-importer-reader/main.go

@@ -114,19 +119,28 @@ func main() {
 		panic(fmt.Sprintf("Error when parsing schemas file: %q", err))
 	}

+	var pos *posTracker
+	if len(*positionFile) > 0 {
+		pos, err = NewPositionKeeper(*positionFile)


i notice that you mix the position"tracker" terminology with position"keeper"

replay requested review from Dieterbe and jtlisi January 9, 2018 13:54

keep import position in whisper reader

6040766

fixes issue #727

replay force-pushed the importer_recover_from_position branch from 278e61f to 6040766 Compare January 10, 2018 07:18

Dieterbe reviewed Jan 10, 2018

View reviewed changes

consistent terminology

762197d

Dieterbe merged commit 29bd120 into master Jan 10, 2018

Dieterbe deleted the importer_recover_from_position branch September 18, 2018 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

keep import position in whisper reader #814

keep import position in whisper reader #814

replay commented Jan 9, 2018

Dieterbe Jan 10, 2018

keep import position in whisper reader #814

keep import position in whisper reader #814

Conversation

replay commented Jan 9, 2018

Dieterbe Jan 10, 2018

Choose a reason for hiding this comment