Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

keep import position in whisper reader #814

Merged
merged 2 commits into from
Jan 10, 2018

Conversation

replay
Copy link
Contributor

@replay replay commented Jan 9, 2018

Tested in my test env. I ran the importer-reader with the following command:

./mt-whisper-importer-reader \
-dst-schemas /home/mst/documents/code/go/src/github.com/grafana/metrictank/scripts/config/storage-schemas.conf \
-http-endpoint http://localhost:5432/chunks \
-name-prefix my.test. \
-orgid 1 \
-threads 3 \
-verbose \
-whisper-directory ziggurat  \
-position-file /tmp/mypos

After a few seconds I aborted with Ctrl+C. The file /tmp/mypos now contains the list of files that have already been processed:

mst@mst-nb1:~/documents/code/go/src/github.com/grafana/metrictank$ cat /tmp/mypos
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_count.wsp
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceFailures_count.wsp
ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_sum.wsp

When I now restart the same command it skips those that have already been completed:

DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceFailures_count.wsp because it was listed as already done 
DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_count.wsp because it was listed as already done 
DEBU[0000] Skipping file ziggurat/GenericJMX/cassandra_ClientRequest/counter/RangeSliceLatency_sum.wsp because it was listed as already done 

fixes issue #727

@replay replay requested review from Dieterbe and jtlisi January 9, 2018 13:54
@replay replay force-pushed the importer_recover_from_position branch from 278e61f to 6040766 Compare January 10, 2018 07:18
@@ -114,19 +119,28 @@ func main() {
panic(fmt.Sprintf("Error when parsing schemas file: %q", err))
}

var pos *posTracker
if len(*positionFile) > 0 {
pos, err = NewPositionKeeper(*positionFile)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i notice that you mix the position"tracker" terminology with position"keeper"

@Dieterbe Dieterbe merged commit 29bd120 into master Jan 10, 2018
@Dieterbe Dieterbe deleted the importer_recover_from_position branch September 18, 2018 09:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants