mt-update-ttl multithreaded + more #843
Conversation
cmd/mt-update-ttl/main.go
Outdated
| return 0.5 - float64(token)/float64(2*minToken) | ||
| } | ||
| return 0.5 + float64(token)/float64(2*maxToken) | ||
| } |
There was a problem hiding this comment.
I think this can be simplified to ((token/maxToken)+1)/2 in both cases.
Dieterbe
force-pushed
the
mt-update-ttl-fixes
branch
from
55581c7
to
6717b6f
Compare
cmd/mt-update-ttl/main.go
Outdated
| "time" | ||
|
|
||
| "github.com/gocql/gocql" | ||
| hostpool "github.com/hailocab/go-hostpool" | ||
| "github.com/raintank/dur" | ||
| ) | ||
|
|
||
| const maxToken = 9223372036854775807 |
There was a problem hiding this comment.
that's a nice way to calculate the completeness, but it took me quite a while to understand how it works, so a comment explaining what this number is and why it can be used to get the completeness based on the return of the token() function would be helpful
There was a problem hiding this comment.
Also could use math.MaxInt64
cmd/mt-update-ttl/main.go
Outdated
|
|
||
| var wg sync.WaitGroup | ||
| for i := 0; i < *numThreads; i++ { | ||
| wg.Add(1) |
There was a problem hiding this comment.
this could be moved before the loop and simply wg.Add(*numThreads)
| return ((float64(token) / float64(maxToken)) + 1) / 2 | ||
| } | ||
|
|
||
| func worker(id int, jobs <-chan string, wg *sync.WaitGroup, session *gocql.Session, startTime, endTime, ttl int, tableIn, tableOut string) { |
| fmt.Println("processed", rownum, "rows") | ||
|
|
||
| wg.Wait() | ||
| log.Printf("DONE. Processed %d keys, %d rows", doneKeys, doneRows) |
There was a problem hiding this comment.
might want to add
if err != nil {
os.Exit(2)
}
after the wg.Wait(), just to make sure that something else calling this tool never falsely assumes everything worked
There was a problem hiding this comment.
that's not important, but why not just
err := keyItr.Close()
wg.Wait()
if err != nil {
fmt.Fprintf(os.Stderr, "ERROR: failed querying %s: %q. processed %d keys, %d rows", tableIn, err, doneKeys, doneRows)
os.Exit(2)
}
log.Printf("DONE. Processed %d keys, %d rows", doneKeys, doneRows)
There was a problem hiding this comment.
you're right, that would save a line. otoh this way has the err handling right after the operation, which is nice for readability. also you can see the error without having to wait for all cassandra operations in the workers to complete.
in particular, if you had extra bogus stuff at the end, did not properly set tableOut
Dieterbe
force-pushed
the
mt-update-ttl-fixes
branch
from
6717b6f
to
63e02b4
Compare
|
all comments adressed. |
@shanson7 did some very nice multi-threading work, resulting in a big speedup. i added a bit of polish on top