fix(raft/log): truncate file and reset offset correctly

[yichengq]

@xiangli-cmu

[xiang90]

lgtm

@philips This fixes the problem that probably causes #829.

[philips]

oh, dang. good fix.

[ongardie]

Sorry I'm a bit late on this one and completely off-topic, but I noticed the notification for this PR and got worried. What I'm worried about is that it'd be dangerous for a server with a corrupt disk to truncate some entries off the end of its log and then continue participating in the cluster.

What's this code trying to do? What are the errors in decoding that result in the file being truncated? A few more comments could help.

[yichengq]

@ongardie
This code mainly focuses on a bug on implementation. Even if a file is truncated, its I/O offset doesn't change and new log entries will be appended to the old point. This could be a big problem when it loads the log next time because the data may confuse the protobuf parsing.

The error comes from failing to parse in protobuf. t is highly possible that the machine was rebooted when sync the latest entries into the disk and only part of it is recorded.

In most cases this should not be a problem for cluster because the others could recover the log.

But it would be terrible if some entries are really lost in the whole cluster. We could print out some warning for the truncation, but I don't know whether there is better solution for that. I could think about some way is to add a flag that indicates log miss when rejoining the cluster to ensure the safety.

[ongardie]

Yeah, it's really hard to tell whether the disk was corrupted before or after acking the write. I'd definitely add a warning here as a minimum.