fix(event_history) fix a bug in event queue

[xiang90]

1. Fix a bug in event_history, which might cause an error when scan through a full queue.
2. Fix etcd stops responding to requests #379
3. Simplify the logic since now we do not have duplicate index in event history

[xiang90]

@philips @bcwaldon @polvi
This also caused the bug you are facing. The previous for loop might become a infinite loop in an edge case while holding a lock.
I confirmed the bug by reproducing it.
And I confirmed it fixed by running previous test 100 times successfully.

[philips]

"I confirmed the bug by reproducing it."

How did you reproduce it?

[xiang90]

Run three etcdbench at the same time for several times.
There is a easy way to reproduce, which requires to change the capacity of
the event queue smaller to increase the possibility of the race.

On Sunday, December 22, 2013, Brandon Philips wrote:

"I confirmed the bug by reproducing it."

How did you reproduce it?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/410#issuecomment-31089513
.

[philips]

The bug seems to be related to coding up a ring buffer by hand. Any reason to not use this? http://golang.org/pkg/container/ring/

[xiang90]

1. We need to serialize/deserialize it when doing snapshot.
2. We need to do random access by index.
   On Monday, December 23, 2013, Brandon Philips wrote:

The bug seems to be related to coding up a ring buffer by hand. Any reason
to not use this? http://golang.org/pkg/container/ring/

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/410#issuecomment-31097057
.

[philips]

I added a test and some comment cleanups on a new PR, here: #413