auth: dramatically improve checkPassword performance

[wswcfan]

Optimize lock scope for CheckPassword, to improve authentication performance in concurrent scenarios 
when enable auth and using authentication based password.

etcd version: 3.4.3

When enable auth and using authentication based password, we found that etcd's read and write performance dropped sharply, and even a large number of timeouts occurred. After our investigation, we found that time is spent on Authenticate.

Furtherly, we found that the CompareHashAndPassword function will take tens of milliseconds. This function will block the lock of CheckPassword, which can cause certain requests to block for tens of seconds when there is thousounds of concurrent connection.
(We have add some debug log in etcd. As shown below, the time unit is millisecond)

Moving CompareHashAndPassword out of the critical section of the lock can greatly improve performance. After we do this, we no longer find timeout requests.

And we developed a tool based on etcd benchmark CLI to test the performance of the Authenticate interface. The performance data is as follows:

We consider a single node etcd cluster with the following hardware configuration:

• 1 machines(etcd server) of v16 CPUs + 32GB Memory + 50GB SSD
• 1 machine(client) of 8 vCPUs + 16GB Memory + 50GB SSD
• Ubuntu 16.04
• etcd Version: 3.5.0-pre, Go Version: go1.13

Before optimization:

benchmark_auth auth --total 3000 --clients 100 --conns 100

benchmark_auth auth --total 3000 --clients 200 --conns 200

With 200 concurrency, a large number of requests time out.

After optimization:

benchmark_auth auth --total 3000 --clients 100 --conns 100

benchmark_auth auth --total 3000 --clients 200 --conns 200

We can see that the performance is improved more than 10 times after optimization. And with the increase of concurrencys, the performance improvement is more and more obvious

[mitake]

Thanks a lot! But could you add a comment about this change to the comment? Not using defer unlock is a little bit unusual so people who read and modify the code should be warned about this (e.g. forgetting about unlocking in error paths).

[wswcfan]

Thanks for your suggestion! I refactored a version that uses closures, so we can still use the defer statement, it may look better than the previous implementation, what do you think? @mitake

[mitake]

@wswcfan LGTM! Yeah your approach based on closure looks great :) Defer to @jingyih

[jingyih]

LGTM. Thanks!