-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
auth: dramatically improve checkPassword performance #11735
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot! But could you add a comment about this change to the comment? Not using defer unlock is a little bit unusual so people who read and modify the code should be warned about this (e.g. forgetting about unlocking in error paths).
b7853d9
to
3f419f9
Compare
Thanks for your suggestion! I refactored a version that uses closures, so we can still use the defer statement, it may look better than the previous implementation, what do you think? @mitake |
to improve authentication performance in concurrent scenarios when enable auth and using authentication based password
3f419f9
to
9cf3162
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks!
…5-origin-release-3.4 Automated cherry pick of #11735 on release-3.4
Optimize lock scope for CheckPassword, to improve authentication performance in concurrent scenarios
when enable auth and using authentication based password.
etcd version: 3.4.3
When enable auth and using authentication based password, we found that etcd's read and write performance dropped sharply, and even a large number of timeouts occurred. After our investigation, we found that time is spent on Authenticate.
![image](https://user-images.githubusercontent.com/7612790/77940903-f1554600-72eb-11ea-896e-eca1090351b7.png)
Furtherly, we found that the CompareHashAndPassword function will take tens of milliseconds. This function will block the lock of CheckPassword, which can cause certain requests to block for tens of seconds when there is thousounds of concurrent connection.
![image](https://user-images.githubusercontent.com/7612790/77940915-f4503680-72eb-11ea-90d2-1d2f8daa1b0c.png)
![image](https://user-images.githubusercontent.com/7612790/77940924-f74b2700-72eb-11ea-811c-36bf367fff15.png)
(We have add some debug log in etcd. As shown below, the time unit is millisecond)
Moving CompareHashAndPassword out of the critical section of the lock can greatly improve performance. After we do this, we no longer find timeout requests.
And we developed a tool based on etcd benchmark CLI to test the performance of the Authenticate interface. The performance data is as follows:
We consider a single node etcd cluster with the following hardware configuration:
Before optimization:
benchmark_auth auth --total 3000 --clients 100 --conns 100
benchmark_auth auth --total 3000 --clients 200 --conns 200
With 200 concurrency, a large number of requests time out.
After optimization:
benchmark_auth auth --total 3000 --clients 100 --conns 100
benchmark_auth auth --total 3000 --clients 200 --conns 200
We can see that the performance is improved more than 10 times after optimization. And with the increase of concurrencys, the performance improvement is more and more obvious