Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reusable async client #639

Merged
merged 10 commits into from
Jan 2, 2024

Conversation

odelmarcelle
Copy link
Contributor

Description

Clears out the session attribute of AsyncHttpConnection and AIOHttpConnection when calling close(), which allow re-using a previously closed AsyncOpensearch instance.

Issues Resolved

Closes #637

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

codecov bot commented Dec 21, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (e323ab2) 72.12% compared to head (d4edaa1) 72.10%.
Report is 1 commits behind head on main.

Files Patch % Lines
opensearchpy/connection/http_urllib3.py 60.00% 4 Missing ⚠️
opensearchpy/connection/http_async.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #639      +/-   ##
==========================================
- Coverage   72.12%   72.10%   -0.02%     
==========================================
  Files          89       89              
  Lines        7939     7949      +10     
==========================================
+ Hits         5726     5732       +6     
- Misses       2213     2217       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks.

Double checking that the synchronous client doesn't have this issue?

Does the client support with (too lazy to look) :))

CHANGELOG.md Outdated Show resolved Hide resolved
@odelmarcelle
Copy link
Contributor Author

Turns out the issue affects clients using Urllib3HttpConnection which discard the pool, but RequestsHttpConnection works fine with my local testing.

I discovered this by adding test for the synchronous client, which apparently uses Urllib3HttpConnection. I'm not too sure where to put a test that would use RequestsHttpConnection.

@saimedhi
Copy link
Collaborator

Turns out the issue affects clients using Urllib3HttpConnection which discard the pool, but RequestsHttpConnection works fine with my local testing.

I discovered this by adding test for the synchronous client, which apparently uses Urllib3HttpConnection. I'm not too sure where to put a test that would use RequestsHttpConnection.

Hello @odelmarcelle , please put your test here for RequestsHttpConnection. And also DCO is failing. Please fix it. Thank you.

Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
…after `close`

Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
@odelmarcelle
Copy link
Contributor Author

Resolved the DCO.

@saimedhi AFAIK, the test won't work there as there's not connection established to an opensearch instance. It should be somewhere under /test_server/, right? I didn't manage to work it out by myself.

One last comment though: Why are there two async connection classes defined? I'm not too sure to understand why AsyncHttpConnection and AIOHttpConnection co-exist. Especially as the former is the one recommended in the guides while most tests are based on the latter.

opensearchpy/connection/http_urllib3.py Outdated Show resolved Hide resolved
test_opensearchpy/test_async/test_server/test_clients.py Outdated Show resolved Hide resolved
test_opensearchpy/test_server/test_clients.py Outdated Show resolved Hide resolved
@odelmarcelle odelmarcelle force-pushed the reusable-async-client branch 2 times, most recently from 80be9a8 to 088e705 Compare December 30, 2023 17:16
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
Signed-off-by: odlmarce <delmarcelle.olivier@gmail.com>
self._create_urllib3_pool()

def _create_urllib3_pool(self) -> None:
self.pool = self._urllib3_pool_factory() # type: ignore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be a bit defensive here and add assert _urllib3_pool_factory is not None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to see how that would help. _urllib3_pool_factory will never be None?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right, its creation is in the constructor.

@dblock
Copy link
Member

dblock commented Jan 2, 2024

LMK if you prefer to merge as is, or if you think adding that assert is a good idea.

@dblock dblock merged commit 3eba72c into opensearch-project:main Jan 2, 2024
53 of 54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Re-create aiohttp.ClientSession after using close() on AsyncOpensearch
3 participants