Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align pool_maxsize for different connection pool implementations. #535

Merged
merged 4 commits into from
Oct 12, 2023

Conversation

dblock
Copy link
Member

@dblock dblock commented Oct 12, 2023

Description

  • Enable pool_maxsize for both Urllib3HttpConnection and RequestsHttpConnection pool implementations.
  • Document how and when to switch between connection pools.
  • Remove RequestsHttpConnection from the documentation when used as a default.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Oct 12, 2023

Codecov Report

Merging #535 (b3a5ed5) into main (84ac172) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #535      +/-   ##
==========================================
+ Coverage   70.72%   70.77%   +0.05%     
==========================================
  Files          83       83              
  Lines        7852     7857       +5     
==========================================
+ Hits         5553     5561       +8     
+ Misses       2299     2296       -3     
Files Coverage Δ
opensearchpy/connection/http_urllib3.py 86.02% <100.00%> (+0.30%) ⬆️
opensearchpy/transport.py 92.16% <100.00%> (+0.14%) ⬆️

... and 1 file with indirect coverage changes

@dblock dblock force-pushed the pool-maxsize branch 3 times, most recently from 36fb442 to 4c1f049 Compare October 12, 2023 19:28
Signed-off-by: dblock <dblock@amazon.com>
Signed-off-by: dblock <dblock@amazon.com>
guides/connection_classes.md Outdated Show resolved Hide resolved
use_ssl = True,
verify_certs = False,
ssl_show_warn = False,
connection_class = Urllib3HttpConnection
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make Urllib3HttpConnection default in code unless a connection_class is explicitly passed in?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's already default here:

DEFAULT_CONNECTION_CLASS = Urllib3HttpConnection

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I missed it, so only the documentation was a bit misleading.

Signed-off-by: dblock <dblock@amazon.com>
Copy link
Collaborator

@VachaShah VachaShah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dblock!

@navneet1v
Copy link

@dblock can we add the recommendation somewhere in doc on what HTTPClient to use? with the benchmarks you have run.

@dblock
Copy link
Member Author

dblock commented Oct 12, 2023

@dblock can we add the recommendation somewhere in doc on what HTTPClient to use? with the benchmarks you have run.

In this PR I had added "We recommend you use the default, unless your application is standardized on requests." LMK if you think it should say something else.

@dblock dblock merged commit 7a638cd into opensearch-project:main Oct 12, 2023
54 checks passed
@dblock dblock deleted the pool-maxsize branch October 12, 2023 22:55
Djcarrillo6 added a commit to Djcarrillo6/opensearch-py that referenced this pull request Oct 14, 2023
Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com>

Updated CHANGELOG

Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com>

Updated CHANGELOG & link to sample.

Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com>

updated changelog (opensearch-project#522)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Bump version to 2.3.2 (opensearch-project#524)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Fix: typos. (opensearch-project#526)

* Fix: typo.

Signed-off-by: dblock <dblock@amazon.com>

* Fix: typo.

Signed-off-by: dblock <dblock@amazon.com>

* Fixed its.

Signed-off-by: dblock <dblock@amazon.com>

* Added Visual Code settings to .gitignore.

Signed-off-by: dblock <dblock@amazon.com>

* Added loop type for async client.

Signed-off-by: dblock <dblock@amazon.com>

---------

Signed-off-by: dblock <dblock@amazon.com>

Modified generator to generate api deprecation warnings (opensearch-project#527)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Generate cat client from API specs (opensearch-project#529)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Generate cluster client from API specs (opensearch-project#530)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Added new guide & sample module for using index templates. (opensearch-project#531)

Added index_template guide and sample

Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com>

Removed EOL Python3.5 & bumped urllib3 version to patch security vulnerability (opensearch-project#533)

Updated CHANGELOG with pull #

Updated CHANGELOG with pull #

Updated CHANGELOG removed section.

Updated CHANGELOG removed section again

Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com>

Align pool_maxsize for different connection pool implementations. (opensearch-project#535)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <dblock@amazon.com>

* Document connection classes and settings.

Signed-off-by: dblock <dblock@amazon.com>

* Undo change in async for backwards compatibility.

Signed-off-by: dblock <dblock@amazon.com>

* Fix: typo.

Signed-off-by: dblock <dblock@amazon.com>

---------

Signed-off-by: dblock <dblock@amazon.com>

Add micro benchmarks. (opensearch-project#537)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <dblock@amazon.com>

* Added benchmarks.

Signed-off-by: dblock <dblock@amazon.com>

* Multi-threaded vs. async benchmarks.

Signed-off-by: dblock <dblock@amazon.com>

* Set pool size to the number of threads.

Signed-off-by: dblock <dblock@amazon.com>

* Added sync/async benchmark.

Signed-off-by: dblock <dblock@amazon.com>

* Report client-side latency.

Signed-off-by: dblock <dblock@amazon.com>

* Various updates to benchmarks, demonstrating threading improves throughput.

Signed-off-by: dblock <dblock@amazon.com>

* Bench info.

Signed-off-by: dblock <dblock@amazon.com>

* Fixup format.

Signed-off-by: dblock <dblock@amazon.com>

* Undo async maxsize.

Signed-off-by: dblock <dblock@amazon.com>

* Moved benchmarks folder.

Signed-off-by: dblock <dblock@amazon.com>

* Updated documentation and project description.

Signed-off-by: dblock <dblock@amazon.com>

---------

Signed-off-by: dblock <dblock@amazon.com>
roma2023 pushed a commit to roma2023/opensearch-py that referenced this pull request Dec 28, 2023
…ensearch-project#535)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <dblock@amazon.com>

* Document connection classes and settings.

Signed-off-by: dblock <dblock@amazon.com>

* Undo change in async for backwards compatibility.

Signed-off-by: dblock <dblock@amazon.com>

* Fix: typo.

Signed-off-by: dblock <dblock@amazon.com>

---------

Signed-off-by: dblock <dblock@amazon.com>
Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants