-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection pooling #21
Comments
Tests would be implemented by mocking out the Connection class so that no actual sockets need to be opened. |
I have a prototype solution working. I'm going to battle test it on production for a week before I come with an official patch. import time
import random
import contextlib
import happybase
from socketpool import ConnectionPool
from socketpool.conn import TcpConnector
class HappybaseConnectionPool(object):
''' singleton to share a connection pool per process '''
pool = None
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(HappybaseConnectionPool, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, host, **options):
if not self.pool:
options['host'] = host
self.pool = ConnectionPool(
factory=HappybaseConnector,
max_size=options.get('max_size', 10),
options=options,
)
def connection(self, **options):
return self.pool.connection(**options)
@contextlib.contextmanager
def table(self, table_name):
with self.pool.connection() as connector:
yield connector.table(table_name)
class HappybaseConnector(TcpConnector):
def __init__(self, host, port, pool=None, **kwargs):
self.host = host
self.port = port
self.connection = happybase.Connection(self.host, self.port)
self._connected = True
# use a 'jiggle' value to make sure there is some
# randomization to expiry, to avoid many conns expiring very
# closely together.
self._life = time.time() - random.randint(0, 10)
self._pool = pool
self.logging = kwargs.get('logging')
def is_connected(self):
if self._connected and self.connection.transport.isOpen():
try:
# isOpen is unreliable, actually try to do something
self.connection.tables()
return True
except:
pass
return False
def handle_exception(self, exception):
if self.logging:
self.logging.error(exception)
else:
print exception
def invalidate(self):
self.connection.close()
self._connected = False
self._life = -1
def open(self):
pass
def close(self):
self.release()
def __getattr__(self, name):
if name in ['table', 'tables', 'create_table', 'delete_table',
'enable_table', 'disable_table', 'is_table_enabled', 'compact_table']:
return getattr(self.connection, name)
else:
raise AttributeError(name) You use it like this: pool = HappybaseConnectionPool('localhost', '9090')
with pool.connection() as connection:
connection.create_table('foobar') |
shouldn't this support multiple thrift servers? pycassa has support for that. |
I'm hitting a bunch of Thrift instances behind a load balancer, which I think makes sense to run externally. If we did load balancing in process, it would mean implementing options like round-robin, least connection, etc. Not sure how you would deal with least-connection between various python processes; they would all be keeping their own connection counts, exclusive of each other. I think it's better left to an external load balancer. |
well long term you could have it aware of regionserver splits for performance. Netflix has a cassandra client that does this http://techblog.netflix.com/2012/01/announcing-astyanax.html |
I think I agree with Chase. Connection pooling is hard, and it adds quite a bit of complexity. Other solutions like load balancers are actually designed to handle this problem on a network level (instead of a process level). |
I actually had a go at this since it also seems the way to go for multi-threading support. I've pushed my current code to a feature branch, which can be seen here: https://github.com/wbolster/happybase/commits/connection-pool Copy/paste from the (w-i-p) docs:
What do you think? I'd appreciate comments/flames/feedback! |
Looks good to me. Probably makes more sense than including a dependency. It would be cool if there was a way of getting a single pool object w/o passing it around everywhere. That's what I'm using a singleton for; but I suppose you could always layer that on top of what you have. |
Okay, I have landed a Connection Pool implementation in the master branch. Please try it out. Comments on the design and API are most welcome. See the API docs at https://happybase.readthedocs.org/en/latest/api.html#happybase.ConnectionPool for more information and example usage. I'm leaving this ticket open since I need to refactor the tutorial/user guide to incorporate some information on the connection pool. |
Fwiw, the feature branch is gone now that this feature has landed on master. I'll need to expand the docs (working on it already) before I consider this issue closed. I'll also cook up a 0.5 release soonish with this feature and some other unreleased enhancements from the master branch. |
Oh, I forgot to mention that I have (privately) received positive test reports about the connection pool, so I have confidence the current implementation is ready for public release. :-) |
HappyBase 0.5 is out! https://twitter.com/wbolster/status/338034468780662784 |
Connection pooling should use a separate pool API, not be completely embedded inside the happybase.Connection class.
Goal:
When using happybase in the context of a web application, it would be useful to re-use connections between page requests. A connection pooling solution should take a MIN, MAX and IDLE count as parameters, and open connections as needed by the application.
Inspiration:
API:
The pool could be instantiated manually per-process in the setup flow of a in a web server framework. For example, in Django, this could be done in settings.py with AUTOCONNECT=False so that connections are not established until the first calls to get_connection().
Retries:
If a connection cannot be established, or is terminated (ie by a timeout), it would attempt to re-establish after RETRY_MS milliseconds.
Errors:
ConnectionPool could thrown an error right away if it can't establish MIN connections immediately. Otherwise, a call to pool.get_connection, will raise various exceptions for things like pool exhaustion (if BLOCK=False), cannot connect to Thrift endpoint, etc.
Other thoughts:
I can't see how we could support connection pooling between multiple python processes except by implementing a separate process to connect through, similar to pgpool.
The text was updated successfully, but these errors were encountered: