Added host failover

[Arttii]

Added a basic mechanism for host failover. You just need to supply a list of other hosts to try. Its a simple solution, but seems to work ok.

[wbolster]

I think this is not worth the complexity in HappyBase, since it's a complex problem that isn't solved properly using a simple for loop. See e.g. #21 (comment) for discussion.

(Furthermore, your code has typos in variable names so I doubt it's actually tested...)

[Arttii]

I do not see what you mean by complexity. This is a very simple solution to the problem. Not the best one, but it works, which is a nicer alternative to having your job just fail if the Thriftserver is dead.
And handling this externally was not the best solution I thought.

Also I do not know what typos you are referring to. The first commit was done from a wrong dir. This code is currently running in multiple jobs.

Just thought it was worth proposing.I am not sold on using a load balancer for simple host rotation, which is the problem I was trying to solve here.I might be wrong though. Anyhow thanks for the consideration I am closing this and will try using HaProxy or Nginx for this.

[wbolster]

There's a few things to note here:

• This PR treats one host as the "preferred" host, and uses the others only as a fallback. This is not a good approach to fail-over or load-balancing. In short, simple fail-over to backup hosts in Connection.open() doesn't properly solve any high availability or load balancing problems. Dedicated solutions such as haproxy do. See the comment and discussion referenced earlier. Maybe @chase-seibert can chime in here and explain?
• Happybase is only a consumer of the Thrift API and its exceptions; it shouldn't raise exceptions "owned" by other libraries (which this PR does), i.e. it touches another's "private parts".

Re. the typos: falis+=1 is a code path that clearly has never been executed.

[wbolster]

Also, the functionality as proposed in this PR can also be implemented outside of Happybase by simply creating a function that wraps Connection.open().

[Arttii]

Yes I understand now, there was not a lot of information on these topics. I looked into HAProxy and I am probably migrating our solution to that now. Thanks for the clarification. Also any thoughts or experiences on using Zookeeper for this?

[wbolster]

Sorry @Arttii can't help you; no experience. But perhaps @chase-seibert can join the discussion?

[chase-seibert]

I would just echo again that an external load balancer is a more complete solution to the same problem. Not only do you get fail-over, but you can also distribute load evenly between two hot hosts, versus having one hot and one passive.

Even if you tried to do that in code, you would run into the problem of some hosts maintaining long-lived connections to a particular server. Other hosts would not be able to take that information into account to implement a truly least-connection load balancing scheme.

[Arttii]

Yes thank you, I realize now I was kinda missing the point with my solution. I implemented the failover with HAProxy now. Works quite well, thanks for the insight.

---

From: Chase Seibertmailto:notifications@github.com
Sent: ‎18.‎02.‎2015 18:31
To: wbolster/happybasemailto:happybase@noreply.github.com
Cc: Arttiimailto:artyom.topchyan@live.com
Subject: Re: [happybase] Added host failover (#81)

I would just echo again that an external load balancer is a more complete solution to the same problem. Not only do you get fail-over, but you can also distribute load evenly between two hot hosts, versus having one hot and one passive.

Even if you tried to do that in code, you would run into the problem of some hosts maintaining long-lived connections to a particular server. Other hosts would not be able to take that information into account to implement a truly least-connection load balancing scheme.

---

Reply to this email directly or view it on GitHub:
#81 (comment)