Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not load page: MasterNotRunningException #19

Open
chenhsiu opened this issue May 6, 2013 · 26 comments
Open

can not load page: MasterNotRunningException #19

chenhsiu opened this issue May 6, 2013 · 26 comments

Comments

@chenhsiu
Copy link

chenhsiu commented May 6, 2013

I have setup conf/hbase-site.xml properly, but get MasterNotRunningException as follows. Any clues?

! @6e7mlmi15 - Internal server error, for request [GET /] ->

play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[MasterNotRunningException: Retried 10 times]]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:134) [play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115) [play_2.9.1-2.0.4.jar:2.0.4]
at akka.actor.Actor$class.apply(Actor.scala:318) [akka-actor-2.0.2.jar:2.0.2] at play.core.ActionInvoker.apply(Invoker.scala:113) [play_2.9.1-2.0.4.jar:2.0.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:626) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.Mailbox.run(Mailbox.scala:179) [akka-actor-2.0.2.jar:2.0.2]
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516) [akka-actor-2.0.2.jar:2.0.2] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [akka-actor-2.0.2.jar:2.0.2]
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [akka-actor-2.0.2.jar:2.0.2]
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:120) ~[hbase-0.90.5.jar:0.90.5]
at models.HBase$class.withAdmin(HBase.scala:42) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.hbase090.HBase090.withAdmin(HBase090.scala:9) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.hbase090.HBase090.eachRegionServer(HBase090.scala:11) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at models.Region$.all(Region.scala:71) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at controllers.Application$$anonfun$index$1.apply(Application.scala:13) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at controllers.Application$$anonfun$index$1.apply(Application.scala:12) ~[hannibal_2.9.1-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at play.api.mvc.Action$$anon$1.apply(Action.scala:170) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.utils.Threads$.withContextClassLoader(Threads.scala:17) ~[play_2.9.1-2.0.4.jar:2.0.4]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:125) [play_2.9.1-2.0.4.jar:2.0.4]
... 11 common frames omitted

@chenhsiu
Copy link
Author

chenhsiu commented May 6, 2013

my logs/log4j.log is empty, how can I turn on the debug log?

@meniku
Copy link
Collaborator

meniku commented May 7, 2013

you can try to change the loglevel from "WARN" to "INFO" in the file "conf/log4j.xml" (line 22)

I recently received a E-Mail with a similar problem. I gave the following suggestions, but did never receive an answer if anything helped:

A few generic questions without looking at the logfiles:

  1. Are you compiling Hannibal right on the host where you run it or do you use the "build_package" script? Please let me know if that's the case, maybe there is still some problem with the script.
  2. Are you running the latest Hannibal version from the master branch? Maybe the problem is already fixed ;-) To update, simply run 'git pull' inside your Hannibal directory.

From the logfile output, I can't exactly tell you what's wrong. I see two possible reasons:
3. the HBase configuration could be wrong (doublecheck that you have the correct zookeeper-quorum in conf/hbase-site.xml).
4. There could be a problem connecting to Zookeeper. Please try to issue this command from your hannibal-host: 'echo "stat" | nc zookeeper-hostname 2181' (for each of your zookeeper-nodes as configured in hbase-site.xml, replace 'zookeeper-hostname' with the correct hostname). The command should immediately return some statistics about zookeeper.

@gdtm86
Copy link

gdtm86 commented May 7, 2013

Hi Nils,

I am seeing a similar issue.

-Guru Medasani

On Tue, May 7, 2013 at 7:58 AM, NiKu notifications@github.com wrote:

you can try to change the loglevel from "WARN" to "INFO" in the file
"conf/log4j.xml" (line 22)

I recently received a E-Mail with a similar problem. I gave the following
suggestions, but did never receive an answer if anything helped:

A few generic questions without looking at the logfiles:

  1. Are you compiling Hannibal right on the host where you run it or do you
    use the "build_package" script? Please let me know if that's the case,
    maybe there is still some problem with the script.
  2. Are you running the latest Hannibal version from the master branch?
    Maybe the problem is already fixed ;-) To update, simply run 'git pull'
    inside your Hannibal directory.

From the logfile output, I can't exactly tell you what's wrong. I see two
possible reasons:
3. the HBase configuration could be wrong (doublecheck that you have the
correct zookeeper-quorum in conf/hbase-site.xml).
4. There could be a problem connecting to Zookeeper. Please try to issue
this command from your hannibal-host: 'echo "stat" | nc zookeeper-hostname
2181' (for each of your zookeeper-nodes as configured in hbase-site.xml,
replace 'zookeeper-hostname' with the correct hostname). The command should
immediately return some statistics about zookeeper.


Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17540333
.

@chenhsiu
Copy link
Author

chenhsiu commented May 7, 2013

@meniku

  1. I use the create_package script
  2. yes. the latest one
  3. I see following logs in log4j.log, after turn on INFO debug level

2013-05-07 16:12:10,228 INFO [application-akka.actor.default-dispatcher-1] ZooKeeper - Session: 0x23e564781076263 closed
2013-05-07 16:12:10,229 INFO [application-akka.actor.default-dispatcher-1] ZooKeeper - Initiating client connection, connectString=spn-d-hdd1.sjdc:2181,spn-d-hdn2.sjdc:2181,spn-d-hdn1.sjdc:2181 sessionTimeout=120000 watcher=hconnection
2013-05-07 16:12:10,230 INFO [application-akka.actor.default-dispatcher-1-EventThread] ClientCnxn - EventThread shut down
2013-05-07 16:12:10,231 INFO [application-akka.actor.default-dispatcher-1-SendThread()] ClientCnxn - Opening socket connection to server spn-d-hdd1.sjdc/10.31.56.35:2181
2013-05-07 16:12:10,232 INFO [application-akka.actor.default-dispatcher-1-SendThread(spn-d-hdd1.sjdc:2181)] ClientCnxn - Socket connection established to spn-d-hdd1.sjdc/10.31.56.35:2181, initiating session
2013-05-07 16:12:10,234 INFO [application-akka.actor.default-dispatcher-1-SendThread(spn-d-hdd1.sjdc:2181)] ClientCnxn - Session establishment complete on server spn-d-hdd1.sjdc/10.31.56.35:2181, sessionid = 0x23e564781076264, negotiated timeout = 120000

@chenhsiu
Copy link
Author

chenhsiu commented May 7, 2013

@meniku BTW, does hannibal support Kerberos authentication?

@gdtm86
Copy link

gdtm86 commented May 7, 2013

Chris,

Which HBase version are you using?

Best Regards,
Guru Medasani

On Tue, May 7, 2013 at 11:15 AM, Chris Huang notifications@git.luolix.topwrote:

@meniku https://github.com/meniku BTW, does hannibal support Kerberos
authentication?


Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17552503
.

@meniku
Copy link
Collaborator

meniku commented May 8, 2013

@gdtm86 From the exception trace I can see that Hannibal is compiled for HBase 0.90

I wonder whether setting the hbase.master in conf/hbase-site.xml could help?

<property>
<name>hbase.master</name>
<value>master-server:60000</value>
</property>

@gdtm86
Copy link

gdtm86 commented May 8, 2013

I tied this and still no luck. Seeing the same errors.

-Guru

On Wed, May 8, 2013 at 7:58 AM, NiKu notifications@github.com wrote:

@gdtm86 https://github.com/gdtm86 From the exception trace I can see
that Hannibal is compiled for HBase 0.90

I wonder whether setting the hbase.master in conf/hbase-site.xml could
help?

hbase.master
master-server:60000


Reply to this email directly or view it on GitHubhttps://github.com//issues/19#issuecomment-17603767
.

@chenhsiu
Copy link
Author

chenhsiu commented May 9, 2013

@gdtm86 I use 0.92
@meniku does hannibal support Kerberos authentication? My Hadoop/HBase cluster has Kerberos enabled. Is that the reason?

@meniku
Copy link
Collaborator

meniku commented May 13, 2013

@chenhsiu your stacktrace indicates that you compile Hannibal for HBase 0.90, for the packaged version, please use the following command to create the package

 HANNIBAL_HBASE_VERSION=0.92 ./create_package

I actually don't know whether Hannibal supports Kerberos authentication since I never used it. From a quick look in the HBase Documentation it seams that you need to setup your hbase-site.xml properly.

@meniku
Copy link
Collaborator

meniku commented May 16, 2013

I just had a similiar issue when I built Hannibal for the wrong version. It did not work directly after I changed the version with HANNIBAL_HBASE_VERSION. I had to do a clean prior to rebuild Hannibal.

The following made it work for me:

./sbt clean
HANNIBAL_HBASE_VERSION=0.92 ./create_package

Does this help?

@keyboardfann
Copy link

Hi, I test it on HBase 0.94 without security and HANNIBAL work fine.
I test it on HBase 0.94 with security HANNIBAL can't work.

Could you give me some idea ?

Below is the information:

TEST CASE 1:
hbase-site.xml without SecureRpcEngine config, hannibal service can start . When it want to update RegionInfo, it show MasterNotRunningException and hmaster log show Incorrect header or version mismatch. It seems RPC version mismatch.

hbase-site.xml :

[root@ci-test-1 ~]# cat hannibal/conf/hbase-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.cluster.distibuted</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ci-test-1.client.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ci-test-1.client.org</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
</configuration>

[hbase@ci-test-1 hannibal]$ ./start

starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /home/hbase/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/home/hbase/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 23711
[debug] c.j.b.BoneCPDataSource - JDBC URL = jdbc:h2:data/metrics.h2, Username = null, partitions = 1, max (per partition) = 30, min (per partition) = 5, helper threads = 3, idle max age = 10 min, idle test period = 1 min
[warn] c.j.b.BoneCPConfig - JDBC username was not set in config!
[warn] c.j.b.BoneCPConfig - JDBC password was not set in config!
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from ehcache.xml found in the classpath: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from URL: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from InputStream
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xmlns:xsi
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xsi:noNamespaceSchemaLocation
[debug] n.s.e.CacheManager - Creating new CacheManager with default config
[debug] n.s.e.u.PropertyUtil - propertiesString is null.
[debug] n.s.e.CacheManager - No disk store path defined. Skipping disk store path conflict test.
[debug] n.s.e.c.ConfigurationHelper - No CacheManagerEventListenerFactory class specified. Skipping...
[debug] n.s.e.Cache - No BootstrapCacheLoaderFactory class specified. Skipping...
[debug] n.s.e.Cache - CacheWriter factory not configured. Skipping...
[debug] n.s.e.c.ConfigurationHelper - No CacheExceptionHandlerFactory class specified. Skipping...
[debug] n.s.e.s.MemoryStore - Initialized net.sf.ehcache.store.NotifyingMemoryStore for play
[debug] n.s.e.Cache - Initialised cache: play
[debug] n.s.e.c.ConfigurationHelper - CacheDecoratorFactory not configured for defaultCache. Skipping for 'play'.
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@257807a, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Wed Dec 04 11:06:37 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 94ms... (Wed Dec 04 11:06:37 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Wed Dec 04 11:06:39 UTC 2013)
[ERROR] [12/04/2013 11:14:24.139] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] Retried 10 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:122)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

[info] application - start initialize Logfile... (Wed Dec 04 11:14:24 UTC 2013)
[ERROR] [12/04/2013 11:22:07.549] [application-akka.actor.default-dispatcher-1] [akka://application/user/updateMetricsActor] Retried 10 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:122)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.LogFile$$anonfun$discoverLogFileUrlPattern$1.apply$mcV$sp(LogFile.scala:119)
    at scala.util.control.Breaks.breakable(Breaks.scala:35)
    at models.LogFile$.discoverLogFileUrlPattern(LogFile.scala:118)
    at models.LogFile$.init(LogFile.scala:156)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$3.apply$mcV$sp(UpdateMetricsActor.scala:85)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:84)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

HBase Master log:

[2013-12-04 11:26:17,384][IPC Reader 9 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7792 got version 3 expected version 4
[2013-12-04 11:26:18,409][IPC Reader 0 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7793 got version 3 expected version 4
[2013-12-04 11:26:19,412][IPC Reader 1 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7794 got version 3 expected version 4
[2013-12-04 11:26:20,420][IPC Reader 2 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7795 got version 3 expected version 4
[2013-12-04 11:26:22,433][IPC Reader 3 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7796 got version 3 expected version 4
[2013-12-04 11:26:24,445][IPC Reader 4 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7797 got version 3 expected version 4
[2013-12-04 11:26:28,486][IPC Reader 5 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7798 got version 3 expected version 4
[2013-12-04 11:26:32,532][IPC Reader 6 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7799 got version 3 expected version 4
[2013-12-04 11:26:40,602][IPC Reader 7 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7800 got version 3 expected version 4

TEST CASE 2:
hbase-site.xml with SecureRpcEngine config, hannibal can't start and show SecureRpcEngine class not found.

[root@ci-test-1 ~]# cat hannibal/conf/hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.cluster.distibuted</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ci-test-1.client.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ci-test-1.client.org</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
   <property>
    <name>hbase.rpc.engine</name>
    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
</configuration>

[hbase@tmh6-ci-test-1 hannibal]$ ./start

starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /home/hbase/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/home/hbase/hannibal/)
(Starting server. Type Ctrl+D to exit logs, the server will remain in background)
Play server process ID is 25281
[debug] c.j.b.BoneCPDataSource - JDBC URL = jdbc:h2:data/metrics.h2, Username = null, partitions = 1, max (per partition) = 30, min (per partition) = 5, helper threads = 3, idle max age = 10 min, idle test period = 1 min
[warn] c.j.b.BoneCPConfig - JDBC username was not set in config!
[warn] c.j.b.BoneCPConfig - JDBC password was not set in config!
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from ehcache.xml found in the classpath: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from URL: jar:file:/home/hbase/.ivy2/cache/play/play_2.9.1/jars/play_2.9.1-2.0.4.jar!/ehcache.xml
[debug] n.s.e.c.ConfigurationFactory - Configuring ehcache from InputStream
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xmlns:xsi
[debug] n.s.e.c.BeanHandler - Ignoring ehcache attribute xsi:noNamespaceSchemaLocation
[debug] n.s.e.CacheManager - Creating new CacheManager with default config
[debug] n.s.e.u.PropertyUtil - propertiesString is null.
[debug] n.s.e.CacheManager - No disk store path defined. Skipping disk store path conflict test.
[debug] n.s.e.c.ConfigurationHelper - No CacheManagerEventListenerFactory class specified. Skipping...
[debug] n.s.e.Cache - No BootstrapCacheLoaderFactory class specified. Skipping...
[debug] n.s.e.Cache - CacheWriter factory not configured. Skipping...
[debug] n.s.e.c.ConfigurationHelper - No CacheExceptionHandlerFactory class specified. Skipping...
[debug] n.s.e.s.MemoryStore - Initialized net.sf.ehcache.store.NotifyingMemoryStore for play
[debug] n.s.e.Cache - Initialised cache: play
[debug] n.s.e.c.ConfigurationHelper - CacheDecoratorFactory not configured for defaultCache. Skipping for 'play'.
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@78497062, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Wed Dec 04 11:28:40 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 133ms... (Wed Dec 04 11:28:41 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Wed Dec 04 11:28:42 UTC 2013)
[ERROR] [12/04/2013 11:28:44.272] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProtocolEngine(HBaseRPC.java:114)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:335)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:312)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:364)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:682)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:110)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.SecureRpcEngine
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
    ... 24 more

@tzzhc4
Copy link

tzzhc4 commented Dec 10, 2013

Was there every an answer about Hannibal supporting Kerberos?

@meniku
Copy link
Collaborator

meniku commented Dec 11, 2013

@tzzhc4
Sorry that I never answered this. Unfortunately I have no idea how to set up a test environment for Kerberos. Maybe it would be better if someone else could test this, maybe someone who already has Kerberos running. Possible the best thing would be if we would have a vagrant setup with kerberos enabled?

@keyboardfann My ideas:

case 1

The log message:

[2013-12-04 11:26:17,384][IPC Reader 9 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:7792 got version 3 expected version 4
...

Could indicate that the used hadoop version 0.20.205.0 by Hannibal is maybe too old. Could you try to change the used Version in project/Build.scala from

      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-core" % "0.20.205.0",
        "org.apache.hbase" % "hbase" % "0.94.3"
      )

to

      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-common" % "0.22.0",
        "org.apache.hbase" % "hbase" % "0.94.14"
      )

and then rebuild the project? I think this is the solution to your first problem and this ticket. Thank you for providing the log-messages.

case 2

I really don't know why the class SecureRpcEngine can not be found. The only thing I found out is, that your second configuration looks not the way it is documented for HBase 0.94.

Instead of

...
   <property>
    <name>hbase.rpc.engine</name>
    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
...

it is documented like

...
      <property>
        <name>hbase.security.authentication</name>
        <value>kerberos</value>
      </property> 

      <property>
        <name>hbase.rpc.protection</name>
        <value>privacy</value>
      </property>
...

Are you sure that your configuration is correct?

@gshively
Copy link

The problem with the resolution of SecureRpcEngine seems to be that HBase 0.94 has a security profile. Seems as though the SecureRpcEngine is defined only when the security profile is defined. I tried to add the 0.94.14-security to the project/Build.scala, where it couldn't find it. But changing back to the 0.94.14 worked - and if I replace the hbase-0.94.14.jar with my hbase-0.94.14-security.jar - SecureRpcEngine is resolved.

I also had to upgrade the hadoop-core, which I did to 1.0.4 prior to seeing your message. Looks like I'm still getting errors, but not sure if it is with my JAAS config or something else.

@keyboardfann
Copy link

Dear @meniku ,
About Case 1, i clone the newest code , modify config and rebuild it , hannibal don't exit because MasterNotRunningException , but it still have Incorrect header or version in HMaster log.

Case 1

[root@ci-test-1 hannibal]# ./start 
starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /root/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/root/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 6543
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@dc67248, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Thu Dec 12 09:28:30 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 79ms... (Thu Dec 12 09:28:30 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Thu Dec 12 09:28:32 UTC 2013)

HMaster log:

[2013-12-12 09:28:57,279][IPC Reader 3 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:26010 got version 3 expected version 4
[2013-12-12 09:28:58,291][IPC Reader 4 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:26011 got version 3 expected version 4
[2013-12-12 09:29:00,313][IPC Reader 5 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:26012 got version 3 expected version 4
[2013-12-12 09:29:02,332][IPC Reader 6 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:26013 got version 3 expected version 4
[2013-12-12 09:29:06,375][IPC Reader 7 on port 8100][WARN ][org.apache.hadoop.ipc.SecureServer]: Incorrect header or version mismatch from 10.1.145.43:26014 got version 3 expected version 4

About case 2 ,
The hbase-site.xml is using on our env , hbase work fine with security. So I don't seem the config have problem.
Here is the config:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ci-test-1.client.tw.trendnet.org:8020/hbase</value>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.security.authorization</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.security.authentication</name>
    <value>kerberos</value>
  </property>
  <property>
    <name>hbase.master.port</name>
    <value>8100</value>
  </property>
  <property>
    <name>hbase.master.info.port</name>
    <value>8110</value>
  </property>
  <property>
    <name>hbase.regionserver.port</name>
    <value>8120</value>
  </property>
  <property>
    <name>hbase.regionserver.info.port</name>
    <value>8130</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ci-test-1.client.tw.trendnet.org,</value>
  </property>
  <property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
  </property>
  <property>
    <name>hbase.master.keytab.file</name>
    <value>/etc/hbase/conf/hbase.keytab</value>
  </property>   
  <property>
    <name>hbase.master.kerberos.principal</name>
    <value>hbase/_HOST@test.com</value>
  </property>
  <property>
    <name>hbase.master.kerberos.https.principal</name>
    <value>host/_HOST@test.com</value>
  </property>
  <property>
    <name>hbase.regionserver.keytab.file</name>
    <value>/etc/hbase/conf/hbase.keytab</value>
  </property>   
  <property>
    <name>hbase.regionserver.kerberos.principal</name>
    <value>hbase/_HOST@test.com</value>
  </property>
  <property>
    <name>hbase.regionserver.kerberos.https.principal</name>
    <value>host/_HOST@test.com</value>
  </property>
  <property>
    <name>hbase.coprocessor.master.classes</name>
    <value>org.apache.hadoop.hbase.security.access.AccessController</value>
  </property>
  <property>
    <name>hbase.coprocessor.region.classes</name>
    <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
  </property>
  <property>
    <name>hbase.rpc.engine</name>
    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.kerberos.removeHostFromPrincipal</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.kerberos.removeRealmFromPrincipal</name>
    <value>true</value>
  </property>  
</configuration>

@meniku
Copy link
Collaborator

meniku commented Dec 12, 2013

Case 1

Weird, I thought Hadoop 0.22 should have IPC version 4 already. Maybe you should try 0.23.9, the latest Hadoop 1 version I found .

Case 2

Sorry, all I can tell is that those class you reference org.apache.hadoop.hbase.ipc.SecureRpcEngine does not seam to be included in the HBase client artifact. Maybe it's included in another artifact? One guess is that it's only available to the HBase server? Is the hbase.rpc.engine really needed for the HBase client or is it only required for IPC between RegionServers?

@keyboardfann
Copy link

Dear @meniku
Thank you for the reply. I remove org.apache.hadoop.hbase.ipc.SecureRpcEngine and try . After change 0.22 to 0.23.9 , rebuild and run . Incorrect header or version problem resolved, but the MasterNotRunningException still happen.

 case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-common" % "0.23.9",
        "org.apache.hbase" % "hbase" % "0.94.14"
      )
[root@ci-test-1 hannibal]# ./start 
starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[root@ci-test-1 hannibal]# kinit -kt /etc/hbase/conf/hbase.keytab hbase/`hostname -f`
[root@ci-test-1 hannibal]# ./start 
starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /root/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/root/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 4410
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@dc67248, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Thu Dec 12 18:40:40 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 145ms... (Thu Dec 12 18:40:41 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Thu Dec 12 18:40:42 UTC 2013)
[error] application - 

! @6ghd0e36o - Internal server error, for request [GET /servers] ->

play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[MasterNotRunningException: Retried 14 times]]
    at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:134) [play_2.9.1-2.0.4.jar:2.0.4]
    at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115) [play_2.9.1-2.0.4.jar:2.0.4]
    at akka.actor.Actor$class.apply(Actor.scala:318) [akka-actor-2.0.2.jar:2.0.2]
    at play.core.ActionInvoker.apply(Invoker.scala:113) [play_2.9.1-2.0.4.jar:2.0.4]
    at akka.actor.ActorCell.invoke(ActorCell.scala:626) [akka-actor-2.0.2.jar:2.0.2]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197) [akka-actor-2.0.2.jar:2.0.2]
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Retried 14 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:139) ~[hbase-0.94.14.jar:0.94.14]
    at models.hbase.HBase$class.withAdmin(HBase.scala:40) ~[classes/:na]
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10) ~[classes/:na]
    at models.hbase.HBase$class.eachTableDescriptor(HBase.scala:31) ~[classes/:na]
    at models.hbase092.HBase092.eachTableDescriptor(HBase092.scala:10) ~[classes/:na]
    at models.Table$.all(Table.scala:16) ~[classes/:na]
[ERROR] [12/12/2013 19:47:53.583] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] Retried 14 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 14 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:139)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

[info] application - start initialize Logfile... (Thu Dec 12 19:47:53 UTC 2013)
[error] application - 

! @6ghd0e36p - Internal server error, for request [GET /servers] ->

play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[ZooKeeperConnectionException: An error is preventing HBase from connecting to ZooKeeper]]
    at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:134) [play_2.9.1-2.0.4.jar:2.0.4]
    at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115) [play_2.9.1-2.0.4.jar:2.0.4]
    at akka.actor.Actor$class.apply(Actor.scala:318) [akka-actor-2.0.2.jar:2.0.2]
    at play.core.ActionInvoker.apply(Invoker.scala:113) [play_2.9.1-2.0.4.jar:2.0.4]
    at akka.actor.ActorCell.invoke(ActorCell.scala:626) [akka-actor-2.0.2.jar:2.0.2]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197) [akka-actor-2.0.2.jar:2.0.2]
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: An error is preventing HBase from connecting to ZooKeeper
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1509) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:713) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:786) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:127) ~[hbase-0.94.14.jar:0.94.14]
    at models.hbase.HBase$class.withAdmin(HBase.scala:40) ~[classes/:na]
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10) ~[classes/:na]
Caused by: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@c33aa9b closed
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1503) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:713) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:786) ~[hbase-0.94.14.jar:0.94.14]
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:127) ~[hbase-0.94.14.jar:0.94.14]
    at models.hbase.HBase$class.withAdmin(HBase.scala:40) ~[classes/:na]
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10) ~[classes/:na]
[ERROR] [12/12/2013 20:55:01.293] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] Retried 14 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 14 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:139)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.LogFile$$anonfun$discoverLogFileUrlPattern$1.apply$mcV$sp(LogFile.scala:119)
    at scala.util.control.Breaks.breakable(Breaks.scala:35)
    at models.LogFile$.discoverLogFileUrlPattern(LogFile.scala:118)
    at models.LogFile$.init(LogFile.scala:156)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$3.apply$mcV$sp(UpdateMetricsActor.scala:85)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:84)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
hbase(main):001:0> scan 't1'
ROW                                                          COLUMN+CELL                                                                                                                                                                    
 r1                                                          column=f1:, timestamp=1386873573899, value=vvvvv            

@gshively
Copy link

I haven't attempted case 1, but regarding the case 2 I resolved the SecureRPC by the following:

  1. Change as suggested the project/Build.scala with an upgraded version(0.94.14).
  2. Run the ./build process to retrieve all of the ivy artifacts.
  3. mkdir -p $HOME/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/ivys
  4. mkdir -p $HOME/.ivy2/local/org.apache.hbase/hbase/jars
  5. Place hbase-0.94.14-security.jar into $HOME/.ivy2/local/org.apache.hbase/hbase/jars
  6. cp $HOME/.ivy2/cache/org.apache.hbase/ivy-0.94.xml $HOME/.ivy2/local/org.apache.hbase/0.94.14-security/ivys/ivy.xml
  7. Change references for 0.94.14 to 0.94.14-security in $HOME/.ivy2/local/org.apache.hbase/0.94.14-security/ivys/ivy.html
  8. Change the suggested project/Build.scala from the earlier change 0.94.14 to 0.94.14-security
  9. Build the project for the 0.94 version

I'm sure there are easier ways to do this, but I'm new to using ivy for the artifacts. My issue looks like it might stem from JAAS - it looks like the JAAS is authenticating with Kerberos, but then com.sun.security.jgss errors with a null principal. I did convert a test HBase cluster to use Kerberos, and Hannibal seems to be working with that test cluster, but not with my original cluster. Not sure if I didm't get everything converted to Kerberos on the test cluster.

@keyboardfann
Copy link

Dear @gshively ,
It's very helpful for me , I try your step and it's success now. Hannibal will scan the default regionserver info port 60030. Is there any config i can change it ?

[root@ci-test-1 hannibal]# ./start 
starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /root/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/root/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 11625
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@dc67248, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Thu Dec 19 15:45:02 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 85ms... (Thu Dec 19 15:45:02 UTC 2013) 
[info] play - Listening for HTTP on port 9000...
[info] application - start update RegionInfo cache... (Thu Dec 19 15:45:04 UTC 2013)
[info] application - completed update RegionInfo cache, took 2385ms... (Thu Dec 19 15:45:06 UTC 2013) 
[info] application - start updating RegionMetrics... (Thu Dec 19 15:45:06 UTC 2013)
[info] application - creating new metric for 35afa3fd12dde375081aa94c9535a8a6 : storefileSizeMB
[info] application - creating new metric for 35afa3fd12dde375081aa94c9535a8a6 : storefiles
[info] application - creating new metric for 35afa3fd12dde375081aa94c9535a8a6 : memstoreSizeMB
[info] application - creating new metric for d9a42287de8f801fce7b3064271f8639 : storefileSizeMB
[info] application - creating new metric for d9a42287de8f801fce7b3064271f8639 : storefiles
[info] application - creating new metric for d9a42287de8f801fce7b3064271f8639 : memstoreSizeMB
[info] application - creating new metric for 7a41aa429a5ea85c894e90bde03d03c9 : storefileSizeMB
[info] application - creating new metric for 7a41aa429a5ea85c894e90bde03d03c9 : storefiles
[info] application - creating new metric for 7a41aa429a5ea85c894e90bde03d03c9 : memstoreSizeMB
[info] application - creating new metric for 1c2edcc934033f9c81cb428921d483ec : storefileSizeMB
[info] application - creating new metric for 1c2edcc934033f9c81cb428921d483ec : storefiles
[info] application - creating new metric for 1c2edcc934033f9c81cb428921d483ec : memstoreSizeMB
[info] application - completed updating 3 RegionMetrics, took 285ms... (Thu Dec 19 15:45:07 UTC 2013) 
[info] application - start initialize Logfile... (Thu Dec 19 15:45:07 UTC 2013)
[ERROR] [12/19/2013 15:45:07.842] [application-akka.actor.default-dispatcher-1] [akka://application/user/updateMetricsActor] Connection refused to http://ci-test-1.client.tw.trendnet.org:60030/logs/
java.net.ConnectException: Connection refused to http://ci-test-1.client.tw.trendnet.org:60030/logs/
    at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:428)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:419)
    at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:381)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:409)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
    ... 3 more

meniku pushed a commit that referenced this issue Dec 19, 2013
…er.info.port' for HBase >= 0.92 (defaulting to 60030) #19
@meniku
Copy link
Collaborator

meniku commented Dec 19, 2013

60030 was hardcoded for HBase >= 0.92, I just pushed a commit to the next branch, which will read the port from the property hbase.regionserver.info.port in hbase-site.xml. Please give it a try @keyboardfann

If I am correct, there are two things left to fix to support Kerberos in Hannibal:

  1. Missing Jar: Unfortunately I cannot find the hbase-XXX-security.jar in the common maven repositories, maybe we have to host it elsewhere to support easy setup of hbase with Kerberos. Another option could be to add enough documentation on how to manually include the hbase-XXX-security.jar in Hannibal. It should be enough to just drop the jar-file to a lib-directory inside Hannibal though. See Unmanaged Dependencies
  2. wrong hbase/hadoop version: As soon as I increase the HBase Version to hbase 0.94.14 and Hadoop version to 0.23.9 my hannibal stops working in my vagrant testing environment. It just locks up when attempting to retrieve the regionserver inormation via the hbase api. I have to dig further in this when I got time before commiting those changes.

@keyboardfann
Copy link

Dear @meniku ,
The next branch is work. Here is the step to run hannibal on secure hbase.

[root@ci-test-1 ~]# git clone https://github.com/sentric/hannibal.git
[root@ci-test-1 hannibal]# git checkout next
[root@ci-test-1 hannibal]# export HANNIBAL_HBASE_VERSION=0.94
Change:
      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-core" % "0.20.205.0",
        "org.apache.hbase" % "hbase" % "0.94.3"
      )

TO:
      case "0.94" => Seq(
        "org.apache.hadoop" % "hadoop-core" % "0.23.9",
        "org.apache.hbase" % "hbase" % "0.94.14"
      )

[root@ci-test-1 hannibal]# ./build 
[root@ci-test-1 hannibal]# mkdir -p $HOME/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/ivys
[root@ci-test-1 hannibal]# mkdir -p $HOME/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/jars
[root@ci-test-1 hannibal]# cp /usr/lib/hbase/hbase-0.94.2-cdh4.2.1-security.jar /root/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/jars/hbase.jar
[root@ci-test-1 hannibal]# cp /root/.ivy2/cache/org.apache.hbase/hbase/ivy-0.94.14.xml /root/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/ivys/ivy.xml
[root@ci-test-1 hannibal]# vim /root/.ivy2/local/org.apache.hbase/hbase/0.94.14-security/ivys/ivy.xml 
Change

TO:
  revi"0.94.14-security"

[root@ci-test-1 hannibal]# vim project/Build.scala
Change
 "org.e.hbase" % "hbase" % "0.94.14"
To:
 "org.e.hbase" % "hbase" % "0.94.14-security"

[root@ci-test-1 hannibal]# ./build 
[root@ci-test-1 hannibal]# ./start 

@keyboardfann
Copy link

But..... I meet a new problem when i deploy hannibal to production cluster . We haver two namenode , two hbase master( it has active standby HA) and five zookeeper, when I start hannibal , I meet org.apache.hadoop.hbase.MasterNotRunningException again..........

Does hannibal support hbase HA?

[root@ci-test-1 hannibal]# ./start 
starting Hannibal on Port 9000 (may be altered by assigning another port-number to HANNIBAL_HTTP_PORT)
[info] Loading project definition from /root/hannibal/project
Configuring for HBase Version: 0.94
[info] Set current project to Hannibal (in build file:/root/hannibal/)

(Starting server. Type Ctrl+D to exit logs, the server will remain in background)

Play server process ID is 1990
[info] play - database [default] connected at jdbc:h2:data/metrics.h2
[info] application - Application has started in Prod-Mode with models.hbase092.HBaseContext092@21270107, starting Update-Metrics-Actor
[info] play - Starting application default Akka system.
[info] play - Application started (Prod)
[info] application - start cleaning metrics and records older than one 86400 seconds... (Fri Dec 20 09:30:40 UTC 2013)
[info] application - cleaned 0 old metrics and 0 old records, took 98ms... (Fri Dec 20 09:30:40 UTC 2013) 
[info] play - Listening for HTTP on port 443...
[info] application - start update RegionInfo cache... (Fri Dec 20 09:30:42 UTC 2013)

[ERROR] [12/20/2013 09:38:25.925] [application-akka.actor.default-dispatcher-1] [akka://application/user/updateMetricsActor] Retried 10 times
org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:133)
    at models.hbase.HBase$class.withAdmin(HBase.scala:40)
    at models.hbase092.HBase092.withAdmin(HBase092.scala:10)
    at models.hbase092.HBase092.eachRegionServer(HBase092.scala:12)
    at models.Region$.updateCache(Region.scala:92)
    at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$apply$1.apply$mcV$sp(UpdateMetricsActor.scala:66)
    at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:139)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:65)
    at actors.UpdateMetricsActor$$anonfun$receive$1.apply(UpdateMetricsActor.scala:62)
    at akka.actor.Actor$class.apply(Actor.scala:318)
    at actors.UpdateMetricsActor.apply(UpdateMetricsActor.scala:60)
    at akka.actor.ActorCell.invoke(ActorCell.scala:626)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
    at akka.dispatch.Mailbox.run(Mailbox.scala:179)
    at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
    at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
    at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
    at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
    at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

HMaster log:

[2013-12-20 10:26:05,851][IPC Reader 7 on port 8100][WARN ][org.apache.hadoop.ipc.HBaseServer]: IPC Server listener on 8100: readAndProcess threw exception org.apache.hadoop.security.AccessControlException: Authentication is required. Count of bytes read: 0
org.apache.hadoop.security.AccessControlException: Authentication is required
    at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:425)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:739)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:530)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:505)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
[2013-12-20 10:26:07,864][IPC Reader 8 on port 8100][WARN ][org.apache.hadoop.ipc.HBaseServer]: IPC Server listener on 8100: readAndProcess threw exception org.apache.hadoop.security.AccessControlException: Authentication is required. Count of bytes read: 0
org.apache.hadoop.security.AccessControlException: Authentication is required
    at org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:425)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:739)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:530)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:505)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

The keytab and auth is ok.

[root@ci-test-1 hannibal]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hbase/ci-test-1@ISPN.TRENDMICRO.COM

Valid starting     Expires            Service principal
12/20/13 09:30:29  12/21/13 09:30:29  krbtgt/ISPN.TRENDMICRO.COM@ISPN.TRENDMICRO.COM
    renew until 12/27/13 09:30:29
[root@ci-test-1 hannibal]# hadoop fs -ls 
Found 31 items
drwxr-xr-x   - hbase hbase          0 2013-12-08 00:00 .Trash
drwxr-xr-x   - hbase hbase          0 2013-06-20 12:24 .oldlogs

@meniku
Copy link
Collaborator

meniku commented Sep 8, 2014

Does the problem still exist with Hannibal 0.10.1? https://github.com/sentric/hannibal/releases/tag/v.0.10.1

@clairvoyant-rohit
Copy link

I am still facing issue with the kerberos and hannibal . Current HBASE version i am using is 1.2 with Cloudera CDH . Any help with steps to install hannibal with kerberos cluster is greatly appreciated.

@clairvoyant-rohit
Copy link

[ERROR] [04/02/2018 07:40:19.653] [application-akka.actor.default-dispatcher-3] [akka://application/user/updateMetricsActor] hconnection-0x362735ef518bf6c Unexpected KeeperException creating base node
org.apache.hadoop.hbase.ZooKeeperConnectionException: hconnection-0x362735ef518bf6c Unexpected KeeperException creating base node
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:168)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1209)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:511)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:502)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:172)
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:92)
at models.hbase.HBase$class.withAdmin(HBase.scala:40)
at models.hbase090.HBase090.withAdmin(HBase090.scala:10)
at models.hbase090.HBase090.eachRegionServer(HBase090.scala:12)
at models.LogFile$$anonfun$discoverLogFileUrlPattern$1.apply$mcV$sp(LogFile.scala:124)
at scala.util.control.Breaks.breakable(Breaks.scala:38)
at models.LogFile$.discoverLogFileUrlPattern(LogFile.scala:123)
at models.LogFile$.init(LogFile.scala:167)
at actors.UpdateMetricsActor$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(UpdateMetricsActor.scala:90)
at actors.UpdateMetricsActor.execute(UpdateMetricsActor.scala:144)
at actors.UpdateMetricsActor$$anonfun$receive$1.applyOrElse(UpdateMetricsActor.scala:89)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at actors.UpdateMetricsActor.aroundReceive(UpdateMetricsActor.scala:65)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/unassigned
at org.apache.zookeeper.KeeperException.create(KeeperException.java:104)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:932)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:164)
... 26 more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants