|
Hi,
I'm setting up an Accumulo cluster, most of the nodes are up and running except for two of the Tablet Server nodes. When either tries to connect, they throw the fallowing error: 24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message from unauthenticatable user: !SYSTEM 24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a service with a mismatched configuration. Please ensure a compatible configuration. org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS - Username or Password is Invalid at org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAuthenticator.java:356) at org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java:149) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1741) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59) at $Proxy2.loadTablet(Unknown Source) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.process(TabletClientService.java:2494) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037) at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631) at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:662) The Master Node, meanwhile, reports: 24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection reset by peer 24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException: Connection refused 24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to stop 10.1.10.19:9997[13c64ff2f8c09ef] I've confirmed that /accumulo/conf/accumulo-site.xml contains the same password, user, and secret for all the nodes, including the ones that won't connect. Any help would be appreciated! |
|
"24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible configuration." I believe that the master will not allow tservers to join the cluster if their configuration is different. Check that the conf and lib directories are the same across the cluster. -----Original Message----- From: Max Metzger [mailto:[hidden email]] Sent: Thursday, January 24, 2013 6:37 PM To: [hidden email] Subject: Error BAD_CREDENTIALS when tablet server tries to connect Hi, I'm setting up an Accumulo cluster, most of the nodes are up and running except for two of the Tablet Server nodes. When either tries to connect, they throw the fallowing error: 24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message from unauthenticatable user: !SYSTEM 24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a service with a mismatched configuration. Please ensure a compatible configuration. org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS - Username or Password is Invalid at org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAu thenticator.java:356) at org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java :149) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.che ckPermission(TabletServer.java:1741) at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loa dTablet(TabletServer.java:1797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 ) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWra p.java:59) at $Proxy2.loadTablet(Unknown Source) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$l oadTablet.process(TabletClientService.java:2494) at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.p rocess(TabletClientService.java:2037) at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerU tils.java:154) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingS erver.java:631) at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TSer verUtils.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:662) The Master Node, meanwhile, reports: 24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection reset by peer 24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException: Connection refused 24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to stop 10.1.10.19:9997[13c64ff2f8c09ef] I've confirmed that /accumulo/conf/accumulo-site.xml contains the same password, user, and secret for all the nodes, including the ones that won't connect. Any help would be appreciated! -- View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tabl et-server-tries-to-connect-tp2640.html Sent from the Users mailing list archive at Nabble.com. |
|
I've confirmed that the /conf and /lib directories are the same for all nodes.
Also, I forgot to mention, the two nodes that can no longer connect (and throw the BAD_CREDENTIALS error) were initially able to connect to the rest of the accumulo cluster, but at some point were dropped or timed out. This error occurs now, when I try and reconnect them. |
|
In reply to this post by Max Metzger
You'll also need to ensure that any configuration properties starting with "instance." are configured identically (string compare) across all nodes. This is because the system password is based on a hash of certain configuration properties that should be set to the same value throughout a cluster in order for Accumulo to behave well.
On Thu, Jan 24, 2013 at 6:36 PM, Max Metzger <[hidden email]> wrote: Hi, |
|
Thanks for your help - that fixed it. The accumulo-site.xml for the affected nodes were listing the Zookeeper nodes in a different order.
|
| Powered by Nabble | Edit this page |
