Quantcast

Error BAD_CREDENTIALS when tablet server tries to connect

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Error BAD_CREDENTIALS when tablet server tries to connect

Max Metzger
Hi,

I'm setting up an Accumulo cluster, most of the nodes are up and running except for two of the Tablet Server nodes. When either tries to connect, they throw the fallowing error:

24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message from unauthenticatable user: !SYSTEM
24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a service with a mismatched configuration. Please ensure a compatible configuration.
org.apache.accumulo.core.client.AccumuloSecurityException: Error BAD_CREDENTIALS - Username or Password is Invalid
        at org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAuthenticator.java:356)
        at org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java:149)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1741)
        at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1797)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy2.loadTablet(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.process(TabletClientService.java:2494)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The Master Node, meanwhile, reports:

24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection reset by peer

24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get tablet server status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException: Connection refused

24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to stop 10.1.10.19:9997[13c64ff2f8c09ef]

I've confirmed that /accumulo/conf/accumulo-site.xml contains the same password, user, and secret for all the nodes, including the ones that won't connect.

Any help would be appreciated!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Error BAD_CREDENTIALS when tablet server tries to connect

dlmarion
"24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration."

 I believe that the master will not allow tservers to join the cluster if
their configuration is different. Check that the conf and lib directories
are the same across the cluster.

-----Original Message-----
From: Max Metzger [mailto:[hidden email]]
Sent: Thursday, January 24, 2013 6:37 PM
To: [hidden email]
Subject: Error BAD_CREDENTIALS when tablet server tries to connect

Hi,

I'm setting up an Accumulo cluster, most of the nodes are up and running
except for two of the Tablet Server nodes. When either tries to connect,
they throw the fallowing error:

24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message
from unauthenticatable user: !SYSTEM
24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration.
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS - Username or Password is Invalid
        at
org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAu
thenticator.java:356)
        at
org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java
:149)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.che
ckPermission(TabletServer.java:1741)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loa
dTablet(TabletServer.java:1797)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWra
p.java:59)
        at $Proxy2.loadTablet(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$l
oadTablet.process(TabletClientService.java:2494)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.p
rocess(TabletClientService.java:2037)
        at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerU
tils.java:154)
        at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingS
erver.java:631)
        at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TSer
verUtils.java:202)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
        at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The Master Node, meanwhile, reports:

24 15:23:30,0088 master:HadoopMaster 1 ERROR unable to get
tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection
reset by peer

24 15:23:50,0109 master:HadoopMaster 3 ERROR unable to get
tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException:
Connection refused

24 15:23:50,0110 master:HadoopMaster 1 WARN attempting to
stop
10.1.10.19:9997[13c64ff2f8c09ef]

I've confirmed that /accumulo/conf/accumulo-site.xml contains the same
password, user, and secret for all the nodes, including the ones that won't
connect.

Any help would be appreciated!



--
View this message in context:
http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tabl
et-server-tries-to-connect-tp2640.html
Sent from the Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Error BAD_CREDENTIALS when tablet server tries to connect

Max Metzger
I've confirmed that the /conf and /lib directories are the same for all nodes.

Also, I forgot to mention, the two nodes that can no longer connect (and throw the BAD_CREDENTIALS error) were initially able to connect to the rest of the accumulo cluster, but at some point were dropped or timed out. This error occurs now, when I try and reconnect them.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error BAD_CREDENTIALS when tablet server tries to connect

Christopher Tubbs-2
In reply to this post by Max Metzger
You'll also need to ensure that any configuration properties starting with "instance." are configured identically (string compare) across all nodes. This is because the system password is based on a hash of certain configuration properties that should be set to the same value throughout a cluster in order for Accumulo to behave well.



--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Thu, Jan 24, 2013 at 6:36 PM, Max Metzger <[hidden email]> wrote:
Hi,

I'm setting up an Accumulo cluster, most of the nodes are up and running
except for two of the Tablet Server nodes. When either tries to connect,
they throw the fallowing error:

24 15:23:47,613 [tabletserver.TabletServer] WARN : Got loadTablet message
from unauthenticatable user: !SYSTEM
24 15:23:47,613 [tabletserver.TabletServer] FATAL: Got message from a
service with a mismatched configuration. Please ensure a compatible
configuration.
org.apache.accumulo.core.client.AccumuloSecurityException: Error
BAD_CREDENTIALS - Username or Password is Invalid
        at
org.apache.accumulo.server.security.ZKAuthenticator.hasSystemPermission(ZKAuthenticator.java:356)
        at
org.apache.accumulo.server.security.Auditor.hasSystemPermission(Auditor.java:149)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1741)
        at
org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1797)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:59)
        at $Proxy2.loadTablet(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.process(TabletClientService.java:2494)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor.process(TabletClientService.java:2037)
        at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:154)
        at
org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
        at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:202)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:662)

The Master Node, meanwhile, reports:

24 15:23:30,0088        master:HadoopMaster     1       ERROR unable to get tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.io.IOException: Connection
reset by peer

24 15:23:50,0109        master:HadoopMaster     3       ERROR unable to get tablet server
status 10.1.10.19:9997[13c64ff2f8c09ef] java.net.ConnectException:
Connection refused

24 15:23:50,0110        master:HadoopMaster     1       WARN attempting to stop
10.1.10.19:9997[13c64ff2f8c09ef]

I've confirmed that /accumulo/conf/accumulo-site.xml contains the same
password, user, and secret for all the nodes, including the ones that won't
connect.

Any help would be appreciated!



--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Error-BAD-CREDENTIALS-when-tablet-server-tries-to-connect-tp2640.html
Sent from the Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Error BAD_CREDENTIALS when tablet server tries to connect

Max Metzger
Thanks for your help - that fixed it. The accumulo-site.xml for the affected nodes were listing the Zookeeper nodes in a different order.

Loading...