Accumulo not starting anymore

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Accumulo not starting anymore

Geoffry Roberts
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was working with 10.7.  But with 10.9 Accumulo began to complain of insufficient file limits and recommended setting maxfiles to 65536, which I did.  

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:host.name=abend.home
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.name=Mac OS X
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.name=gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1456c089e9b0008, negotiated timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways, 

Geoffry Roberts
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Sean Busbey
Which Accumulo version is this?

Can you upload the rest of hte logs somewhere (including .out / .err files)?


On Wed, Apr 16, 2014 at 3:15 PM, Geoffry Roberts <[hidden email]> wrote:
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was working with 10.7.  But with 10.9 Accumulo began to complain of insufficient file limits and recommended setting maxfiles to 65536, which I did.  

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:host.name=abend.home
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.name=Mac OS X
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.name=gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1456c089e9b0008, negotiated timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways, 

Geoffry Roberts



--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Mike Drob-3
In reply to this post by Geoffry Roberts
Can you verify that the accumulo files are still present in HDFS?

hdfs -ls /accumulo


On Wed, Apr 16, 2014 at 4:15 PM, Geoffry Roberts <[hidden email]> wrote:
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was working with 10.7.  But with 10.9 Accumulo began to complain of insufficient file limits and recommended setting maxfiles to 65536, which I did.  

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:host.name=abend.home
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.name=Mac OS X
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.name=gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1456c089e9b0008, negotiated timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways, 

Geoffry Roberts

Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Geoffry Roberts
In reply to this post by Sean Busbey

Sorry for neglecting the version. It's Accumulo 1.5.1.

On Apr 16, 2014 4:25 PM, "Sean Busbey" <[hidden email]> wrote:
Which Accumulo version is this?

Can you upload the rest of hte logs somewhere (including .out / .err files)?


On Wed, Apr 16, 2014 at 3:15 PM, Geoffry Roberts <[hidden email]> wrote:
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was working with 10.7.  But with 10.9 Accumulo began to complain of insufficient file limits and recommended setting maxfiles to 65536, which I did.  

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:host.name=abend.home
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.name=Mac OS X
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.name=gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1456c089e9b0008, negotiated timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways, 

Geoffry Roberts



--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Geoffry Roberts
In reply to this post by Mike Drob-3

Yes, the init files are showing in dfshealth page.

On Apr 16, 2014 4:30 PM, "Mike Drob" <[hidden email]> wrote:
Can you verify that the accumulo files are still present in HDFS?

hdfs -ls /accumulo


On Wed, Apr 16, 2014 at 4:15 PM, Geoffry Roberts <[hidden email]> wrote:
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was working with 10.7.  But with 10.9 Accumulo began to complain of insufficient file limits and recommended setting maxfiles to 65536, which I did.  

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:host.name=abend.home
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.name=Mac OS X
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.name=gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection established to localhost/127.0.0.1:2181, initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1456c089e9b0008, negotiated timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to be initialized

--
There are ways and there are ways, 

Geoffry Roberts

Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Josh Elser
In reply to this post by Geoffry Roberts
Are you sure ZooKeeper is running and wasn't wiped?

See what kind of results come from `ls /accumulo` and `ls
/accumulo/instances` in the ZooKeeper shell
($ZOOKEEPER_HOME/bin/zkCli.sh). '/accumulo' should be non-empty, and you
should see the instance name you provided when running `accumulo init`
inside '/accumulo/instances'

On 4/16/14, 4:15 PM, Geoffry Roberts wrote:

> All,
>
> Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is
> there a way to troubleshoot this?
>
> The back story is I upgraded from OSX 10.7 to 10.9.  Everything was
> working with 10.7.  But with 10.9 Accumulo began to complain of
> insufficient file limits and recommended setting maxfiles to 65536,
> which I did.
>
> Hadoop starts -- version 2.3.0
> Zookeeper starts -- version 3.4.6
> Java -- version 1.7.55
>
> I've included part of a log file just in case.
>
> Thanks mucho
>
> From:  master_abend.home.debug.log
>
> t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
> 2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
> d9f3a06a-ef06-4860-a08d-9cff805a9249
> 2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
> 2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to
> zookeeper
> 2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to
> localhost:2181 with timeout 30000 with auth
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:host.name <http://host.name>=abend.home
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.version=1.7.0_55
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.vendor=Oracle Corporation
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.library.path=/usr/local/hadoop/lib/native
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:java.compiler=<NA>
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:os.name <http://os.name>=Mac OS X
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:os.arch=x86_64
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:os.version=10.9.2
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:user.name <http://user.name>=gcr
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:user.home=/Users/gcr
> 2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
> environment:user.dir=/Users/gcr
> 2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client
> connection, connectString=localhost:2181 sessionTimeout=30000
> watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
> 2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket
> connection to server localhost/127.0.0.1:2181 <http://127.0.0.1:2181>.
> Will not attempt to authenticate using SASL (unknown error)
> 2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection
> established to localhost/127.0.0.1:2181 <http://127.0.0.1:2181>,
> initiating session
> 2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session
> establishment complete on server localhost/127.0.0.1:2181
> <http://127.0.0.1:2181>, sessionid = 0x1456c089e9b0008, negotiated
> timeout = 30000
> 2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch
> for Monitor Log4j watcher
> 2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to
> be initialized
> 2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to
> be initialized
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Geoffry Roberts
Here's what I get:

Listing from hdfs:


drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/instance_id

drwxr-xr-x   - gcr supergroup          0 2014-04-15 13:11 /accumulo/recovery

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/tables

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/version

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:43 /accumulo/wal


Zookeeper shell:

[zk: localhost:2181(CONNECTED) 0] ls /accumulo

Node does not exist: /accumulo

[zk: localhost:2181(CONNECTED) 1] ls /accumulo/instances

Node does not exist: /accumulo/instances

[zk: localhost:2181(CONNECTED) 2] 




On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser <[hidden email]> wrote:
Are you sure ZooKeeper is running and wasn't wiped?

See what kind of results come from `ls /accumulo` and `ls /accumulo/instances` in the ZooKeeper shell ($ZOOKEEPER_HOME/bin/zkCli.sh). '/accumulo' should be non-empty, and you should see the instance name you provided when running `accumulo init` inside '/accumulo/instances'


On 4/16/14, 4:15 PM, Geoffry Roberts wrote:
All,

Suddenly, Accumulo will no longer start.  Log files are not helpful.  Is
there a way to troubleshoot this?

The back story is I upgraded from OSX 10.7 to 10.9.  Everything was
working with 10.7.  But with 10.9 Accumulo began to complain of
insufficient file limits and recommended setting maxfiles to 65536,
which I did.

Hadoop starts -- version 2.3.0
Zookeeper starts -- version 3.4.6
Java -- version 1.7.55

I've included part of a log file just in case.

Thanks mucho

From:  master_abend.home.debug.log

t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
d9f3a06a-ef06-4860-a08d-9cff805a9249
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to talk to
zookeeper
2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to
localhost:2181 with timeout 30000 with auth
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:host.name <http://host.name>=abend.home

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.version=1.7.0_55
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.vendor=Oracle Corporation
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.class.path=/usr/local/accumulo/conf:/usr/local/accumulo/lib/accumulo-start.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.library.path=/usr/local/hadoop/lib/native
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.io.tmpdir=/var/folders/sb/g6bpj4cd401c1sw566x2r41m0000gn/T/
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:java.compiler=<NA>
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.name <http://os.name>=Mac OS X

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.arch=x86_64
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:os.version=10.9.2
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.name <http://user.name>=gcr

2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.home=/Users/gcr
2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
environment:user.dir=/Users/gcr
2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating client
connection, connectString=localhost:2181 sessionTimeout=30000
watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@14731467
2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket
connection to server localhost/127.0.0.1:2181 <http://127.0.0.1:2181>.

Will not attempt to authenticate using SASL (unknown error)
2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket connection
established to localhost/127.0.0.1:2181 <http://127.0.0.1:2181>,

initiating session
2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session
establishment complete on server localhost/127.0.0.1:2181
<http://127.0.0.1:2181>, sessionid = 0x1456c089e9b0008, negotiated

timeout = 30000
2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set watch
for Monitor Log4j watcher
2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for accumulo to
be initialized
2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for accumulo to
be initialized

--
There are ways and there are ways,

Geoffry Roberts



--
There are ways and there are ways, 

Geoffry Roberts
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Josh Elser
Alright, well that's your problem! Your ZooKeeper is empty. Accumulo
uses the information it places in ZooKeeper to bootstrap and find the
data in HDFS.

Are you sure you configured the ZooKeeper dataDir to be something other
than /tmp? In $ZOOKEEPER_HOME/conf/zoo.cfg, something like:

   dataDir=/var/lib/zookeeper/

On 4/16/14, 7:21 PM, Geoffry Roberts wrote:

> Here's what I get:
>
> Listing from hdfs:
>
>
> drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42
> /accumulo/instance_id
>
> drwxr-xr-x   - gcr supergroup          0 2014-04-15 13:11 /accumulo/recovery
>
> drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/tables
>
> drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/version
>
> drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:43 /accumulo/wal
>
>
> Zookeeper shell:
>
> [zk: localhost:2181(CONNECTED) 0] ls /accumulo
>
> Node does not exist: /accumulo
>
> [zk: localhost:2181(CONNECTED) 1] ls /accumulo/instances
>
> Node does not exist: /accumulo/instances
>
> [zk: localhost:2181(CONNECTED) 2]
>
>
>
>
> On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Are you sure ZooKeeper is running and wasn't wiped?
>
>     See what kind of results come from `ls /accumulo` and `ls
>     /accumulo/instances` in the ZooKeeper shell
>     ($ZOOKEEPER_HOME/bin/zkCli.sh)__. '/accumulo' should be non-empty,
>     and you should see the instance name you provided when running
>     `accumulo init` inside '/accumulo/instances'
>
>
>     On 4/16/14, 4:15 PM, Geoffry Roberts wrote:
>
>         All,
>
>         Suddenly, Accumulo will no longer start.  Log files are not
>         helpful.  Is
>         there a way to troubleshoot this?
>
>         The back story is I upgraded from OSX 10.7 to 10.9.  Everything was
>         working with 10.7.  But with 10.9 Accumulo began to complain of
>         insufficient file limits and recommended setting maxfiles to 65536,
>         which I did.
>
>         Hadoop starts -- version 2.3.0
>         Zookeeper starts -- version 3.4.6
>         Java -- version 1.7.55
>
>         I've included part of a log file just in case.
>
>         Thanks mucho
>
>         From:  master_abend.home.debug.log
>
>         t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
>         2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
>         d9f3a06a-ef06-4860-a08d-__9cff805a9249
>         2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
>         2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to
>         talk to
>         zookeeper
>         2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to
>         localhost:2181 with timeout 30000 with auth
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:zookeeper.version=__3.4.6-1569965, built on
>         02/20/2014 09:09 GMT
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:host.name <http://host.name>
>         <http://host.name>=abend.home
>
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.version=1.7.__0_55
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.vendor=Oracle Corporation
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.home=/__Library/Java/__JavaVirtualMachines/jdk1.7.0___55.jdk/Contents/Home/jre
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.class.path=/__usr/local/accumulo/conf:/usr/__local/accumulo/lib/accumulo-__start.jar:/usr/local/hadoop/__share/hadoop/common/lib/log4j-__1.2.17.jar
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.library.path=__/usr/local/hadoop/lib/native
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.io.tmpdir=/__var/folders/sb/__g6bpj4cd401c1sw566x2r41m0000gn__/T/
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:java.compiler=<NA>
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:os.name <http://os.name> <http://os.name>=Mac OS X
>
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:os.arch=x86_64
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:os.version=10.9.2
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:user.name <http://user.name> <http://user.name>=gcr
>
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:user.home=/Users/__gcr
>         2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>         environment:user.dir=/Users/__gcr
>         2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating
>         client
>         connection, connectString=localhost:2181 sessionTimeout=30000
>         watcher=org.apache.accumulo.__fate.zookeeper.ZooSession$__ZooWatcher@14731467
>         2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket
>         connection to server localhost/127.0.0.1:2181
>         <http://127.0.0.1:2181> <http://127.0.0.1:2181>.
>
>         Will not attempt to authenticate using SASL (unknown error)
>         2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket
>         connection
>         established to localhost/127.0.0.1:2181 <http://127.0.0.1:2181>
>         <http://127.0.0.1:2181>,
>
>         initiating session
>         2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session
>         establishment complete on server localhost/127.0.0.1:2181
>         <http://127.0.0.1:2181>
>         <http://127.0.0.1:2181>, sessionid = 0x1456c089e9b0008, negotiated
>
>         timeout = 30000
>         2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set
>         watch
>         for Monitor Log4j watcher
>         2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for
>         accumulo to
>         be initialized
>         2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for
>         accumulo to
>         be initialized
>
>         --
>         There are ways and there are ways,
>
>         Geoffry Roberts
>
>
>
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Geoffry Roberts
Firstly, thanks Josh. I forgot I reinstalled Zookeeper--it had been a casualty of my OSX upgrade--and neglected to set the data path.  I did that, but Accumlo still wouldn't start.  I then deleted the /ACCUMULO directory in HDFS and did a re-init.  That fixed it.  Accumulo appears to be running and Zookeeper is listing something that looks right.  

I think I'll declare victory--well see.


On Wed, Apr 16, 2014 at 7:51 PM, Josh Elser <[hidden email]> wrote:
Alright, well that's your problem! Your ZooKeeper is empty. Accumulo uses the information it places in ZooKeeper to bootstrap and find the data in HDFS.

Are you sure you configured the ZooKeeper dataDir to be something other than /tmp? In $ZOOKEEPER_HOME/conf/zoo.cfg, something like:

  dataDir=/var/lib/zookeeper/


On 4/16/14, 7:21 PM, Geoffry Roberts wrote:
Here's what I get:

Listing from hdfs:


drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42
/accumulo/instance_id

drwxr-xr-x   - gcr supergroup          0 2014-04-15 13:11 /accumulo/recovery

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/tables

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42 /accumulo/version

drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:43 /accumulo/wal


Zookeeper shell:

[zk: localhost:2181(CONNECTED) 0] ls /accumulo

Node does not exist: /accumulo

[zk: localhost:2181(CONNECTED) 1] ls /accumulo/instances

Node does not exist: /accumulo/instances

[zk: localhost:2181(CONNECTED) 2]




On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser <[hidden email]
<mailto:[hidden email]>> wrote:

    Are you sure ZooKeeper is running and wasn't wiped?

    See what kind of results come from `ls /accumulo` and `ls
    /accumulo/instances` in the ZooKeeper shell
    ($ZOOKEEPER_HOME/bin/zkCli.sh)__. '/accumulo' should be non-empty,

    and you should see the instance name you provided when running
    `accumulo init` inside '/accumulo/instances'


    On 4/16/14, 4:15 PM, Geoffry Roberts wrote:

        All,

        Suddenly, Accumulo will no longer start.  Log files are not
        helpful.  Is
        there a way to troubleshoot this?

        The back story is I upgraded from OSX 10.7 to 10.9.  Everything was
        working with 10.7.  But with 10.9 Accumulo began to complain of
        insufficient file limits and recommended setting maxfiles to 65536,
        which I did.

        Hadoop starts -- version 2.3.0
        Zookeeper starts -- version 3.4.6
        Java -- version 1.7.55

        I've included part of a log file just in case.

        Thanks mucho

        From:  master_abend.home.debug.log

        t2014-04-16 15:59:07,250 [server.Accumulo] INFO : master starting
        2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
        d9f3a06a-ef06-4860-a08d-__9cff805a9249

        2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data Version 5
        2014-04-16 15:59:07,254 [server.Accumulo] INFO : Attempting to
        talk to
        zookeeper
        2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG: Connecting to
        localhost:2181 with timeout 30000 with auth
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:zookeeper.version=__3.4.6-1569965, built on

        02/20/2014 09:09 GMT
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:host.name <http://host.name>
        <http://host.name>=abend.home

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.version=1.7.__0_55

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.vendor=Oracle Corporation
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.home=/__Library/Java/__JavaVirtualMachines/jdk1.7.0___55.jdk/Contents/Home/jre

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.class.path=/__usr/local/accumulo/conf:/usr/__local/accumulo/lib/accumulo-__start.jar:/usr/local/hadoop/__share/hadoop/common/lib/log4j-__1.2.17.jar

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.library.path=__/usr/local/hadoop/lib/native

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.io.tmpdir=/__var/folders/sb/__g6bpj4cd401c1sw566x2r41m0000gn__/T/

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:java.compiler=<NA>
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:os.name <http://os.name> <http://os.name>=Mac OS X


        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:os.arch=x86_64
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:os.version=10.9.2
        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:user.name <http://user.name> <http://user.name>=gcr


        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:user.home=/Users/__gcr

        2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
        environment:user.dir=/Users/__gcr

        2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO : Initiating
        client
        connection, connectString=localhost:2181 sessionTimeout=30000
        watcher=org.apache.accumulo.__fate.zookeeper.ZooSession$__ZooWatcher@14731467

        2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO : Opening socket
        connection to server localhost/127.0.0.1:2181
        <http://127.0.0.1:2181> <http://127.0.0.1:2181>.


        Will not attempt to authenticate using SASL (unknown error)
        2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO : Socket
        connection
        established to localhost/127.0.0.1:2181 <http://127.0.0.1:2181>
        <http://127.0.0.1:2181>,

        initiating session
        2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO : Session
        establishment complete on server localhost/127.0.0.1:2181
        <http://127.0.0.1:2181>
        <http://127.0.0.1:2181>, sessionid = 0x1456c089e9b0008, negotiated

        timeout = 30000
        2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher] INFO : Set
        watch
        for Monitor Log4j watcher
        2014-04-16 15:59:07,399 [server.Accumulo] INFO : Waiting for
        accumulo to
        be initialized
        2014-04-16 15:59:08,401 [server.Accumulo] INFO : Waiting for
        accumulo to
        be initialized

        --
        There are ways and there are ways,

        Geoffry Roberts




--
There are ways and there are ways,

Geoffry Roberts



--
There are ways and there are ways, 

Geoffry Roberts
Reply | Threaded
Open this post in threaded view
|

Re: Accumulo not starting anymore

Josh Elser
Excellent.

As long as you didn't have anything you needed to preserve, deleting the
data and re-init'ing is definitely the easiest. I do quite a bit of work
on OSX again these days -- you shouldn't have too many issues :)

On 4/17/14, 10:44 AM, Geoffry Roberts wrote:

> Firstly, thanks Josh. I forgot I reinstalled Zookeeper--it had been a
> casualty of my OSX upgrade--and neglected to set the data path.  I did
> that, but Accumlo still wouldn't start.  I then deleted the /ACCUMULO
> directory in HDFS and did a re-init.  That fixed it.  Accumulo appears
> to be running and Zookeeper is listing something that looks right.
>
> I think I'll declare victory--well see.
>
>
> On Wed, Apr 16, 2014 at 7:51 PM, Josh Elser <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Alright, well that's your problem! Your ZooKeeper is empty. Accumulo
>     uses the information it places in ZooKeeper to bootstrap and find
>     the data in HDFS.
>
>     Are you sure you configured the ZooKeeper dataDir to be something
>     other than /tmp? In $ZOOKEEPER_HOME/conf/zoo.cfg, something like:
>
>        dataDir=/var/lib/zookeeper/
>
>
>     On 4/16/14, 7:21 PM, Geoffry Roberts wrote:
>
>         Here's what I get:
>
>         Listing from hdfs:
>
>
>         drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42
>         /accumulo/instance_id
>
>         drwxr-xr-x   - gcr supergroup          0 2014-04-15 13:11
>         /accumulo/recovery
>
>         drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42
>         /accumulo/tables
>
>         drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:42
>         /accumulo/version
>
>         drwxr-xr-x   - gcr supergroup          0 2014-04-15 11:43
>         /accumulo/wal
>
>
>         Zookeeper shell:
>
>         [zk: localhost:2181(CONNECTED) 0] ls /accumulo
>
>         Node does not exist: /accumulo
>
>         [zk: localhost:2181(CONNECTED) 1] ls /accumulo/instances
>
>         Node does not exist: /accumulo/instances
>
>         [zk: localhost:2181(CONNECTED) 2]
>
>
>
>
>         On Wed, Apr 16, 2014 at 6:31 PM, Josh Elser
>         <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>
>              Are you sure ZooKeeper is running and wasn't wiped?
>
>              See what kind of results come from `ls /accumulo` and `ls
>              /accumulo/instances` in the ZooKeeper shell
>              ($ZOOKEEPER_HOME/bin/zkCli.sh)____. '/accumulo' should be
>         non-empty,
>
>              and you should see the instance name you provided when running
>              `accumulo init` inside '/accumulo/instances'
>
>
>              On 4/16/14, 4:15 PM, Geoffry Roberts wrote:
>
>                  All,
>
>                  Suddenly, Accumulo will no longer start.  Log files are not
>                  helpful.  Is
>                  there a way to troubleshoot this?
>
>                  The back story is I upgraded from OSX 10.7 to 10.9.
>           Everything was
>                  working with 10.7.  But with 10.9 Accumulo began to
>         complain of
>                  insufficient file limits and recommended setting
>         maxfiles to 65536,
>                  which I did.
>
>                  Hadoop starts -- version 2.3.0
>                  Zookeeper starts -- version 3.4.6
>                  Java -- version 1.7.55
>
>                  I've included part of a log file just in case.
>
>                  Thanks mucho
>
>                  From:  master_abend.home.debug.log
>
>                  t2014-04-16 15:59:07,250 [server.Accumulo] INFO :
>         master starting
>                  2014-04-16 15:59:07,251 [server.Accumulo] INFO : Instance
>                  d9f3a06a-ef06-4860-a08d-____9cff805a9249
>
>                  2014-04-16 15:59:07,254 [server.Accumulo] INFO : Data
>         Version 5
>                  2014-04-16 15:59:07,254 [server.Accumulo] INFO :
>         Attempting to
>                  talk to
>                  zookeeper
>                  2014-04-16 15:59:07,264 [zookeeper.ZooSession] DEBUG:
>         Connecting to
>                  localhost:2181 with timeout 30000 with auth
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:zookeeper.version=____3.4.6-1569965, built on
>
>                  02/20/2014 09:09 GMT
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:host.name <http://host.name> <http://host.name>
>                  <http://host.name>=abend.home
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:java.version=1.7.____0_55
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:java.vendor=Oracle Corporation
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>
>         environment:java.home=/____Library/Java/____JavaVirtualMachines/jdk1.7.0_____55.jdk/Contents/Home/jre
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>
>         environment:java.class.path=/____usr/local/accumulo/conf:/usr/____local/accumulo/lib/accumulo-____start.jar:/usr/local/hadoop/____share/hadoop/common/lib/__log4j-__1.2.17.jar
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>
>         environment:java.library.path=____/usr/local/hadoop/lib/native
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>
>         environment:java.io.tmpdir=/____var/folders/sb/____g6bpj4cd401c1sw566x2r41m0000gn____/T/
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:java.compiler=<NA>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:os.name <http://os.name> <http://os.name>
>         <http://os.name>=Mac OS X
>
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:os.arch=x86_64
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:os.version=10.9.2
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:user.name <http://user.name>
>         <http://user.name> <http://user.name>=gcr
>
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:user.home=/Users/____gcr
>
>                  2014-04-16 15:59:07,271 [zookeeper.ZooKeeper] INFO : Client
>                  environment:user.dir=/Users/____gcr
>
>                  2014-04-16 15:59:07,272 [zookeeper.ZooKeeper] INFO :
>         Initiating
>                  client
>                  connection, connectString=localhost:2181
>         sessionTimeout=30000
>
>         watcher=org.apache.accumulo.____fate.zookeeper.ZooSession$____ZooWatcher@14731467
>
>                  2014-04-16 15:59:07,288 [zookeeper.ClientCnxn] INFO :
>         Opening socket
>                  connection to server localhost/127.0.0.1:2181
>         <http://127.0.0.1:2181>
>                  <http://127.0.0.1:2181> <http://127.0.0.1:2181>.
>
>
>                  Will not attempt to authenticate using SASL (unknown error)
>                  2014-04-16 15:59:07,289 [zookeeper.ClientCnxn] INFO :
>         Socket
>                  connection
>                  established to localhost/127.0.0.1:2181
>         <http://127.0.0.1:2181> <http://127.0.0.1:2181>
>                  <http://127.0.0.1:2181>,
>
>                  initiating session
>                  2014-04-16 15:59:07,294 [zookeeper.ClientCnxn] INFO :
>         Session
>                  establishment complete on server
>         localhost/127.0.0.1:2181 <http://127.0.0.1:2181>
>                  <http://127.0.0.1:2181>
>                  <http://127.0.0.1:2181>, sessionid = 0x1456c089e9b0008,
>         negotiated
>
>                  timeout = 30000
>                  2014-04-16 15:59:07,394 [watcher.MonitorLog4jWatcher]
>         INFO : Set
>                  watch
>                  for Monitor Log4j watcher
>                  2014-04-16 15:59:07,399 [server.Accumulo] INFO :
>         Waiting for
>                  accumulo to
>                  be initialized
>                  2014-04-16 15:59:08,401 [server.Accumulo] INFO :
>         Waiting for
>                  accumulo to
>                  be initialized
>
>                  --
>                  There are ways and there are ways,
>
>                  Geoffry Roberts
>
>
>
>
>         --
>         There are ways and there are ways,
>
>         Geoffry Roberts
>
>
>
>
> --
> There are ways and there are ways,
>
> Geoffry Roberts