scan of small table stopped working (hangs)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

scan of small table stopped working (hangs)

o haya
Hi,

I have a small Accumulo test instance, that I installed following this page:

https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04

I've been doing testing and was able to successfully run the SimpleIngestClient from the minicluster project and create a table, testTable.

After that, I could use the Accumulo shell and scan the table, after logging in as a user, "demo" and running "scan -t testTable".

However, tonight, after I brought up the machine and started Hadoop (/apps/hadoop/sbin/start-dfs.sh), then zookeeper (/apps/zookeeper/bin/zkServer.sh start) and finally, Accumulo (/apps/accumulo/bin/start-all.sh).

And then I started the Accumulo shell and logged in as "demo" and did "scan -t testTable" and it just hangs, with no output.

I found another thread that suggested enabling "debug on" and here is the output:

root@accumulo1> debug on
2017-05-23 01:44:37,524 [admin.TableOperations] TRACE: tid=13 Fetching list of tables...
2017-05-23 01:44:37,526 [admin.TableOperations] TRACE: tid=13 Fetched 5 table names in 0.001 secs
2017-05-23 01:44:37,526 [rpc.ThriftUtil] TRACE: Opening normal transport
2017-05-23 01:44:37,532 [impl.ThriftTransportPool] TRACE: Creating new connection to connection to localhost:9997
2017-05-23 01:44:37,534 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 170
2017-05-23 01:44:37,534 [admin.TableOperations] TRACE: tid=13 Fetching list of namespaces...
2017-05-23 01:44:37,534 [admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces in 0.000 secs
root@accumulo1> scan -t testTable
2017-05-23 01:44:51,547 [admin.TableOperations] TRACE: tid=13 Checking if table testTable exists...
2017-05-23 01:44:51,547 [admin.TableOperations] TRACE: tid=13 Checked existance of true in 0.000 secs
2017-05-23 01:44:51,548 [rpc.ThriftUtil] TRACE: Opening normal transport
2017-05-23 01:44:51,549 [impl.ThriftTransportPool] TRACE: Creating new connection to connection to localhost:9997
2017-05-23 01:44:51,565 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,565 [admin.TableOperations] TRACE: tid=13 Checking if table testTable exists...
2017-05-23 01:44:51,566 [admin.TableOperations] TRACE: tid=13 Checked existance of true in 0.000 secs
2017-05-23 01:44:51,566 [impl.ThriftTransportPool] TRACE: Using existing connection to localhost:9997
2017-05-23 01:44:51,582 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,583 [impl.ThriftTransportPool] TRACE: Using existing connection to localhost:9997
2017-05-23 01:44:51,594 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,594 [admin.TableOperations] TRACE: tid=13 Checking if table testTable exists...
2017-05-23 01:44:51,594 [admin.TableOperations] TRACE: tid=13 Checked existance of true in 0.000 secs
2017-05-23 01:44:51,594 [impl.ThriftTransportPool] TRACE: Using existing connection to localhost:9997
2017-05-23 01:44:51,608 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 6808
2017-05-23 01:44:51,609 [impl.ThriftTransportPool] TRACE: Using existing connection to localhost:9997
2017-05-23 01:44:51,614 [impl.ThriftTransportPool] TRACE: Returned connection localhost:9997 (120000) ioCount: 130
2017-05-23 01:44:51,620 [shell.Shell] DEBUG: Found no scan iterators to set
2017-05-23 01:44:51,644 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,644 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,644 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:51,658 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.014 secs
2017-05-23 01:44:51,659 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.014 secs
2017-05-23 01:44:51,659 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.015 secs
2017-05-23 01:44:51,659 [impl.ThriftScanner] DEBUG: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:51,759 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,759 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,759 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:51,759 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:51,759 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:51,760 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:51,760 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:51,952 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:51,952 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:51,952 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:51,952 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:51,952 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:51,953 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:51,953 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:52,302 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:52,302 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:52,302 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:52,303 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:52,303 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:52,303 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.001 secs
2017-05-23 01:44:52,303 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:52,960 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:52,960 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:52,960 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:52,960 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:52,960 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:52,960 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:52,960 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:54,282 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:54,282 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:54,282 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:54,282 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:54,282 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:54,282 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:54,282 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:44:56,898 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:44:56,898 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:44:56,898 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:44:56,898 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:44:56,898 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:56,898 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:44:56,898 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:02,204 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:02,204 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:02,204 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:02,204 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:02,204 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:02,205 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:02,205 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:07,560 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:07,560 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:07,560 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:07,560 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:07,561 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:07,561 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:07,561 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:13,002 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:13,002 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:13,002 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:13,002 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:13,002 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:13,002 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:13,002 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:17,530 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:17,530 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:17,530 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:17,530 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:17,530 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:17,530 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:17,530 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:22,802 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:22,803 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:22,803 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:22,803 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:22,803 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:22,803 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:22,803 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:28,247 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=7 row= skipRow=false retry=false
2017-05-23 01:45:28,247 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet  table=!0 row=7; skipRow=false retry=false
2017-05-23 01:45:28,247 [impl.RootTabletLocator] TRACE: tid=13 Looking up root tablet location in zookeeper.
2017-05-23 01:45:28,248 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at null in 0.000 secs
2017-05-23 01:45:28,248 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:28,248 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null at null in 0.000 secs
2017-05-23 01:45:28,248 [impl.ThriftScanner] TRACE: Failed to locate tablet for table : 7 row :
2017-05-23 01:45:30,781 [ipc.Client] DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client] DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client] DEBUG: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@a95842c
2017-05-23 01:45:30,782 [ipc.Client] DEBUG: Stopping client


Can anyone tell from the above why the scan command is no longer working?

Thanks,
Jim
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scan of small table stopped working (hangs)

o haya
Hi,

I noticed that in the monitor_nodejs.whatever.com.debug.log file I am seeing these messages:

2017-05-23 01:56:51,035 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_
2017-05-23 01:56:57,173 [monitor.Monitor] INFO :  Failed to obtain problem reports
java.lang.RuntimeException: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
        at org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
        at org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
        at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
        at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
        ... 6 more
2017-05-23 01:56:57,512 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_
2017-05-23 01:57:03,604 [monitor.Monitor] INFO :  Failed to obtain problem reports
java.lang.RuntimeException: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
        at org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
        at org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
        at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
        at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
        ... 6 more
2017-05-23 01:57:03,944 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_




--------------------------------------------
On Tue, 5/23/17, o haya <[hidden email]> wrote:

 Subject: scan of small table stopped working (hangs)
 To: [hidden email]
 Date: Tuesday, May 23, 2017, 1:56 AM
 
 Hi,
 
 I have a small Accumulo test instance,
 that I installed following this page:
 
 https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04
 
 I've been doing testing and was able to
 successfully run the SimpleIngestClient from the minicluster
 project and create a table, testTable.
 
 After that, I could use the Accumulo
 shell and scan the table, after logging in as a user, "demo"
 and running "scan -t testTable".
 
 However, tonight, after I brought up
 the machine and started Hadoop
 (/apps/hadoop/sbin/start-dfs.sh), then zookeeper
 (/apps/zookeeper/bin/zkServer.sh start) and finally,
 Accumulo (/apps/accumulo/bin/start-all.sh).
 
 And then I started the Accumulo shell
 and logged in as "demo" and did "scan -t testTable" and it
 just hangs, with no output.
 
 I found another thread that suggested
 enabling "debug on" and here is the output:
 
 root@accumulo1>
 debug on
 2017-05-23 01:44:37,524
 [admin.TableOperations] TRACE: tid=13 Fetching list of
 tables...
 2017-05-23 01:44:37,526
 [admin.TableOperations] TRACE: tid=13 Fetched 5 table names
 in 0.001 secs
 2017-05-23 01:44:37,526
 [rpc.ThriftUtil] TRACE: Opening normal transport
 2017-05-23 01:44:37,532
 [impl.ThriftTransportPool] TRACE: Creating new connection to
 connection to localhost:9997
 2017-05-23 01:44:37,534
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 170
 2017-05-23 01:44:37,534
 [admin.TableOperations] TRACE: tid=13 Fetching list of
 namespaces...
 2017-05-23 01:44:37,534
 [admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces
 in 0.000 secs
 root@accumulo1>
 scan -t testTable
 2017-05-23 01:44:51,547
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,547
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,548
 [rpc.ThriftUtil] TRACE: Opening normal transport
 2017-05-23 01:44:51,549
 [impl.ThriftTransportPool] TRACE: Creating new connection to
 connection to localhost:9997
 2017-05-23 01:44:51,565
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,565
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,566
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,566
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,582
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,583
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,594
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,594
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,594
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,594
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,608
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,609
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,614
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 130
 2017-05-23 01:44:51,620 [shell.Shell]
 DEBUG: Found no scan iterators to set
 2017-05-23 01:44:51,644
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,644
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,644
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,658
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.014 secs
 2017-05-23 01:44:51,659
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.014 secs
 2017-05-23 01:44:51,659
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.015 secs
 2017-05-23 01:44:51,659
 [impl.ThriftScanner] DEBUG: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,759
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,759
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,760
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,760
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,952
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,952
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,953
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,953
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:52,302
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:52,302
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:52,302
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:52,303
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:52,303
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,303
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.001 secs
 2017-05-23 01:44:52,303
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:52,960
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:52,960
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:54,282
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:54,282
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:56,898
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:56,898
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:02,204
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:02,204
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:02,205
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:02,205
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:07,560
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:07,560
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:07,560
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:07,560
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:13,002
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:13,002
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:17,530
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:17,530
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:22,802
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:22,803
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:22,803
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:28,247
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:28,247
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:28,247
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:28,248
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:30,781 [ipc.Client]
 DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: stopping actual client because no more references
 remain: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: Stopping client
 
 
 Can anyone tell from the above why the
 scan command is no longer working?
 
 Thanks,
 Jim
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: scan of small table stopped working (hangs)

Michael Wall
Jim

I suspect your installation used files in /tmp that have been cleaned up.  Look at your hdfs-site.xml and ensure the dfs.name.dir and dfs.data.dir are not in /tmp.  Then look at your zoo.cfg directory and ensure the dataDir is defined and not /tmp.

Mike

On Tue, May 23, 2017 at 2:02 AM o haya <[hidden email]> wrote:
Hi,

I noticed that in the monitor_nodejs.whatever.com.debug.log file I am seeing these messages:

2017-05-23 01:56:51,035 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_
2017-05-23 01:56:57,173 [monitor.Monitor] INFO :  Failed to obtain problem reports
java.lang.RuntimeException: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
        at org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
        at org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
        at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
        at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
        ... 6 more
2017-05-23 01:56:57,512 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_
2017-05-23 01:57:03,604 [monitor.Monitor] INFO :  Failed to obtain problem reports
java.lang.RuntimeException: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
        at org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
        at org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
        at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
        at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
        at org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
        at org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
        at org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
        ... 6 more
2017-05-23 01:57:03,944 [impl.ThriftScanner] DEBUG:  Failed to locate tablet for table : !0 row : ~err_




--------------------------------------------
On Tue, 5/23/17, o haya <[hidden email]> wrote:

 Subject: scan of small table stopped working (hangs)
 To: [hidden email]
 Date: Tuesday, May 23, 2017, 1:56 AM

 Hi,

 I have a small Accumulo test instance,
 that I installed following this page:

 https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04

 I've been doing testing and was able to
 successfully run the SimpleIngestClient from the minicluster
 project and create a table, testTable.

 After that, I could use the Accumulo
 shell and scan the table, after logging in as a user, "demo"
 and running "scan -t testTable".

 However, tonight, after I brought up
 the machine and started Hadoop
 (/apps/hadoop/sbin/start-dfs.sh), then zookeeper
 (/apps/zookeeper/bin/zkServer.sh start) and finally,
 Accumulo (/apps/accumulo/bin/start-all.sh).

 And then I started the Accumulo shell
 and logged in as "demo" and did "scan -t testTable" and it
 just hangs, with no output.

 I found another thread that suggested
 enabling "debug on" and here is the output:

 root@accumulo1>
 debug on
 2017-05-23 01:44:37,524
 [admin.TableOperations] TRACE: tid=13 Fetching list of
 tables...
 2017-05-23 01:44:37,526
 [admin.TableOperations] TRACE: tid=13 Fetched 5 table names
 in 0.001 secs
 2017-05-23 01:44:37,526
 [rpc.ThriftUtil] TRACE: Opening normal transport
 2017-05-23 01:44:37,532
 [impl.ThriftTransportPool] TRACE: Creating new connection to
 connection to localhost:9997
 2017-05-23 01:44:37,534
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 170
 2017-05-23 01:44:37,534
 [admin.TableOperations] TRACE: tid=13 Fetching list of
 namespaces...
 2017-05-23 01:44:37,534
 [admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces
 in 0.000 secs
 root@accumulo1>
 scan -t testTable
 2017-05-23 01:44:51,547
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,547
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,548
 [rpc.ThriftUtil] TRACE: Opening normal transport
 2017-05-23 01:44:51,549
 [impl.ThriftTransportPool] TRACE: Creating new connection to
 connection to localhost:9997
 2017-05-23 01:44:51,565
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,565
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,566
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,566
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,582
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,583
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,594
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,594
 [admin.TableOperations] TRACE: tid=13 Checking if table
 testTable exists...
 2017-05-23 01:44:51,594
 [admin.TableOperations] TRACE: tid=13 Checked existance of
 true in 0.000 secs
 2017-05-23 01:44:51,594
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,608
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 6808
 2017-05-23 01:44:51,609
 [impl.ThriftTransportPool] TRACE: Using existing connection
 to localhost:9997
 2017-05-23 01:44:51,614
 [impl.ThriftTransportPool] TRACE: Returned connection
 localhost:9997 (120000) ioCount: 130
 2017-05-23 01:44:51,620 [shell.Shell]
 DEBUG: Found no scan iterators to set
 2017-05-23 01:44:51,644
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,644
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,644
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,658
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.014 secs
 2017-05-23 01:44:51,659
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.014 secs
 2017-05-23 01:44:51,659
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.015 secs
 2017-05-23 01:44:51,659
 [impl.ThriftScanner] DEBUG: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,759
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,759
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:51,759
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,760
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,760
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:51,952
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:51,952
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:51,952
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,953
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:51,953
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:52,302
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:52,302
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:52,302
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:52,303
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:52,303
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,303
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.001 secs
 2017-05-23 01:44:52,303
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:52,960
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:52,960
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:52,960
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:54,282
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:54,282
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:54,282
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:44:56,898
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:44:56,898
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:44:56,898
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:02,204
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:02,204
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:02,204
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:02,205
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:02,205
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:07,560
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:07,560
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:07,560
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:07,560
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:07,561
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:13,002
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:13,002
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:13,002
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:17,530
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:17,530
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:17,530
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:22,802
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:22,803
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:22,803
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:22,803
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:28,247
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=7 row= skipRow=false retry=false
 2017-05-23 01:45:28,247
 [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet 
 table=!0 row=7; skipRow=false retry=false
 2017-05-23 01:45:28,247
 [impl.RootTabletLocator] TRACE: tid=13 Looking up root
 tablet location in zookeeper.
 2017-05-23 01:45:28,248
 [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
 null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
 at null in 0.000 secs
 2017-05-23 01:45:28,248
 [impl.ThriftScanner] TRACE: Failed to locate tablet for
 table : 7 row :
 2017-05-23 01:45:30,781 [ipc.Client]
 DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: stopping actual client because no more references
 remain: org.apache.hadoop.ipc.Client@a95842c
 2017-05-23 01:45:30,782 [ipc.Client]
 DEBUG: Stopping client


 Can anyone tell from the above why the
 scan command is no longer working?

 Thanks,
 Jim

Loading...