Getting IOExceptions in internalRead!

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Getting IOExceptions in internalRead!

Suresh Prajapati
Hello Everyone,

I am trying to query an accumulo table using geomesa, everything works
fine, I get the correct result of the query but getting warnings
logged about "Error closing output stream". Here is sample log:

[WARN]  2017-05-04 11:57:03 TIOStreamTransport:112 - Error closing
output stream.

java.io.IOException: The stream is closed

at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)

at org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)

at org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)

at org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)

at org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)

at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:710)

at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:353)

at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)

at java.lang.Thread.run(Thread.java:745)


I am also getting logs on accumulo web interface:

Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)

I searched regarding the same but found that it's saying something
related to overloading your clusters. How to solve this?

Thank You
Suresh Prajapati
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Getting IOExceptions in internalRead!

David Boyd
Suresh
      I have seen this also. Look at the list archives as I recall Emilio stated this was a bug 



Sent from my Samsung Galaxy smartphone.


-------- Original message --------
From: Suresh Prajapati <[hidden email]>
Date: 04/05/2017 02:48 (GMT-05:00)
Subject: Getting IOExceptions in internalRead!

Hello Everyone,

I am trying to query an accumulo table using geomesa, everything works
fine, I get the correct result of the query but getting warnings
logged about "Error closing output stream". Here is sample log:

[WARN]  2017-05-04 11:57:03 TIOStreamTransport:112 - Error closing
output stream.

java.io.IOException: The stream is closed

at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)

at org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)

at org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)

at org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)

at org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)

at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:710)

at org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:353)

at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)

at java.lang.Thread.run(Thread.java:745)


I am also getting logs on accumulo web interface:

Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)

I searched regarding the same but found that it's saying something
related to overloading your clusters. How to solve this?

Thank You
Suresh Prajapati
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Getting IOExceptions in internalRead!

Suresh Prajapati
Yes, Got that.

On Thu, May 4, 2017 at 3:52 PM, Dave Boyd <[hidden email]> wrote:

> Suresh
>       I have seen this also. Look at the list archives as I recall Emilio
> stated this was a bug
>
>
>
> Sent from my Samsung Galaxy smartphone.
>
>
> -------- Original message --------
> From: Suresh Prajapati <[hidden email]>
> Date: 04/05/2017 02:48 (GMT-05:00)
> To: [hidden email]
> Subject: Getting IOExceptions in internalRead!
>
> Hello Everyone,
>
> I am trying to query an accumulo table using geomesa, everything works
> fine, I get the correct result of the query but getting warnings
> logged about "Error closing output stream". Here is sample log:
>
> [WARN]  2017-05-04 11:57:03 TIOStreamTransport:112 - Error closing
> output stream.
>
> java.io.IOException: The stream is closed
>
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
>
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
>
> at
> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)
>
> at
> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)
>
> at
> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)
>
> at org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:710)
>
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:353)
>
> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> I am also getting logs on accumulo web interface:
>
> Got an IOException in internalRead!
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at
> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
> at
> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203)
> at
> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)
>
> I searched regarding the same but found that it's saying something
> related to overloading your clusters. How to solve this?
>
> Thank You
> Suresh Prajapati
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Getting IOExceptions in internalRead!

Emilio Lahr-Vivaz
Hi Suresh, David,

I think I speculated that it *might* be a bug in GeoMesa, but I haven't
seen any evidence of that yet. David, do you still see those warnings
after cleaning up your close methods? In normal operation, I don't see
those, but possibly the access patterns you're using are different.
Suresh, make sure that you're closing any readers you open.

Thanks,

Emilio

On 05/04/2017 06:40 AM, Suresh Prajapati wrote:

> Yes, Got that.
>
> On Thu, May 4, 2017 at 3:52 PM, Dave Boyd <[hidden email]> wrote:
>> Suresh
>>        I have seen this also. Look at the list archives as I recall Emilio
>> stated this was a bug
>>
>>
>>
>> Sent from my Samsung Galaxy smartphone.
>>
>>
>> -------- Original message --------
>> From: Suresh Prajapati <[hidden email]>
>> Date: 04/05/2017 02:48 (GMT-05:00)
>> To: [hidden email]
>> Subject: Getting IOExceptions in internalRead!
>>
>> Hello Everyone,
>>
>> I am trying to query an accumulo table using geomesa, everything works
>> fine, I get the correct result of the query but getting warnings
>> logged about "Error closing output stream". Here is sample log:
>>
>> [WARN]  2017-05-04 11:57:03 TIOStreamTransport:112 - Error closing
>> output stream.
>>
>> java.io.IOException: The stream is closed
>>
>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
>>
>> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>
>> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
>>
>> at
>> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)
>>
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)
>>
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)
>>
>> at org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)
>>
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:710)
>>
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:353)
>>
>> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> I am also getting logs on accumulo web interface:
>>
>> Got an IOException in internalRead!
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> at
>> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142)
>> at
>> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539)
>> at
>> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>> at
>> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>> at
>> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203)
>> at
>> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)
>>
>> I searched regarding the same but found that it's saying something
>> related to overloading your clusters. How to solve this?
>>
>> Thank You
>> Suresh Prajapati

Loading...