exception from thrift

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

exception from thrift

z11373
Hi,
I have c++ client talking to Accumulo via Thrift.
The client app is simply running in the loop, and inside the loop it creates batch writer, add mutation with random length of string data, call that batch writer flush() and close() methods. The code is running fine, except when running it with multiple instances, and after some number of iterations, some of the instances may start hitting this exception:

org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1

Do you know what causing this exception? Does it have anything to do with the fact the client is accessing Accumulo via Thrift, or problem with the Thrift proxy itself?


Thanks,
Z
Reply | Threaded
Open this post in threaded view
|

Re: exception from thrift

Marc P.
Could be a number of causes. If you are not using the proxy I'd ensure the
tserver is still hosting after a split. If you are you can get more info
from the exception.

In my CPP client I take failed mutations and requeue them depending on the
exception details. If the extent locations have changed I update. If a
security issue then I return an error.

On Tue, Feb 14, 2017, 11:11 AM z11373 <[hidden email]> wrote:

> Hi,
> I have c++ client talking to Accumulo via Thrift.
> The client app is simply running in the loop, and inside the loop it
> creates
> batch writer, add mutation with random length of string data, call that
> batch writer flush() and close() methods. The code is running fine, except
> when running it with multiple instances, and after some number of
> iterations, some of the instances may start hitting this exception:
>
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0  security codes: {}  # server errors 0 # exceptions 1
>
> Do you know what causing this exception? Does it have anything to do with
> the fact the client is accessing Accumulo via Thrift, or problem with the
> Thrift proxy itself?
>
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/exception-from-thrift-tp19919.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: exception from thrift

z11373
Thanks Marc for quick response.
The code throws exception when its calling flush(), and I was told since we are using older version of Thrift C++ files, complete exception info may not be returned. Here is snippet of my code to catch it:

        try {
            writer.addMutation(mutation);
            writer.flush();
        }
        catch (const accumulo::AccumuloSecurityException& ase) {
            std::cerr << stage << " Hit accumulo security exception: " << ase.msg << std::endl;
        }

and "org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations : 0  security codes: {}  # server errors 0 # exceptions 1" is the only exception message I got.


I can write code to retry when hitting that exception, do I just retry calling flush(), or I need to re-create the batch writer object (and add mutation again)?

Thanks,
Z
Reply | Threaded
Open this post in threaded view
|

Re: exception from thrift

Marc P.
Z,
  Is this your custom thrift RPC call or an object against the thrift
proxy? If you are connecting to the proxy, then perhaps explore the
exception further there or within Accumulo's Monitor. Depending on the
problem that's occurring a retry will likely yield the same result.
Do you have access to either to determine the problem? If you are using the
proxy I would venture to guess that the rejection is likely
one that isn't to be ignored.

If you are doing your own thrift RPC calls, then I may be able to provide
some code snippets to help as you get different
information from writing your own calls; however, I don't want to assume
you have gone down this route and provide incorrect information.

On Tue, Feb 14, 2017 at 5:22 PM, z11373 <[hidden email]> wrote:

> Thanks Marc for quick response.
> The code throws exception when its calling flush(), and I was told since we
> are using older version of Thrift C++ files, complete exception info may
> not
> be returned. Here is snippet of my code to catch it:
>
>         try {
>             writer.addMutation(mutation);
>             writer.flush();
>         }
>         catch (const accumulo::AccumuloSecurityException& ase) {
>             std::cerr << stage << " Hit accumulo security exception: " <<
> ase.msg << std::endl;
>         }
>
> and "org.apache.accumulo.core.client.MutationsRejectedException: #
> constraint violations : 0  security codes: {}  # server errors 0 #
> exceptions 1" is the only exception message I got.
>
>
> I can write code to retry when hitting that exception, do I just retry
> calling flush(), or I need to re-create the batch writer object (and add
> mutation again)?
>
> Thanks,
> Z
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345
> .n5.nabble.com/exception-from-thrift-tp19919p19921.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: exception from thrift

Josh Elser
I think he's using the Thrift proxy, Marc (given the first message).

Z -- I would check the Thrift proxy's log to see if the exception was
logged there, or the TabletServer(s) log otherwise. It might just be a
defect that the Thrift proxy server doesn't pass along that information
to your client correctly..

Marc P. wrote:

> Z,
>    Is this your custom thrift RPC call or an object against the thrift
> proxy? If you are connecting to the proxy, then perhaps explore the
> exception further there or within Accumulo's Monitor. Depending on the
> problem that's occurring a retry will likely yield the same result.
> Do you have access to either to determine the problem? If you are using the
> proxy I would venture to guess that the rejection is likely
> one that isn't to be ignored.
>
> If you are doing your own thrift RPC calls, then I may be able to provide
> some code snippets to help as you get different
> information from writing your own calls; however, I don't want to assume
> you have gone down this route and provide incorrect information.
>
> On Tue, Feb 14, 2017 at 5:22 PM, z11373<[hidden email]>  wrote:
>
>> Thanks Marc for quick response.
>> The code throws exception when its calling flush(), and I was told since we
>> are using older version of Thrift C++ files, complete exception info may
>> not
>> be returned. Here is snippet of my code to catch it:
>>
>>          try {
>>              writer.addMutation(mutation);
>>              writer.flush();
>>          }
>>          catch (const accumulo::AccumuloSecurityException&  ase) {
>>              std::cerr<<  stage<<  " Hit accumulo security exception: "<<
>> ase.msg<<  std::endl;
>>          }
>>
>> and "org.apache.accumulo.core.client.MutationsRejectedException: #
>> constraint violations : 0  security codes: {}  # server errors 0 #
>> exceptions 1" is the only exception message I got.
>>
>>
>> I can write code to retry when hitting that exception, do I just retry
>> calling flush(), or I need to re-create the batch writer object (and add
>> mutation again)?
>>
>> Thanks,
>> Z
>>
>>
>>
>> --
>> View this message in context: http://apache-accumulo.1065345
>> .n5.nabble.com/exception-from-thrift-tp19919p19921.html
>> Sent from the Developers mailing list archive at Nabble.com.
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: exception from thrift

z11373
Thanks Josh! You are right, I am using the Thrift proxy with the proxy server came with Accumulo installation. Actually, now it's getting obvious that the problem is likely not Thrift itself, but Accumulo side.
That exception was due to timeout (thanks to Mike Moss who helped me pointing the issue), and looking at the code, I passed 10s timeout setting to batch writer. That setting may not be long enough as this test client (with 50 instances running) will write tenths of MBs of data, and when Accumulo server is busy.

To proof that, I wrote a simple Java client program that mimic same thing like that C++ client, and it also hit the same issue when I set 10s timeout. I don't see the exception when I tried with 60s timeout.

I have side question though, for this case when timeout happens and write failed, to retry the write I need to re-create the connection too, right?


Thanks,
Z