Does a hard shutdown of accumulo clear all fates in zookeeper

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Does a hard shutdown of accumulo clear all fates in zookeeper

Jeff Kubina
I had three fates in the accumulo fate queue that had no tasks assigned to them and could not be deleted or failed. All create/delete/clone etc commands were timing out. I did a hard shutdown of all services. I had planned to do "accumulo org.apache.accumulo.server.fate.Admin kill <txid>" but that command is deprecated and replaced with the fate shell command. Since the fate shell command failed does a hard shutdown ensure that all the fates in zookeeper are removed or do I need to delete them manually?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does a hard shutdown of accumulo clear all fates in zookeeper

Josh Elser
No, by design, FATE operations wouldn't be removed on shutdown as it
would leave the system in an inconsistent state.

Think back to the common problem we're trying to solve with FATE: what
happens to a partial table creation during unexpected shutdown. We the
FATE runner comes back up (the master), it needs to be able to rollback
to the last known "good" state of the FATE op, and then proceed. The
FATE op _should_ just proceed on its own after a restart.

However, it sounds like you might have run into a bug with FATE which is
causing this situation to happen. Have you done any more digging into
why these operations were "stuck" and timing out? If you are confident
that these are orphaned and the system isn't properly managing them, you
would have to kill them yourself.

Jeff Kubina wrote:
> I had three fates in the accumulo fate queue that had no tasks assigned
> to them and could not be deleted or failed. All create/delete/clone etc
> commands were timing out. I did a hard shutdown of all services. I had
> planned to do "accumulo org.apache.accumulo.server.fate.Admin kill
> <txid>" but that command is deprecated and replaced with the fate shell
> command. Since the fate shell command failed does a hard shutdown ensure
> that all the fates in zookeeper are removed or do I need to delete them
> manually?
Loading...