Adding empty string split causes consistency check to failed for merge command

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding empty string split causes consistency check to failed for merge command

Dong Zhou-2
Hi all,

I noticed that adding an empty string to as a split will cause merge command failed the consistency check. 

2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441 unassigned 442 v.unassigned 441 verify.total 441
2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check failed 1b<<

Also, is there a way to remove an unwanted split from table. Merge is one way, but with empty string does not work. 

Cheers,

-Dong Zhou
Reply | Threaded
Open this post in threaded view
|

Re: Adding empty string split causes consistency check to failed for merge command

Keith Turner
I replicated this locally and then did the following surgery to fix it.

Create a table and determine its table id.  The table foo is table id
2.  This tableid is used for rows in metadata table.

  root@uno> createtable foo
  root@uno foo> tables -l
  accumulo.metadata    =>        !0
  accumulo.replication =>      +rep
  accumulo.root        =>        +r
  foo                  =>         2
  trace                =>         1

Add the empty split and another split.

  root@uno foo> addsplits -t foo '' abc

Inpect the metdata table.

  root@uno foo> scan -t accumulo.metadata -b 2;
  2; loc:1000174d2400003 []    localhost:9997
  2; srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000005
  2; srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
  2; srv:time []    M0
  2; ~tab:~pr []    \x00
  2;abc loc:1000174d2400003 []    localhost:9997
  2;abc srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000006
  2;abc srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
  2;abc srv:time []    M0
  2;abc ~tab:~pr []    \x01
  2< loc:1000174d2400003 []    localhost:9997
  2< srv:dir []    hdfs://localhost:8020/accumulo/tables/2/default_tablet
  2< srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
  2< srv:time []    M0
  2< ~tab:~pr []    \x01abc

Insert some data

  root@uno foo> insert a f q v1
  root@uno foo> insert b f q v2

Offline table and grant write permission to metadata.

  root@uno foo> offline -t foo -w
  root@uno foo> grant Table.WRITE -u root -t accumulo.metadata

Delete the tablet with an empty end row

  root@uno foo> deletemany -r 2; -t accumulo.metadata
  Delete { 2; srv:dir [] } ? y
  [DELETED] 2; srv:dir []
  Delete { 2; srv:lock [] } ? y
  [DELETED] 2; srv:lock []
  Delete { 2; srv:time [] } ? y
  [DELETED] 2; srv:time []
  Delete { 2; ~tab:~pr [] } ? y
  [DELETED] 2; ~tab:~pr []

Fix the tablet with end row "abc" to indicate no previous tablet

  root@uno foo> table accumulo.metadata
  root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00

Bring table online and scan it

  root@uno accumulo.metadata> online foo
  root@uno accumulo.metadata> scan -t foo
  a f:q []    v1
  b f:q []    v2

Remove write permission to metadata table.

  root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata

On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <[hidden email]> wrote:

> Hi all,
>
> I noticed that adding an empty string to as a split will cause merge command
> failed the consistency check.
>
> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441
> unassigned 442 v.unassigned 441 verify.total 441
> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check
> failed 1b<<
>
> Also, is there a way to remove an unwanted split from table. Merge is one
> way, but with empty string does not work.
>
> Cheers,
>
> -Dong Zhou
Reply | Threaded
Open this post in threaded view
|

Re: Adding empty string split causes consistency check to failed for merge command

Keith Turner
I omitted something in with my previous email. I assumed the the
tablet with an empty row could have no data.  However, it can have
data.  If the tablet has files, they should not be deleted without
inserting those files into the next tablet.

  root@uno accumulo.metadata> addsplits -t foo '' abc
  root@uno foo> insert '' f q v3
  root@uno foo> scan
   f:q []    v3
  a f:q []    v1
  b f:q []    v2

Now we see the empty end row tablet has a file.  This file would need
to be inserted into the next tablet before deleting.

  root@uno foo> flush -t foo
  root@uno foo> scan -t accumulo.metadata -c file -r 2;
  2; file:hdfs://localhost:8020/accumulo/tables/2/t-0000015/F0000016.rf
[]    185,1


On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <[hidden email]> wrote:

> I replicated this locally and then did the following surgery to fix it.
>
> Create a table and determine its table id.  The table foo is table id
> 2.  This tableid is used for rows in metadata table.
>
>   root@uno> createtable foo
>   root@uno foo> tables -l
>   accumulo.metadata    =>        !0
>   accumulo.replication =>      +rep
>   accumulo.root        =>        +r
>   foo                  =>         2
>   trace                =>         1
>
> Add the empty split and another split.
>
>   root@uno foo> addsplits -t foo '' abc
>
> Inpect the metdata table.
>
>   root@uno foo> scan -t accumulo.metadata -b 2;
>   2; loc:1000174d2400003 []    localhost:9997
>   2; srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000005
>   2; srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2; srv:time []    M0
>   2; ~tab:~pr []    \x00
>   2;abc loc:1000174d2400003 []    localhost:9997
>   2;abc srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000006
>   2;abc srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2;abc srv:time []    M0
>   2;abc ~tab:~pr []    \x01
>   2< loc:1000174d2400003 []    localhost:9997
>   2< srv:dir []    hdfs://localhost:8020/accumulo/tables/2/default_tablet
>   2< srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2< srv:time []    M0
>   2< ~tab:~pr []    \x01abc
>
> Insert some data
>
>   root@uno foo> insert a f q v1
>   root@uno foo> insert b f q v2
>
> Offline table and grant write permission to metadata.
>
>   root@uno foo> offline -t foo -w
>   root@uno foo> grant Table.WRITE -u root -t accumulo.metadata
>
> Delete the tablet with an empty end row
>
>   root@uno foo> deletemany -r 2; -t accumulo.metadata
>   Delete { 2; srv:dir [] } ? y
>   [DELETED] 2; srv:dir []
>   Delete { 2; srv:lock [] } ? y
>   [DELETED] 2; srv:lock []
>   Delete { 2; srv:time [] } ? y
>   [DELETED] 2; srv:time []
>   Delete { 2; ~tab:~pr [] } ? y
>   [DELETED] 2; ~tab:~pr []
>
> Fix the tablet with end row "abc" to indicate no previous tablet
>
>   root@uno foo> table accumulo.metadata
>   root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00
>
> Bring table online and scan it
>
>   root@uno accumulo.metadata> online foo
>   root@uno accumulo.metadata> scan -t foo
>   a f:q []    v1
>   b f:q []    v2
>
> Remove write permission to metadata table.
>
>   root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
>
> On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <[hidden email]> wrote:
>> Hi all,
>>
>> I noticed that adding an empty string to as a split will cause merge command
>> failed the consistency check.
>>
>> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441
>> unassigned 442 v.unassigned 441 verify.total 441
>> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check
>> failed 1b<<
>>
>> Also, is there a way to remove an unwanted split from table. Merge is one
>> way, but with empty string does not work.
>>
>> Cheers,
>>
>> -Dong Zhou
Reply | Threaded
Open this post in threaded view
|

Re: Adding empty string split causes consistency check to failed for merge command

Dong Zhou-2
Hi Keith,

Thanks for the remedy steps, I tried the instruction and it worked. Yay.

I was trying to fix the issue using similar steps, but I missed out offline and online the table part, which leads me to not able to run merge command with following error:
> 2018-03-05 19:10:06,037 [master.Master] INFO : Asking tserver1:10011[161d36db90563c3] to chop 3;a<
> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : 21 tablets are chopped 3<<
> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Waiting for 21 chopped tablets to be 22 3<<
> 2018-03-05 19:10:06,151 [master.Master] INFO : Asking  tserver1:10011[161d36db90563c3] to chop 3;a<
> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Computing next merge state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : 21 tablets are chopped 3<<
> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Waiting for 21 chopped tablets to be 22 3<<

One more question for the supplementary part that the "" split has files.
Would it be a good idea to run hdfs command to physically move the file into the next tablet folder, and then insert an entry in metadata for it.
Or it does not matter, I can just reuse the folder that was created for "" split.

Thanks,

-Dong Zhou



On Mon, Mar 5, 2018 at 9:34 AM, Keith Turner <[hidden email]> wrote:
I omitted something in with my previous email. I assumed the the
tablet with an empty row could have no data.  However, it can have
data.  If the tablet has files, they should not be deleted without
inserting those files into the next tablet.

  root@uno accumulo.metadata> addsplits -t foo '' abc
  root@uno foo> insert '' f q v3
  root@uno foo> scan
   f:q []    v3
  a f:q []    v1
  b f:q []    v2

Now we see the empty end row tablet has a file.  This file would need
to be inserted into the next tablet before deleting.

  root@uno foo> flush -t foo
  root@uno foo> scan -t accumulo.metadata -c file -r 2;
  2; file:hdfs://localhost:8020/accumulo/tables/2/t-0000015/F0000016.rf
[]    185,1


On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <[hidden email]> wrote:
> I replicated this locally and then did the following surgery to fix it.
>
> Create a table and determine its table id.  The table foo is table id
> 2.  This tableid is used for rows in metadata table.
>
>   root@uno> createtable foo
>   root@uno foo> tables -l
>   accumulo.metadata    =>        !0
>   accumulo.replication =>      +rep
>   accumulo.root        =>        +r
>   foo                  =>         2
>   trace                =>         1
>
> Add the empty split and another split.
>
>   root@uno foo> addsplits -t foo '' abc
>
> Inpect the metdata table.
>
>   root@uno foo> scan -t accumulo.metadata -b 2;
>   2; loc:1000174d2400003 []    localhost:9997
>   2; srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000005
>   2; srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2; srv:time []    M0
>   2; ~tab:~pr []    \x00
>   2;abc loc:1000174d2400003 []    localhost:9997
>   2;abc srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000006
>   2;abc srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2;abc srv:time []    M0
>   2;abc ~tab:~pr []    \x01
>   2< loc:1000174d2400003 []    localhost:9997
>   2< srv:dir []    hdfs://localhost:8020/accumulo/tables/2/default_tablet
>   2< srv:lock []    tservers/localhost:9997/zlock-0000000000$1000174d2400003
>   2< srv:time []    M0
>   2< ~tab:~pr []    \x01abc
>
> Insert some data
>
>   root@uno foo> insert a f q v1
>   root@uno foo> insert b f q v2
>
> Offline table and grant write permission to metadata.
>
>   root@uno foo> offline -t foo -w
>   root@uno foo> grant Table.WRITE -u root -t accumulo.metadata
>
> Delete the tablet with an empty end row
>
>   root@uno foo> deletemany -r 2; -t accumulo.metadata
>   Delete { 2; srv:dir [] } ? y
>   [DELETED] 2; srv:dir []
>   Delete { 2; srv:lock [] } ? y
>   [DELETED] 2; srv:lock []
>   Delete { 2; srv:time [] } ? y
>   [DELETED] 2; srv:time []
>   Delete { 2; ~tab:~pr [] } ? y
>   [DELETED] 2; ~tab:~pr []
>
> Fix the tablet with end row "abc" to indicate no previous tablet
>
>   root@uno foo> table accumulo.metadata
>   root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00
>
> Bring table online and scan it
>
>   root@uno accumulo.metadata> online foo
>   root@uno accumulo.metadata> scan -t foo
>   a f:q []    v1
>   b f:q []    v2
>
> Remove write permission to metadata table.
>
>   root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
>
> On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <[hidden email]> wrote:
>> Hi all,
>>
>> I noticed that adding an empty string to as a split will cause merge command
>> failed the consistency check.
>>
>> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441
>> unassigned 442 v.unassigned 441 verify.total 441
>> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check
>> failed 1b<<
>>
>> Also, is there a way to remove an unwanted split from table. Merge is one
>> way, but with empty string does not work.
>>
>> Cheers,
>>
>> -Dong Zhou

Reply | Threaded
Open this post in threaded view
|

Re: Adding empty string split causes consistency check to failed for merge command

Keith Turner
On Mon, Mar 5, 2018 at 2:16 PM, Dong Zhou <[hidden email]> wrote:

> Hi Keith,
>
> Thanks for the remedy steps, I tried the instruction and it worked. Yay.
>
> I was trying to fix the issue using similar steps, but I missed out offline
> and online the table part, which leads me to not able to run merge command
> with following error:
>> 2018-03-05 19:10:06,037 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>> 2018-03-05 19:10:06,151 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>
> One more question for the supplementary part that the "" split has files.
> Would it be a good idea to run hdfs command to physically move the file into

That may be helpful because I am not sure if the Accumulo Garbage
Collector will ever delete the directory when it becomes empty.  You
could compact the tablets referencing the file and then delete the dir
when its empty.

If moving the file, then you need to be careful because multiple files
may reference the tablet.  You can grep the metadata table for the
file to be sure.

> the next tablet folder, and then insert an entry in metadata for it.
> Or it does not matter, I can just reuse the folder that was created for ""
> split.
>
> Thanks,
>
> -Dong Zhou
>
>
>
> On Mon, Mar 5, 2018 at 9:34 AM, Keith Turner <[hidden email]> wrote:
>>
>> I omitted something in with my previous email. I assumed the the
>> tablet with an empty row could have no data.  However, it can have
>> data.  If the tablet has files, they should not be deleted without
>> inserting those files into the next tablet.
>>
>>   root@uno accumulo.metadata> addsplits -t foo '' abc
>>   root@uno foo> insert '' f q v3
>>   root@uno foo> scan
>>    f:q []    v3
>>   a f:q []    v1
>>   b f:q []    v2
>>
>> Now we see the empty end row tablet has a file.  This file would need
>> to be inserted into the next tablet before deleting.
>>
>>   root@uno foo> flush -t foo
>>   root@uno foo> scan -t accumulo.metadata -c file -r 2;
>>   2; file:hdfs://localhost:8020/accumulo/tables/2/t-0000015/F0000016.rf
>> []    185,1
>>
>>
>> On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <[hidden email]> wrote:
>> > I replicated this locally and then did the following surgery to fix it.
>> >
>> > Create a table and determine its table id.  The table foo is table id
>> > 2.  This tableid is used for rows in metadata table.
>> >
>> >   root@uno> createtable foo
>> >   root@uno foo> tables -l
>> >   accumulo.metadata    =>        !0
>> >   accumulo.replication =>      +rep
>> >   accumulo.root        =>        +r
>> >   foo                  =>         2
>> >   trace                =>         1
>> >
>> > Add the empty split and another split.
>> >
>> >   root@uno foo> addsplits -t foo '' abc
>> >
>> > Inpect the metdata table.
>> >
>> >   root@uno foo> scan -t accumulo.metadata -b 2;
>> >   2; loc:1000174d2400003 []    localhost:9997
>> >   2; srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000005
>> >   2; srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2; srv:time []    M0
>> >   2; ~tab:~pr []    \x00
>> >   2;abc loc:1000174d2400003 []    localhost:9997
>> >   2;abc srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000006
>> >   2;abc srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2;abc srv:time []    M0
>> >   2;abc ~tab:~pr []    \x01
>> >   2< loc:1000174d2400003 []    localhost:9997
>> >   2< srv:dir []
>> > hdfs://localhost:8020/accumulo/tables/2/default_tablet
>> >   2< srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2< srv:time []    M0
>> >   2< ~tab:~pr []    \x01abc
>> >
>> > Insert some data
>> >
>> >   root@uno foo> insert a f q v1
>> >   root@uno foo> insert b f q v2
>> >
>> > Offline table and grant write permission to metadata.
>> >
>> >   root@uno foo> offline -t foo -w
>> >   root@uno foo> grant Table.WRITE -u root -t accumulo.metadata
>> >
>> > Delete the tablet with an empty end row
>> >
>> >   root@uno foo> deletemany -r 2; -t accumulo.metadata
>> >   Delete { 2; srv:dir [] } ? y
>> >   [DELETED] 2; srv:dir []
>> >   Delete { 2; srv:lock [] } ? y
>> >   [DELETED] 2; srv:lock []
>> >   Delete { 2; srv:time [] } ? y
>> >   [DELETED] 2; srv:time []
>> >   Delete { 2; ~tab:~pr [] } ? y
>> >   [DELETED] 2; ~tab:~pr []
>> >
>> > Fix the tablet with end row "abc" to indicate no previous tablet
>> >
>> >   root@uno foo> table accumulo.metadata
>> >   root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00
>> >
>> > Bring table online and scan it
>> >
>> >   root@uno accumulo.metadata> online foo
>> >   root@uno accumulo.metadata> scan -t foo
>> >   a f:q []    v1
>> >   b f:q []    v2
>> >
>> > Remove write permission to metadata table.
>> >
>> >   root@uno accumulo.metadata> revoke Table.WRITE -u root -t
>> > accumulo.metadata
>> >
>> > On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <[hidden email]> wrote:
>> >> Hi all,
>> >>
>> >> I noticed that adding an empty string to as a split will cause merge
>> >> command
>> >> failed the consistency check.
>> >>
>> >> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped
>> >> 441
>> >> unassigned 442 v.unassigned 441 verify.total 441
>> >> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency
>> >> check
>> >> failed 1b<<
>> >>
>> >> Also, is there a way to remove an unwanted split from table. Merge is
>> >> one
>> >> way, but with empty string does not work.
>> >>
>> >> Cheers,
>> >>
>> >> -Dong Zhou
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Adding empty string split causes consistency check to failed for merge command

Dong Zhou-2
right, will scan through the metadata with column "file" to be sure.
Thanks a lot for the help. :)

-Dong

On Mon, Mar 5, 2018 at 1:50 PM, Keith Turner <[hidden email]> wrote:
On Mon, Mar 5, 2018 at 2:16 PM, Dong Zhou <[hidden email]> wrote:
> Hi Keith,
>
> Thanks for the remedy steps, I tried the instruction and it worked. Yay.
>
> I was trying to fix the issue using similar steps, but I missed out offline
> and online the table part, which leads me to not able to run merge command
> with following error:
>> 2018-03-05 19:10:06,037 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>> 2018-03-05 19:10:06,151 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>
> One more question for the supplementary part that the "" split has files.
> Would it be a good idea to run hdfs command to physically move the file into

That may be helpful because I am not sure if the Accumulo Garbage
Collector will ever delete the directory when it becomes empty.  You
could compact the tablets referencing the file and then delete the dir
when its empty.

If moving the file, then you need to be careful because multiple files
may reference the tablet.  You can grep the metadata table for the
file to be sure.

> the next tablet folder, and then insert an entry in metadata for it.
> Or it does not matter, I can just reuse the folder that was created for ""
> split.
>
> Thanks,
>
> -Dong Zhou
>
>
>
> On Mon, Mar 5, 2018 at 9:34 AM, Keith Turner <[hidden email]> wrote:
>>
>> I omitted something in with my previous email. I assumed the the
>> tablet with an empty row could have no data.  However, it can have
>> data.  If the tablet has files, they should not be deleted without
>> inserting those files into the next tablet.
>>
>>   root@uno accumulo.metadata> addsplits -t foo '' abc
>>   root@uno foo> insert '' f q v3
>>   root@uno foo> scan
>>    f:q []    v3
>>   a f:q []    v1
>>   b f:q []    v2
>>
>> Now we see the empty end row tablet has a file.  This file would need
>> to be inserted into the next tablet before deleting.
>>
>>   root@uno foo> flush -t foo
>>   root@uno foo> scan -t accumulo.metadata -c file -r 2;
>>   2; file:hdfs://localhost:8020/accumulo/tables/2/t-0000015/F0000016.rf
>> []    185,1
>>
>>
>> On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <[hidden email]> wrote:
>> > I replicated this locally and then did the following surgery to fix it.
>> >
>> > Create a table and determine its table id.  The table foo is table id
>> > 2.  This tableid is used for rows in metadata table.
>> >
>> >   root@uno> createtable foo
>> >   root@uno foo> tables -l
>> >   accumulo.metadata    =>        !0
>> >   accumulo.replication =>      +rep
>> >   accumulo.root        =>        +r
>> >   foo                  =>         2
>> >   trace                =>         1
>> >
>> > Add the empty split and another split.
>> >
>> >   root@uno foo> addsplits -t foo '' abc
>> >
>> > Inpect the metdata table.
>> >
>> >   root@uno foo> scan -t accumulo.metadata -b 2;
>> >   2; loc:1000174d2400003 []    localhost:9997
>> >   2; srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000005
>> >   2; srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2; srv:time []    M0
>> >   2; ~tab:~pr []    \x00
>> >   2;abc loc:1000174d2400003 []    localhost:9997
>> >   2;abc srv:dir []    hdfs://localhost:8020/accumulo/tables/2/t-0000006
>> >   2;abc srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2;abc srv:time []    M0
>> >   2;abc ~tab:~pr []    \x01
>> >   2< loc:1000174d2400003 []    localhost:9997
>> >   2< srv:dir []
>> > hdfs://localhost:8020/accumulo/tables/2/default_tablet
>> >   2< srv:lock []
>> > tservers/localhost:9997/zlock-0000000000$1000174d2400003
>> >   2< srv:time []    M0
>> >   2< ~tab:~pr []    \x01abc
>> >
>> > Insert some data
>> >
>> >   root@uno foo> insert a f q v1
>> >   root@uno foo> insert b f q v2
>> >
>> > Offline table and grant write permission to metadata.
>> >
>> >   root@uno foo> offline -t foo -w
>> >   root@uno foo> grant Table.WRITE -u root -t accumulo.metadata
>> >
>> > Delete the tablet with an empty end row
>> >
>> >   root@uno foo> deletemany -r 2; -t accumulo.metadata
>> >   Delete { 2; srv:dir [] } ? y
>> >   [DELETED] 2; srv:dir []
>> >   Delete { 2; srv:lock [] } ? y
>> >   [DELETED] 2; srv:lock []
>> >   Delete { 2; srv:time [] } ? y
>> >   [DELETED] 2; srv:time []
>> >   Delete { 2; ~tab:~pr [] } ? y
>> >   [DELETED] 2; ~tab:~pr []
>> >
>> > Fix the tablet with end row "abc" to indicate no previous tablet
>> >
>> >   root@uno foo> table accumulo.metadata
>> >   root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00
>> >
>> > Bring table online and scan it
>> >
>> >   root@uno accumulo.metadata> online foo
>> >   root@uno accumulo.metadata> scan -t foo
>> >   a f:q []    v1
>> >   b f:q []    v2
>> >
>> > Remove write permission to metadata table.
>> >
>> >   root@uno accumulo.metadata> revoke Table.WRITE -u root -t
>> > accumulo.metadata
>> >
>> > On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <[hidden email]> wrote:
>> >> Hi all,
>> >>
>> >> I noticed that adding an empty string to as a split will cause merge
>> >> command
>> >> failed the consistency check.
>> >>
>> >> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped
>> >> 441
>> >> unassigned 442 v.unassigned 441 verify.total 441
>> >> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency
>> >> check
>> >> failed 1b<<
>> >>
>> >> Also, is there a way to remove an unwanted split from table. Merge is
>> >> one
>> >> way, but with empty string does not work.
>> >>
>> >> Cheers,
>> >>
>> >> -Dong Zhou
>
>