tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max
Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

dlmarion
Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max

 
Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max



From:        Dave Marion <[hidden email]>
To:        [hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        23/03/2017 14:55
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]> wrote:

Hi All,


I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:


table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30


tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8


My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:


default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8


default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50


Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.


Regards,
Max





Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Michael Wall
Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <[hidden email]> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max



From:        Dave Marion <[hidden email]>
To:        [hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        23/03/2017 14:55
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]> wrote:

Hi All,


I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:


table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30


tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8


My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:


default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8


default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50


Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.


Regards,
Max





Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
One table, 9 pre-split tablets, 3 tablets per server and the data is uniform distributed among each tablet.
Max




From:        Michael Wall <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 16:28
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@...> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <dlmarion@...>
To:        
user@..., Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
MASSIMIL@...> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max







Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Marc P.
In reply to this post by Michael Wall
Max,
  Additionally, what are your max open files? If you are under heavy ingest you may exceed this with compactions if they cannot keep up. Additionally, how many files per compaction are you allowing? 

On Thu, Mar 23, 2017 at 10:27 AM, Michael Wall <[hidden email]> wrote:
Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <[hidden email]> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max



From:        Dave Marion <[hidden email]>
To:        [hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        23/03/2017 14:55
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]> wrote:

Hi All,


I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:


table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30


tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8


My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:


default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8


default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50


Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.


Regards,
Max






Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

dlmarion
In reply to this post by Massimilian Mattetti
The number of concurrent major compactions per tserver should be directly related to the number of tablets hosted on each tablet server. During normal operations, major compactions for a tablet should be initiated by the applying the value of `table.compaction.major.ratio` to `table.file.max` and the actual number of files. Additionally, the major compaction process in the tablet server is controlled by the `tserver.compaction.major.*` properties. If you want to test whether or not your `tserver.compaction.major.concurrent.max` property is being honored, go into the shell and compact the table manually using the compact command. This will perform a full compaction and you should see 8 concurrent MajC's occurring on each tserver. If this is the case, but you are not seeing 8 concurrent MajC's at ingest time, then I would think that you are not pushing it hard enough.

I am less familiar with the minor compaction initiation process, but in general if you are not seeing hold times, push harder. You could also test your `tserver.compaction.minor.concurrent.max` setting by performing a flush command on the table from the shell.

On March 23, 2017 at 10:13 AM Massimilian Mattetti <[hidden email]> wrote:

With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max



From:        Dave Marion <[hidden email]>
To:        [hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        23/03/2017 14:55
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]> wrote:

Hi All,


I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:


table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30


tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8


My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:


default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8


default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50


Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.


Regards,
Max






 
Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Michael Wall
In reply to this post by Massimilian Mattetti
Max,

So your max major compactions will be 3 per tablet server.  Accumulo will not run 2 majors on the same tablet concurrently.

Mike

On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <[hidden email]> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is uniform distributed among each tablet.
Max




From:        Michael Wall <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 16:28
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <[hidden email]> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <[hidden email]>
To:        
[hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
[hidden email]> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max







Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
In reply to this post by Marc P.
The tserver.compaction.major.thread.files.open.max=10 and I have a maximum of 30 files per tablet (table.file.max=30). What property are you referring to for the max open files?
Thanks,
Max




From:        "Marc P." <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 16:37
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,
  Additionally, what are your max open files? If you are under heavy ingest you may exceed this with compactions if they cannot keep up. Additionally, how many files per compaction are you allowing? 

On Thu, Mar 23, 2017 at 10:27 AM, Michael Wall <mjwall@...> wrote:
Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@...> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <dlmarion@...>
To:        
user@..., Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
MASSIMIL@...> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max








Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
In reply to this post by Michael Wall
I wasn't aware of such constrain. So I can just increase the number of tablets per server and it will perform more major compactions.
Thanks,
Max




From:        Michael Wall <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 17:09
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

So your max major compactions will be 3 per tablet server.  Accumulo will not run 2 majors on the same tablet concurrently.

Mike

On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <MASSIMIL@...> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is uniform distributed among each tablet.
Max





From:        
Michael Wall <mjwall@...>
To:        
user@...
Date:        
23/03/2017 16:28
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <
MASSIMIL@...> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <dlmarion@...>
To:        
user@..., Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
MASSIMIL@...> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max









Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Michael Wall
Yes, until you hit another constraint like Marc and Dave were asking about.

Mike

On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <[hidden email]> wrote:
I wasn't aware of such constrain. So I can just increase the number of tablets per server and it will perform more major compactions.
Thanks,
Max




From:        Michael Wall <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 17:09
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

So your max major compactions will be 3 per tablet server.  Accumulo will not run 2 majors on the same tablet concurrently.

Mike

On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <[hidden email]> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is uniform distributed among each tablet.
Max





From:        
Michael Wall <[hidden email]>
To:        
[hidden email]
Date:        
23/03/2017 16:28
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <
[hidden email]> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <[hidden email]>
To:        
[hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
[hidden email]> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max









Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Massimilian Mattetti
I just tested the ingestion with 10 tablets per Server and now the system achieve up to 24 concurrent major compactions.

I have another issue with the tserver.memory.maps.max property. I am play with the size of the native map to track how its size affect the ingestion performance. I set the values of table.compaction.minor.logs.threshold and tserver.walog.max.size big enough so I can increase the map size up to 40GB without problems. At the beginning changing the size of the map (using the shell) didn't bring any benefit to the ingestion process. Monitoring the servers I noticed that their RAM consumption was constant. Eventually I tried to restart the tablet servers after changing the tserver.memory.maps.max and the RAM usage on each node increased as I expected. From the documentation I know that any change to the tserver.memory.maps.max should take effect without restarting the tablet servers, is that always true? (tserver.memory.maps.native.enabled has always been true and from the logs I see that the shared library has been correctly loaded).
Thanks!

Best Regards,
Max



From:        Michael Wall <[hidden email]>
To:        [hidden email]
Date:        23/03/2017 17:49
Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Yes, until you hit another constraint like Marc and Dave were asking about.

Mike

On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <MASSIMIL@...> wrote:
I wasn't aware of such constrain. So I can just increase the number of tablets per server and it will perform more major compactions.
Thanks,
Max





From:        
Michael Wall <mjwall@...>
To:        
user@...
Date:        
23/03/2017 17:09
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

So your max major compactions will be 3 per tablet server.  Accumulo will not run 2 majors on the same tablet concurrently.

Mike

On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <
MASSIMIL@...> wrote:
One table, 9 pre-split tablets, 3 tablets per server and the data is uniform distributed among each tablet.
Max





From:        
Michael Wall <mjwall@...>
To:        
user@...
Date:        
23/03/2017 16:28
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Max,

On you 3 node cluster, how many tables are you ingesting into?  How many tablets are in each table?  Are the tablets equally spread amongst the 3 tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <
MASSIMIL@...> wrote:
With the configuration I presented before the concurrent major compactions are never more than 3 per tablet server while the minor are under the 4 per node. Can one of the other configurations be the cause of this behavior?

Regards,
Max




From:        
Dave Marion <dlmarion@...>
To:        
user@..., Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        
23/03/2017 14:55
Subject:        
Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1




Can you explain more what you mean by "My problem is that both the minor and major compactions do not overcome their default max values?" I have done some testing with 1.8.1 and specifically modifying tserver.compaction.major.concurrent.max to a higher number and I have seen it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <
MASSIMIL@...> wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome their default max values. I checked the config from the shell and it looks fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override ........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override ........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the previous version.
Thanks.

Regards,
Max











Reply | Threaded
Open this post in threaded view
|

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

Keith Turner
On Thu, Mar 23, 2017 at 12:37 PM, Massimilian Mattetti
<[hidden email]> wrote:

> I just tested the ingestion with 10 tablets per Server and now the system
> achieve up to 24 concurrent major compactions.
>
> I have another issue with the tserver.memory.maps.max property. I am play
> with the size of the native map to track how its size affect the ingestion
> performance. I set the values of table.compaction.minor.logs.threshold and
> tserver.walog.max.size big enough so I can increase the map size up to 40GB
> without problems. At the beginning changing the size of the map (using the
> shell) didn't bring any benefit to the ingestion process. Monitoring the
> servers I noticed that their RAM consumption was constant. Eventually I
> tried to restart the tablet servers after changing the
> tserver.memory.maps.max and the RAM usage on each node increased as I
> expected. From the documentation I know that any change to the
> tserver.memory.maps.max should take effect without restarting the tablet
> servers, is that always true? (tserver.memory.maps.native.enabled has always
> been true and from the logs I see that the shared library has been correctly
> loaded).
> Thanks!

I took a quick look at the 1.6.5 code and it does not seem to update
after tserver start in that version.

>
> Best Regards,
> Max
>
>
>
> From:        Michael Wall <[hidden email]>
> To:        [hidden email]
> Date:        23/03/2017 17:49
>
> Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Yes, until you hit another constraint like Marc and Dave were asking about.
>
> Mike
>
> On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti <[hidden email]>
> wrote:
> I wasn't aware of such constrain. So I can just increase the number of
> tablets per server and it will perform more major compactions.
> Thanks,
> Max
>
>
>
>
> From:        Michael Wall <[hidden email]>
> To:        [hidden email]
> Date:        23/03/2017 17:09
> Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Max,
>
> So your max major compactions will be 3 per tablet server.  Accumulo will
> not run 2 majors on the same tablet concurrently.
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti <[hidden email]>
> wrote:
> One table, 9 pre-split tablets, 3 tablets per server and the data is uniform
> distributed among each tablet.
> Max
>
>
>
>
> From:        Michael Wall <[hidden email]>
> To:        [hidden email]
> Date:        23/03/2017 16:28
> Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into?  How many
> tablets are in each table?  Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <[hidden email]>
> wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From:        Dave Marion <[hidden email]>
> To:        [hidden email], Massimilian Mattetti/Haifa/IBM@IBMIL
> Date:        23/03/2017 14:55
> Subject:        Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> ________________________________
>
>
>
> Can you explain more what you mean by "My problem is that both the minor and
> major compactions do not overcome their default max values?" I have done
> some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti <[hidden email]>
> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default    | tserver.compaction.major.concurrent.max ................ | 3
> system    |                     @override
> ........................................... | 8
>
> default    | tserver.compaction.minor.concurrent.max ............... | 4
> system    |                     @override
> ........................................... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous version.
> Thanks.
>
> Regards,
> Max
>
>
>
>
>
>
>
>
>
>