How to Perform an True Update of a Record?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How to Perform an True Update of a Record?

gtotsline
Hi -

I have an Accumulo table with records that for certain situations I would
like to do a true update on, i.e. I want to:
1. Scan or Batch Scan a record
2. Make some updates to that record
3. Write the record so that it *replaces *the current Accumulo version of
that record (and not create a new version).

The VersioningIterator is removed from this table and while I understand I
can add it dynamically to the scan, our real objective is to not pile up a
bunch of versioned records. We need to use Accumulo versioning for some use
cases for this table, but in other use cases do not want Accumulo to create
a new version on a mutate, we want to effectively update the latest version
of the record.  Any thoughts on how to accomplish this?

Thanks!



--
Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

Mike Miller-2
Perhaps you want to use conditional mutations for your use case?  Here is
an example from the Accumulo Tour which uses the ConditionalWriter:
https://accumulo.apache.org/tour/conditional-writer/


On Wed, Jan 23, 2019 at 1:03 PM gtotsline <[hidden email]> wrote:

> Hi -
>
> I have an Accumulo table with records that for certain situations I would
> like to do a true update on, i.e. I want to:
> 1. Scan or Batch Scan a record
> 2. Make some updates to that record
> 3. Write the record so that it *replaces *the current Accumulo version of
> that record (and not create a new version).
>
> The VersioningIterator is removed from this table and while I understand I
> can add it dynamically to the scan, our real objective is to not pile up a
> bunch of versioned records. We need to use Accumulo versioning for some use
> cases for this table, but in other use cases do not want Accumulo to create
> a new version on a mutate, we want to effectively update the latest version
> of the record.  Any thoughts on how to accomplish this?
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

gtotsline
Hi Mike -

Thanks for responding so quickly, it's greatly appreciated.  The
ConditionalWriter does not appear to address our use case where we actually
want to suppress Accumulo versioning  of a record based on the value in the
record that was read vs. input data our system receives.  Is there a way to
dynamically suppress Accumulo record versioning?



--
Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

Mike Miller-2
> Is there a way to dynamically suppress Accumulo record versioning?

Not that I know of but perhaps others have come up with ways to do this.

What if you set the maxVersions on your table to 1 (config -t mytable -s
table.iterator.majc.vers.opt.maxVersions=1) and use conditional mutation to
determine whether to update or not? Then you wouldn't have to worry about
versions, just updating the data based on input.

On Wed, Jan 23, 2019 at 1:27 PM gtotsline <[hidden email]> wrote:

> Hi Mike -
>
> Thanks for responding so quickly, it's greatly appreciated.  The
> ConditionalWriter does not appear to address our use case where we actually
> want to suppress Accumulo versioning  of a record based on the value in the
> record that was read vs. input data our system receives.  Is there a way to
> dynamically suppress Accumulo record versioning?
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

Josh Elser-2
In reply to this post by gtotsline
Why are you trying to do this in the first place? When you write a new
version of a cell, you are essentially replacing the old value. Leaving
the old value in the table and lazily removing it (via compaction) is a
core optimization to the write-path for Accumulo (from BigTable itself).

I'm having a hard time understanding why you're trying to do what you're
asking.

On 1/23/19 1:26 PM, gtotsline wrote:

> Hi Mike -
>
> Thanks for responding so quickly, it's greatly appreciated.  The
> ConditionalWriter does not appear to address our use case where we actually
> want to suppress Accumulo versioning  of a record based on the value in the
> record that was read vs. input data our system receives.  Is there a way to
> dynamically suppress Accumulo record versioning?
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

gtotsline
Josh Elser-2 wrote
> Why are you trying to do this in the first place? When you write a new
> version of a cell, you are essentially replacing the old value. Leaving
> the old value in the table and lazily removing it (via compaction) is a
> core optimization to the write-path for Accumulo (from BigTable itself).
>
> I'm having a hard time understanding why you're trying to do what you're
> asking.

Hi Josh -

The reason we are trying to do this is that we have two competing
requirements:
1. Maintain an unbounded versioned history of a certain subset or types of
updates (for audit and forensics purposes) for a row
2. Do not add to that row's versioned history for a certain other subset or
types of updates.

So in the same table we want to take advantage of Accumulo's versioning for
some change cases for a record, while for others we would like to avoid new
versions being created because of the very high volume of changes
anticipated.

Thanks.




--
Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

gtotsline
In reply to this post by Mike Miller-2
Mike Miller-2 wrote
>> Is there a way to dynamically suppress Accumulo record versioning?
>
> Not that I know of but perhaps others have come up with ways to do this.
>
> What if you set the maxVersions on your table to 1 (config -t mytable -s
> table.iterator.majc.vers.opt.maxVersions=1) and use conditional mutation
> to
> determine whether to update or not? Then you wouldn't have to worry about
> versions, just updating the data based on input.

The problem is, we want versions of a record to be created for some change
cases for the record while in others change cases for the same record we do
not (some changes are very high volume so we want to avoid filling up the DB
while still using it to maintain a version history select set of updates).  



--
Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

Mike Miller-2
What about creating a separate table to keep the history, with versioning
and write to this table when appropriate?

On Wed, Jan 23, 2019 at 2:26 PM gtotsline <[hidden email]> wrote:

> Mike Miller-2 wrote
> >> Is there a way to dynamically suppress Accumulo record versioning?
> >
> > Not that I know of but perhaps others have come up with ways to do this.
> >
> > What if you set the maxVersions on your table to 1 (config -t mytable -s
> > table.iterator.majc.vers.opt.maxVersions=1) and use conditional mutation
> > to
> > determine whether to update or not? Then you wouldn't have to worry about
> > versions, just updating the data based on input.
>
> The problem is, we want versions of a record to be created for some change
> cases for the record while in others change cases for the same record we do
> not (some changes are very high volume so we want to avoid filling up the
> DB
> while still using it to maintain a version history select set of
> updates).
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to Perform an True Update of a Record?

gtotsline
Mike Miller-2 wrote
> What about creating a separate table to keep the history, with versioning
> and write to this table when appropriate?

Mike - brilliant, we just arrived at the same conclusion, two tables:
1. Max Versions is set to 1 and holds the "true updates" we are seeking
without filling up the DB with version records.
2. No limit on versions (i.e. VersionIterator is removed from table) and
holds the changes cases for each record that we have to support for
forensics and audit purposes.



--
Sent from: http://apache-accumulo.1065345.n5.nabble.com/Developers-f3.html