Dynamic set of visibility labels / Trusted query

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Dynamic set of visibility labels / Trusted query

Rob Verkuylen
Hi,

I'm designing a table where the ColumnVisibility is based off of labels within the data as it comes in. I know the formatting, but not the exact label names and the labels change over time.

Based on the labels of the data which I can use for ColumnVisibility and the user's access to certain labels(determined via LDAP group memberchip) I want to use the labels as a filtering mechanism.

Questions are: Is this the best approach? Obviously this works client side, but I want to push the filtering to the server. It seems that I need to assign every possible label to a user, to get access to it. Ideally I would have a trusted used with all access and the query engine extracts the labels from the current user and uses them as authorisation labels.

I guess this is also possible with (custom?) filtering on the server, but this would seem the ideal use case for visibility labels with Accumulo to me.

Thnx, Rob
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic set of visibility labels / Trusted query

Josh Elser-2
I think you know this already, but I'm not 100% sure based on your
message: trying to change the labels on the data is a bad idea. If you
need to handle a case where a label means one thing on day1 and another
thing on day2, you would need to build the logic to handle that.

The only other thing I thought of is that you will have to write code
that updates your "trusted user" to have all of the authorization labels
that you might use. Even the Accumulo "superuser" must have the proper
authorizations set that you want to use.

On 9/10/18 9:40 AM, Rob Verkuylen wrote:

> Hi,
>
> I'm designing a table where the ColumnVisibility is based off of labels
> within the data as it comes in. I know the formatting, but not the exact
> label names and the labels change over time.
>
> Based on the labels of the data which I can use for ColumnVisibility and
> the user's access to certain labels(determined via LDAP group
> memberchip) I want to use the labels as a filtering mechanism.
>
> Questions are: Is this the best approach? Obviously this works client
> side, but I want to push the filtering to the server. It seems that I
> need to assign every possible label to a user, to get access to it.
> Ideally I would have a trusted used with all access and the query engine
> extracts the labels from the current user and uses them as authorisation
> labels.
>
> I guess this is also possible with (custom?) filtering on the server,
> but this would seem the ideal use case for visibility labels with
> Accumulo to me.
>
> Thnx, Rob
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic set of visibility labels / Trusted query

Rob Verkuylen
Thnx Josh, 

You're right, I wont change the labels or their meaning. In this use case I have sensor data coming in from X amount of sensors, all labeled SensorX. New sensor data comes in every day from existing sources, but also from newly installed sensors with a previously unknown but unique name. I want to label the data with the sensorname, so I can give ppl access to the sensordata belonging to them specifically or their team. 

The amount of sensors can run into the thousands, meaning that amount of labels added to the user. I get that adding auth labels makes sense when the labels are pretty much static, but what of use cases like these? Would a filtering iterator be a better match? 

Thnx, Rob

On Mon, Sep 10, 2018 at 6:04 PM, Josh Elser <[hidden email]> wrote:
I think you know this already, but I'm not 100% sure based on your message: trying to change the labels on the data is a bad idea. If you need to handle a case where a label means one thing on day1 and another thing on day2, you would need to build the logic to handle that.

The only other thing I thought of is that you will have to write code that updates your "trusted user" to have all of the authorization labels that you might use. Even the Accumulo "superuser" must have the proper authorizations set that you want to use.


On 9/10/18 9:40 AM, Rob Verkuylen wrote:
Hi,

I'm designing a table where the ColumnVisibility is based off of labels within the data as it comes in. I know the formatting, but not the exact label names and the labels change over time.

Based on the labels of the data which I can use for ColumnVisibility and the user's access to certain labels(determined via LDAP group memberchip) I want to use the labels as a filtering mechanism.

Questions are: Is this the best approach? Obviously this works client side, but I want to push the filtering to the server. It seems that I need to assign every possible label to a user, to get access to it. Ideally I would have a trusted used with all access and the query engine extracts the labels from the current user and uses them as authorisation labels.

I guess this is also possible with (custom?) filtering on the server, but this would seem the ideal use case for visibility labels with Accumulo to me.

Thnx, Rob

Reply | Threaded
Open this post in threaded view
|

Re: Dynamic set of visibility labels / Trusted query

Josh Elser-2
Remember that visibility labels are effectively post-filters. If you
want to support access patterns of data based on specific data sources,
you would need to structure your data in such a way that you can find
all of the data from that sensor fast.

If the kind of data you get from one sensor is static (e.g. always of a
specific type), I'd suggest you build labels based on that. Labels
describing the data itself, instead of the sensor which that data came from.

If you need to filter data purely based on the sensor the data came
from, I can't think of a better approach than tagging each sensor. This
will have limitations when it comes to the built-in ZooKeeper-backed
visibility labels implementation which you will have to eventually
consider replacing (the ZK impl can probably only scale to the
10000s-100000s of labels)

On 9/11/18 10:01 AM, Rob Verkuylen wrote:

> Thnx Josh,
>
> You're right, I wont change the labels or their meaning. In this use
> case I have sensor data coming in from X amount of sensors, all labeled
> SensorX. New sensor data comes in every day from existing sources, but
> also from newly installed sensors with a previously unknown but unique
> name. I want to label the data with the sensorname, so I can give ppl
> access to the sensordata belonging to them specifically or their team.
>
> The amount of sensors can run into the thousands, meaning that amount of
> labels added to the user. I get that adding auth labels makes sense when
> the labels are pretty much static, but what of use cases like these?
> Would a filtering iterator be a better match?
>
> Thnx, Rob
>
> On Mon, Sep 10, 2018 at 6:04 PM, Josh Elser <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I think you know this already, but I'm not 100% sure based on your
>     message: trying to change the labels on the data is a bad idea. If
>     you need to handle a case where a label means one thing on day1 and
>     another thing on day2, you would need to build the logic to handle that.
>
>     The only other thing I thought of is that you will have to write
>     code that updates your "trusted user" to have all of the
>     authorization labels that you might use. Even the Accumulo
>     "superuser" must have the proper authorizations set that you want to
>     use.
>
>
>     On 9/10/18 9:40 AM, Rob Verkuylen wrote:
>
>         Hi,
>
>         I'm designing a table where the ColumnVisibility is based off of
>         labels within the data as it comes in. I know the formatting,
>         but not the exact label names and the labels change over time.
>
>         Based on the labels of the data which I can use for
>         ColumnVisibility and the user's access to certain
>         labels(determined via LDAP group memberchip) I want to use the
>         labels as a filtering mechanism.
>
>         Questions are: Is this the best approach? Obviously this works
>         client side, but I want to push the filtering to the server. It
>         seems that I need to assign every possible label to a user, to
>         get access to it. Ideally I would have a trusted used with all
>         access and the query engine extracts the labels from the current
>         user and uses them as authorisation labels.
>
>         I guess this is also possible with (custom?) filtering on the
>         server, but this would seem the ideal use case for visibility
>         labels with Accumulo to me.
>
>         Thnx, Rob
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Dynamic set of visibility labels / Trusted query

Rob Verkuylen
Alright given your feedback I think a slightly modified filtering iterator can do the job as well. In reality I need to not only give acces based on source/sensor, but also based off of other labels like legal, enrichments, flags etc. Some of there are more static, so maybe a combination of both Auth/ColViz and the custom filter.  Thnx

On Tue, Sep 11, 2018 at 4:40 PM, Josh Elser <[hidden email]> wrote:
Remember that visibility labels are effectively post-filters. If you want to support access patterns of data based on specific data sources, you would need to structure your data in such a way that you can find all of the data from that sensor fast.

If the kind of data you get from one sensor is static (e.g. always of a specific type), I'd suggest you build labels based on that. Labels describing the data itself, instead of the sensor which that data came from.

If you need to filter data purely based on the sensor the data came from, I can't think of a better approach than tagging each sensor. This will have limitations when it comes to the built-in ZooKeeper-backed visibility labels implementation which you will have to eventually consider replacing (the ZK impl can probably only scale to the 10000s-100000s of labels)

On 9/11/18 10:01 AM, Rob Verkuylen wrote:
Thnx Josh,

You're right, I wont change the labels or their meaning. In this use case I have sensor data coming in from X amount of sensors, all labeled SensorX. New sensor data comes in every day from existing sources, but also from newly installed sensors with a previously unknown but unique name. I want to label the data with the sensorname, so I can give ppl access to the sensordata belonging to them specifically or their team.

The amount of sensors can run into the thousands, meaning that amount of labels added to the user. I get that adding auth labels makes sense when the labels are pretty much static, but what of use cases like these? Would a filtering iterator be a better match?

Thnx, Rob

On Mon, Sep 10, 2018 at 6:04 PM, Josh Elser <[hidden email] <mailto:[hidden email]>> wrote:

    I think you know this already, but I'm not 100% sure based on your
    message: trying to change the labels on the data is a bad idea. If
    you need to handle a case where a label means one thing on day1 and
    another thing on day2, you would need to build the logic to handle that.

    The only other thing I thought of is that you will have to write
    code that updates your "trusted user" to have all of the
    authorization labels that you might use. Even the Accumulo
    "superuser" must have the proper authorizations set that you want to
    use.


    On 9/10/18 9:40 AM, Rob Verkuylen wrote:

        Hi,

        I'm designing a table where the ColumnVisibility is based off of
        labels within the data as it comes in. I know the formatting,
        but not the exact label names and the labels change over time.

        Based on the labels of the data which I can use for
        ColumnVisibility and the user's access to certain
        labels(determined via LDAP group memberchip) I want to use the
        labels as a filtering mechanism.

        Questions are: Is this the best approach? Obviously this works
        client side, but I want to push the filtering to the server. It
        seems that I need to assign every possible label to a user, to
        get access to it. Ideally I would have a trusted used with all
        access and the query engine extracts the labels from the current
        user and uses them as authorisation labels.

        I guess this is also possible with (custom?) filtering on the
        server, but this would seem the ideal use case for visibility
        labels with Accumulo to me.

        Thnx, Rob