[DISCUSS] Hadoop 3 and our dependencies generally

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[DISCUSS] Hadoop 3 and our dependencies generally

Sean Busbey
Hi Folks!

I think we need to start being more formal in planning for Hadoop 3.
They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].

What do folks think about starting to push on an Accumulo 2.0 release that
only supports Hadoop 3? It would let us move faster, which we'll need to do
if we want to get any API changes in to the Hadoop 3 line.

If we get started soon we can probably make it parity on beta/GA status
with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
GA by end of the year.

Going to just Hadoop 3+ would also be a sufficient dependency break that we
could do a pass updating any lagging dependencies to latest major version.


[1]: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
--
busbey
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Michael Hogue
All,

   For what it's worth, I'd attempted to run both 1.7.x and 1.8.x on Hadoop
3 and ran into a fairly straightforward dependency issue [1] that, when
addressed, should allow current Accumulo versions to run on Hadoop 3.
Hopefully this means that it's not a large lift to get to the point you're
aiming for.

Thanks,
Mike

[1] https://issues.apache.org/jira/browse/ACCUMULO-4611


On Thu, Aug 3, 2017 at 10:09 AM Sean Busbey <[hidden email]> wrote:

> Hi Folks!
>
> I think we need to start being more formal in planning for Hadoop 3.
> They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].
>
> What do folks think about starting to push on an Accumulo 2.0 release that
> only supports Hadoop 3? It would let us move faster, which we'll need to do
> if we want to get any API changes in to the Hadoop 3 line.
>
> If we get started soon we can probably make it parity on beta/GA status
> with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
> GA by end of the year.
>
> Going to just Hadoop 3+ would also be a sufficient dependency break that we
> could do a pass updating any lagging dependencies to latest major version.
>
>
> [1]:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> --
> busbey
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Keith Turner
In reply to this post by Sean Busbey
I am in favor of going to Hadoop 3 for Accumulo 2.  If we do this then
Accumulo 2 can not release until after Hadoop 3 does.  Any idea when
Hadoop 3 will release?

On Thu, Aug 3, 2017 at 10:08 AM, Sean Busbey <[hidden email]> wrote:

> Hi Folks!
>
> I think we need to start being more formal in planning for Hadoop 3.
> They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].
>
> What do folks think about starting to push on an Accumulo 2.0 release that
> only supports Hadoop 3? It would let us move faster, which we'll need to do
> if we want to get any API changes in to the Hadoop 3 line.
>
> If we get started soon we can probably make it parity on beta/GA status
> with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
> GA by end of the year.
>
> Going to just Hadoop 3+ would also be a sufficient dependency break that we
> could do a pass updating any lagging dependencies to latest major version.
>
>
> [1]: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> --
> busbey
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Josh Elser-2
In reply to this post by Sean Busbey
+1 sounds like a good idea to me.

On 8/3/17 10:08 AM, Sean Busbey wrote:

> Hi Folks!
>
> I think we need to start being more formal in planning for Hadoop 3.
> They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].
>
> What do folks think about starting to push on an Accumulo 2.0 release that
> only supports Hadoop 3? It would let us move faster, which we'll need to do
> if we want to get any API changes in to the Hadoop 3 line.
>
> If we get started soon we can probably make it parity on beta/GA status
> with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
> GA by end of the year.
>
> Going to just Hadoop 3+ would also be a sufficient dependency break that we
> could do a pass updating any lagging dependencies to latest major version.
>
>
> [1]: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Christopher Tubbs-2
+1 from me, too, but I'd like to review what actually changes in master for
the migration to happen. I don't know much about Hadoop 3. I'm curious what
the releases will look like (AFAIK, it's only snapshot builds right now; is
that correct?), how our dependencies will change, and what API stability
guarantees it offers.

On Thu, Aug 3, 2017 at 12:59 PM Josh Elser <[hidden email]> wrote:

> +1 sounds like a good idea to me.
>
> On 8/3/17 10:08 AM, Sean Busbey wrote:
> > Hi Folks!
> >
> > I think we need to start being more formal in planning for Hadoop 3.
> > They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].
> >
> > What do folks think about starting to push on an Accumulo 2.0 release
> that
> > only supports Hadoop 3? It would let us move faster, which we'll need to
> do
> > if we want to get any API changes in to the Hadoop 3 line.
> >
> > If we get started soon we can probably make it parity on beta/GA status
> > with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
> > GA by end of the year.
> >
> > Going to just Hadoop 3+ would also be a sufficient dependency break that
> we
> > could do a pass updating any lagging dependencies to latest major
> version.
> >
> >
> > [1]:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Mike Drob-4
There is a 3.0.0-alpha4 release currently available as a non-snapshot
version.

I'm not sure it comes with API stability guarantees at all, IIRC the Hadoop
community is planning on providing that for their betas.

Mike

On Thu, Aug 3, 2017 at 12:17 PM, Christopher <[hidden email]> wrote:

> +1 from me, too, but I'd like to review what actually changes in master for
> the migration to happen. I don't know much about Hadoop 3. I'm curious what
> the releases will look like (AFAIK, it's only snapshot builds right now; is
> that correct?), how our dependencies will change, and what API stability
> guarantees it offers.
>
> On Thu, Aug 3, 2017 at 12:59 PM Josh Elser <[hidden email]> wrote:
>
> > +1 sounds like a good idea to me.
> >
> > On 8/3/17 10:08 AM, Sean Busbey wrote:
> > > Hi Folks!
> > >
> > > I think we need to start being more formal in planning for Hadoop 3.
> > > They're up to 3.0.0-alpha4 and are pushing towards API-locking
> betas[1].
> > >
> > > What do folks think about starting to push on an Accumulo 2.0 release
> > that
> > > only supports Hadoop 3? It would let us move faster, which we'll need
> to
> > do
> > > if we want to get any API changes in to the Hadoop 3 line.
> > >
> > > If we get started soon we can probably make it parity on beta/GA status
> > > with the Hadoop 3 line. That would give us a beta for Accumulo Summit
> and
> > > GA by end of the year.
> > >
> > > Going to just Hadoop 3+ would also be a sufficient dependency break
> that
> > we
> > > could do a pass updating any lagging dependencies to latest major
> > version.
> > >
> > >
> > > [1]:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Hadoop 3 and our dependencies generally

Sean Busbey
In reply to this post by Keith Turner
the RM for Hadoop 3.0.0 has been periodically updating when he aims to have
release votes done:

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release

Obviously not guaranteed, since it is an active ASF project. ;)

We can have Accumulo releases, just not GA releases. So we could mimic
Hadoop's versioning and vote out 2.0.0-alpha-X, 2.0.0-beta-X, etc. We would
expressly call out that our normal compatibility promises wouldn't apply to
these. Or that only a reduced form applies if we have some assurances from
Hadoop.

As for migration, maybe after we know what Accumulo 2.0 on Hadoop 3.0 looks
like (I guess beta timeframe?) we can then work out if we should add
Hadoop2 and Hadoop 3 support in a future 1.y release and require
double-rolling-upgrade for no downtime upgrade.

On Thu, Aug 3, 2017 at 10:28 AM, Keith Turner <[hidden email]> wrote:

> I am in favor of going to Hadoop 3 for Accumulo 2.  If we do this then
> Accumulo 2 can not release until after Hadoop 3 does.  Any idea when
> Hadoop 3 will release?
>
> On Thu, Aug 3, 2017 at 10:08 AM, Sean Busbey <[hidden email]> wrote:
> > Hi Folks!
> >
> > I think we need to start being more formal in planning for Hadoop 3.
> > They're up to 3.0.0-alpha4 and are pushing towards API-locking betas[1].
> >
> > What do folks think about starting to push on an Accumulo 2.0 release
> that
> > only supports Hadoop 3? It would let us move faster, which we'll need to
> do
> > if we want to get any API changes in to the Hadoop 3 line.
> >
> > If we get started soon we can probably make it parity on beta/GA status
> > with the Hadoop 3 line. That would give us a beta for Accumulo Summit and
> > GA by end of the year.
> >
> > Going to just Hadoop 3+ would also be a sufficient dependency break that
> we
> > could do a pass updating any lagging dependencies to latest major
> version.
> >
> >
> > [1]: https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+3.0.0+release
> > --
> > busbey
>



--
busbey
Loading...