[infinispan-dev] [!] Reorganization of dependencies & release process

[Adrian Nistor]

+1 for moving Infinispan lucene directory out

But why move Query components out? And which ones did you have in mind?

On 05/14/2014 12:50 AM, Sanne Grinovero wrote:
> This is a reboot of the thread previously started on both the
> infinispan-dev and the hibernate-dev mailing list as "Handling of
> mutual dependency with Infinispan" [1].
> We discussed further during the Hibernate fortnightly meeting [2], and
> came to the conclusion that we need Infinispan to change how some
> repositories are organised and how the release is assembled.
>
> # The problem
>
> To restate the issue, as you might painfully remember, every time
> there is a need for a Lucene update or a Search update we need to sync
> up for a complex dance of releases in both projects to accommodate for
> a small-step iterative process to handle the circular dependency.
> This problem is not too bad today as since a year we're releasing the
> Lucene Directory in an unusual - and very unmaintainable - temporary
> solution to be compatible with two different major versions of Apache
> Lucene; namely what Infinispan Query needs and what Hibernate Search
> needs are different modules.
> But the party is over, and I want to finally drop support for Lucene 3
> and cleanup the unusual and unmaintainable build mess targeting a
> single Lucene version only.
> As soon as we converge to building a single version however - we're
> back to the complex problem we had when we supported a single version
> which is handling of a circular dependency - just that the problem has
> worsened lately the Lucene project has been more active and more
> inclined than what it used to be to break both internal and public
> APIs.
>
> In short, we have a circular dependency between Hibernate Search and
> Infinispan which we've been able to handle via hacks and some luck,
> but it imposes a serious threat to development flexibility, and the
> locked-in release process is not desirable either.
>
> # The solution
>
> we think in conclusion there's a single "proper" way out, and it also
> happens to provide some very interesting side effects in terms of
> maintenance overhead for everyone: Infinispan Core needs to release
> independently from the non-core modules.
> This would have the Lucene Directory depend on a released tag of
> infinispan-core, and be able to be released independently.
> Minor situations with benefit:
>   - we often don't make any change in the Lucene Directory, still we
> need to release it.
>   - when I actually need a release of it, I'm currently begging for a
> quick release of Infinispan: very costly
> The Big Ones:
>   - we can manage the Lucene Directory to provide support for different
> versions of Lucene without necessarily breaking other modules
>   - we can release quickly what's needed to move Search ahead in terms
> of Lucene versions without needing to make the Infinispan Query module
> compatible at the same time (in case you haven't followed this area:
> this seems to be my main activity rather than making valuable stuff).
>
> The goal is of course to linearise the dependencies; it seems to also
> simplify some of our tasks which is a welcome side-effect. I expect it
> also to make the project less scary for new contributors.
>
> # How does it impact users
>
> ## Maven users
> modules will continue to be modules.. I guess nobody will notice,
> other than we might have a different versioning scheme, but we help
> people out via the Infinispan BOM.
>
> ## Distribution users
> There should be no difference, other than (as well) some jars might
> not be aligned in terms of version. But that's probably even less of a
> problem, as I expect distribution users to just put what they get on
> their classpath.
>
> # How it impacts us
>
> 1) I'll move the Lucene Directory project to an different repository;
> same for the Query related components.
> I think you should/could consider the same for other components, based
> on ad-hoc considerations of the trade offs, but I'd expect ultimately
> to see a more frequent and "core only" release.
>
> 2) We'll have different kinds of releases: the "core only" and the
> "full releases".
> I think we'll also see components being released independently, but
> these are either Maven-only or meant for preparation of other
> components, or preparation for a "full release".
>
> 3) Tests (!)
> Such a move should in no way relax the regression-safety of
> infinispan-core: we need to still consider it unacceptable for a core
> change to break one of the modules moving out of the main tree.
> Personally I think I've pushed many tests about problems found in the
> "query modules" as unit tests in core, so that should be relatively
> safe, but it also happened that someone would "tune" these.
> I realise it's not practical to expect people to run tests of
> downstream modules, so we'll have to automate most of these tasks in
> CI.
> Careful on perception: if today there are three levels of defence
> against a regression (the author, the reviewer and CI all running the
> suite for each change), in such an organisation you have only one. So
> ignoring a CI failure as a "probable hiccup" could be much more
> dangerous than usual.
>
> # When
>
> Doing this _might_ be a blocker for any Lucene update; so since one
> just happened I'll probably have no urgent need for a couple of weeks
> at least.
> But we shouldn't be in a position in which an update could not be
> possible, so I hope we'll agree to implement this sooner rather than
> later, so we won't have to do it during an emergency.
>
> Also while this might sound a bit crazy at first, I see many
> flexibility benefits which can't hurt now that the project is getting
> larger and more complex to release.
> Not least, having a micro release of "Infinispan essentials" would be
> very welcome in terms of lowing the initial barrier; this was proposed
> at various meetings and highly endorsed by many but just never
> happened.
>
> Any comment please? I hope I covered it all, and sorry for that :D
>
> Cheers,
> Sanne
>
>
> 1 - http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011419.html
> 2 - http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2014/hibernate-dev.2014-05-13-13.24.log.html
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev