[infinispan-dev] [!] Reorganization of dependencies & release process

[Sanne Grinovero]

This is a reboot of the thread previously started on both the
infinispan-dev and the hibernate-dev mailing list as "Handling of
mutual dependency with Infinispan" [1].
We discussed further during the Hibernate fortnightly meeting [2], and
came to the conclusion that we need Infinispan to change how some
repositories are organised and how the release is assembled.

# The problem

To restate the issue, as you might painfully remember, every time
there is a need for a Lucene update or a Search update we need to sync
up for a complex dance of releases in both projects to accommodate for
a small-step iterative process to handle the circular dependency.
This problem is not too bad today as since a year we're releasing the
Lucene Directory in an unusual - and very unmaintainable - temporary
solution to be compatible with two different major versions of Apache
Lucene; namely what Infinispan Query needs and what Hibernate Search
needs are different modules.
But the party is over, and I want to finally drop support for Lucene 3
and cleanup the unusual and unmaintainable build mess targeting a
single Lucene version only.
As soon as we converge to building a single version however - we're
back to the complex problem we had when we supported a single version
which is handling of a circular dependency - just that the problem has
worsened lately the Lucene project has been more active and more
inclined than what it used to be to break both internal and public
APIs.

In short, we have a circular dependency between Hibernate Search and
Infinispan which we've been able to handle via hacks and some luck,
but it imposes a serious threat to development flexibility, and the
locked-in release process is not desirable either.

# The solution

we think in conclusion there's a single "proper" way out, and it also
happens to provide some very interesting side effects in terms of
maintenance overhead for everyone: Infinispan Core needs to release
independently from the non-core modules.
This would have the Lucene Directory depend on a released tag of
infinispan-core, and be able to be released independently.
Minor situations with benefit:
 - we often don't make any change in the Lucene Directory, still we
need to release it.
 - when I actually need a release of it, I'm currently begging for a
quick release of Infinispan: very costly
The Big Ones:
 - we can manage the Lucene Directory to provide support for different
versions of Lucene without necessarily breaking other modules
 - we can release quickly what's needed to move Search ahead in terms
of Lucene versions without needing to make the Infinispan Query module
compatible at the same time (in case you haven't followed this area:
this seems to be my main activity rather than making valuable stuff).

The goal is of course to linearise the dependencies; it seems to also
simplify some of our tasks which is a welcome side-effect. I expect it
also to make the project less scary for new contributors.

# How does it impact users

## Maven users
modules will continue to be modules.. I guess nobody will notice,
other than we might have a different versioning scheme, but we help
people out via the Infinispan BOM.

## Distribution users
There should be no difference, other than (as well) some jars might
not be aligned in terms of version. But that's probably even less of a
problem, as I expect distribution users to just put what they get on
their classpath.

# How it impacts us

1) I'll move the Lucene Directory project to an different repository;
same for the Query related components.
I think you should/could consider the same for other components, based
on ad-hoc considerations of the trade offs, but I'd expect ultimately
to see a more frequent and "core only" release.

2) We'll have different kinds of releases: the "core only" and the
"full releases".
I think we'll also see components being released independently, but
these are either Maven-only or meant for preparation of other
components, or preparation for a "full release".

3) Tests (!)
Such a move should in no way relax the regression-safety of
infinispan-core: we need to still consider it unacceptable for a core
change to break one of the modules moving out of the main tree.
Personally I think I've pushed many tests about problems found in the
"query modules" as unit tests in core, so that should be relatively
safe, but it also happened that someone would "tune" these.
I realise it's not practical to expect people to run tests of
downstream modules, so we'll have to automate most of these tasks in
CI.
Careful on perception: if today there are three levels of defence
against a regression (the author, the reviewer and CI all running the
suite for each change), in such an organisation you have only one. So
ignoring a CI failure as a "probable hiccup" could be much more
dangerous than usual.

# When

Doing this _might_ be a blocker for any Lucene update; so since one
just happened I'll probably have no urgent need for a couple of weeks
at least.
But we shouldn't be in a position in which an update could not be
possible, so I hope we'll agree to implement this sooner rather than
later, so we won't have to do it during an emergency.

Also while this might sound a bit crazy at first, I see many
flexibility benefits which can't hurt now that the project is getting
larger and more complex to release.
Not least, having a micro release of "Infinispan essentials" would be
very welcome in terms of lowing the initial barrier; this was proposed
at various meetings and highly endorsed by many but just never
happened.

Any comment please? I hope I covered it all, and sorry for that :D

Cheers,
Sanne


1 - http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011419.html
2 - http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2014/hibernate-dev.2014-05-13-13.24.log.html