[infinispan-dev] [!] Reorganization of dependencies & release process

[Sanne Grinovero]

On 14 May 2014 12:20, Dan Berindei <dan.berindei at gmail.com> wrote:
> I don't see a lot of value in doing core-only releases. Releases are
> expensive because we have to update the website and documentation, and we
> have to announce the release everywhere. Releasing only the core won't
> change that.
>
> Also, we don't try to maintain backwards compatibility between Alpha/Beta
> releases. So releasing only the core is only practical for minor/micro
> releases.
>
> OTOH, doing a maven-only release is just a matter of starting the release
> script on the CI machine, and doing a couple clicks an hour later in the
> Nexus UI to release the staging repository. Doing a core-only maven-only
> release would have about the same overhead.
>
> Wouldn't it be enough to move the Lucene directory to a separate repository
> (and release schedule)? We could easily do a couple maven-only releases to
> prepare for Search updgrades, I don't see any problems with that.

Infinispan Query uses the Lucene Directory. So if you move out only LD
- and keep Query - but make any change in Infinispan Core which breaks
the Directory (and this isn't as unlikely as we'd want to.. but even
if it was, the problem is that it's not impossible) - then you
wouldn't be able to ship an Infinispan Core release as the Query
functionality would be broken.

An obvious answer would be to move Query out as well.. but more and
more modules are depending on it every day.
Not sure if that's a good thing, but I'm sure that making our release
process easier is not a good reason to avoid providing useful
features.

At some point to simplify configuration parsing and validation we even
considered moving Query into Infinispan Core - just to remind how
pervasive this is.

We can only break the cycle if you allow for an acyclic dependency
graph, so nothing in the Infinispan release which is created by
tagging infinispan-core can depend on infinispan-lucene-directory
(directly or indirectly).
To me that implies necessarily that a lot of modules - at least all of
those somehow depending on queries / lucene - need to be released
separately from infinispan-core.

And as mentioned this has other welcome side-effects; for one the
highly discussed, desired "small" release of Infinispan would I think
make life much easier to newcomers, avoiding to scare away users but
also contributors.
I also like the fact that we can release smaller things more
frequently, and when major APIs have to change, we can split the work
in smaller iterations rather than one developer having to do all the
heavy lifting across all modules.

Sanne

>
> Cheers
> Dan
>
>
> On Wed, May 14, 2014 at 12:50 AM, Sanne Grinovero <sanne at infinispan.org>
> wrote:
>>
>> This is a reboot of the thread previously started on both the
>> infinispan-dev and the hibernate-dev mailing list as "Handling of
>> mutual dependency with Infinispan" [1].
>> We discussed further during the Hibernate fortnightly meeting [2], and
>> came to the conclusion that we need Infinispan to change how some
>> repositories are organised and how the release is assembled.
>>
>> # The problem
>>
>> To restate the issue, as you might painfully remember, every time
>> there is a need for a Lucene update or a Search update we need to sync
>> up for a complex dance of releases in both projects to accommodate for
>> a small-step iterative process to handle the circular dependency.
>> This problem is not too bad today as since a year we're releasing the
>> Lucene Directory in an unusual - and very unmaintainable - temporary
>> solution to be compatible with two different major versions of Apache
>> Lucene; namely what Infinispan Query needs and what Hibernate Search
>> needs are different modules.
>> But the party is over, and I want to finally drop support for Lucene 3
>> and cleanup the unusual and unmaintainable build mess targeting a
>> single Lucene version only.
>> As soon as we converge to building a single version however - we're
>> back to the complex problem we had when we supported a single version
>> which is handling of a circular dependency - just that the problem has
>> worsened lately the Lucene project has been more active and more
>> inclined than what it used to be to break both internal and public
>> APIs.
>>
>> In short, we have a circular dependency between Hibernate Search and
>> Infinispan which we've been able to handle via hacks and some luck,
>> but it imposes a serious threat to development flexibility, and the
>> locked-in release process is not desirable either.
>>
>> # The solution
>>
>> we think in conclusion there's a single "proper" way out, and it also
>> happens to provide some very interesting side effects in terms of
>> maintenance overhead for everyone: Infinispan Core needs to release
>> independently from the non-core modules.
>> This would have the Lucene Directory depend on a released tag of
>> infinispan-core, and be able to be released independently.
>> Minor situations with benefit:
>>  - we often don't make any change in the Lucene Directory, still we
>> need to release it.
>>  - when I actually need a release of it, I'm currently begging for a
>> quick release of Infinispan: very costly
>> The Big Ones:
>>  - we can manage the Lucene Directory to provide support for different
>> versions of Lucene without necessarily breaking other modules
>>  - we can release quickly what's needed to move Search ahead in terms
>> of Lucene versions without needing to make the Infinispan Query module
>> compatible at the same time (in case you haven't followed this area:
>> this seems to be my main activity rather than making valuable stuff).
>>
>> The goal is of course to linearise the dependencies; it seems to also
>> simplify some of our tasks which is a welcome side-effect. I expect it
>> also to make the project less scary for new contributors.
>>
>> # How does it impact users
>>
>> ## Maven users
>> modules will continue to be modules.. I guess nobody will notice,
>> other than we might have a different versioning scheme, but we help
>> people out via the Infinispan BOM.
>>
>> ## Distribution users
>> There should be no difference, other than (as well) some jars might
>> not be aligned in terms of version. But that's probably even less of a
>> problem, as I expect distribution users to just put what they get on
>> their classpath.
>>
>> # How it impacts us
>>
>> 1) I'll move the Lucene Directory project to an different repository;
>> same for the Query related components.
>> I think you should/could consider the same for other components, based
>> on ad-hoc considerations of the trade offs, but I'd expect ultimately
>> to see a more frequent and "core only" release.
>>
>> 2) We'll have different kinds of releases: the "core only" and the
>> "full releases".
>> I think we'll also see components being released independently, but
>> these are either Maven-only or meant for preparation of other
>> components, or preparation for a "full release".
>>
>> 3) Tests (!)
>> Such a move should in no way relax the regression-safety of
>> infinispan-core: we need to still consider it unacceptable for a core
>> change to break one of the modules moving out of the main tree.
>> Personally I think I've pushed many tests about problems found in the
>> "query modules" as unit tests in core, so that should be relatively
>> safe, but it also happened that someone would "tune" these.
>> I realise it's not practical to expect people to run tests of
>> downstream modules, so we'll have to automate most of these tasks in
>> CI.
>> Careful on perception: if today there are three levels of defence
>> against a regression (the author, the reviewer and CI all running the
>> suite for each change), in such an organisation you have only one. So
>> ignoring a CI failure as a "probable hiccup" could be much more
>> dangerous than usual.
>>
>> # When
>>
>> Doing this _might_ be a blocker for any Lucene update; so since one
>> just happened I'll probably have no urgent need for a couple of weeks
>> at least.
>> But we shouldn't be in a position in which an update could not be
>> possible, so I hope we'll agree to implement this sooner rather than
>> later, so we won't have to do it during an emergency.
>>
>> Also while this might sound a bit crazy at first, I see many
>> flexibility benefits which can't hurt now that the project is getting
>> larger and more complex to release.
>> Not least, having a micro release of "Infinispan essentials" would be
>> very welcome in terms of lowing the initial barrier; this was proposed
>> at various meetings and highly endorsed by many but just never
>> happened.
>>
>> Any comment please? I hope I covered it all, and sorry for that :D
>>
>> Cheers,
>> Sanne
>>
>>
>> 1 - http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011419.html
>> 2 -
>> http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2014/hibernate-dev.2014-05-13-13.24.log.html
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev