[infinispan-dev] [!] Reorganization of dependencies & release process

[Emmanuel Bernard]

Let's not overload the subject here. 

We can first approach the problem like Dan mentions by doing a tag for core in git and release in maven. 
Then every other Infinispan bit is tagged and released as one and depend on that core tag. That makes for 1 marketing release (website, blog ect).

You an also consider that releasing Infinispan core marketing wise makes sense but that is another story that we should keep separate. 

> On 14 mai 2014, at 14:24, Sanne Grinovero <sanne at infinispan.org> wrote:
> 
>> On 14 May 2014 12:20, Dan Berindei <dan.berindei at gmail.com> wrote:
>> I don't see a lot of value in doing core-only releases. Releases are
>> expensive because we have to update the website and documentation, and we
>> have to announce the release everywhere. Releasing only the core won't
>> change that.
>> 
>> Also, we don't try to maintain backwards compatibility between Alpha/Beta
>> releases. So releasing only the core is only practical for minor/micro
>> releases.
>> 
>> OTOH, doing a maven-only release is just a matter of starting the release
>> script on the CI machine, and doing a couple clicks an hour later in the
>> Nexus UI to release the staging repository. Doing a core-only maven-only
>> release would have about the same overhead.
>> 
>> Wouldn't it be enough to move the Lucene directory to a separate repository
>> (and release schedule)? We could easily do a couple maven-only releases to
>> prepare for Search updgrades, I don't see any problems with that.
> 
> Infinispan Query uses the Lucene Directory. So if you move out only LD
> - and keep Query - but make any change in Infinispan Core which breaks
> the Directory (and this isn't as unlikely as we'd want to.. but even
> if it was, the problem is that it's not impossible) - then you
> wouldn't be able to ship an Infinispan Core release as the Query
> functionality would be broken.
> 
> An obvious answer would be to move Query out as well.. but more and
> more modules are depending on it every day.
> Not sure if that's a good thing, but I'm sure that making our release
> process easier is not a good reason to avoid providing useful
> features.
> 
> At some point to simplify configuration parsing and validation we even
> considered moving Query into Infinispan Core - just to remind how
> pervasive this is.
> 
> We can only break the cycle if you allow for an acyclic dependency
> graph, so nothing in the Infinispan release which is created by
> tagging infinispan-core can depend on infinispan-lucene-directory
> (directly or indirectly).
> To me that implies necessarily that a lot of modules - at least all of
> those somehow depending on queries / lucene - need to be released
> separately from infinispan-core.
> 
> And as mentioned this has other welcome side-effects; for one the
> highly discussed, desired "small" release of Infinispan would I think
> make life much easier to newcomers, avoiding to scare away users but
> also contributors.
> I also like the fact that we can release smaller things more
> frequently, and when major APIs have to change, we can split the work
> in smaller iterations rather than one developer having to do all the
> heavy lifting across all modules.
> 
> Sanne
> 
>> 
>> Cheers
>> Dan
>> 
>> 
>> On Wed, May 14, 2014 at 12:50 AM, Sanne Grinovero <sanne at infinispan.org>
>> wrote:
>>> 
>>> This is a reboot of the thread previously started on both the
>>> infinispan-dev and the hibernate-dev mailing list as "Handling of
>>> mutual dependency with Infinispan" [1].
>>> We discussed further during the Hibernate fortnightly meeting [2], and
>>> came to the conclusion that we need Infinispan to change how some
>>> repositories are organised and how the release is assembled.
>>> 
>>> # The problem
>>> 
>>> To restate the issue, as you might painfully remember, every time
>>> there is a need for a Lucene update or a Search update we need to sync
>>> up for a complex dance of releases in both projects to accommodate for
>>> a small-step iterative process to handle the circular dependency.
>>> This problem is not too bad today as since a year we're releasing the
>>> Lucene Directory in an unusual - and very unmaintainable - temporary
>>> solution to be compatible with two different major versions of Apache
>>> Lucene; namely what Infinispan Query needs and what Hibernate Search
>>> needs are different modules.
>>> But the party is over, and I want to finally drop support for Lucene 3
>>> and cleanup the unusual and unmaintainable build mess targeting a
>>> single Lucene version only.
>>> As soon as we converge to building a single version however - we're
>>> back to the complex problem we had when we supported a single version
>>> which is handling of a circular dependency - just that the problem has
>>> worsened lately the Lucene project has been more active and more
>>> inclined than what it used to be to break both internal and public
>>> APIs.
>>> 
>>> In short, we have a circular dependency between Hibernate Search and
>>> Infinispan which we've been able to handle via hacks and some luck,
>>> but it imposes a serious threat to development flexibility, and the
>>> locked-in release process is not desirable either.
>>> 
>>> # The solution
>>> 
>>> we think in conclusion there's a single "proper" way out, and it also
>>> happens to provide some very interesting side effects in terms of
>>> maintenance overhead for everyone: Infinispan Core needs to release
>>> independently from the non-core modules.
>>> This would have the Lucene Directory depend on a released tag of
>>> infinispan-core, and be able to be released independently.
>>> Minor situations with benefit:
>>> - we often don't make any change in the Lucene Directory, still we
>>> need to release it.
>>> - when I actually need a release of it, I'm currently begging for a
>>> quick release of Infinispan: very costly
>>> The Big Ones:
>>> - we can manage the Lucene Directory to provide support for different
>>> versions of Lucene without necessarily breaking other modules
>>> - we can release quickly what's needed to move Search ahead in terms
>>> of Lucene versions without needing to make the Infinispan Query module
>>> compatible at the same time (in case you haven't followed this area:
>>> this seems to be my main activity rather than making valuable stuff).
>>> 
>>> The goal is of course to linearise the dependencies; it seems to also
>>> simplify some of our tasks which is a welcome side-effect. I expect it
>>> also to make the project less scary for new contributors.
>>> 
>>> # How does it impact users
>>> 
>>> ## Maven users
>>> modules will continue to be modules.. I guess nobody will notice,
>>> other than we might have a different versioning scheme, but we help
>>> people out via the Infinispan BOM.
>>> 
>>> ## Distribution users
>>> There should be no difference, other than (as well) some jars might
>>> not be aligned in terms of version. But that's probably even less of a
>>> problem, as I expect distribution users to just put what they get on
>>> their classpath.
>>> 
>>> # How it impacts us
>>> 
>>> 1) I'll move the Lucene Directory project to an different repository;
>>> same for the Query related components.
>>> I think you should/could consider the same for other components, based
>>> on ad-hoc considerations of the trade offs, but I'd expect ultimately
>>> to see a more frequent and "core only" release.
>>> 
>>> 2) We'll have different kinds of releases: the "core only" and the
>>> "full releases".
>>> I think we'll also see components being released independently, but
>>> these are either Maven-only or meant for preparation of other
>>> components, or preparation for a "full release".
>>> 
>>> 3) Tests (!)
>>> Such a move should in no way relax the regression-safety of
>>> infinispan-core: we need to still consider it unacceptable for a core
>>> change to break one of the modules moving out of the main tree.
>>> Personally I think I've pushed many tests about problems found in the
>>> "query modules" as unit tests in core, so that should be relatively
>>> safe, but it also happened that someone would "tune" these.
>>> I realise it's not practical to expect people to run tests of
>>> downstream modules, so we'll have to automate most of these tasks in
>>> CI.
>>> Careful on perception: if today there are three levels of defence
>>> against a regression (the author, the reviewer and CI all running the
>>> suite for each change), in such an organisation you have only one. So
>>> ignoring a CI failure as a "probable hiccup" could be much more
>>> dangerous than usual.
>>> 
>>> # When
>>> 
>>> Doing this _might_ be a blocker for any Lucene update; so since one
>>> just happened I'll probably have no urgent need for a couple of weeks
>>> at least.
>>> But we shouldn't be in a position in which an update could not be
>>> possible, so I hope we'll agree to implement this sooner rather than
>>> later, so we won't have to do it during an emergency.
>>> 
>>> Also while this might sound a bit crazy at first, I see many
>>> flexibility benefits which can't hurt now that the project is getting
>>> larger and more complex to release.
>>> Not least, having a micro release of "Infinispan essentials" would be
>>> very welcome in terms of lowing the initial barrier; this was proposed
>>> at various meetings and highly endorsed by many but just never
>>> happened.
>>> 
>>> Any comment please? I hope I covered it all, and sorry for that :D
>>> 
>>> Cheers,
>>> Sanne
>>> 
>>> 
>>> 1 - http://lists.jboss.org/pipermail/hibernate-dev/2014-May/011419.html
>>> 2 -
>>> http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2014/hibernate-dev.2014-05-13-13.24.log.html
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev