[infinispan-dev] moving (some) cache stores in a different github repository

Fri May 10 05:06:42 EDT 2013

There seems to be a bit of confusion on this thread.  The things I hope to achieve here are:

1.  De-coupled release cycle.
Most of our releases include new versions of XYZCacheStore, even though there are no changes to it.  This creates noise, IMO.  Cache Stores should only be released when there are changes made to it.  Now this wasn't so much of a problem when we just had a small handful of cache stores, but as this increases, this becomes even more noisy/confusing to end-users.

2.  Smaller download.
Not everyone uses all cache stores; not including everything in a zip ball will reduce download size.  But as pointed out before, this can be achieved via other techniques.

3.  Scalability.
Moving cache stores to separate repos will allow us to add more cache stores, accept more contribs for experimental cache stores, build out a richer ecosystem.  Right now, we restrict the number of cache store impls to prevent bloat of the core distribution.

This does *not* impact the developer at all, IMO.  CI (and test) runs on core will still involve testing all *non-experimental* cache stores.  I think this should happen every time and not just daily.

In terms of a compatibility matrix (which cache stores work with which versions of core), we'd need to devise a scheme.  For example, match on major.minor, like CacheStore 5.3.x will work with any version of core 5.3.x.

- M

On 10 May 2013, at 00:48, Sanne Grinovero <sanne at infinispan.org> wrote:

> On 10 May 2013 00:24, Mircea Markus <mmarkus at redhat.com> wrote:
>> 
>> On 9 May 2013, at 23:47, Sanne Grinovero wrote:
>> 
>>> On 9 May 2013 23:18, Mircea Markus <mmarkus at redhat.com> wrote:
>>>> 
>>>> On 9 May 2013, at 22:13, Sanne Grinovero wrote:
>>>> 
>>>>> On 9 May 2013 21:38, Mircea Markus <mmarkus at redhat.com> wrote:
>>>>>> 
>>>>>> On 9 May 2013, at 16:03, Sanne Grinovero wrote:
>>>>>> 
>>>>>>> On 9 May 2013 15:10, Manik Surtani <msurtani at redhat.com> wrote:
>>>>>>>> This is something we discussed last year.  IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos.  They will also have their own release cycles, so will only be released as and when there is a change in their code.
>>>>>>> 
>>>>>>> I remember the discussion, but never agreed on it being a good idea.
>>>>>>> I'm not strictly against it, just that I am not understanding the
>>>>>>> point if it, while there are drawbacks.
>>>>>> pros: reduce distribution size, keep build time for the core manageable, allow a per/cachestore upgrade.
>>>>> 
>>>>> Is that the only advantage?
>>>> indeed the first one I mentioned could be achieved through packaging, but not the last two.
>>> 
>>> I don't understand the merit on the last two either:
>>> 
>>> # "keep build time for the core manageable"
>>> 
>>> Manik mentioned that all CacheStore s would be tested after each core
>>> change, even rising the question of how to let the CORE build fail in
>>> case one would have problems, so you still have to run tests for all
>>> modules: the build time doesn't change neither for pull request
>>> reviewers nor CI server.
>> They shouldn't be tested on every build the same we we don't test 3rd party cache stores on every build. Once a day should be enough IMO.
>> Also we'll have the FileCacheStore and the RemoteCacheStore that run on every build.
> 
> Ok that clarifies the goal. Use profiles?
> 
>>> 
>>> Admit it: you plan to not run all tests all the time, or if a
>>> CacheStore fails to fix it "later".
>> no :-) fix it as soon as it is spotted and ad a test in the core suite to make sure it won't happen again.
>> 
>>> At the very best such a fix
>>> happens after the damage is done.
>> Possibly. Things should improve with time though, as unit tests are added to catch the failures.
>> 
>>> 
>>> Note that the MongoDB CacheStore - even if just in pull request state
>>> - self activates its own build (and tests) only if the runtime
>>> provides a MongoDB instance., so partially applying this idea to
>>> simplify build requirements (i.e. not mandating a MongoDB installation
>>> to build infinispan core tests). Of course this would be still
>>> expected from team members and CI server.
>> that's quite clever
>>> It's an example of another approach to disable some modules - if you
>>> really want that - without changing the source structure.
>>> 
>>> # "allow a per/cachestore upgrade"
>>> 
>>> I'm confused on this one. Do you mean "allow the team to forget
>>> updating some cachestore" ?
>> The main reason is: if we fix a critical bug e.g. JDBC cache store we can release it immediately without an full release cycle and people can take advantage of that sooner.
> 
> -1
> Release it all often.
> 
>>> To me it looks like a proposal to downgrade importance of maintenance
>>> of some (all?) cachestores;
>>> we could discuss that if you prefer? It
>>> just looks like a different subject.
>>> 
>>>>> Because the source control should have
>>>>> nothing to do with how the distribution is assembled.
>>>>> I still don't see why it's worth the effort,
>>>> it's a pretty straight forward task to move them out. Same to add CI job :-)
>>>>> and the pain.
>>>> If you're referring to the possibility of the cache store build to fail, I don't think this is negative thing as the core test suite should only get better.
>>> 
>>> Not every change is an improvement; I don't think the test suite will
>>> improve automagically if we make the wrong choices. But by "pain" I
>>> was referring to the need to keep multiple repositories in sync
>>> without cross-tagging capabilities nor anything similar; you'll need
>>> releases of one to move forward with the other, etc..
>>> Or is the plan to use git modules?
>>> Usually people resolve such problems by merging multiple projects in a
>>> single repo..
>>> 
>>> In summary, I'm not strictly against it as technically I won't
>>> maintain either side of the integration, just that I see *exclusively*
>>> disadvantages for you so I'm quite confused by the suggestion.
>>> 
>>> Cheers,
>>> Sanne
>>> 
>>>> 
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
twitter.com/maniksurtani

Platform Architect, JBoss Data Grid
http://red.ht/data-grid