Hi, Because of the hight number of dependencies and large distribution size (build time as well) there were discussions about moving some of the cache stores into an independent git repository under the infinispan project. Now that we have two more cache stores to be integrated (JPA and MongoDB), it's a good opportunity to start a discussion around this. The cache stores I'm thinking about are: JPA, MongoDB (integrated them directly in the new repository), Cassandra, Cloud, HBase, jdbm and bdbje. Or thinking more about it, why not all of them but the file cache store - which is dependency free and it's nice to ship it . There are several options when it comes to releasing these stores. I think the simplest is by to a maven repo (i.e. no .zip file). If there'll be demand for specific distribution files we can always add it at a further point in time. wdyt? Cheers,

With independent release numbers (and days) ? I realize it makes developer's life easier as you can break them all without needing to care, still it won't make users life easier. What do you propose we should do when an Infinispan (core project) change breaks some tests in the CacheStore? On 9 May 2013 14:07, Adrian Nistor <anistor(a)redhat.com&gt; wrote: > I think it makes sense to move all of them to a separate project, except > FileCacheStore. I doubt there is real need for a zip distribution for > this new project. > > We can continue to provide a zip for the main ispn project, but I still > think few people actually use it compared to the maven artifacts. I > believe the main reason for a zip distro is to be able to accurately > count the number of downloads :). > > On 05/09/2013 03:22 PM, Mircea Markus wrote: >> Hi, >> >> Because of the hight number of dependencies and large distribution size (build time as well) there were discussions about moving some of the cache stores into an independent git repository under the infinispan project. Now that we have two more cache stores to be integrated (JPA and MongoDB), it's a good opportunity to start a discussion around this. >> >> The cache stores I'm thinking about are: JPA, MongoDB (integrated them directly in the new repository), Cassandra, Cloud, HBase, jdbm and bdbje. Or thinking more about it, why not all of them but the file cache store - which is dependency free and it's nice to ship it . >> >> There are several options when it comes to releasing these stores. I think the simplest is by to a maven repo (i.e. no .zip file). If there'll be demand for specific distribution files we can always add it at a further point in time. >> >> wdyt? >> >> Cheers, > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 9 May 2013 15:10, Manik Surtani <msurtani(a)redhat.com&gt; wrote: > This is something we discussed last year. IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos. They will also have their own release cycles, so will only be released as and when there is a change in their code. I remember the discussion, but never agreed on it being a good idea. I'm not strictly against it, just that I am not understanding the point if it, while there are drawbacks.

> Now when it comes to breaking stuff, this means that our CI will get more complex. Any changes to core will cause a build and re-test of all "consuming" projects (cache stores and hot rod clients). Failures on any of these "consuming" projects should cause a failure in core's CI suite. I like the sound of that, but to play devil's advocate that's much easier to implement by keeping them all in the same repository. Also you need to answer how non-Maven users will know which versions are going to be compatible. "everyone should use Maven" is not a fair reply ;-)

On 9 May 2013 21:38, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On 9 May 2013, at 16:03, Sanne Grinovero wrote: > >> On 9 May 2013 15:10, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >>> This is something we discussed last year. IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos. They will also have their own release cycles, so will only be released as and when there is a change in their code. >> >> I remember the discussion, but never agreed on it being a good idea. >> I'm not strictly against it, just that I am not understanding the >> point if it, while there are drawbacks. > pros: reduce distribution size, keep build time for the core manageable, allow a per/cachestore upgrade. Is that the only advantage?

Because the source control should have nothing to do with how the distribution is assembled. I still don't see why it's worth the effort,

and the pain.

On 9 May 2013 23:18, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On 9 May 2013, at 22:13, Sanne Grinovero wrote: > >> On 9 May 2013 21:38, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>> >>> On 9 May 2013, at 16:03, Sanne Grinovero wrote: >>> >>>> On 9 May 2013 15:10, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >>>>> This is something we discussed last year. IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos. They will also have their own release cycles, so will only be released as and when there is a change in their code. >>>> >>>> I remember the discussion, but never agreed on it being a good idea. >>>> I'm not strictly against it, just that I am not understanding the >>>> point if it, while there are drawbacks. >>> pros: reduce distribution size, keep build time for the core manageable, allow a per/cachestore upgrade. >> >> Is that the only advantage? > indeed the first one I mentioned could be achieved through packaging, but not the last two. I don't understand the merit on the last two either: # "keep build time for the core manageable" Manik mentioned that all CacheStore s would be tested after each core change, even rising the question of how to let the CORE build fail in case one would have problems, so you still have to run tests for all modules: the build time doesn't change neither for pull request reviewers nor CI server.

Admit it: you plan to not run all tests all the time, or if a CacheStore fails to fix it "later".

At the very best such a fix happens after the damage is done.

Note that the MongoDB CacheStore - even if just in pull request state - self activates its own build (and tests) only if the runtime provides a MongoDB instance., so partially applying this idea to simplify build requirements (i.e. not mandating a MongoDB installation to build infinispan core tests). Of course this would be still expected from team members and CI server.

It's an example of another approach to disable some modules - if you really want that - without changing the source structure. # "allow a per/cachestore upgrade" I'm confused on this one. Do you mean "allow the team to forget updating some cachestore" ?

To me it looks like a proposal to downgrade importance of maintenance of some (all?) cachestores; we could discuss that if you prefer? It just looks like a different subject. >> Because the source control should have >> nothing to do with how the distribution is assembled. >> I still don't see why it's worth the effort, > it's a pretty straight forward task to move them out. Same to add CI job :-) >> and the pain. > If you're referring to the possibility of the cache store build to fail, I don't think this is negative thing as the core test suite should only get better. Not every change is an improvement; I don't think the test suite will improve automagically if we make the wrong choices. But by "pain" I was referring to the need to keep multiple repositories in sync without cross-tagging capabilities nor anything similar; you'll need releases of one to move forward with the other, etc.. Or is the plan to use git modules? Usually people resolve such problems by merging multiple projects in a single repo.. In summary, I'm not strictly against it as technically I won't maintain either side of the integration, just that I see *exclusively* disadvantages for you so I'm quite confused by the suggestion. Cheers, Sanne > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10 May 2013 00:24, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On 9 May 2013, at 23:47, Sanne Grinovero wrote: > >> On 9 May 2013 23:18, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>> >>> On 9 May 2013, at 22:13, Sanne Grinovero wrote: >>> >>>> On 9 May 2013 21:38, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>>>> >>>>> On 9 May 2013, at 16:03, Sanne Grinovero wrote: >>>>> >>>>>> On 9 May 2013 15:10, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >>>>>>> This is something we discussed last year. IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos. They will also have their own release cycles, so will only be released as and when there is a change in their code. >>>>>> >>>>>> I remember the discussion, but never agreed on it being a good idea. >>>>>> I'm not strictly against it, just that I am not understanding the >>>>>> point if it, while there are drawbacks. >>>>> pros: reduce distribution size, keep build time for the core manageable, allow a per/cachestore upgrade. >>>> >>>> Is that the only advantage? >>> indeed the first one I mentioned could be achieved through packaging, but not the last two. >> >> I don't understand the merit on the last two either: >> >> # "keep build time for the core manageable" >> >> Manik mentioned that all CacheStore s would be tested after each core >> change, even rising the question of how to let the CORE build fail in >> case one would have problems, so you still have to run tests for all >> modules: the build time doesn't change neither for pull request >> reviewers nor CI server. > They shouldn't be tested on every build the same we we don't test 3rd party cache stores on every build. Once a day should be enough IMO. > Also we'll have the FileCacheStore and the RemoteCacheStore that run on every build. Ok that clarifies the goal. Use profiles? >> >> Admit it: you plan to not run all tests all the time, or if a >> CacheStore fails to fix it "later". > no :-) fix it as soon as it is spotted and ad a test in the core suite to make sure it won't happen again. > >> At the very best such a fix >> happens after the damage is done. > Possibly. Things should improve with time though, as unit tests are added to catch the failures. > >> >> Note that the MongoDB CacheStore - even if just in pull request state >> - self activates its own build (and tests) only if the runtime >> provides a MongoDB instance., so partially applying this idea to >> simplify build requirements (i.e. not mandating a MongoDB installation >> to build infinispan core tests). Of course this would be still >> expected from team members and CI server. > that's quite clever >> It's an example of another approach to disable some modules - if you >> really want that - without changing the source structure. >> >> # "allow a per/cachestore upgrade" >> >> I'm confused on this one. Do you mean "allow the team to forget >> updating some cachestore" ? > The main reason is: if we fix a critical bug e.g. JDBC cache store we can release it immediately without an full release cycle and people can take advantage of that sooner. -1 Release it all often. >> To me it looks like a proposal to downgrade importance of maintenance >> of some (all?) cachestores; >> we could discuss that if you prefer? It >> just looks like a different subject. >> >>>> Because the source control should have >>>> nothing to do with how the distribution is assembled. >>>> I still don't see why it's worth the effort, >>> it's a pretty straight forward task to move them out. Same to add CI job :-) >>>> and the pain. >>> If you're referring to the possibility of the cache store build to fail, I don't think this is negative thing as the core test suite should only get better. >> >> Not every change is an improvement; I don't think the test suite will >> improve automagically if we make the wrong choices. But by "pain" I >> was referring to the need to keep multiple repositories in sync >> without cross-tagging capabilities nor anything similar; you'll need >> releases of one to move forward with the other, etc.. >> Or is the plan to use git modules? >> Usually people resolve such problems by merging multiple projects in a >> single repo.. >> >> In summary, I'm not strictly against it as technically I won't >> maintain either side of the integration, just that I see *exclusively* >> disadvantages for you so I'm quite confused by the suggestion. >> >> Cheers, >> Sanne >> >>> >>> Cheers, >>> -- >>> Mircea Markus >>> Infinispan lead (www.infinispan.org) >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10 May 2013, at 10:06, Manik Surtani wrote: > There seems to be a bit of confusion on this thread. The things I hope to achieve here are: > > 1. De-coupled release cycle. > Most of our releases include new versions of XYZCacheStore, even though there are no changes to it. This creates noise, IMO. Cache Stores should only be released when there are changes made to it. Now this wasn't so much of a problem when we just had a small handful of cache stores, but as this increases, this becomes even more noisy/confusing to end-users. > > 2. Smaller download. > Not everyone uses all cache stores; not including everything in a zip ball will reduce download size. But as pointed out before, this can be achieved via other techniques. > > 3. Scalability. > Moving cache stores to separate repos will allow us to add more cache stores, accept more contribs for experimental cache stores, build out a richer ecosystem. Right now, we restrict the number of cache store impls to prevent bloat of the core distribution. well summarised. > > This does *not* impact the developer at all, IMO. CI (and test) runs on core will still involve testing all *non-experimental* cache stores. I think this should happen every time and not just daily. I don't think we need to do it on every build. But this is more of an configuration option and we can do it if needed.

> > In terms of a compatibility matrix (which cache stores work with which versions of core), we'd need to devise a scheme. For example, match on major.minor, like CacheStore 5.3.x will work with any version of core 5.3.x. +1

On 10 May 2013 00:24, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On 9 May 2013, at 23:47, Sanne Grinovero wrote: > >> On 9 May 2013 23:18, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>> >>> On 9 May 2013, at 22:13, Sanne Grinovero wrote: >>> >>>> On 9 May 2013 21:38, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>>>> >>>>> On 9 May 2013, at 16:03, Sanne Grinovero wrote: >>>>> >>>>>> On 9 May 2013 15:10, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >>>>>>> This is something we discussed last year. IIRC we agreed that all cache stores (except the 0-dep FCS) and hot rod clients will move out of the infinispan repo, and have their own repos. They will also have their own release cycles, so will only be released as and when there is a change in their code. >>>>>> >>>>>> I remember the discussion, but never agreed on it being a good idea. >>>>>> I'm not strictly against it, just that I am not understanding the >>>>>> point if it, while there are drawbacks. >>>>> pros: reduce distribution size, keep build time for the core manageable, allow a per/cachestore upgrade. >>>> >>>> Is that the only advantage? >>> indeed the first one I mentioned could be achieved through packaging, but not the last two. >> >> I don't understand the merit on the last two either: >> >> # "keep build time for the core manageable" >> >> Manik mentioned that all CacheStore s would be tested after each core >> change, even rising the question of how to let the CORE build fail in >> case one would have problems, so you still have to run tests for all >> modules: the build time doesn't change neither for pull request >> reviewers nor CI server. > They shouldn't be tested on every build the same we we don't test 3rd party cache stores on every build. Once a day should be enough IMO. > Also we'll have the FileCacheStore and the RemoteCacheStore that run on every build. Ok that clarifies the goal. Use profiles? >> >> Admit it: you plan to not run all tests all the time, or if a >> CacheStore fails to fix it "later". > no :-) fix it as soon as it is spotted and ad a test in the core suite to make sure it won't happen again. > >> At the very best such a fix >> happens after the damage is done. > Possibly. Things should improve with time though, as unit tests are added to catch the failures. > >> >> Note that the MongoDB CacheStore - even if just in pull request state >> - self activates its own build (and tests) only if the runtime >> provides a MongoDB instance., so partially applying this idea to >> simplify build requirements (i.e. not mandating a MongoDB installation >> to build infinispan core tests). Of course this would be still >> expected from team members and CI server. > that's quite clever >> It's an example of another approach to disable some modules - if you >> really want that - without changing the source structure. >> >> # "allow a per/cachestore upgrade" >> >> I'm confused on this one. Do you mean "allow the team to forget >> updating some cachestore" ? > The main reason is: if we fix a critical bug e.g. JDBC cache store we can release it immediately without an full release cycle and people can take advantage of that sooner. -1 Release it all often.

With independent release numbers (and days) ? I realize it makes developer's life easier as you can break them all without needing to care,

still it won't make users life easier.

What do you propose we should do when an Infinispan (core project) change breaks some tests in the CacheStore?

Are you planning to create a repository for all the cache store or one for each cache store?

From a user perspective, I think that is a good idea because I believe the cache store code rarely changes. However, it will make the developer life a little bit more complex :)