Hi, Sounds like a reasonable plan. I am just wondering what happened with the idea of creating a standalone repository for the Infinispan Lucene Directory (I guess that implies moving things from Inifinispan and Search into a new repo)? I think this was our most favoured plan until now. IMO this has even more flexibility as the move to Infinispan.

Also it should be in the interest of Infinispan as well to split things out, given how many modules they already have. I think smaller independent "projects" is more favourable than this huge monolithic multi-module setups. It also comes with additional flexibility when it comes to releasing.

In case we co ahead with moving the code to Infinispan: > A) Should the Maven group id be changed? I'd expect it to be > transferred to "org.infinispan" group at least, and probably need a > better artifact id too. +1 At least group id needs top change. > B) License. Our code is LGPL, most of Infinispan is ASL - but not all > of it. So I expect it would be possible to keep the existing license > at least for now, and defer eventual license changes as a separate > step (if people feel need for any change at all). Would it not be easier to change to ASL?

> C) Documentation. Besides the needed updates in Maven coordinates / > download sources, I don't expect much of a difference: we'd still > explain how to set this integration up. +1 > I think we should drop these from our distribution +1 --Hardy

Hi, +1 for addressing this, the cyclic dependency has been a pain point for a long time. Moving the directory provider code out of HS is the right thing to do. But moving it into ISPN creates the release dependency you describe: After a HS release there will be a time where people potentially cannot use it with the ISPN directory provider until there has been a new ISPN release which adapts to the HS (and, transitively, Lucene) changes. So it kind of ties the ISPN release cycle to HS, although ISPN itself might be fine using the previousi HSEARCH version for its own querying purposes.

To me the problem appears to be that ISPN a) uses HS (for querying) but b) also extends it (as index storage) at the same time. For a) ISPN could live with older HS versions AFAIU, whereas for b) following HS releases closely is needed in order to keep that integration usable as HS (and Lucene) advances.

So alternatively the ISPN Lucene directory as well as the HS directory (and related integration tests) could be moved into a separate repository, with its own release lifecycle. Then when a new HS release comes out, only that separate integration project needs to be adapted and released (which of course could be prepared during Alpha/Beta phases, so it could be released on the same day as HS), whereas no new version of ISPN itself would be needed. Also the dependency cycle would be nicely untangled:

* ISPN depends on HS * "Lucene directory + provider" depends on HS and ISPN * Users wishing to store their indexes in ISPN would add the "Lucene directory + provider" dependency One issue I can see though is that the "Lucene directory + provider" component could not be easily part of a ISPN/JDG distribution as the dependency would be the other way around. Not sure whether that's actually a problem.

> - It will be easier to have new Infinispan code take advantage of > features exposed on the Infinispan Lucene Directory This makes me curious; There is ISPN code which directly depends on that directory implementation? What is this about, I assumed no such dependency to exist. If so, having the directory and provider in the ISPN repo itself indeed seems the only way, with the consequence of ISPN having to follow HS releases in a timely manner.

--Gunnar 2015-01-25 22:19 GMT+01:00 Sanne Grinovero <sanne(a)hibernate.org&gt;: > > All, > we've discussed several times the issues we have because of the > circular dependency between Hibernate Search and Infinispan. > Normally the pain point we aim to address with such discussions is the > need to carefully coordinate between releases [of Search and > Infinispan] in our quest for a final stable release, and brings some > release pressure. > > More recently it has also become a compatibility problem, as the two > projects target different platforms/environments, and have different > life-cycle expectations - creating more maintenance work such as lots > of back-porting of patches.. distracting from our goals. > > We already discussed some solutions, but none too convincing. I have a > new fresh proposal which I feel is more interesting: > > # New plan > > 1) the module "/infinispan" from the Hibernate Search source tree is > moved to the Infinispan project into some "hibernate-search-directory" > Maven module. > > 2) Hibernate Search drops any dependency to Infinispan > > Reminder: this "/infinispan" module we have contains just a couple of > classes, and represents the "DirectoryProvider" implementation which > integrates with the Hibernate Search autodiscovery and creates a > Directory instance (whose implementation always lived in Infinispan). > Most of the code is about applying configuration properties, > integration tests. > > It's a very simple plan, but has some valuable consequences: > - Search can move on without ever needing to wait for Infinispan - > and vice versa. > - There is no longer a circular dependency > - Search would be in control of its Lucene dependency, and can > upgrade as needed. We could experiment with different Lucene versions > without necessarily wait for Infinispan to solve compatiblity issues > first - and often more complex as that means then for Infinispan to be > able to guarantee a compatibility with a *range* of Lucene versions, > to include both the target Lucene and the version currently consumed > via Infinispan Query / Hibernate Search Engine. > - Infinispan can *opt* to stick with an older version of Search - or > update - provided it can satisfy both a) integration with possible > changes to the Search SPI b) an update for possible new requirements > of the new Lucene version (the Lucene Directory might need > compatibility fixes) > - The dependency structure would better match the one as provided by > our Enterprise Products. For example, it's the JDG distribution - not > Hibernate Search - to provide these integration bits, so it makes more > sense to build the Directory against the specific version of > Infinispan than against the specific version of Search. > - It will be easier to have new Infinispan code take advantage of > features exposed on the Infinispan Lucene Directory - if all changed > parts are in the same [Infinispan] repository. This is actually being > a problem right now, as it's holding back a POC meant to deliver great > improvements with the Directory implementation. > > And not least we'll have faster builds ;-) > > # Drawbacks > > First one is we'll probably have our users need to change some details > of how they build. > Infinispan might need to fix some SPI related code before being able > to upgrade, but historically our SPI has been extremely stable. > The real problem is that when such a thing happens, after we've > released a version of Search there might be some time before a > compatible version of Infinispan is made available as well. > > In practice this means the gap of time in which we have to catch up on > API changes is "exposed" to end users wanting to use our latest and > possibly blocked - but while they would then see the tip of the > iceberg of our integration, I believe it would still take the same > amount of waiting time in terms of calendar dates in which the working > duo is available to them - as with the current model in such a > situation we need to wait for the same Infinispan release to happen > before we can release ours. > So: same time, but we'd have a leaner process, and possibly quicker > releases for all users not interested in that - or just benefits in > all those scenarios in which we don't break APIs which is very common. > I've not identified other problems, so my opinion is that these are > well worth the benefits. > > # Consequences for our users > > Not much. Even today we expect our users to depend on several jar > files provided by the Infinispan team; this would be just one more. > Opens some questions though: > > A) Should the Maven group id be changed? I'd expect it to be > transferred to "org.infinispan" group at least, and probably need a > better artifact id too. > > B) License. Our code is LGPL, most of Infinispan is ASL - but not all > of it. So I expect it would be possible to keep the existing license > at least for now, and defer eventual license changes as a separate > step (if people feel need for any change at all). > > C) Documentation. Besides the needed updates in Maven coordinates / > download sources, I don't expect much of a difference: we'd still > explain how to set this integration up. > > D) Distribution. Today we distribute this module, and its > dependencies, in our release bundles. Which implies we distribute a > copy of various Infinispan jars. > I think we should drop these from our distribution - even though it > might seem counter-intuitive: > while it might seem convenient to have these included, the whole point > of the change would be that there would be more flexibility in which > versions of Infinispan would work with Search. And actually the > integration tests and this specific knowledge would be responsibility > of Infinispan. > > Am I failing to see a more critical issue? > How would you all feel about our code being transferred to the > different project? > > Sanne > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev

On 25 Jan 2015, at 22:19, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: All, we've discussed several times the issues we have because of the circular dependency between Hibernate Search and Infinispan. Normally the pain point we aim to address with such discussions is the need to carefully coordinate between releases [of Search and Infinispan] in our quest for a final stable release, and brings some release pressure. More recently it has also become a compatibility problem, as the two projects target different platforms/environments, and have different life-cycle expectations - creating more maintenance work such as lots of back-porting of patches.. distracting from our goals. We already discussed some solutions, but none too convincing. I have a new fresh proposal which I feel is more interesting: # New plan 1) the module "/infinispan" from the Hibernate Search source tree is moved to the Infinispan project into some "hibernate-search-directory" Maven module. 2) Hibernate Search drops any dependency to Infinispan Reminder: this "/infinispan" module we have contains just a couple of classes, and represents the "DirectoryProvider" implementation which integrates with the Hibernate Search autodiscovery and creates a Directory instance (whose implementation always lived in Infinispan). Most of the code is about applying configuration properties, integration tests. It's a very simple plan, but has some valuable consequences: - Search can move on without ever needing to wait for Infinispan - and vice versa. - There is no longer a circular dependency - Search would be in control of its Lucene dependency, and can upgrade as needed. We could experiment with different Lucene versions without necessarily wait for Infinispan to solve compatiblity issues first - and often more complex as that means then for Infinispan to be able to guarantee a compatibility with a *range* of Lucene versions, to include both the target Lucene and the version currently consumed via Infinispan Query / Hibernate Search Engine. - Infinispan can *opt* to stick with an older version of Search - or update - provided it can satisfy both a) integration with possible changes to the Search SPI b) an update for possible new requirements of the new Lucene version (the Lucene Directory might need compatibility fixes) - The dependency structure would better match the one as provided by our Enterprise Products. For example, it's the JDG distribution - not Hibernate Search - to provide these integration bits, so it makes more sense to build the Directory against the specific version of Infinispan than against the specific version of Search. - It will be easier to have new Infinispan code take advantage of features exposed on the Infinispan Lucene Directory - if all changed parts are in the same [Infinispan] repository. This is actually being a problem right now, as it's holding back a POC meant to deliver great improvements with the Directory implementation. And not least we'll have faster builds ;-) # Drawbacks First one is we'll probably have our users need to change some details of how they build. Infinispan might need to fix some SPI related code before being able to upgrade, but historically our SPI has been extremely stable. The real problem is that when such a thing happens, after we've released a version of Search there might be some time before a compatible version of Infinispan is made available as well. In practice this means the gap of time in which we have to catch up on API changes is "exposed" to end users wanting to use our latest and possibly blocked - but while they would then see the tip of the iceberg of our integration, I believe it would still take the same amount of waiting time in terms of calendar dates in which the working duo is available to them - as with the current model in such a situation we need to wait for the same Infinispan release to happen before we can release ours. So: same time, but we'd have a leaner process, and possibly quicker releases for all users not interested in that - or just benefits in all those scenarios in which we don't break APIs which is very common. I've not identified other problems, so my opinion is that these are well worth the benefits. # Consequences for our users Not much. Even today we expect our users to depend on several jar files provided by the Infinispan team; this would be just one more. Opens some questions though: A) Should the Maven group id be changed? I'd expect it to be transferred to "org.infinispan" group at least, and probably need a better artifact id too. B) License. Our code is LGPL, most of Infinispan is ASL - but not all of it. So I expect it would be possible to keep the existing license at least for now, and defer eventual license changes as a separate step (if people feel need for any change at all). C) Documentation. Besides the needed updates in Maven coordinates / download sources, I don't expect much of a difference: we'd still explain how to set this integration up. D) Distribution. Today we distribute this module, and its dependencies, in our release bundles. Which implies we distribute a copy of various Infinispan jars. I think we should drop these from our distribution - even though it might seem counter-intuitive: while it might seem convenient to have these included, the whole point of the change would be that there would be more flexibility in which versions of Infinispan would work with Search. And actually the integration tests and this specific knowledge would be responsibility of Infinispan. Am I failing to see a more critical issue? How would you all feel about our code being transferred to the different project? Sanne _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

On 26 Jan 2015, at 17:08, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: On 26 January 2015 at 15:05, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > One angle that is a bit unclear to me is how easy is this consumable and discoverable by our users. Discovery would be the same as today, same configuration property. We have your "resolution strategy" which translates from short name to fully qualified name, which is already "pluggable" in the case of the IndexManager lookup. So we have plenty of options A) look for the new FQCN B) plug a custom resolver strategy for the DirectoryProvider names C)use the custom IndexManager: we had already decided at our f2f meeting that it would be far easier for people to only have to choose the IndexManager implementation, which would be in charge of setting up the right backend/directory couples as a "profile". WildFly users don't even need to know about changes in the module names / jar names / maven coordinates as the instructions would still be: download both modules, drop them in your AS -> works. In short, we can keep user changes to minimal. > Some in this thread mentioned creating a separate repository. I see this as a massive ease of use problem if the user is exposed to finding the right version of each of these three libs. Likewise for documentation, project identity, etc. +1 not my preferred choice either. I think we'll stick to my plan, at least until I'm proven this proposal is utterly broken. > I think the merge of this piece of code into Infinispan does not create too much of a problem in that regard. The main issue is that as an Hibernate Search user, I don’t know which Infinispan to target. And an Infinispan user does not really care for the most part about an explicit HSearch usage outside Infinispan query. We'll have to document that. Indeed quite tricky, as there might be cases in which there is no compatible version of Infinispan whatsoever (for example each time we migrate to a new Lucene there would be a gap).. but my hope is that would be some Alpha/Beta, and by the time of a Final release we'd have a viable suggestion, with more options while the branch matures in maintenance mode. > BTW will the documentation on HSearch integration with Infinispan Lucene Directory remain in HSearch’s doc or be copied in Infinispan’s? I'd prefer to have a single consistent document for our users to read: we'd have the high level description from the point of view of an Hibernate user, and exposing to the subset of tuning options which make sense to him. Infinispan should have a different copy, especially as it's becoming clear that tuning and configuration for this specific component is quite different in the two different use cases. Ultimately both "engines" should be able to guess and auto-configure several options which are manual today, and these guessing and custom backends are different. The low level details of the Directory tuning should probably live in Infinispan only, in close coupling to the code changes, to make sure there is a document set kept in synch with the specific version. From experience we know the new configuration options needed by Infinispan are rarely in the "need to know" for Hibernate users. Sanne > > Emmanuel > >> On 25 Jan 2015, at 22:19, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: >> >> All, >> we've discussed several times the issues we have because of the >> circular dependency between Hibernate Search and Infinispan. >> Normally the pain point we aim to address with such discussions is the >> need to carefully coordinate between releases [of Search and >> Infinispan] in our quest for a final stable release, and brings some >> release pressure. >> >> More recently it has also become a compatibility problem, as the two >> projects target different platforms/environments, and have different >> life-cycle expectations - creating more maintenance work such as lots >> of back-porting of patches.. distracting from our goals. >> >> We already discussed some solutions, but none too convincing. I have a >> new fresh proposal which I feel is more interesting: >> >> # New plan >> >> 1) the module "/infinispan" from the Hibernate Search source tree is >> moved to the Infinispan project into some "hibernate-search-directory" >> Maven module. >> >> 2) Hibernate Search drops any dependency to Infinispan >> >> Reminder: this "/infinispan" module we have contains just a couple of >> classes, and represents the "DirectoryProvider" implementation which >> integrates with the Hibernate Search autodiscovery and creates a >> Directory instance (whose implementation always lived in Infinispan). >> Most of the code is about applying configuration properties, >> integration tests. >> >> It's a very simple plan, but has some valuable consequences: >> - Search can move on without ever needing to wait for Infinispan - >> and vice versa. >> - There is no longer a circular dependency >> - Search would be in control of its Lucene dependency, and can >> upgrade as needed. We could experiment with different Lucene versions >> without necessarily wait for Infinispan to solve compatiblity issues >> first - and often more complex as that means then for Infinispan to be >> able to guarantee a compatibility with a *range* of Lucene versions, >> to include both the target Lucene and the version currently consumed >> via Infinispan Query / Hibernate Search Engine. >> - Infinispan can *opt* to stick with an older version of Search - or >> update - provided it can satisfy both a) integration with possible >> changes to the Search SPI b) an update for possible new requirements >> of the new Lucene version (the Lucene Directory might need >> compatibility fixes) >> - The dependency structure would better match the one as provided by >> our Enterprise Products. For example, it's the JDG distribution - not >> Hibernate Search - to provide these integration bits, so it makes more >> sense to build the Directory against the specific version of >> Infinispan than against the specific version of Search. >> - It will be easier to have new Infinispan code take advantage of >> features exposed on the Infinispan Lucene Directory - if all changed >> parts are in the same [Infinispan] repository. This is actually being >> a problem right now, as it's holding back a POC meant to deliver great >> improvements with the Directory implementation. >> >> And not least we'll have faster builds ;-) >> >> # Drawbacks >> >> First one is we'll probably have our users need to change some details >> of how they build. >> Infinispan might need to fix some SPI related code before being able >> to upgrade, but historically our SPI has been extremely stable. >> The real problem is that when such a thing happens, after we've >> released a version of Search there might be some time before a >> compatible version of Infinispan is made available as well. >> >> In practice this means the gap of time in which we have to catch up on >> API changes is "exposed" to end users wanting to use our latest and >> possibly blocked - but while they would then see the tip of the >> iceberg of our integration, I believe it would still take the same >> amount of waiting time in terms of calendar dates in which the working >> duo is available to them - as with the current model in such a >> situation we need to wait for the same Infinispan release to happen >> before we can release ours. >> So: same time, but we'd have a leaner process, and possibly quicker >> releases for all users not interested in that - or just benefits in >> all those scenarios in which we don't break APIs which is very common. >> I've not identified other problems, so my opinion is that these are >> well worth the benefits. >> >> # Consequences for our users >> >> Not much. Even today we expect our users to depend on several jar >> files provided by the Infinispan team; this would be just one more. >> Opens some questions though: >> >> A) Should the Maven group id be changed? I'd expect it to be >> transferred to "org.infinispan" group at least, and probably need a >> better artifact id too. >> >> B) License. Our code is LGPL, most of Infinispan is ASL - but not all >> of it. So I expect it would be possible to keep the existing license >> at least for now, and defer eventual license changes as a separate >> step (if people feel need for any change at all). >> >> C) Documentation. Besides the needed updates in Maven coordinates / >> download sources, I don't expect much of a difference: we'd still >> explain how to set this integration up. >> >> D) Distribution. Today we distribute this module, and its >> dependencies, in our release bundles. Which implies we distribute a >> copy of various Infinispan jars. >> I think we should drop these from our distribution - even though it >> might seem counter-intuitive: >> while it might seem convenient to have these included, the whole point >> of the change would be that there would be more flexibility in which >> versions of Infinispan would work with Search. And actually the >> integration tests and this specific knowledge would be responsibility >> of Infinispan. >> >> Am I failing to see a more critical issue? >> How would you all feel about our code being transferred to the >> different project? >> >> Sanne >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >