[hibernate-dev] development sprint start: Hibernate Search

[Emmanuel Bernard]

Notes from the discussion on roadmap in decreasing order. Parallelize up to 4 tasks max.

### Must do

Lucene 5

Highly available master re-election
- JMS messages within the transaction
- high available master re-election
- Kafka? if JMS clustered MDB does not work

Compatible with Java 9
- should be minor

### Should have

Timeout exception for MassIndexer
ElasticSearch backend
Free Form

Emmanuel

> On 29 Jun 2015, at 01:43, Sanne Grinovero <sanne at hibernate.org> wrote:
> 
> Hello,
> welcome to Hibernate Search time!
> 
> [for those unaware: some of are now experimenting to work on 2-3 week
> sprints fully focused on a single Hibernate project, rotating the
> subject. We decided this privately as it's a matter of time-management
> for us, but I'm now opening the conversation up to all developers and
> contributors as it affects the project evolution and technical
> discussion; essentially it means we'll be focused on Hibernate Search
> more than other projects in the next few weeks, and aim at get some
> significant stuff done]
> 
> My first and foremost goal for the next couple of weeks would be to
> drive forward a pain point which is all of:
> - showing active interest from several power-contributors [1,2,3]
> - highly demanded from product perspective
> - had lots of people *begging* for better solutions in the past
> 
> You might have guessed: I'm talking about the backend configuration
> complexity in a clustered environment: both the JGroups and the JMS
> solutions expose the user to various complex system settings.
> With Emmanuel and Hardy I had some hints of conversations about it,
> but essentially to start this subject I'm proposing a meeting to
> discuss these; we can try and make it open to everyone, I might even
> make a couple of slides.
> 
> # What do we want
> During our last meeting, a scary point was to hear that Emmanuel was
> considering the priority to be free form. It never was for me, and
> while we didn't dig during that call, we better clarify this soon.
> Let's please find a moment on IRC to discuss the goals, especially as
> I need to update the project roadmap.
> 
> # How do we want it
> I've been hoping for a clear/formal set of requirements to be provided
> by some users, as there are many ways to look at the problem.
> But this never came, and I'm concluding that:
> A) if a paying customer or other kind of sponsor will want to discuss
> these requirements I better fly to them and talk face to face.
> B) I'm being lazy and selfish in expecting externals to clarify all
> details.. I shouldn't try to deflect this hard problem.
> 
> I've been thinking of several possible ways, and there are lots of
> options, and some tradeoffs to choose from.
> One of these options is to use a distributed consensus - since we
> already use JGroups in various projects, JGroups RAFT [4] seems a
> natural candidate but while I'd love the excuse to play with it, it's
> a very new codebase.
> Another option would be the more mature Apache Kafka - great for log
> based replication so might even be complementary to the JGroups RAFT
> implementation - or just improve JMS (via the standard or via Apache
> Camel) to have it integrate with Transactions [5: just got a
> contribution!] and provide better failover options.
> Not least, I just heard that WildFly 10 is going to provide some form
> of automatic HA/JMS singleton consumer.. I just heard about it and
> will need to find more about it.
> 
> While it's tempting to implement our own custom super clever backend,
> we should prioritize for an off-the-shelf method with high return on
> investment to solve the pain point.
> Also, as suggested by Hardy some months ago, it would be awesome to
> have the so called "Hibernate Search master node" not need any entity
> classes nor depend at all to the deployed application (nor its
> extensions like analyzers), so that if the solution still needs a
> "master role", we could simply provide a master app which doesn't need
> changes on application changes. This would necessarily be a change for
> 6.0, but let's either prepare for that, or get rid of the "master
> node" concept altogether.
> 
> We have been sitting and thinking about the problem since a while, I'd
> love now to see some empirical progress: merge them as experimental,
> and have some natural selection happen while these would also help us
> to refine the requirements.
> 
> ## Other topics
> Of course this isn't the only thing we'll be working on. The primary
> goal of current branch is still to deliver an Hibernate ORM 5
> compatible version, but we're in an horrible position with that since
> WildFly 10 just released another alpha tag which still doesn't use
> Hibernate  ORM v.5; Since it's [currently and temporarily] hard to run
> WildFly overriding the Hibernate ORM version, we won't be able to
> close in to a CR or Final release until at least the next Wildfly tag.
> In the meantime we can do some of research needed for the above topic,
> and make progress with the many issues open for 5.4 [6].
> 
> Another subject which we really should work on in this sprint, is to
> avoid transaction timeout on MassIndexer within a container [7]
> 
> So, for tomorrow: to get started, JIRA is updated and you have all
> tasks assigned already. Let's start from there, and then schedule a
> meeting to discuss the above.
> 
> Thanks!
> Sanne
> 
> References :
> 1 - https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend
> 2 - https://github.com/mrobson/hibernate-search-infinispan-jms
> 3 - https://forum.hibernate.org/viewtopic.php?f=9&t=1040179
> 4 - https://github.com/belaban/jgroups-raft
> 5 - https://hibernate.atlassian.net/browse/HSEARCH-668
> 6 - https://hibernate.atlassian.net/issues/?filter=12266
> 7 - https://hibernate.atlassian.net/browse/HSEARCH-1474
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev