Re: [hibernate-dev] [Hibernate Search] Database back end worker

2015-08-05 23:38 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org&gt;: > > Hi Flemming, > welcome on this list! > > I waited a bit to reply myself, as you already know I like the > proposal. Unfortunately many others are on holidays, so other feedback > might be slow. > > Still I wouldn't let that slow you down and start the works for > merging it; I already anticipated over chat that this would come and > we all agree that the concept is useful! > I don't think others looked at the details yet, but if it comes to > concerns at that level, we can address smaller issues incrementally. > (I also didn't look at micro-details, as it's easier to comment on > those on a pull request). > > I had the same question as Martin regarding clustering: with the > current implementation you expect something like the master/slave > configuration, or Infinispan to be used as storage, correct? > I also think it would be interesting to explore the approach further > to also - optionally - serve as a replacement for these, but that's > another feature which is easier to experiment with after the core > concept is merged. > > Yes that's correct. To start with it was written very specific to the use > of Infinispan as directory. And I agree on we should explore other cluster configuration and I have some ideas how we can implement it. > > . > > > In short, I would simply merge your backend as a new module in > Hibernate Search! Fork our repository, and send a pull request. > > # Code layout / Modules > > In terms of code structure, you might have noticed that the module > 'hibernate-search-engine' (/engine in the source code) does not depend > on JPA nor Hibernate ORM; the reason is that other projects reuse the > core indexing strategy and the backends. Since it would be nice to > allow them to optionally use your backend, still not mandate a > dependency on ORM for those who don't, I think this should be a new > Maven module. > > We already have > /backends/jgroups > /backends/jms > > So we could add (name to be refined?) : > /backends/relationaldb sure, no problem :) > > > Also, your integration tests probably should be moved together with > our other integration tests. They are currently running WildFly > 10.0.0.Alpha6, but that shouldn't be a problem. > > # Code Style > > We use tabs ;-) > And also have various other "exotic" conventions regarding white-space > usage, right header files, etc.. > We use CheckStyle to keep it tidy, it will give you lots of errors and > when there are many it's not very helpful; I would suggest to take the > formatting templates attached at the following link and use your IDE's > formatting capabilities, resort to checkstyle just for the final > validation: > - http://hibernate.org/search/contribute/ > > # JDK > > It looks like your extension requires Java 8; if you could convert it > to Java 7 that would be nice. Don't think it will be an issue. As far I remember we don't use any Java8 specific functionality > > > # Rebasing to latest > > I'm afraid we're now aiming at Hibernate ORM 5, so some details might > need to be updated; probably just in the configuration area. We're > also in the process of upgrading to Apache Lucene 5, but that > shouldn't affect you at all. > > # Some improvement ideas > > While we should support the case in which Hibernate Search is not > being run as an extension of Hibernate ORM, that's likely the most > common one. > In that scenario I think it would be nice to be able to lookup the > existing ORM services so that users don't need to repeat for example > the datasource configuration. > > We might also be able to reuse all of the SessionFactory, but I'm not > sure how to include your model without it potentially interfering with > the end user's model; I'd say let's start by sharing some services > from ORM and then see what kind of improvements we can build into ORM > for this use case; for example this might simplify some of the > TransactionManager configuration code I'm seeing in your repository. > > Of course your existing configuration properties are useful too, > especially for the non-ORM case as we'll need be able to reuse the ORM > services. > > Also, you might have noticed we are now able to optionally include the > backend operations in the same transaction. That's not the default, as > commonly people don't want that, but it would be very interesting to > evolve this backend to support that option too, you wouldn't even > require XA when storing the entity in the same database! > - http://in.relation.to/2015/07/09/hibernate-search-jms-transaction/ > Yes, it's very nice feature and fit perfectly with relationaldb > > I'd be happy to help with this, feel free to share non-working and/or > intermediate experimental branches when having questions or just > stuck. > Please start by creating a JIRA, you can leave the target version > undefined: we'll merge it when it's ready. > For all the task you have listed can I create sub task to the JIRA, or how do you track tasks? > > Thanks, > Sanne > > > On 5 August 2015 at 20:05, Flemming Harms <flemming.harms(a)gmail.com&gt; > wrote: > > Hi Martin > > > > For this version the AbstractDatabaseHibernateSearchController is not > > able > > to process Lucene workers simultaneously. When we build it our initial > > requirement was only one node should process the workers at a time, but > > the > > “master” was floating. We use Quartz to get this type of functionality > > and > > it will synchronizing the execution between the nodes. But you could > > also > > use an HA-singleton to dedicate a specific node to process the workers. > > > > We had been playing with an idea where we stamp the LuceneDatabaseWork > > with > > the known cluster nodes, and then the last node will remove it from the > > database or a scheduled job can take care of it. The advance of this > > solution is it will make Infinispan optional, and it can store the > > indexes > > on each node instead in a shared cache. > > > > Your idea and work look very nice. Pretty awesome feature to support > > different JPA providers. > > > > -- > > cheers > > Flemming > > > > > > 2015-08-05 11:57 GMT+02:00 Martin Braun <martinbraun123(a)aol.com&gt;: > > > >> Hi, > >> > >> > >> Note: I am no core developer of Hibernate Search, but I am currently > >> working on something > >> that looks quite similar to what you are doing :). One part of it is an > >> updating mechanism based on triggers > >> that uses the database as a event-storage as well. It's not the exact > >> same > >> thing, but related. > >> > >> > >> https://github.com/Hotware/Hibernate-Search-JPA > >> > >> > >> > >> The idea is quite nice, but after looking at the sourcecode I am > >> wondering > >> how the different nodes are able to work together, because in > >> AbstractDatabaseHibernateSearchController you remove the entity > >> from the persistence context and I wasn't able to find code that would > >> make up for that. > >> > >> > >> Doesn't that mean that the other workers will not be able to read that > >> entity? > >> Or will users of this need to implement their own synchronization > >> mechanism between > >> the different nodes? > >> > >> > >> Martin Braun > >> martinbraun123(a)aol.com > >> www.github.com/s4ke > >> > >> > >> > >> > >> -----Original Message----- > >> From: Flemming Harms <flemming.harms(a)gmail.com&gt; > >> To: Hibernate.org <hibernate-dev(a)lists.jboss.org&gt; > >> Sent: Tue, Aug 4, 2015 6:40 pm > >> Subject: [hibernate-dev] [Hibernate Search] Database back end worker > >> > >> > >> Hey guys > >> > >> I want to introduce myself and a new database back-end worker, me > >> and > >> another guy have build for hibernate search. I already had some initial > >> talk > >> with Sanne regarding if this could be interested to the hibernate > >> search > >> project. > >> > >> I have been working with Hibernate Search from some time and actually > >> done > >> various small custom modification to search since 3.x, especial > >> around > >> running in a cluster and indexing. To make a long story short when > >> we > >> upgraded Hibernate search we thought it would be ideal to use a SQL > >> database > >> as storage for lucene workers for 3 main reasons. > >> > >> - The database was shared > >> between the nodes > >> - The workers was persistent in case of a node crash. > >> - No > >> master/slave > >> > >> > >> *In some way it’s very similar to the JMS back-end worker, where > >> the user > >> also have to implement a MDB that process the workers. In our case > >> they > >> will have to implement a job using something like quartz or a > >> timer > >> service. * > >> > >> *We are using JPA as persistence layer for the database, even > >> it’s a fairly > >> simple entity we persistent, but it make sense for supporting > >> various > >> databases and schema update out of the box. We have tried to make it’s > >> as > >> easy as possible to set-up by minimizing the number of properties, and > >> it’s > >> all configurable from the persistence.xml* > >> > >> *The actually work can* be > >> *find > >> here > >> https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend > >> > >> <https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend>* > >> > >> > >> > >> *So > >> based on this introduction and the code, is this something you could > >> use? (of > >> course with the modification it requires to follow the design, > >> style, docs etc > >> for the search)*-- > >> > >> Kind regards > >> Flemming > >> Harms > >> _______________________________________________ > >> hibernate-dev mailing > >> list > >> hibernate-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > >> > >> > >> _______________________________________________ > >> hibernate-dev mailing list > >> hibernate-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > > > > > > -- > > > > Kind regards / Med Venlig Hilsen > > Flemming Harms > > > > - > > > > https://twitter.com/fnharms > > https://dk.linkedin.com/in/fharms > > _______________________________________________ > > hibernate-dev mailing list > > hibernate-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev -- Kind regards / Med Venlig Hilsen Flemming Harms https://twitter.com/fnharms https://dk.linkedin.com/in/fharms