Hi, Note: I am no core developer of Hibernate Search, but I am currently working on something that looks quite similar to what you are doing :). One part of it is an updating mechanism based on triggers that uses the database as a event-storage as well. It's not the exact same thing, but related. https://github.com/Hotware/Hibernate-Search-JPA The idea is quite nice, but after looking at the sourcecode I am wondering how the different nodes are able to work together, because in AbstractDatabaseHibernateSearchController you remove the entity from the persistence context and I wasn't able to find code that would make up for that. Doesn't that mean that the other workers will not be able to read that entity? Or will users of this need to implement their own synchronization mechanism between the different nodes? Martin Braun martinbraun123(a)aol.com www.github.com/s4ke -----Original Message----- From: Flemming Harms <flemming.harms(a)gmail.com&gt; To: Hibernate.org <hibernate-dev(a)lists.jboss.org&gt; Sent: Tue, Aug 4, 2015 6:40 pm Subject: [hibernate-dev] [Hibernate Search] Database back end worker Hey guys I want to introduce myself and a new database back-end worker, me and another guy have build for hibernate search. I already had some initial talk with Sanne regarding if this could be interested to the hibernate search project. I have been working with Hibernate Search from some time and actually done various small custom modification to search since 3.x, especial around running in a cluster and indexing. To make a long story short when we upgraded Hibernate search we thought it would be ideal to use a SQL database as storage for lucene workers for 3 main reasons. - The database was shared between the nodes - The workers was persistent in case of a node crash. - No master/slave *In some way it’s very similar to the JMS back-end worker, where the user also have to implement a MDB that process the workers. In our case they will have to implement a job using something like quartz or a timer service. * *We are using JPA as persistence layer for the database, even it’s a fairly simple entity we persistent, but it make sense for supporting various databases and schema update out of the box. We have tried to make it’s as easy as possible to set-up by minimizing the number of properties, and it’s all configurable from the persistence.xml* *The actually work can* be *find here https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend <https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend>* *So based on this introduction and the code, is this something you could use? (of course with the modification it requires to follow the design, style, docs etc for the search)*-- Kind regards Flemming Harms _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

Hi Flemming, welcome on this list! I waited a bit to reply myself, as you already know I like the proposal. Unfortunately many others are on holidays, so other feedback might be slow. Still I wouldn't let that slow you down and start the works for merging it; I already anticipated over chat that this would come and we all agree that the concept is useful! I don't think others looked at the details yet, but if it comes to concerns at that level, we can address smaller issues incrementally. (I also didn't look at micro-details, as it's easier to comment on those on a pull request). I had the same question as Martin regarding clustering: with the current implementation you expect something like the master/slave configuration, or Infinispan to be used as storage, correct? I also think it would be interesting to explore the approach further to also - optionally - serve as a replacement for these, but that's another feature which is easier to experiment with after the core concept is merged.

Yes that's correct. To start with it was written very specific to the use of Infinispan as directory.

.

In short, I would simply merge your backend as a new module in Hibernate Search! Fork our repository, and send a pull request. # Code layout / Modules In terms of code structure, you might have noticed that the module 'hibernate-search-engine' (/engine in the source code) does not depend on JPA nor Hibernate ORM; the reason is that other projects reuse the core indexing strategy and the backends. Since it would be nice to allow them to optionally use your backend, still not mandate a dependency on ORM for those who don't, I think this should be a new Maven module. We already have /backends/jgroups /backends/jms So we could add (name to be refined?) : /backends/relationaldb

Also, your integration tests probably should be moved together with our other integration tests. They are currently running WildFly 10.0.0.Alpha6, but that shouldn't be a problem. # Code Style We use tabs ;-) And also have various other "exotic" conventions regarding white-space usage, right header files, etc.. We use CheckStyle to keep it tidy, it will give you lots of errors and when there are many it's not very helpful; I would suggest to take the formatting templates attached at the following link and use your IDE's formatting capabilities, resort to checkstyle just for the final validation: - http://hibernate.org/search/contribute/ # JDK It looks like your extension requires Java 8; if you could convert it to Java 7 that would be nice.

# Rebasing to latest I'm afraid we're now aiming at Hibernate ORM 5, so some details might need to be updated; probably just in the configuration area. We're also in the process of upgrading to Apache Lucene 5, but that shouldn't affect you at all. # Some improvement ideas While we should support the case in which Hibernate Search is not being run as an extension of Hibernate ORM, that's likely the most common one. In that scenario I think it would be nice to be able to lookup the existing ORM services so that users don't need to repeat for example the datasource configuration. We might also be able to reuse all of the SessionFactory, but I'm not sure how to include your model without it potentially interfering with the end user's model; I'd say let's start by sharing some services from ORM and then see what kind of improvements we can build into ORM for this use case; for example this might simplify some of the TransactionManager configuration code I'm seeing in your repository. Of course your existing configuration properties are useful too, especially for the non-ORM case as we'll need be able to reuse the ORM services. Also, you might have noticed we are now able to optionally include the backend operations in the same transaction. That's not the default, as commonly people don't want that, but it would be very interesting to evolve this backend to support that option too, you wouldn't even require XA when storing the entity in the same database! - http://in.relation.to/2015/07/09/hibernate-search-jms-transaction/ Yes, it's very nice feature and fit perfectly with relationaldb

I'd be happy to help with this, feel free to share non-working and/or intermediate experimental branches when having questions or just stuck. Please start by creating a JIRA, you can leave the target version undefined: we'll merge it when it's ready. For all the task you have listed can I create sub task to the JIRA, or how

Thanks, Sanne On 5 August 2015 at 20:05, Flemming Harms <flemming.harms(a)gmail.com&gt; wrote: > Hi Martin > > For this version the AbstractDatabaseHibernateSearchController is not able > to process Lucene workers simultaneously. When we build it our initial > requirement was only one node should process the workers at a time, but the > “master” was floating. We use Quartz to get this type of functionality and > it will synchronizing the execution between the nodes. But you could also > use an HA-singleton to dedicate a specific node to process the workers. > > We had been playing with an idea where we stamp the LuceneDatabaseWork with > the known cluster nodes, and then the last node will remove it from the > database or a scheduled job can take care of it. The advance of this > solution is it will make Infinispan optional, and it can store the indexes > on each node instead in a shared cache. > > Your idea and work look very nice. Pretty awesome feature to support > different JPA providers. > > -- > cheers > Flemming > > > 2015-08-05 11:57 GMT+02:00 Martin Braun <martinbraun123(a)aol.com&gt;: > >> Hi, >> >> >> Note: I am no core developer of Hibernate Search, but I am currently >> working on something >> that looks quite similar to what you are doing :). One part of it is an >> updating mechanism based on triggers >> that uses the database as a event-storage as well. It's not the exact same >> thing, but related. >> >> >> https://github.com/Hotware/Hibernate-Search-JPA >> >> >> >> The idea is quite nice, but after looking at the sourcecode I am wondering >> how the different nodes are able to work together, because in >> AbstractDatabaseHibernateSearchController you remove the entity >> from the persistence context and I wasn't able to find code that would >> make up for that. >> >> >> Doesn't that mean that the other workers will not be able to read that >> entity? >> Or will users of this need to implement their own synchronization >> mechanism between >> the different nodes? >> >> >> Martin Braun >> martinbraun123(a)aol.com >> www.github.com/s4ke >> >> >> >> >> -----Original Message----- >> From: Flemming Harms <flemming.harms(a)gmail.com&gt; >> To: Hibernate.org <hibernate-dev(a)lists.jboss.org&gt; >> Sent: Tue, Aug 4, 2015 6:40 pm >> Subject: [hibernate-dev] [Hibernate Search] Database back end worker >> >> >> Hey guys >> >> I want to introduce myself and a new database back-end worker, me >> and >> another guy have build for hibernate search. I already had some initial >> talk >> with Sanne regarding if this could be interested to the hibernate >> search >> project. >> >> I have been working with Hibernate Search from some time and actually >> done >> various small custom modification to search since 3.x, especial >> around >> running in a cluster and indexing. To make a long story short when >> we >> upgraded Hibernate search we thought it would be ideal to use a SQL >> database >> as storage for lucene workers for 3 main reasons. >> >> - The database was shared >> between the nodes >> - The workers was persistent in case of a node crash. >> - No >> master/slave >> >> >> *In some way it’s very similar to the JMS back-end worker, where >> the user >> also have to implement a MDB that process the workers. In our case >> they >> will have to implement a job using something like quartz or a >> timer >> service. * >> >> *We are using JPA as persistence layer for the database, even >> it’s a fairly >> simple entity we persistent, but it make sense for supporting >> various >> databases and schema update out of the box. We have tried to make it’s >> as >> easy as possible to set-up by minimizing the number of properties, and >> it’s >> all configurable from the persistence.xml* >> >> *The actually work can* be >> *find >> here >> https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend >> <https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend >* >> >> >> >> *So >> based on this introduction and the code, is this something you could >> use? (of >> course with the modification it requires to follow the design, >> style, docs etc >> for the search)*-- >> >> Kind regards >> Flemming >> Harms >> _______________________________________________ >> hibernate-dev mailing >> list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> >> >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > -- > > Kind regards / Med Venlig Hilsen > Flemming Harms > > - > > https://twitter.com/fnharms > https://dk.linkedin.com/in/fharms > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev

Hi, yes creating issues on JIRA is what we normally do, feel free to create them! They don't have to be sub-stasks; we use subtasks when there are several "blocker" steps which need to be accomplished to get a feature and it's so large that it needs to be split. In this case I'd make a single JIRA for the new feature - which gets resolved when we'll merge it in its most essential form - and further improvement ideas can be created now or as needed as independent improvements. Thanks! Sanne On 9 August 2015 at 06:56, Flemming Harms <flemming.harms(a)gmail.com&gt; wrote: > > > 2015-08-05 23:38 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org&gt;: >> >> Hi Flemming, >> welcome on this list! >> >> I waited a bit to reply myself, as you already know I like the >> proposal. Unfortunately many others are on holidays, so other feedback >> might be slow. >> >> Still I wouldn't let that slow you down and start the works for >> merging it; I already anticipated over chat that this would come and >> we all agree that the concept is useful! >> I don't think others looked at the details yet, but if it comes to >> concerns at that level, we can address smaller issues incrementally. >> (I also didn't look at micro-details, as it's easier to comment on >> those on a pull request). >> >> I had the same question as Martin regarding clustering: with the >> current implementation you expect something like the master/slave >> configuration, or Infinispan to be used as storage, correct? >> I also think it would be interesting to explore the approach further >> to also - optionally - serve as a replacement for these, but that's >> another feature which is easier to experiment with after the core >> concept is merged. > > >> >> Yes that's correct. To start with it was written very specific to the use >> of Infinispan as directory. > > And I agree on we should explore other cluster configuration and I have some > ideas how we can implement it. >> >> . >> >> >> In short, I would simply merge your backend as a new module in >> Hibernate Search! Fork our repository, and send a pull request. >> >> # Code layout / Modules >> >> In terms of code structure, you might have noticed that the module >> 'hibernate-search-engine' (/engine in the source code) does not depend >> on JPA nor Hibernate ORM; the reason is that other projects reuse the >> core indexing strategy and the backends. Since it would be nice to >> allow them to optionally use your backend, still not mandate a >> dependency on ORM for those who don't, I think this should be a new >> Maven module. >> >> We already have >> /backends/jgroups >> /backends/jms >> >> So we could add (name to be refined?) : >> /backends/relationaldb > > sure, no problem :) >> >> >> Also, your integration tests probably should be moved together with >> our other integration tests. They are currently running WildFly >> 10.0.0.Alpha6, but that shouldn't be a problem. >> >> # Code Style >> >> We use tabs ;-) >> And also have various other "exotic" conventions regarding white-space >> usage, right header files, etc.. >> We use CheckStyle to keep it tidy, it will give you lots of errors and >> when there are many it's not very helpful; I would suggest to take the >> formatting templates attached at the following link and use your IDE's >> formatting capabilities, resort to checkstyle just for the final >> validation: >> - http://hibernate.org/search/contribute/ >> >> # JDK >> >> It looks like your extension requires Java 8; if you could convert it >> to Java 7 that would be nice. > > Don't think it will be an issue. As far I remember we don't use any Java8 > specific functionality >> >> >> # Rebasing to latest >> >> I'm afraid we're now aiming at Hibernate ORM 5, so some details might >> need to be updated; probably just in the configuration area. We're >> also in the process of upgrading to Apache Lucene 5, but that >> shouldn't affect you at all. >> >> # Some improvement ideas >> >> While we should support the case in which Hibernate Search is not >> being run as an extension of Hibernate ORM, that's likely the most >> common one. >> In that scenario I think it would be nice to be able to lookup the >> existing ORM services so that users don't need to repeat for example >> the datasource configuration. >> >> We might also be able to reuse all of the SessionFactory, but I'm not >> sure how to include your model without it potentially interfering with >> the end user's model; I'd say let's start by sharing some services >> from ORM and then see what kind of improvements we can build into ORM >> for this use case; for example this might simplify some of the >> TransactionManager configuration code I'm seeing in your repository. >> >> Of course your existing configuration properties are useful too, >> especially for the non-ORM case as we'll need be able to reuse the ORM >> services. >> >> Also, you might have noticed we are now able to optionally include the >> backend operations in the same transaction. That's not the default, as >> commonly people don't want that, but it would be very interesting to >> evolve this backend to support that option too, you wouldn't even >> require XA when storing the entity in the same database! >> - http://in.relation.to/2015/07/09/hibernate-search-jms-transaction/ >> > Yes, it's very nice feature and fit perfectly with relationaldb >> >> I'd be happy to help with this, feel free to share non-working and/or >> intermediate experimental branches when having questions or just >> stuck. >> Please start by creating a JIRA, you can leave the target version >> undefined: we'll merge it when it's ready. >> > For all the task you have listed can I create sub task to the JIRA, or how > do you track tasks? > >> >> Thanks, >> Sanne >> >> >> On 5 August 2015 at 20:05, Flemming Harms <flemming.harms(a)gmail.com&gt; >> wrote: >> > Hi Martin >> > >> > For this version the AbstractDatabaseHibernateSearchController is not >> > able >> > to process Lucene workers simultaneously. When we build it our initial >> > requirement was only one node should process the workers at a time, but >> > the >> > “master” was floating. We use Quartz to get this type of functionality >> > and >> > it will synchronizing the execution between the nodes. But you could >> > also >> > use an HA-singleton to dedicate a specific node to process the workers. >> > >> > We had been playing with an idea where we stamp the LuceneDatabaseWork >> > with >> > the known cluster nodes, and then the last node will remove it from the >> > database or a scheduled job can take care of it. The advance of this >> > solution is it will make Infinispan optional, and it can store the >> > indexes >> > on each node instead in a shared cache. >> > >> > Your idea and work look very nice. Pretty awesome feature to support >> > different JPA providers. >> > >> > -- >> > cheers >> > Flemming >> > >> > >> > 2015-08-05 11:57 GMT+02:00 Martin Braun <martinbraun123(a)aol.com&gt;: >> > >> >> Hi, >> >> >> >> >> >> Note: I am no core developer of Hibernate Search, but I am currently >> >> working on something >> >> that looks quite similar to what you are doing :). One part of it is an >> >> updating mechanism based on triggers >> >> that uses the database as a event-storage as well. It's not the exact >> >> same >> >> thing, but related. >> >> >> >> >> >> https://github.com/Hotware/Hibernate-Search-JPA >> >> >> >> >> >> >> >> The idea is quite nice, but after looking at the sourcecode I am >> >> wondering >> >> how the different nodes are able to work together, because in >> >> AbstractDatabaseHibernateSearchController you remove the entity >> >> from the persistence context and I wasn't able to find code that would >> >> make up for that. >> >> >> >> >> >> Doesn't that mean that the other workers will not be able to read that >> >> entity? >> >> Or will users of this need to implement their own synchronization >> >> mechanism between >> >> the different nodes? >> >> >> >> >> >> Martin Braun >> >> martinbraun123(a)aol.com >> >> www.github.com/s4ke >> >> >> >> >> >> >> >> >> >> -----Original Message----- >> >> From: Flemming Harms <flemming.harms(a)gmail.com&gt; >> >> To: Hibernate.org <hibernate-dev(a)lists.jboss.org&gt; >> >> Sent: Tue, Aug 4, 2015 6:40 pm >> >> Subject: [hibernate-dev] [Hibernate Search] Database back end worker >> >> >> >> >> >> Hey guys >> >> >> >> I want to introduce myself and a new database back-end worker, me >> >> and >> >> another guy have build for hibernate search. I already had some initial >> >> talk >> >> with Sanne regarding if this could be interested to the hibernate >> >> search >> >> project. >> >> >> >> I have been working with Hibernate Search from some time and actually >> >> done >> >> various small custom modification to search since 3.x, especial >> >> around >> >> running in a cluster and indexing. To make a long story short when >> >> we >> >> upgraded Hibernate search we thought it would be ideal to use a SQL >> >> database >> >> as storage for lucene workers for 3 main reasons. >> >> >> >> - The database was shared >> >> between the nodes >> >> - The workers was persistent in case of a node crash. >> >> - No >> >> master/slave >> >> >> >> >> >> *In some way it’s very similar to the JMS back-end worker, where >> >> the user >> >> also have to implement a MDB that process the workers. In our case >> >> they >> >> will have to implement a job using something like quartz or a >> >> timer >> >> service. * >> >> >> >> *We are using JPA as persistence layer for the database, even >> >> it’s a fairly >> >> simple entity we persistent, but it make sense for supporting >> >> various >> >> databases and schema update out of the box. We have tried to make it’s >> >> as >> >> easy as possible to set-up by minimizing the number of properties, and >> >> it’s >> >> all configurable from the persistence.xml* >> >> >> >> *The actually work can* be >> >> *find >> >> here >> >> https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend >> >> >> >> <https://github.com/umbrew/org.umbrew.hibernate.database.worker.backend>* >> >> >> >> >> >> >> >> *So >> >> based on this introduction and the code, is this something you could >> >> use? (of >> >> course with the modification it requires to follow the design, >> >> style, docs etc >> >> for the search)*-- >> >> >> >> Kind regards >> >> Flemming >> >> Harms >> >> _______________________________________________ >> >> hibernate-dev mailing >> >> list >> >> hibernate-dev(a)lists.jboss.org >> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> >> >> >> >> >> _______________________________________________ >> >> hibernate-dev mailing list >> >> hibernate-dev(a)lists.jboss.org >> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> > >> > >> > >> > >> > -- >> > >> > Kind regards / Med Venlig Hilsen >> > Flemming Harms >> > >> > - >> > >> > https://twitter.com/fnharms >> > https://dk.linkedin.com/in/fharms >> > _______________________________________________ >> > hibernate-dev mailing list >> > hibernate-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > -- > > Kind regards / Med Venlig Hilsen > Flemming Harms > > https://twitter.com/fnharms > https://dk.linkedin.com/in/fharms > _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev