yes Steve I'm more familiar with Antlr4 ( but not 3) and I gave a look at your poc. Apart some problems to fully understand the semantic model (due to my lack of a complete knowledge of the domain problem), I agree with you about the simplicity and elegance of the grammar for HQL recognition and semantic model building. What I don't like it's the necessity to build our own semantic model walker/s in order to produce the final SQL. On 14 August 2015 at 16:32, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > We've had a few discussions about this in the past. As 5.0 is getting > close to Final (next week), its time to start contemplating our next major > tasks. The consensus pick for that has been the idea of a "unified SQL > generation engine" along with a shared project for the semantic analysis > of > HQL/JPQL (and recently it was decided to include JPA Criteria > interpretation here as well). > > The central premise is this. Take the roughly 6 or 7 different top-level > ways Hibernate generates SQL and combine that into one "engine" based on > the input of a "semantic tree". The mentioned HQL/JPQL/Criteria shared > project will be one producer of such semantic trees. Others would include > persisters (for insert/update/delete requests) and loaders (for load > requests). > > We have a lot of tasks for this overall goal still remaining. > > We still have to finalize the design for the HQL/JPQL/Criteria to semantic > tree translator. One option is to proceed with the Antlr 4 based approach > I started a PoC for. John has been helping me some lately with that. The > first task here is to come to a consensus whether Antlr 4 is the way we > want to proceed here. We've been over the pros and cons before in detail. > In summary, there is a lot to love with Antlr 4. Our grammar for HQL > recognition and semantic tree building is very simple and elegant imo. > The > drawback is clearly the lack of tree walking, meaning that we are > responsible for writing by hand our walker for the semantic tree. In fact > multiple, since each consumer (orm, ogm, search) would need to write their > own. And if we decide to build another AST while walking the semantic > tree, we'd end up having to hand-write yet another walker for those. > > What I mean by that last part is that there are 2 ways we might choose to > deal with the semantic tree. For the purpose of discussion, let's look at > the ORM case. The first approach is to simply generate the SQL as we walk > the semantic tree; this would be a 2 phase interpretation approach (input > -> semantic tree -> SQL). That works in many cases. However it breaks > down in other cases. This is exactly the approach our existing HQL > translator uses. The other approach is to use a 3-phase translation > (input > -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint to > one > of the major problems. One source "semantic" query will often correspond > to multiple SQL queries; that is hard to manage in the 2-phase approach. > And not to mention integrating things like follow-on fetches and other > enhancements we want to gain from this. My vote is definitely for 3 or > more phases of interpretation. The problem is that this is exactly where > Antlr 4 sort of falls down. > > So first things first... we need to decide on Antlr 3 versus Antlr 4 > (versus some other parser solution). > > Next, on the ORM side (every "backend" can decide this individually) we > need to decide on the approach for semantic-tree to SQL translation, which > somewhat depends on the Antlr 3 versus Antlr 4 decision. > > We really need to decide these things ASAP and get moving on them as soon > as ORM 5.0 is finished. > > Also, this is a massive undertaking with huge gain potentials for not just > ORM. As such we need to understand who will be working on this. Sanne, > Gunnar... I know y'all have a vested interest and a desire to work on it. > John, I know the same is true for you. Andrea? Have you had a chance to > look over the poc and/or get more familiar with Antlr? > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev >

http://www.antlr2.org/article/1170602723163/treewalkers.html Not sure if y'all have seen this. Its an old article advocating manual tree walking (what we are facing here) over using generated tree walkers. On Wed, Aug 19, 2015 at 12:27 PM Steve Ebersole <steve(a)hibernate.org&gt; wrote: > I agree. Its my biggest hang up with regard to using Antlr 4. Actually, > its my only hang up with Antlr 4, but its a huge one. > > On Tue, Aug 18, 2015 at 9:30 AM andrea boriero <dreborier(a)gmail.com&gt; > wrote: > >> yes Steve I'm more familiar with Antlr4 ( but not 3) and I gave a look >> at your poc. >> >> Apart some problems to fully understand the semantic model (due to my >> lack of a complete knowledge of the domain problem), >> I agree with you about the simplicity and elegance of the grammar for >> HQL recognition and semantic model building. >> >> What I don't like it's the necessity to build our own semantic model >> walker/s in order to produce the final SQL. >> >> >> >> >> >> >> >> >> >> On 14 August 2015 at 16:32, Steve Ebersole <steve(a)hibernate.org&gt; wrote: >> >>> We've had a few discussions about this in the past. As 5.0 is getting >>> close to Final (next week), its time to start contemplating our next >>> major >>> tasks. The consensus pick for that has been the idea of a "unified SQL >>> generation engine" along with a shared project for the semantic >>> analysis of >>> HQL/JPQL (and recently it was decided to include JPA Criteria >>> interpretation here as well). >>> >>> The central premise is this. Take the roughly 6 or 7 different >>> top-level >>> ways Hibernate generates SQL and combine that into one "engine" based on >>> the input of a "semantic tree". The mentioned HQL/JPQL/Criteria shared >>> project will be one producer of such semantic trees. Others would >>> include >>> persisters (for insert/update/delete requests) and loaders (for load >>> requests). >>> >>> We have a lot of tasks for this overall goal still remaining. >>> >>> We still have to finalize the design for the HQL/JPQL/Criteria to >>> semantic >>> tree translator. One option is to proceed with the Antlr 4 based >>> approach >>> I started a PoC for. John has been helping me some lately with that. >>> The >>> first task here is to come to a consensus whether Antlr 4 is the way we >>> want to proceed here. We've been over the pros and cons before in >>> detail. >>> In summary, there is a lot to love with Antlr 4. Our grammar for HQL >>> recognition and semantic tree building is very simple and elegant imo. >>> The >>> drawback is clearly the lack of tree walking, meaning that we are >>> responsible for writing by hand our walker for the semantic tree. In >>> fact >>> multiple, since each consumer (orm, ogm, search) would need to write >>> their >>> own. And if we decide to build another AST while walking the semantic >>> tree, we'd end up having to hand-write yet another walker for those. >>> >>> What I mean by that last part is that there are 2 ways we might choose >>> to >>> deal with the semantic tree. For the purpose of discussion, let's look >>> at >>> the ORM case. The first approach is to simply generate the SQL as we >>> walk >>> the semantic tree; this would be a 2 phase interpretation approach >>> (input >>> -> semantic tree -> SQL). That works in many cases. However it breaks >>> down in other cases. This is exactly the approach our existing HQL >>> translator uses. The other approach is to use a 3-phase translation >>> (input >>> -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint to >>> one >>> of the major problems. One source "semantic" query will often >>> correspond >>> to multiple SQL queries; that is hard to manage in the 2-phase approach. >>> And not to mention integrating things like follow-on fetches and other >>> enhancements we want to gain from this. My vote is definitely for 3 or >>> more phases of interpretation. The problem is that this is exactly >>> where >>> Antlr 4 sort of falls down. >>> >>> So first things first... we need to decide on Antlr 3 versus Antlr 4 >>> (versus some other parser solution). >>> >>> Next, on the ORM side (every "backend" can decide this individually) we >>> need to decide on the approach for semantic-tree to SQL translation, >>> which >>> somewhat depends on the Antlr 3 versus Antlr 4 decision. >>> >>> We really need to decide these things ASAP and get moving on them as >>> soon >>> as ORM 5.0 is finished. >>> >>> Also, this is a massive undertaking with huge gain potentials for not >>> just >>> ORM. As such we need to understand who will be working on this. Sanne, >>> Gunnar... I know y'all have a vested interest and a desire to work on >>> it. >>> John, I know the same is true for you. Andrea? Have you had a chance >>> to >>> look over the poc and/or get more familiar with Antlr? >>> >> _______________________________________________ >>> hibernate-dev mailing list >>> hibernate-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>> >> >>

Just a heads up that I started a major refactoring of the antlr4 poc project in preparation for starting to look at this next sql-gen step. First I am making it into a multi-module project. We will have the hql-parser module, but then also an orm-sql-gen module to be able to play with that part. This makes sure we are not blending orm concerns into the pure hql parser. Also, I started working on splitting the "semantic query" model out into a separate module as well. There are a few reasons for this. I wont go into them all here. The main one being that HQL is just one producer of this semantic model. Rather than another long name I went with the acronym SQM (Semantic Query Model) here. The top package being org.hibernate.sqm. These changes already illustrated some tighter couplings then I had intended, so it was a good exercise. I'll push once I get those couplings cleaned up. On Fri, Aug 21, 2015 at 2:35 PM andrea boriero <dreborier(a)gmail.com&gt; wrote: > I haven't seen it, I'm going to read it. > > On 21 August 2015 at 16:54, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > >> http://www.antlr2.org/article/1170602723163/treewalkers.html >> >> Not sure if y'all have seen this. Its an old article advocating manual >> tree walking (what we are facing here) over using generated tree walkers. >> >> >> >> On Wed, Aug 19, 2015 at 12:27 PM Steve Ebersole <steve(a)hibernate.org&gt; >> wrote: >> >>> I agree. Its my biggest hang up with regard to using Antlr 4. >>> Actually, its my only hang up with Antlr 4, but its a huge one. >>> >>> On Tue, Aug 18, 2015 at 9:30 AM andrea boriero <dreborier(a)gmail.com&gt; >>> wrote: >>> >>>> yes Steve I'm more familiar with Antlr4 ( but not 3) and I gave a look >>>> at your poc. >>>> >>>> Apart some problems to fully understand the semantic model (due to my >>>> lack of a complete knowledge of the domain problem), >>>> I agree with you about the simplicity and elegance of the grammar >>>> for HQL recognition and semantic model building. >>>> >>>> What I don't like it's the necessity to build our own semantic model >>>> walker/s in order to produce the final SQL. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 14 August 2015 at 16:32, Steve Ebersole <steve(a)hibernate.org&gt; >>>> wrote: >>>> >>>>> We've had a few discussions about this in the past. As 5.0 is getting >>>>> close to Final (next week), its time to start contemplating our next >>>>> major >>>>> tasks. The consensus pick for that has been the idea of a "unified >>>>> SQL >>>>> generation engine" along with a shared project for the semantic >>>>> analysis of >>>>> HQL/JPQL (and recently it was decided to include JPA Criteria >>>>> interpretation here as well). >>>>> >>>>> The central premise is this. Take the roughly 6 or 7 different >>>>> top-level >>>>> ways Hibernate generates SQL and combine that into one "engine" based >>>>> on >>>>> the input of a "semantic tree". The mentioned HQL/JPQL/Criteria >>>>> shared >>>>> project will be one producer of such semantic trees. Others would >>>>> include >>>>> persisters (for insert/update/delete requests) and loaders (for load >>>>> requests). >>>>> >>>>> We have a lot of tasks for this overall goal still remaining. >>>>> >>>>> We still have to finalize the design for the HQL/JPQL/Criteria to >>>>> semantic >>>>> tree translator. One option is to proceed with the Antlr 4 based >>>>> approach >>>>> I started a PoC for. John has been helping me some lately with >>>>> that. The >>>>> first task here is to come to a consensus whether Antlr 4 is the way >>>>> we >>>>> want to proceed here. We've been over the pros and cons before in >>>>> detail. >>>>> In summary, there is a lot to love with Antlr 4. Our grammar for HQL >>>>> recognition and semantic tree building is very simple and elegant >>>>> imo. The >>>>> drawback is clearly the lack of tree walking, meaning that we are >>>>> responsible for writing by hand our walker for the semantic tree. In >>>>> fact >>>>> multiple, since each consumer (orm, ogm, search) would need to write >>>>> their >>>>> own. And if we decide to build another AST while walking the semantic >>>>> tree, we'd end up having to hand-write yet another walker for those. >>>>> >>>>> What I mean by that last part is that there are 2 ways we might >>>>> choose to >>>>> deal with the semantic tree. For the purpose of discussion, let's >>>>> look at >>>>> the ORM case. The first approach is to simply generate the SQL as we >>>>> walk >>>>> the semantic tree; this would be a 2 phase interpretation approach >>>>> (input >>>>> -> semantic tree -> SQL). That works in many cases. However it >>>>> breaks >>>>> down in other cases. This is exactly the approach our existing HQL >>>>> translator uses. The other approach is to use a 3-phase translation >>>>> (input >>>>> -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint >>>>> to one >>>>> of the major problems. One source "semantic" query will often >>>>> correspond >>>>> to multiple SQL queries; that is hard to manage in the 2-phase >>>>> approach. >>>>> And not to mention integrating things like follow-on fetches and other >>>>> enhancements we want to gain from this. My vote is definitely for 3 >>>>> or >>>>> more phases of interpretation. The problem is that this is exactly >>>>> where >>>>> Antlr 4 sort of falls down. >>>>> >>>>> So first things first... we need to decide on Antlr 3 versus Antlr 4 >>>>> (versus some other parser solution). >>>>> >>>>> Next, on the ORM side (every "backend" can decide this individually) >>>>> we >>>>> need to decide on the approach for semantic-tree to SQL translation, >>>>> which >>>>> somewhat depends on the Antlr 3 versus Antlr 4 decision. >>>>> >>>>> We really need to decide these things ASAP and get moving on them as >>>>> soon >>>>> as ORM 5.0 is finished. >>>>> >>>>> Also, this is a massive undertaking with huge gain potentials for not >>>>> just >>>>> ORM. As such we need to understand who will be working on this. >>>>> Sanne, >>>>> Gunnar... I know y'all have a vested interest and a desire to work on >>>>> it. >>>>> John, I know the same is true for you. Andrea? Have you had a >>>>> chance to >>>>> look over the poc and/or get more familiar with Antlr? >>>>> >>>> _______________________________________________ >>>>> hibernate-dev mailing list >>>>> hibernate-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>>>> >>>> >>>> >

Another point I want to discuss up from because it affects tree structure. Specifically the idea of an "unbounded implicit inheritance" query. These are queries like "from java.lang.Object". Queries where the from clause pulls in "unmapped inheritance". These are fine, to an extent. Hibernate has natively supported these since way back[1]. What is problematic is cases where we have more than one "unmapped inheritance" reference. E.g. "from java.lang.Object o1, java.lang.Object o2". In fact its the same difficulty as an unbounded cartesian product, but here in terms of the number of SQL queries we need to produce/execute. So I propose that we allow just one "unmapped inheritance" reference per query. [1] Reminder to self... another "strict JPQL compliance" consideration. On Sat, Aug 22, 2015 at 1:16 PM Steve Ebersole <steve(a)hibernate.org&gt; wrote: > I got that initial refactoring pushed to my fork... > > On Fri, Aug 21, 2015 at 3:51 PM Steve Ebersole <steve(a)hibernate.org&gt; > wrote: > >> Just a heads up that I started a major refactoring of the antlr4 poc >> project in preparation for starting to look at this next sql-gen step. >> >> First I am making it into a multi-module project. We will have the >> hql-parser module, but then also an orm-sql-gen module to be able to play >> with that part. This makes sure we are not blending orm concerns into the >> pure hql parser. >> >> Also, I started working on splitting the "semantic query" model out into >> a separate module as well. There are a few reasons for this. I wont go >> into them all here. The main one being that HQL is just one producer of >> this semantic model. Rather than another long name I went with the >> acronym SQM (Semantic Query Model) here. The top package being >> org.hibernate.sqm. >> >> These changes already illustrated some tighter couplings then I had >> intended, so it was a good exercise. I'll push once I get those couplings >> cleaned up. >> >> On Fri, Aug 21, 2015 at 2:35 PM andrea boriero <dreborier(a)gmail.com&gt; >> wrote: >> >>> I haven't seen it, I'm going to read it. >>> >>> On 21 August 2015 at 16:54, Steve Ebersole <steve(a)hibernate.org&gt; wrote: >>> >>>> http://www.antlr2.org/article/1170602723163/treewalkers.html >>>> >>>> Not sure if y'all have seen this. Its an old article advocating >>>> manual tree walking (what we are facing here) over using generated tree >>>> walkers. >>>> >>>> >>>> >>>> On Wed, Aug 19, 2015 at 12:27 PM Steve Ebersole <steve(a)hibernate.org&gt; >>>> wrote: >>>> >>>>> I agree. Its my biggest hang up with regard to using Antlr 4. >>>>> Actually, its my only hang up with Antlr 4, but its a huge one. >>>>> >>>>> On Tue, Aug 18, 2015 at 9:30 AM andrea boriero <dreborier(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> yes Steve I'm more familiar with Antlr4 ( but not 3) and I gave a >>>>>> look at your poc. >>>>>> >>>>>> Apart some problems to fully understand the semantic model (due to >>>>>> my lack of a complete knowledge of the domain problem), >>>>>> I agree with you about the simplicity and elegance of the grammar >>>>>> for HQL recognition and semantic model building. >>>>>> >>>>>> What I don't like it's the necessity to build our own semantic model >>>>>> walker/s in order to produce the final SQL. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 14 August 2015 at 16:32, Steve Ebersole <steve(a)hibernate.org&gt; >>>>>> wrote: >>>>>> >>>>>>> We've had a few discussions about this in the past. As 5.0 is >>>>>>> getting >>>>>>> close to Final (next week), its time to start contemplating our >>>>>>> next major >>>>>>> tasks. The consensus pick for that has been the idea of a "unified >>>>>>> SQL >>>>>>> generation engine" along with a shared project for the semantic >>>>>>> analysis of >>>>>>> HQL/JPQL (and recently it was decided to include JPA Criteria >>>>>>> interpretation here as well). >>>>>>> >>>>>>> The central premise is this. Take the roughly 6 or 7 different >>>>>>> top-level >>>>>>> ways Hibernate generates SQL and combine that into one "engine" >>>>>>> based on >>>>>>> the input of a "semantic tree". The mentioned HQL/JPQL/Criteria >>>>>>> shared >>>>>>> project will be one producer of such semantic trees. Others would >>>>>>> include >>>>>>> persisters (for insert/update/delete requests) and loaders (for load >>>>>>> requests). >>>>>>> >>>>>>> We have a lot of tasks for this overall goal still remaining. >>>>>>> >>>>>>> We still have to finalize the design for the HQL/JPQL/Criteria to >>>>>>> semantic >>>>>>> tree translator. One option is to proceed with the Antlr 4 based >>>>>>> approach >>>>>>> I started a PoC for. John has been helping me some lately with >>>>>>> that. The >>>>>>> first task here is to come to a consensus whether Antlr 4 is the >>>>>>> way we >>>>>>> want to proceed here. We've been over the pros and cons before in >>>>>>> detail. >>>>>>> In summary, there is a lot to love with Antlr 4. Our grammar for >>>>>>> HQL >>>>>>> recognition and semantic tree building is very simple and elegant >>>>>>> imo. The >>>>>>> drawback is clearly the lack of tree walking, meaning that we are >>>>>>> responsible for writing by hand our walker for the semantic tree. >>>>>>> In fact >>>>>>> multiple, since each consumer (orm, ogm, search) would need to >>>>>>> write their >>>>>>> own. And if we decide to build another AST while walking the >>>>>>> semantic >>>>>>> tree, we'd end up having to hand-write yet another walker for those. >>>>>>> >>>>>>> What I mean by that last part is that there are 2 ways we might >>>>>>> choose to >>>>>>> deal with the semantic tree. For the purpose of discussion, let's >>>>>>> look at >>>>>>> the ORM case. The first approach is to simply generate the SQL as >>>>>>> we walk >>>>>>> the semantic tree; this would be a 2 phase interpretation approach >>>>>>> (input >>>>>>> -> semantic tree -> SQL). That works in many cases. However it >>>>>>> breaks >>>>>>> down in other cases. This is exactly the approach our existing HQL >>>>>>> translator uses. The other approach is to use a 3-phase >>>>>>> translation (input >>>>>>> -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a >>>>>>> hint to one >>>>>>> of the major problems. One source "semantic" query will often >>>>>>> correspond >>>>>>> to multiple SQL queries; that is hard to manage in the 2-phase >>>>>>> approach. >>>>>>> And not to mention integrating things like follow-on fetches and >>>>>>> other >>>>>>> enhancements we want to gain from this. My vote is definitely for >>>>>>> 3 or >>>>>>> more phases of interpretation. The problem is that this is exactly >>>>>>> where >>>>>>> Antlr 4 sort of falls down. >>>>>>> >>>>>>> So first things first... we need to decide on Antlr 3 versus Antlr 4 >>>>>>> (versus some other parser solution). >>>>>>> >>>>>>> Next, on the ORM side (every "backend" can decide this >>>>>>> individually) we >>>>>>> need to decide on the approach for semantic-tree to SQL >>>>>>> translation, which >>>>>>> somewhat depends on the Antlr 3 versus Antlr 4 decision. >>>>>>> >>>>>>> We really need to decide these things ASAP and get moving on them >>>>>>> as soon >>>>>>> as ORM 5.0 is finished. >>>>>>> >>>>>>> Also, this is a massive undertaking with huge gain potentials for >>>>>>> not just >>>>>>> ORM. As such we need to understand who will be working on this. >>>>>>> Sanne, >>>>>>> Gunnar... I know y'all have a vested interest and a desire to work >>>>>>> on it. >>>>>>> John, I know the same is true for you. Andrea? Have you had a >>>>>>> chance to >>>>>>> look over the poc and/or get more familiar with Antlr? >>>>>>> >>>>>> _______________________________________________ >>>>>>> hibernate-dev mailing list >>>>>>> hibernate-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>>>>>> >>>>>> >>>>>> >>>

In practice, when are unmapped inheritance queries typically used? I tend to see them only for bulk deletions, IIRC. But in general, I'd assume they're a product of "doing something incorrectly", especially if the query includes more than one. On 08/24/2015 08:40 AM, andrea boriero wrote: > I have nothing against your proposal so +1 > > On 24 August 2015 at 04:55, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > >> Another point I want to discuss up from because it affects tree >> structure. Specifically the idea of an "unbounded implicit inheritance" >> query. These are queries like "from java.lang.Object". Queries where the >> from clause pulls in "unmapped inheritance". These are fine, to an >> extent. Hibernate has natively supported these since way back[1]. >> >> What is problematic is cases where we have more than one "unmapped >> inheritance" reference. E.g. "from java.lang.Object o1, java.lang.Object >> o2". In fact its the same difficulty as an unbounded cartesian product, >> but here in terms of the number of SQL queries we need to produce/execute. >> >> So I propose that we allow just one "unmapped inheritance" reference per >> query. >> >> [1] Reminder to self... another "strict JPQL compliance" consideration. >> >> On Sat, Aug 22, 2015 at 1:16 PM Steve Ebersole <steve(a)hibernate.org&gt; >> wrote: >> >>> I got that initial refactoring pushed to my fork... >>> >>> On Fri, Aug 21, 2015 at 3:51 PM Steve Ebersole <steve(a)hibernate.org&gt; >>> wrote: >>> >>>> Just a heads up that I started a major refactoring of the antlr4 poc >>>> project in preparation for starting to look at this next sql-gen step. >>>> >>>> First I am making it into a multi-module project. We will have the >>>> hql-parser module, but then also an orm-sql-gen module to be able to play >>>> with that part. This makes sure we are not blending orm concerns into the >>>> pure hql parser. >>>> >>>> Also, I started working on splitting the "semantic query" model out into >>>> a separate module as well. There are a few reasons for this. I wont go >>>> into them all here. The main one being that HQL is just one producer of >>>> this semantic model. Rather than another long name I went with the >>>> acronym SQM (Semantic Query Model) here. The top package being >>>> org.hibernate.sqm. >>>> >>>> These changes already illustrated some tighter couplings then I had >>>> intended, so it was a good exercise. I'll push once I get those couplings >>>> cleaned up. >>>> >>>> On Fri, Aug 21, 2015 at 2:35 PM andrea boriero <dreborier(a)gmail.com&gt; >>>> wrote: >>>> >>>>> I haven't seen it, I'm going to read it. >>>>> >>>>> On 21 August 2015 at 16:54, Steve Ebersole <steve(a)hibernate.org&gt; wrote: >>>>> >>>>>> http://www.antlr2.org/article/1170602723163/treewalkers.html >>>>>> >>>>>> Not sure if y'all have seen this. Its an old article advocating >>>>>> manual tree walking (what we are facing here) over using generated tree >>>>>> walkers. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 19, 2015 at 12:27 PM Steve Ebersole < steve(a)hibernate.org&gt; >>>>>> wrote: >>>>>> >>>>>>> I agree. Its my biggest hang up with regard to using Antlr 4. >>>>>>> Actually, its my only hang up with Antlr 4, but its a huge one. >>>>>>> >>>>>>> On Tue, Aug 18, 2015 at 9:30 AM andrea boriero < dreborier(a)gmail.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> yes Steve I'm more familiar with Antlr4 ( but not 3) and I gave a >>>>>>>> look at your poc. >>>>>>>> >>>>>>>> Apart some problems to fully understand the semantic model (due to >>>>>>>> my lack of a complete knowledge of the domain problem), >>>>>>>> I agree with you about the simplicity and elegance of the grammar >>>>>>>> for HQL recognition and semantic model building. >>>>>>>> >>>>>>>> What I don't like it's the necessity to build our own semantic model >>>>>>>> walker/s in order to produce the final SQL. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 14 August 2015 at 16:32, Steve Ebersole <steve(a)hibernate.org&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> We've had a few discussions about this in the past. As 5.0 is >>>>>>>>> getting >>>>>>>>> close to Final (next week), its time to start contemplating our >>>>>>>>> next major >>>>>>>>> tasks. The consensus pick for that has been the idea of a "unified >>>>>>>>> SQL >>>>>>>>> generation engine" along with a shared project for the semantic >>>>>>>>> analysis of >>>>>>>>> HQL/JPQL (and recently it was decided to include JPA Criteria >>>>>>>>> interpretation here as well). >>>>>>>>> >>>>>>>>> The central premise is this. Take the roughly 6 or 7 different >>>>>>>>> top-level >>>>>>>>> ways Hibernate generates SQL and combine that into one "engine" >>>>>>>>> based on >>>>>>>>> the input of a "semantic tree". The mentioned HQL/JPQL/Criteria >>>>>>>>> shared >>>>>>>>> project will be one producer of such semantic trees. Others would >>>>>>>>> include >>>>>>>>> persisters (for insert/update/delete requests) and loaders (for load >>>>>>>>> requests). >>>>>>>>> >>>>>>>>> We have a lot of tasks for this overall goal still remaining. >>>>>>>>> >>>>>>>>> We still have to finalize the design for the HQL/JPQL/Criteria to >>>>>>>>> semantic >>>>>>>>> tree translator. One option is to proceed with the Antlr 4 based >>>>>>>>> approach >>>>>>>>> I started a PoC for. John has been helping me some lately with >>>>>>>>> that. The >>>>>>>>> first task here is to come to a consensus whether Antlr 4 is the >>>>>>>>> way we >>>>>>>>> want to proceed here. We've been over the pros and cons before in >>>>>>>>> detail. >>>>>>>>> In summary, there is a lot to love with Antlr 4. Our grammar for >>>>>>>>> HQL >>>>>>>>> recognition and semantic tree building is very simple and elegant >>>>>>>>> imo. The >>>>>>>>> drawback is clearly the lack of tree walking, meaning that we are >>>>>>>>> responsible for writing by hand our walker for the semantic tree. >>>>>>>>> In fact >>>>>>>>> multiple, since each consumer (orm, ogm, search) would need to >>>>>>>>> write their >>>>>>>>> own. And if we decide to build another AST while walking the >>>>>>>>> semantic >>>>>>>>> tree, we'd end up having to hand-write yet another walker for those. >>>>>>>>> >>>>>>>>> What I mean by that last part is that there are 2 ways we might >>>>>>>>> choose to >>>>>>>>> deal with the semantic tree. For the purpose of discussion, let's >>>>>>>>> look at >>>>>>>>> the ORM case. The first approach is to simply generate the SQL as >>>>>>>>> we walk >>>>>>>>> the semantic tree; this would be a 2 phase interpretation approach >>>>>>>>> (input >>>>>>>>> -> semantic tree -> SQL). That works in many cases. However it >>>>>>>>> breaks >>>>>>>>> down in other cases. This is exactly the approach our existing HQL >>>>>>>>> translator uses. The other approach is to use a 3-phase >>>>>>>>> translation (input >>>>>>>>> -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a >>>>>>>>> hint to one >>>>>>>>> of the major problems. One source "semantic" query will often >>>>>>>>> correspond >>>>>>>>> to multiple SQL queries; that is hard to manage in the 2-phase >>>>>>>>> approach. >>>>>>>>> And not to mention integrating things like follow-on fetches and >>>>>>>>> other >>>>>>>>> enhancements we want to gain from this. My vote is definitely for >>>>>>>>> 3 or >>>>>>>>> more phases of interpretation. The problem is that this is exactly >>>>>>>>> where >>>>>>>>> Antlr 4 sort of falls down. >>>>>>>>> >>>>>>>>> So first things first... we need to decide on Antlr 3 versus Antlr 4 >>>>>>>>> (versus some other parser solution). >>>>>>>>> >>>>>>>>> Next, on the ORM side (every "backend" can decide this >>>>>>>>> individually) we >>>>>>>>> need to decide on the approach for semantic-tree to SQL >>>>>>>>> translation, which >>>>>>>>> somewhat depends on the Antlr 3 versus Antlr 4 decision. >>>>>>>>> >>>>>>>>> We really need to decide these things ASAP and get moving on them >>>>>>>>> as soon >>>>>>>>> as ORM 5.0 is finished. >>>>>>>>> >>>>>>>>> Also, this is a massive undertaking with huge gain potentials for >>>>>>>>> not just >>>>>>>>> ORM. As such we need to understand who will be working on this. >>>>>>>>> Sanne, >>>>>>>>> Gunnar... I know y'all have a vested interest and a desire to work >>>>>>>>> on it. >>>>>>>>> John, I know the same is true for you. Andrea? Have you had a >>>>>>>>> chance to >>>>>>>>> look over the poc and/or get more familiar with Antlr? >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>>> hibernate-dev mailing list >>>>>>>>> hibernate-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>>>>>>>> >>>>>>>> > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

The split makes a lot of sense, should we have another module for the criteria query SQM translation?

hibernate-hql-parser-antlr4poc module?

The other approach is to use a 3-phase translation (input -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint to one of the major problems. One source "semantic" query will often correspond to multiple SQL queries; that is hard to manage in the 2-phase approach.

Hi Steve, > The other approach is to use a 3-phase translation (input > -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint to one > of the major problems. One source "semantic" query will often correspond > to multiple SQL queries; that is hard to manage in the 2-phase approach. In which situations will this happen? I can see inheritance where a HQL query targeting a super-type needs to be translated into a SQL query per sub-type table. What others are there?

as we phase the same task with non-SQL backends in SQL. I.e. i'd be beneficial to have input -> semantic-tree -> semantic-output-query-tree(s) -> (SQL|non-SQL query). There "semantic-output-query-tree(s)" would be an abstract representation of the queries to be executed, e.g. referencing the table name(s). But it would be unaware of SQL specifics.

2015-08-26 14:41 GMT+02:00 Steve Ebersole <steve(a)hibernate.org&gt;: > On Wed, Aug 26, 2015 at 2:10 AM Gunnar Morling <gunnar(a)hibernate.org&gt; wrote: >> >> Hi Steve, >> >> > The other approach is to use a 3-phase translation (input >> > -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint to >> > one >> > of the major problems. One source "semantic" query will often >> > correspond >> > to multiple SQL queries; that is hard to manage in the 2-phase approach. >> >> In which situations will this happen? I can see inheritance where a >> HQL query targeting a super-type needs to be translated into a SQL >> query per sub-type table. What others are there? > > > For ORM the only time this happens today for a SELECT query is in the "split > query" case I mentioned elsewhere (a query like 'from java.lang.Object'). > SQM does this much better than we do in ORM today. in SQM we build a > semantic tree that encodes the "unmapped polymorphism" such that we get a > tree with 'java.lang.Object' as the root from element. But it is a > FromElement with a special type of EntityTypeDescriptor (which comes from > the caller remember): PolymorphicEntityTypeDescriptor. On the ORM side then > I have a QuerySplitter that takes that query and makes a copy of that entire > SQM tree, one for each mapped implementor of the specified class. FWIW, ORM > does this today, albeit in a different way. Today we split the query based > on String manip and then feed it parser. Here we feed it to the parser and > use the tree to split it; much less brittle :) > > Really the cases where this would happen (one "concrete SQM" -> multiple > SQL) would be UPDATE and DELETE queries against "multi-table structures" > (inheritance, secondary tables). > > >> For the purposes of OGM this phase ideally would not be tied to SQL, >> as we phase the same task with non-SQL backends in SQL. I.e. i'd be >> beneficial to have input -> semantic-tree -> >> semantic-output-query-tree(s) -> (SQL|non-SQL query). There >> "semantic-output-query-tree(s)" would be an abstract representation of >> the queries to be executed, e.g. referencing the table name(s). But it >> would be unaware of SQL specifics. > > > OGM would be doing this. This SQM is the end result of the shared library. > WHat each caller does with the SQM is up to that particular caller. We > should consider moving QuerySplitter (its in my PoC, which now acts as the > PoC for using this in ORM) into the hibernate-sqm module. Any caller > wanting to support those unmapped class references will need to do the same > thing. Yes, that'd be good I think. We'd have to apply the same rules for splitting as ORM. > > BTW, another cool thing to note is the (still expanding) support for "strict > JPQL compliance" enforcement.

Nevermind. I will not do that. I think I have found a still-easyish way to do it.

On Thu, Aug 27, 2015 at 10:57 AM Steve Ebersole <steve(a)hibernate.org&gt; wrote: > I do want to pull ORM in to the hibernate-sqm module as a test dependency > to be able to more easily set up the ModelMetadata stuff based on a > SessionFactory. That is possibly awkward later when we then use > hibernate-sqm in ORM in terms of having 2 different versions of ORM. I am > open to alternatives that don't involve *me* developing a real(ish) ModelMetadata > impl from scratch. > > On Thu, Aug 27, 2015 at 8:45 AM Gunnar Morling <gunnar(a)hibernate.org&gt; > wrote: > >> 2015-08-26 14:41 GMT+02:00 Steve Ebersole <steve(a)hibernate.org&gt;: >> > On Wed, Aug 26, 2015 at 2:10 AM Gunnar Morling <gunnar(a)hibernate.org&gt; >> wrote: >> >> >> >> Hi Steve, >> >> >> >> > The other approach is to use a 3-phase translation (input >> >> > -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a hint >> to >> >> > one >> >> > of the major problems. One source "semantic" query will often >> >> > correspond >> >> > to multiple SQL queries; that is hard to manage in the 2-phase >> approach. >> >> >> >> In which situations will this happen? I can see inheritance where a >> >> HQL query targeting a super-type needs to be translated into a SQL >> >> query per sub-type table. What others are there? >> > >> > >> > For ORM the only time this happens today for a SELECT query is in the >> "split >> > query" case I mentioned elsewhere (a query like 'from >> java.lang.Object'). >> > SQM does this much better than we do in ORM today. in SQM we build a >> > semantic tree that encodes the "unmapped polymorphism" such that we get >> a >> > tree with 'java.lang.Object' as the root from element. But it is a >> > FromElement with a special type of EntityTypeDescriptor (which comes >> from >> > the caller remember): PolymorphicEntityTypeDescriptor. On the ORM side >> then >> > I have a QuerySplitter that takes that query and makes a copy of that >> entire >> > SQM tree, one for each mapped implementor of the specified class. >> FWIW, ORM >> > does this today, albeit in a different way. Today we split the query >> based >> > on String manip and then feed it parser. Here we feed it to the parser >> and >> > use the tree to split it; much less brittle :) >> > >> > Really the cases where this would happen (one "concrete SQM" -> multiple >> > SQL) would be UPDATE and DELETE queries against "multi-table structures" >> > (inheritance, secondary tables). >> > >> > >> >> For the purposes of OGM this phase ideally would not be tied to SQL, >> >> as we phase the same task with non-SQL backends in SQL. I.e. i'd be >> >> beneficial to have input -> semantic-tree -> >> >> semantic-output-query-tree(s) -> (SQL|non-SQL query). There >> >> "semantic-output-query-tree(s)" would be an abstract representation of >> >> the queries to be executed, e.g. referencing the table name(s). But it >> >> would be unaware of SQL specifics. >> > >> > >> > OGM would be doing this. This SQM is the end result of the shared >> library. >> > WHat each caller does with the SQM is up to that particular caller. We >> > should consider moving QuerySplitter (its in my PoC, which now acts as >> the >> > PoC for using this in ORM) into the hibernate-sqm module. Any caller >> > wanting to support those unmapped class references will need to do the >> same >> > thing. >> >> Yes, that'd be good I think. We'd have to apply the same rules for >> splitting as ORM. >> >> > >> > BTW, another cool thing to note is the (still expanding) support for >> "strict >> > JPQL compliance" enforcement. >> > _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

Getting back to this work this weekend and next week. Initially I am working on the basics for SQL generation from a SQM model. There was an open question before with regards to whether it made sense to render the SQM directly into SQL (String) or whether it instead made sense to render the SQM into a "SQL model" which we'd walk afterwards to generate the SQL. So let's investigate this second approach... CONS The major drawback is that we'd end up walking the tree (a different tree, the "SQL model") again which would have both a performance and some complexity concerns. The performance concern may or may not be a big deal. We'd end up having to walk parts of the SQM tree multiple times anyway. Part of the complexity comes from having to write yet another walker for walking this "SQL model" tree. It's not a major issue as we have already seen it work in the SQM walkers. A complication here is that it does limit how we apply Dialect intervention and the types of intervention a Dialect can achieve. Rendering to a String is pretty free-form and the Dialect can do all kinds of things. When we render to a tree, the Dialect has to adhere to available constructs we have encoded into the tree structure. A silly example is query that the Oracle dialect wants to render using a CONNECT-BY query. Rendering that to a String directly is not a big deal. But if we render that to a tree, the tree needs to be capable of understanding that construct and substructure; that or we allow for "free-form nodes" in the tree, but that gets tricky for walking. So if we stick to "general SQL structure", rendering to a SQL tree should be very doable. PROS The main advantage is to better leverage contextual information. Duh, that's always the reason a structure is better than a BLOB/String :) Here, specifically it would better allow us to render smarter SQL. The one that I was initially working though which prompted this email is the case of "repeating expressions". Consider a SQL query like "select a+b from XYZ where a+b > 2". The more efficient way to write that query is "select a+b as c from XYZ where c > 2". We could technically even do that kind of normalization while rendering the SQM directly to String. But it is easier and less error prone if we rendered to a SQL tree first. Other things include leaving off un-needed joins, etc. On Fri, Aug 28, 2015 at 6:07 AM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > On 27 August 2015 at 18:30, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > > Nevermind. I will not do that. I think I have found a still-easyish > way > > to do it. > > Great! Highly appreciate that. > > > > > > On Thu, Aug 27, 2015 at 10:57 AM Steve Ebersole <steve(a)hibernate.org&gt; > wrote: > > > >> I do want to pull ORM in to the hibernate-sqm module as a test > dependency > >> to be able to more easily set up the ModelMetadata stuff based on a > >> SessionFactory. That is possibly awkward later when we then use > >> hibernate-sqm in ORM in terms of having 2 different versions of ORM. > I am > >> open to alternatives that don't involve *me* developing a real(ish) > ModelMetadata > >> impl from scratch. > >> > >> On Thu, Aug 27, 2015 at 8:45 AM Gunnar Morling <gunnar(a)hibernate.org&gt; > >> wrote: > >> > >>> 2015-08-26 14:41 GMT+02:00 Steve Ebersole <steve(a)hibernate.org&gt;: > >>> > On Wed, Aug 26, 2015 at 2:10 AM Gunnar Morling < > gunnar(a)hibernate.org&gt; > >>> wrote: > >>> >> > >>> >> Hi Steve, > >>> >> > >>> >> > The other approach is to use a 3-phase translation (input > >>> >> > -> semantic-tree -> semantic-SQL-tree(s) -> SQL). This gives a > hint > >>> to > >>> >> > one > >>> >> > of the major problems. One source "semantic" query will often > >>> >> > correspond > >>> >> > to multiple SQL queries; that is hard to manage in the 2-phase > >>> approach. > >>> >> > >>> >> In which situations will this happen? I can see inheritance where a > >>> >> HQL query targeting a super-type needs to be translated into a SQL > >>> >> query per sub-type table. What others are there? > >>> > > >>> > > >>> > For ORM the only time this happens today for a SELECT query is in > the > >>> "split > >>> > query" case I mentioned elsewhere (a query like 'from > >>> java.lang.Object'). > >>> > SQM does this much better than we do in ORM today. in SQM we build > a > >>> > semantic tree that encodes the "unmapped polymorphism" such that we > get > >>> a > >>> > tree with 'java.lang.Object' as the root from element. But it is a > >>> > FromElement with a special type of EntityTypeDescriptor (which comes > >>> from > >>> > the caller remember): PolymorphicEntityTypeDescriptor. On the ORM > side > >>> then > >>> > I have a QuerySplitter that takes that query and makes a copy of > that > >>> entire > >>> > SQM tree, one for each mapped implementor of the specified class. > >>> FWIW, ORM > >>> > does this today, albeit in a different way. Today we split the > query > >>> based > >>> > on String manip and then feed it parser. Here we feed it to the > parser > >>> and > >>> > use the tree to split it; much less brittle :) > >>> > > >>> > Really the cases where this would happen (one "concrete SQM" -> > multiple > >>> > SQL) would be UPDATE and DELETE queries against "multi-table > structures" > >>> > (inheritance, secondary tables). > >>> > > >>> > > >>> >> For the purposes of OGM this phase ideally would not be tied to > SQL, > >>> >> as we phase the same task with non-SQL backends in SQL. I.e. i'd be > >>> >> beneficial to have input -> semantic-tree -> > >>> >> semantic-output-query-tree(s) -> (SQL|non-SQL query). There > >>> >> "semantic-output-query-tree(s)" would be an abstract > representation of > >>> >> the queries to be executed, e.g. referencing the table name(s). > But it > >>> >> would be unaware of SQL specifics. > >>> > > >>> > > >>> > OGM would be doing this. This SQM is the end result of the shared > >>> library. > >>> > WHat each caller does with the SQM is up to that particular > caller. We > >>> > should consider moving QuerySplitter (its in my PoC, which now acts > as > >>> the > >>> > PoC for using this in ORM) into the hibernate-sqm module. Any > caller > >>> > wanting to support those unmapped class references will need to do > the > >>> same > >>> > thing. > >>> > >>> Yes, that'd be good I think. We'd have to apply the same rules for > >>> splitting as ORM. > >>> > >>> > > >>> > BTW, another cool thing to note is the (still expanding) support for > >>> "strict > >>> > JPQL compliance" enforcement. > >>> > >> > > _______________________________________________ > > hibernate-dev mailing list > > hibernate-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev >