12:42 p.m.
As most of you know already, we are planning to redesign the current
Antlr-based HQL/JPQL parser in ORM for a variety of reasons.
The current approach in the translator (Antlr 2 based, although Antlr 3
supports the same model) is that we actually define multiple
grammars/parsers which progressively re-write the tree adding more and more
semantic information; think of this as multiple passes or phases. The
current code has 3 phases:
1) parsing - we simply parse the HQL/JPQL query into an AST, although we do
do one interesting (and uber-important!) re-write here where we "hoist" the
from clause in front of all other clauses.
2) rough semantic analysis - the current code, to be honest, sucks here.
The end result of this phase is a tree that mixes normalized semantic
information with lots of SQL fragments. It is extremely fugly
3) rendering to SQL
The idea of phases is still the best way to attack this translation imo. I
just think we did not implement the phases very well before; we were just
learning Antlr at the time. So part of the redesign here is to leverage
our better understanding of Antlr and design some better trees. The other
big reason is to centralize the generation of SQL into one place rather
than the 3 different places we do it today (not to mention the many, many
places we render SQL fragments).
Part of the process here is to decide which parser to use. Antlr 2 is
ancient :) I used Antlr 3 in the initial prototyping of this redesign
because it was the most recent release at that time. In the interim Antlr
4 has been released.
I have been evaluating whether Antlr 4 is appropriate for our needs there.
Antlr 4 is a pretty big conceptual deviation from Antlr 2/3 in quite a few
ways. Generally speaking, Antlr 4 is geared more towards interpreting
rather than translating/transforming. It can handle "transformation" if
the transformation is the final step in the process. Transformations is
where tree re-writing comes in handy.
First lets step back and look at the "conceptual model" of Antlr 4. The
grammar is used to produce:
1) the parser - takes the input and builds a "parse tree" based on the
rules of the lexer and grammar.
2) listener/visitor for parse-tree traversal - can optionally generate
listeners or visitors (or both) for traversing the parse tree (output from
parser).
There are 2 highly-related changes that negatively impact us:
1) no tree grammars/parsers
2) no tree re-writing
Our existing translator is fundamentally built on the concepts of tree
parsers and tree re-writing. Even the initial prototypes for the redesign
(and the current state of hql-parser which Sanne and Gunnar picked up from
there) are built on those concepts. So moving to Antlr 4 in that regard
does represent a risk. How big of a risk, and whether that risk is worth
it, is what we need to determine.
What does all this mean in simple, practical terms? Let's look at a simple
query: "select c.headquarters.state.code from Company c". Simple syntactic
analysis will produce a tree something like:
[QUERY]
[SELECT]
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
[FROM]
[SPACE]
[SPACE_ROOT]
[IDENT, "Customer"]
[IDENT, "c"]
There is not a lot of semantic (meaning) information here. A more semantic
representation of the query would look something like:
[QUERY]
[SELECT]
[ATTRIBUTE_REF]
[ALIAS_REF, "<gen:1>"]
[IDENT, "code"]
[FROM]
[SPACE]
[PERSISTER_REF]
[ENTITY_NAME, "com.acme.Customer"]
[ALIAS, "c"]
[JOIN]
[INNER]
[ATTRIBUTE_JOIN]
[IDENT, "headquarters"]
[ALIAS, "<gen:0>"]
[JOIN]
[INNER]
[ATTRIBUTE_JOIN]
[IDENT, "state"]
[ALIAS, "<gen:1>"]
Notice especially the difference in the tree rules. This is tree
re-writing, and is the major difference affecting us. Consider a specific
thing like the "c.headquarters.state.code" DOT-IDENT sequence. Essentially
Antlr 4 would make us deal with that as a DOT-IDENT sequence through all
the phases - even SQL generation. Quite fugly. The intent of Antlr 4 in
cases like this is to build up an external state table (external to the
tree itself) or what Antlr folks typically refer to as "iterative tree
decoration"[1]. So with Antlr 4, in generating the SQL, we would still be
handling calls in terms of "c.headquarters.state.code" in the SELECT clause
and resolving that through the external symbol tables. Again, with Antlr 4
we would always be walking that initial (non-semantic) tree. Unless I am
missing something. I would be happy to be corrected, if anyone knows Antlr
4 better. I have also asked as part of the antlr-discussion group[2].
In my opinion though, if it comes down to us needing to walk the tree in
that first form across all phases I just do not see the benefit to moving
to Antlr 4.
P.S. When I say SQL above I really just mean the target query language for
the back-end data store whether that be SQL targeting a RDBMS for ORM or a
NoSQL store for OGM.
[1] I still have not fully grokked this paradigm, so I may be missing
something, but... AFAICT even in this paradigm the listener/visitor rules
are defined in terms of the initial parse tree rules rather than more
[2] https://groups.google.com/forum/#!topic/antlr-discussion/hzF_YrzfDKo
3:56 a.m.
Hi Steve,
Did you ever have a chance to apply the "decorated parse tree" approach to
your Antlr4 PoC?
What I like about the Antlr4 approach is the fact that you don't need a set
of several quite similar grammars as you'd do with the tree transformation
approach. Also using the current version of Antlr instead of 3 appears
attractive to me wrt. to bugfixes and future development of the tool.
Based on what I understand from your discussions on the Antlr mailing list,
I'd assume the parse tree and the external state it references to look
roughly like so (---> indicates a reference to state built up during
sub-sequential walks, maybe in some external "table", maybe stored within
the (typed) tree nodes themselves):
[QUERY]
[SELECT]
[ATTRIBUTE_REF] ---> AttributeReference("<gen:1>",
"code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
[FROM]
[SPACE]
[SPACE_ROOT] ---> InnerJoin( InnerJoin ( PersisterRef( "c",
"com.acme.Customer" ), TableRef ( "<gen:0>",
"headquarters" ) ), TableRef (
"<gen:1>", "state" ) ) )
[IDENT, "Customer"]
[IDENT, "c"]
I.e. instead of transforming the tree itself, the state required for output
generation would be added as "decorators" to nodes of the original parse
tree itself. That's just the basic idea as I understand it, surely the
specific types of the decorator elements (AttributeReference, InnerJoin etc.)
may look different. During "query rendering" we'd have to inspect the
decorator state of the parse tree nodes and interpret it accordingly.
So I believe the issue of alias resolution and implicit join conversion
could be handled without tree transformations (at least conceptually, I
could not code an actual implementation out of my head right away). But
maybe there are other cases where tree transformations are more strictly
needed?
--Gunnar
2014-11-13 19:42 GMT+01:00 Steve Ebersole <steve(a)hibernate.org>:
As most of you know already, we are planning to redesign the current
Antlr-based HQL/JPQL parser in ORM for a variety of reasons.
The current approach in the translator (Antlr 2 based, although Antlr 3
supports the same model) is that we actually define multiple
grammars/parsers which progressively re-write the tree adding more and more
semantic information; think of this as multiple passes or phases. The
current code has 3 phases:
1) parsing - we simply parse the HQL/JPQL query into an AST, although we do
do one interesting (and uber-important!) re-write here where we "hoist" the
from clause in front of all other clauses.
2) rough semantic analysis - the current code, to be honest, sucks here.
The end result of this phase is a tree that mixes normalized semantic
information with lots of SQL fragments. It is extremely fugly
3) rendering to SQL
The idea of phases is still the best way to attack this translation imo. I
just think we did not implement the phases very well before; we were just
learning Antlr at the time. So part of the redesign here is to leverage
our better understanding of Antlr and design some better trees. The other
big reason is to centralize the generation of SQL into one place rather
than the 3 different places we do it today (not to mention the many, many
places we render SQL fragments).
Part of the process here is to decide which parser to use. Antlr 2 is
ancient :) I used Antlr 3 in the initial prototyping of this redesign
because it was the most recent release at that time. In the interim Antlr
4 has been released.
I have been evaluating whether Antlr 4 is appropriate for our needs there.
Antlr 4 is a pretty big conceptual deviation from Antlr 2/3 in quite a few
ways. Generally speaking, Antlr 4 is geared more towards interpreting
rather than translating/transforming. It can handle "transformation" if
the transformation is the final step in the process. Transformations is
where tree re-writing comes in handy.
First lets step back and look at the "conceptual model" of Antlr 4. The
grammar is used to produce:
1) the parser - takes the input and builds a "parse tree" based on the
rules of the lexer and grammar.
2) listener/visitor for parse-tree traversal - can optionally generate
listeners or visitors (or both) for traversing the parse tree (output from
parser).
There are 2 highly-related changes that negatively impact us:
1) no tree grammars/parsers
2) no tree re-writing
Our existing translator is fundamentally built on the concepts of tree
parsers and tree re-writing. Even the initial prototypes for the redesign
(and the current state of hql-parser which Sanne and Gunnar picked up from
there) are built on those concepts. So moving to Antlr 4 in that regard
does represent a risk. How big of a risk, and whether that risk is worth
it, is what we need to determine.
What does all this mean in simple, practical terms? Let's look at a simple
query: "select c.headquarters.state.code from Company c". Simple syntactic
analysis will produce a tree something like:
[QUERY]
[SELECT]
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
[FROM]
[SPACE]
[SPACE_ROOT]
[IDENT, "Customer"]
[IDENT, "c"]
There is not a lot of semantic (meaning) information here. A more semantic
representation of the query would look something like:
[QUERY]
[SELECT]
[ATTRIBUTE_REF]
[ALIAS_REF, "<gen:1>"]
[IDENT, "code"]
[FROM]
[SPACE]
[PERSISTER_REF]
[ENTITY_NAME, "com.acme.Customer"]
[ALIAS, "c"]
[JOIN]
[INNER]
[ATTRIBUTE_JOIN]
[IDENT, "headquarters"]
[ALIAS, "<gen:0>"]
[JOIN]
[INNER]
[ATTRIBUTE_JOIN]
[IDENT, "state"]
[ALIAS, "<gen:1>"]
Notice especially the difference in the tree rules. This is tree
re-writing, and is the major difference affecting us. Consider a specific
thing like the "c.headquarters.state.code" DOT-IDENT sequence. Essentially
Antlr 4 would make us deal with that as a DOT-IDENT sequence through all
the phases - even SQL generation. Quite fugly. The intent of Antlr 4 in
cases like this is to build up an external state table (external to the
tree itself) or what Antlr folks typically refer to as "iterative tree
decoration"[1]. So with Antlr 4, in generating the SQL, we would still be
handling calls in terms of "c.headquarters.state.code" in the SELECT clause
and resolving that through the external symbol tables. Again, with Antlr 4
we would always be walking that initial (non-semantic) tree. Unless I am
missing something. I would be happy to be corrected, if anyone knows Antlr
4 better. I have also asked as part of the antlr-discussion group[2].
In my opinion though, if it comes down to us needing to walk the tree in
that first form across all phases I just do not see the benefit to moving
to Antlr 4.
P.S. When I say SQL above I really just mean the target query language for
the back-end data store whether that be SQL targeting a RDBMS for ORM or a
NoSQL store for OGM.
[1] I still have not fully grokked this paradigm, so I may be missing
something, but... AFAICT even in this paradigm the listener/visitor rules
are defined in terms of the initial parse tree rules rather than more
[2] https://groups.google.com/forum/#!topic/antlr-discussion/hzF_YrzfDKo
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
5:49 a.m.
On 9 June 2015 at 09:56, Gunnar Morling <gunnar(a)hibernate.org> wrote:
Hi Steve,
Did you ever have a chance to apply the "decorated parse tree" approach to
your Antlr4 PoC?
What I like about the Antlr4 approach is the fact that you don't need a set
of several quite similar grammars as you'd do with the tree transformation
approach. Also using the current version of Antlr instead of 3 appears
attractive to me wrt. to bugfixes and future development of the tool.
Based on what I understand from your discussions on the Antlr mailing list,
I'd assume the parse tree and the external state it references to look
roughly like so (---> indicates a reference to state built up during
sub-sequential walks, maybe in some external "table", maybe stored within
the (typed) tree nodes themselves):
[QUERY]
[SELECT]
[ATTRIBUTE_REF] ---> AttributeReference("<gen:1>",
"code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
[FROM]
[SPACE]
[SPACE_ROOT] ---> InnerJoin( InnerJoin ( PersisterRef( "c",
"com.acme.Customer" ), TableRef ( "<gen:0>",
"headquarters" ) ), TableRef (
"<gen:1>", "state" ) ) )
[IDENT, "Customer"]
[IDENT, "c"]
I.e. instead of transforming the tree itself, the state required for output
generation would be added as "decorators" to nodes of the original parse
tree itself. That's just the basic idea as I understand it, surely the
specific types of the decorator elements (AttributeReference, InnerJoin etc.)
may look different. During "query rendering" we'd have to inspect the
decorator state of the parse tree nodes and interpret it accordingly.
So I believe the issue of alias resolution and implicit join conversion
could be handled without tree transformations (at least conceptually, I
could not code an actual implementation out of my head right away). But
maybe there are other cases where tree transformations are more strictly
needed?
Do you mean that you would be ok to "navigate" all the [DOT] nodes to
get to the decorated attachments?
In that case while you might be fine to translate each fragment into a
different fragment, it's not straight forward to transform it into a
different structure, say with sub-trees in different orders or nodes
which don't have a 1:1 match.
It's of course doable if you are filling in your own builder while
navigating these (like we do with the Lucene DSL output), but it
doesn't help you with multiple phases which is what Steve is pointing
out.
I would highly prefer to feed the semantic representation of the tree
to our query generating backends, especially so if we could all share
the same initial smart phases to do some basic validations and
optimisations DRY. But then the consuming backends will likely have
some additional validations and optimisations which need to be
backend-specific (dialect-specific or technology specific in case of
OGM).
Steve, you mentioned that ANTLR4 handles transformations but only when
it's the last step. What prevents us to chain multiple such
transformations, applying the "last step" approach multiple times?
I didn't look at it at all, so take this just as an high level,
conceptual question. I guess one would need to clearly define all
intermediate data types rather than have ANTLR generate them like it
does with tokens, but that could be the lesser trouble?
Thanks,
Sanne
--Gunnar
2014-11-13 19:42 GMT+01:00 Steve Ebersole <steve(a)hibernate.org>:
> As most of you know already, we are planning to redesign the current
> Antlr-based HQL/JPQL parser in ORM for a variety of reasons.
>
> The current approach in the translator (Antlr 2 based, although Antlr 3
> supports the same model) is that we actually define multiple
> grammars/parsers which progressively re-write the tree adding more and more
> semantic information; think of this as multiple passes or phases. The
> current code has 3 phases:
> 1) parsing - we simply parse the HQL/JPQL query into an AST, although we do
> do one interesting (and uber-important!) re-write here where we "hoist"
the
> from clause in front of all other clauses.
> 2) rough semantic analysis - the current code, to be honest, sucks here.
> The end result of this phase is a tree that mixes normalized semantic
> information with lots of SQL fragments. It is extremely fugly
> 3) rendering to SQL
>
> The idea of phases is still the best way to attack this translation imo. I
> just think we did not implement the phases very well before; we were just
> learning Antlr at the time. So part of the redesign here is to leverage
> our better understanding of Antlr and design some better trees. The other
> big reason is to centralize the generation of SQL into one place rather
> than the 3 different places we do it today (not to mention the many, many
> places we render SQL fragments).
>
> Part of the process here is to decide which parser to use. Antlr 2 is
> ancient :) I used Antlr 3 in the initial prototyping of this redesign
> because it was the most recent release at that time. In the interim Antlr
> 4 has been released.
>
> I have been evaluating whether Antlr 4 is appropriate for our needs there.
> Antlr 4 is a pretty big conceptual deviation from Antlr 2/3 in quite a few
> ways. Generally speaking, Antlr 4 is geared more towards interpreting
> rather than translating/transforming. It can handle "transformation" if
> the transformation is the final step in the process. Transformations is
> where tree re-writing comes in handy.
>
> First lets step back and look at the "conceptual model" of Antlr 4. The
> grammar is used to produce:
> 1) the parser - takes the input and builds a "parse tree" based on the
> rules of the lexer and grammar.
> 2) listener/visitor for parse-tree traversal - can optionally generate
> listeners or visitors (or both) for traversing the parse tree (output from
> parser).
>
> There are 2 highly-related changes that negatively impact us:
> 1) no tree grammars/parsers
> 2) no tree re-writing
>
> Our existing translator is fundamentally built on the concepts of tree
> parsers and tree re-writing. Even the initial prototypes for the redesign
> (and the current state of hql-parser which Sanne and Gunnar picked up from
> there) are built on those concepts. So moving to Antlr 4 in that regard
> does represent a risk. How big of a risk, and whether that risk is worth
> it, is what we need to determine.
>
> What does all this mean in simple, practical terms? Let's look at a simple
> query: "select c.headquarters.state.code from Company c". Simple
syntactic
> analysis will produce a tree something like:
>
> [QUERY]
> [SELECT]
> [DOT]
> [DOT]
> [DOT]
> [IDENT, "c"]
> [IDENT, "headquarters"]
> [IDENT, "state"]
> [IDENT, "code"]
> [FROM]
> [SPACE]
> [SPACE_ROOT]
> [IDENT, "Customer"]
> [IDENT, "c"]
>
> There is not a lot of semantic (meaning) information here. A more semantic
> representation of the query would look something like:
>
> [QUERY]
> [SELECT]
> [ATTRIBUTE_REF]
> [ALIAS_REF, "<gen:1>"]
> [IDENT, "code"]
> [FROM]
> [SPACE]
> [PERSISTER_REF]
> [ENTITY_NAME, "com.acme.Customer"]
> [ALIAS, "c"]
> [JOIN]
> [INNER]
> [ATTRIBUTE_JOIN]
> [IDENT, "headquarters"]
> [ALIAS, "<gen:0>"]
> [JOIN]
> [INNER]
> [ATTRIBUTE_JOIN]
> [IDENT, "state"]
> [ALIAS, "<gen:1>"]
>
>
> Notice especially the difference in the tree rules. This is tree
> re-writing, and is the major difference affecting us. Consider a specific
> thing like the "c.headquarters.state.code" DOT-IDENT sequence.
Essentially
> Antlr 4 would make us deal with that as a DOT-IDENT sequence through all
> the phases - even SQL generation. Quite fugly. The intent of Antlr 4 in
> cases like this is to build up an external state table (external to the
> tree itself) or what Antlr folks typically refer to as "iterative tree
> decoration"[1]. So with Antlr 4, in generating the SQL, we would still be
> handling calls in terms of "c.headquarters.state.code" in the SELECT
clause
> and resolving that through the external symbol tables. Again, with Antlr 4
> we would always be walking that initial (non-semantic) tree. Unless I am
> missing something. I would be happy to be corrected, if anyone knows Antlr
> 4 better. I have also asked as part of the antlr-discussion group[2].
>
> In my opinion though, if it comes down to us needing to walk the tree in
> that first form across all phases I just do not see the benefit to moving
> to Antlr 4.
>
> P.S. When I say SQL above I really just mean the target query language for
> the back-end data store whether that be SQL targeting a RDBMS for ORM or a
> NoSQL store for OGM.
>
> [1] I still have not fully grokked this paradigm, so I may be missing
> something, but... AFAICT even in this paradigm the listener/visitor rules
> are defined in terms of the initial parse tree rules rather than more
> [2] https://groups.google.com/forum/#!topic/antlr-discussion/hzF_YrzfDKo
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
6:11 a.m.
2015-06-09 12:49 GMT+02:00 Sanne Grinovero <sanne(a)hibernate.org>:
On 9 June 2015 at 09:56, Gunnar Morling <gunnar(a)hibernate.org>
wrote:
> Hi Steve,
>
> Did you ever have a chance to apply the "decorated parse tree" approach
to
> your Antlr4 PoC?
>
> What I like about the Antlr4 approach is the fact that you don't need a
set
> of several quite similar grammars as you'd do with the tree
transformation
> approach. Also using the current version of Antlr instead of 3 appears
> attractive to me wrt. to bugfixes and future development of the tool.
>
> Based on what I understand from your discussions on the Antlr mailing
list,
> I'd assume the parse tree and the external state it references to look
> roughly like so (---> indicates a reference to state built up during
> sub-sequential walks, maybe in some external "table", maybe stored within
> the (typed) tree nodes themselves):
>
> [QUERY]
> [SELECT]
> [ATTRIBUTE_REF] ---> AttributeReference("<gen:1>",
"code")
> [DOT]
> [DOT]
> [DOT]
> [IDENT, "c"]
> [IDENT, "headquarters"]
> [IDENT, "state"]
> [IDENT, "code"]
> [FROM]
> [SPACE]
> [SPACE_ROOT] ---> InnerJoin( InnerJoin ( PersisterRef( "c",
> "com.acme.Customer" ), TableRef ( "<gen:0>",
"headquarters" ) ),
TableRef (
> "<gen:1>", "state" ) ) )
> [IDENT, "Customer"]
> [IDENT, "c"]
>
> I.e. instead of transforming the tree itself, the state required for
output
> generation would be added as "decorators" to nodes of the original parse
> tree itself. That's just the basic idea as I understand it, surely the
> specific types of the decorator elements (AttributeReference, InnerJoin
etc.)
> may look different. During "query rendering" we'd have to inspect the
> decorator state of the parse tree nodes and interpret it accordingly.
>
> So I believe the issue of alias resolution and implicit join conversion
> could be handled without tree transformations (at least conceptually, I
> could not code an actual implementation out of my head right away). But
> maybe there are other cases where tree transformations are more strictly
> needed?
Do you mean that you would be ok to "navigate" all the [DOT] nodes to
get to the decorated attachments?
In that case while you might be fine to translate each fragment into a
different fragment, it's not straight forward to transform it into a
different structure, say with sub-trees in different orders or nodes
which don't have a 1:1 match.
It's of course doable if you are filling in your own builder while
navigating these (like we do with the Lucene DSL output), but it
doesn't help you with multiple phases which is what Steve is pointing
out.
No, what I mean is to add attachments to nodes up in the tree, based on
information either a) from sub-nodes of that tree or b) nodes somewhere
else in the tree. E.g. a) is the case for the attribute reference, which is
represented by an attachment at the ATTRIBUTE_REF node (it has been created
by prior visits to the DOT sub-nodes) and b) is the case for the implicit
join syntax: It is declared by sub-nodes of the SELECT clause, but the
attachments representing the join are added beneath the FROM clause.
The query generation would work based on these "semantic" attachments, it
would not visit the individual DOT nodes for instance.
I would highly prefer to feed the semantic representation of the tree
to our query generating backends, especially so if we could all
share
the same initial smart phases to do some basic validations and
optimisations DRY. But then the consuming backends will likely have
some additional validations and optimisations which need to be
backend-specific (dialect-specific or technology specific in case of
OGM).
Yes, of course that's my preference as well. But collecting semantic
attachments on higher-level nodes (using one more several visits on the
parse tree) should not be in the way of that.
The difference to incrementally altering the structure of the tree is that
this approach attaches the required state to nodes of the original tree
itself. E.g. in a first pass you could register all alias definitions (e.g.
"c" = PersisterRef(Customer)) in some look-up table. Then in a second pass
you could resolve alias uses against these definitions and attach that
resolved information to the node representing the original reference. So
"semantic representation" would be in node attachments (again, likely in
aggregated forms on super-nodes or nodes somewhere else in the tree)
instead of nodes themselves.
At least that's how I understand things to work in Antlr4 based on their
docs and the user group discussions initiated by Steve.
Steve, you mentioned that ANTLR4 handles transformations but only when
it's the last step. What prevents us to chain multiple such
transformations, applying the "last step" approach multiple times?
I didn't look at it at all, so take this just as an high level,
conceptual question. I guess one would need to clearly define all
intermediate data types rather than have ANTLR generate them like it
does with tokens, but that could be the lesser trouble?
Thanks,
Sanne
>
> --Gunnar
>
>
>
>
>
>
>
> 2014-11-13 19:42 GMT+01:00 Steve Ebersole <steve(a)hibernate.org>:
>
>> As most of you know already, we are planning to redesign the current
>> Antlr-based HQL/JPQL parser in ORM for a variety of reasons.
>>
>> The current approach in the translator (Antlr 2 based, although Antlr 3
>> supports the same model) is that we actually define multiple
>> grammars/parsers which progressively re-write the tree adding more and
more
>> semantic information; think of this as multiple passes or phases. The
>> current code has 3 phases:
>> 1) parsing - we simply parse the HQL/JPQL query into an AST, although
we do
>> do one interesting (and uber-important!) re-write here where we
"hoist"
the
>> from clause in front of all other clauses.
>> 2) rough semantic analysis - the current code, to be honest, sucks here.
>> The end result of this phase is a tree that mixes normalized semantic
>> information with lots of SQL fragments. It is extremely fugly
>> 3) rendering to SQL
>>
>> The idea of phases is still the best way to attack this translation
imo. I
>> just think we did not implement the phases very well before; we were
just
>> learning Antlr at the time. So part of the redesign here is to leverage
>> our better understanding of Antlr and design some better trees. The
other
>> big reason is to centralize the generation of SQL into one place rather
>> than the 3 different places we do it today (not to mention the many,
many
>> places we render SQL fragments).
>>
>> Part of the process here is to decide which parser to use. Antlr 2 is
>> ancient :) I used Antlr 3 in the initial prototyping of this redesign
>> because it was the most recent release at that time. In the interim
Antlr
>> 4 has been released.
>>
>> I have been evaluating whether Antlr 4 is appropriate for our needs
there.
>> Antlr 4 is a pretty big conceptual deviation from Antlr 2/3 in quite a
few
>> ways. Generally speaking, Antlr 4 is geared more towards interpreting
>> rather than translating/transforming. It can handle "transformation"
if
>> the transformation is the final step in the process. Transformations is
>> where tree re-writing comes in handy.
>>
>> First lets step back and look at the "conceptual model" of Antlr 4.
The
>> grammar is used to produce:
>> 1) the parser - takes the input and builds a "parse tree" based on
the
>> rules of the lexer and grammar.
>> 2) listener/visitor for parse-tree traversal - can optionally generate
>> listeners or visitors (or both) for traversing the parse tree (output
from
>> parser).
>>
>> There are 2 highly-related changes that negatively impact us:
>> 1) no tree grammars/parsers
>> 2) no tree re-writing
>>
>> Our existing translator is fundamentally built on the concepts of tree
>> parsers and tree re-writing. Even the initial prototypes for the
redesign
>> (and the current state of hql-parser which Sanne and Gunnar picked up
from
>> there) are built on those concepts. So moving to Antlr 4 in that regard
>> does represent a risk. How big of a risk, and whether that risk is
worth
>> it, is what we need to determine.
>>
>> What does all this mean in simple, practical terms? Let's look at a
simple
>> query: "select c.headquarters.state.code from Company c". Simple
syntactic
>> analysis will produce a tree something like:
>>
>> [QUERY]
>> [SELECT]
>> [DOT]
>> [DOT]
>> [DOT]
>> [IDENT, "c"]
>> [IDENT, "headquarters"]
>> [IDENT, "state"]
>> [IDENT, "code"]
>> [FROM]
>> [SPACE]
>> [SPACE_ROOT]
>> [IDENT, "Customer"]
>> [IDENT, "c"]
>>
>> There is not a lot of semantic (meaning) information here. A more
semantic
>> representation of the query would look something like:
>>
>> [QUERY]
>> [SELECT]
>> [ATTRIBUTE_REF]
>> [ALIAS_REF, "<gen:1>"]
>> [IDENT, "code"]
>> [FROM]
>> [SPACE]
>> [PERSISTER_REF]
>> [ENTITY_NAME, "com.acme.Customer"]
>> [ALIAS, "c"]
>> [JOIN]
>> [INNER]
>> [ATTRIBUTE_JOIN]
>> [IDENT, "headquarters"]
>> [ALIAS, "<gen:0>"]
>> [JOIN]
>> [INNER]
>> [ATTRIBUTE_JOIN]
>> [IDENT, "state"]
>> [ALIAS, "<gen:1>"]
>>
>>
>> Notice especially the difference in the tree rules. This is tree
>> re-writing, and is the major difference affecting us. Consider a
specific
>> thing like the "c.headquarters.state.code" DOT-IDENT sequence.
Essentially
>> Antlr 4 would make us deal with that as a DOT-IDENT sequence through all
>> the phases - even SQL generation. Quite fugly. The intent of Antlr 4
in
>> cases like this is to build up an external state table (external to the
>> tree itself) or what Antlr folks typically refer to as "iterative tree
>> decoration"[1]. So with Antlr 4, in generating the SQL, we would still
be
>> handling calls in terms of "c.headquarters.state.code" in the SELECT
clause
>> and resolving that through the external symbol tables. Again, with
Antlr 4
>> we would always be walking that initial (non-semantic) tree. Unless I
am
>> missing something. I would be happy to be corrected, if anyone knows
Antlr
>> 4 better. I have also asked as part of the antlr-discussion group[2].
>>
>> In my opinion though, if it comes down to us needing to walk the tree in
>> that first form across all phases I just do not see the benefit to
moving
>> to Antlr 4.
>>
>> P.S. When I say SQL above I really just mean the target query language
for
>> the back-end data store whether that be SQL targeting a RDBMS for ORM
or a
>> NoSQL store for OGM.
>>
>> [1] I still have not fully grokked this paradigm, so I may be missing
>> something, but... AFAICT even in this paradigm the listener/visitor
rules
>> are defined in terms of the initial parse tree rules rather than more
>> [2]
https://groups.google.com/forum/#!topic/antlr-discussion/hzF_YrzfDKo
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
9:51 a.m.
On Tue, Jun 9, 2015 at 5:50 AM Sanne Grinovero <sanne(a)hibernate.org> wrote:
Do you mean that you would be ok to "navigate" all the [DOT] nodes to
get to the decorated attachments?
In that case while you might be fine to translate each fragment into a
different fragment, it's not straight forward to transform it into a
different structure, say with sub-trees in different orders or nodes
which don't have a 1:1 match.
It's of course doable if you are filling in your own builder while
navigating these (like we do with the Lucene DSL output), but it
doesn't help you with multiple phases which is what Steve is pointing
out.
I would highly prefer to feed the semantic representation of the tree
to our query generating backends, especially so if we could all share
the same initial smart phases to do some basic validations and
optimisations DRY. But then the consuming backends will likely have
some additional validations and optimisations which need to be
backend-specific (dialect-specific or technology specific in case of
OGM).
Steve, you mentioned that ANTLR4 handles transformations but only
when it's the last step. What prevents us to chain multiple such
transformations, applying the "last step" approach multiple times?
I didn't look at it at all, so take this just as an high level,
conceptual question. I guess one would need to clearly define all
intermediate data types rather than have ANTLR generate them like it
does with tokens, but that could be the lesser trouble?
The "problem" is that the Antlr listeners/visitors are always based on the
original parse tree. The transformation is not the concern. The concern
is how you match up the listener/visitor calls based on the original parse
tree into actions on the semantic tree.
The thing to keep in mind is that the Antlr listeners/visitors are based on
that parse tree. Going back to the Customer-headquarters query and the
original parse and semantic trees, given a call to process the "dot node"
that represents the root of the select expression, how do you "map" that to
the attributeReference node in the semantic tree?
Once the trees start to deviate you have basically lost the ability to
drive processing of that "subsequent tree structure" based on Antlr. At
least easily. There are probably some approaches we could use to allow
that. Off the top of my head, I could see assigning each and every node in
the parse tree a uid and then maintaining a "node replacement map" based on
those uids. But that all seems like a lot of work.
Another option I have seen Antlr folks mention is to write a second grammar
defined based on your semantic tree. It would produce listeners/visitors
based on the structure we ultimately expect in the semantic tree. Antlr
would just not provide use the tree re-writing anymore; we'd do that
manually. Other than that, everything else (10,000 foot view) should
remain the same. Of course devils's in the details :)
We could also, which I think is what you are suggesting Sanne, have the
query parser project produce the semantic tree and then it would just be up
to the consumers of that semantic tree to do with it whats it wants.
Combining this with the idea of a second grammar for the semantic tree, we
could say that the query parser project provides:
1) Antlr 4 listeners and visitors based on that semantic tree grammar
2) An API for converting HQL to such semantic trees.
9:02 a.m.
On Tue, Jun 9, 2015 at 3:57 AM Gunnar Morling <gunnar(a)hibernate.org> wrote:
What I like about the Antlr4 approach is the fact that you don't need a set
of several quite similar grammars as you'd do with the tree
transformation
approach. Also using the current version of Antlr instead of 3 appears
attractive to me wrt. to bugfixes and future development of the tool.
Understand that we would all "like" to use Antlr 4 for many reasons, myself
included. But it has to work for our needs. There are just so many open
questions (for me) as to whether that is the case.
Based on what I understand from your discussions on the Antlr mailing
list, I'd assume the parse tree and the external state it references to
look roughly like so (---> indicates a reference to state built up during
sub-sequential walks, maybe in some external "table", maybe stored within
the (typed) tree nodes themselves):
[QUERY]
[SELECT]
[ATTRIBUTE_REF] ---> AttributeReference("<gen:1>",
"code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
[FROM]
[SPACE]
[SPACE_ROOT] ---> InnerJoin( InnerJoin ( PersisterRef( "c",
"com.acme.Customer" ), TableRef ( "<gen:0>",
"headquarters" ) ), TableRef (
"<gen:1>", "state" ) ) )
[IDENT, "Customer"]
[IDENT, "c"]
I.e. instead of transforming the tree itself, the state required for
output generation would be added as "decorators" to nodes of the original
parse tree itself. That's just the basic idea as I understand it, surely
the specific types of the decorator elements (AttributeReference,
InnerJoin etc.) may look different. During "query rendering" we'd have to
inspect the decorator state of the parse tree nodes and interpret it
accordingly.
Well, see you do something "tricky" here that is actually one of my
concerns with Antlr 4 :) You mix a parse tree and a semantic tree.
Specifically this part of your tree:
[ATTRIBUTE_REF] ---> AttributeReference("<gen:1>", "code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
The idea of "ATTRIBUTE_REF" is a semantic concept. The DOT-IDENT struct
is your parse tree. Antlr 4 does allow mixing these based on left
refactoring of the rules, *but* there is an assumption there... that the
branches in such a left-refactored rule can be resolved unambiguously. I
am not so sure we can do that.
In simpler terms... Antlr 4 needs you to be able to apply those semantic
resolutions (attributeRef versus javaLiteralRef versus
oraclePackagedProcedure versus ...) up front.
So take the input that produces that tree: select c.headquarters.state.code
Syntactically that dot-ident structure could represent any number of
things. And semantically we just simply do not have enough information.
We *could* eliminate it being a javaLiteralRef if we
made javaLiteralRef the highest precedence branch in the left-factored rule
that produces this, but that has serious drawbacks:
1) we are checking each and every dot-ident path as a possible
javaLiteralRef first, which means reflection (perf)
2) it is not a fool-proof approach. The problem is that javaLiteralRef
should really have very low precedence. There are conceivably cases where
the expression could resolve to either a javaLiteralRef or an attributeRef,
and in those cases the resolution should be routed through attributeRef not
javaLiteralRef
The ultimate problem there is that we cannot possibly know much of the
information we need to know for proper semantic analysis until after we
have seen the FROM clause. We got around that with older Antlr versions
specifically via tree-rewriting: we re-write the tree to "hoist" FROM
before the other clauses.
So I believe the issue of alias resolution and implicit join conversion
could be handled without tree transformations (at least conceptually,
I
could not code an actual implementation out of my head right away). But
maybe there are other cases where tree transformations are more strictly
needed?
Well I just illustrated above how that is actually a problem that does need
either tree transformations or at least delayed processing of the sub-tree.
Also get out of your head this idea that we can encode the semantic
resolution of dot-ident paths into the tree. We simply will not be able to
(I believe). And I think that starts to show my reservations about Antlr
4. Basically every pass over this tree we will need to deal with
[[DOT][IDENT]] as opposed to [ATTRIBUTE_REFERENCE]
10:14 a.m.
2015-06-09 16:02 GMT+02:00 Steve Ebersole <steve(a)hibernate.org>:
On Tue, Jun 9, 2015 at 3:57 AM Gunnar Morling
<gunnar(a)hibernate.org>
wrote:
What I like about the Antlr4 approach is the fact that you don't need a
> set of several quite similar grammars as you'd do with the tree
> transformation approach. Also using the current version of Antlr instead of
> 3 appears attractive to me wrt. to bugfixes and future development of the
> tool.
>
Understand that we would all "like" to use Antlr 4 for many reasons,
myself included. But it has to work for our needs. There are just so many
open questions (for me) as to whether that is the case.
Sure, that's what we need to find out.
> Based on what I understand from your discussions on the Antlr
mailing
> list, I'd assume the parse tree and the external state it references to
> look roughly like so (---> indicates a reference to state built up during
> sub-sequential walks, maybe in some external "table", maybe stored within
> the (typed) tree nodes themselves):
>
> [QUERY]
> [SELECT]
> [ATTRIBUTE_REF] ---> AttributeReference("<gen:1>",
"code")
> [DOT]
> [DOT]
> [DOT]
> [IDENT, "c"]
> [IDENT, "headquarters"]
> [IDENT, "state"]
> [IDENT, "code"]
> [FROM]
> [SPACE]
> [SPACE_ROOT] ---> InnerJoin( InnerJoin ( PersisterRef( "c",
> "com.acme.Customer" ), TableRef ( "<gen:0>",
"headquarters" ) ), TableRef (
> "<gen:1>", "state" ) ) )
> [IDENT, "Customer"]
> [IDENT, "c"]
>
> I.e. instead of transforming the tree itself, the state required for
> output generation would be added as "decorators" to nodes of the original
> parse tree itself. That's just the basic idea as I understand it, surely
> the specific types of the decorator elements (AttributeReference,
> InnerJoin etc.) may look different. During "query rendering" we'd have
> to inspect the decorator state of the parse tree nodes and interpret it
> accordingly.
>
Well, see you do something "tricky" here that is actually one of my
concerns with Antlr 4 :) You mix a parse tree and a semantic tree.
Specifically this part of your tree:
[ATTRIBUTE_REF] ---> AttributeReference("<gen:1>", "code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
The idea of "ATTRIBUTE_REF" is a semantic concept. The DOT-IDENT struct
is your parse tree. Antlr 4 does allow mixing these based on left
refactoring of the rules, *but* there is an assumption there... that the
branches in such a left-refactored rule can be resolved unambiguously. I
am not so sure we can do that.
Yes, indeed I cheated here a bit. Probably it should be the following
instead:
[DOT] ---> AttributeReference("<gen:1>", "code")
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
Or maybe something like:
[SELECTION_PARTICLE] ---> AttributeReference("<gen:1>",
"code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
Where SELECTION_PARTICLE would be an abstract representation of anything
that can be selected (attribute ref, Java literal ref etc.) and the
decorator element added in a later pass would specify its actual semantics
based on the alias definitions etc. discovered before.
Bottom line being, that decorators providing semantics are attached to the
nodes of the parse tree based on information gathered in previous passes.
In simpler terms... Antlr 4 needs you to be able to apply those semantic
resolutions (attributeRef versus javaLiteralRef versus
oraclePackagedProcedure versus ...) up front.
So take the input that produces that tree: select c.headquarters.state.code
Syntactically that dot-ident structure could represent any number of
things. And semantically we just simply do not have enough information.
We *could* eliminate it being a javaLiteralRef if we
made javaLiteralRef the highest precedence branch in the left-factored rule
that produces this, but that has serious drawbacks:
1) we are checking each and every dot-ident path as a possible
javaLiteralRef first, which means reflection (perf)
2) it is not a fool-proof approach. The problem is that javaLiteralRef
should really have very low precedence. There are conceivably cases where
the expression could resolve to either a javaLiteralRef or an attributeRef,
and in those cases the resolution should be routed through attributeRef not
javaLiteralRef
The ultimate problem there is that we cannot possibly know much of the
information we need to know for proper semantic analysis until after we
have seen the FROM clause. We got around that with older Antlr versions
specifically via tree-rewriting: we re-write the tree to "hoist" FROM
before the other clauses.
So I believe the issue of alias resolution and implicit join conversion
> could be handled without tree transformations (at least conceptually, I
> could not code an actual implementation out of my head right away). But
> maybe there are other cases where tree transformations are more strictly
> needed?
>
Well I just illustrated above how that is actually a problem that does
need either tree transformations or at least delayed processing of the
sub-tree.
Also get out of your head this idea that we can encode the semantic
resolution of dot-ident paths into the tree. We simply will not be able to
(I believe).
Not into the tree itself, but we can encode that semantic resolution into
decorators (node attachments).
And I think that starts to show my reservations about Antlr 4.
Basically
every pass over this tree we will need to deal with [[DOT][IDENT]] as
opposed to [ATTRIBUTE_REFERENCE]
Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
semantic decorators attached previously. During rendering I would expect
mainly those attachments to be of importance for the query creation.
Admittedly, that's all quite "high level", but so far it seems doable to me
in principle. It doesn't answer of course actual tree transformations such
as (x + 0) -> x. I am not sure whether there are cases like this.
10:47 a.m.
On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling <gunnar(a)hibernate.org> wrote:
Yes, indeed I cheated here a bit. Probably it should be the following
instead:
[DOT] ---> AttributeReference("<gen:1>", "code")
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
How do you identify one DOT as referring to something else versus any of
the other DOTs?
Or maybe something like:
[SELECTION_PARTICLE] ---> AttributeReference("<gen:1>",
"code")
[DOT]
[DOT]
[DOT]
[IDENT, "c"]
[IDENT, "headquarters"]
[IDENT, "state"]
[IDENT, "code"]
Where SELECTION_PARTICLE would be an abstract representation of anything
that can be selected (attribute ref, Java literal ref etc.) and the
decorator element added in a later pass would specify its actual semantics
based on the alias definitions etc. discovered before.
Bottom line being, that decorators providing semantics are attached to the
nodes of the parse tree based on information gathered in previous passes.
And what does that look like in real, practical terms? That's what
concerns me :) I don't know, and you are just speaking in generalities.
So what does that look like in practice?
Not into the tree itself, but we can encode that semantic resolution into
decorators (node attachments).
Again, what do these "node attachments" look like in practice? I have
zero clue and based on my discussions with Antlr folks its not pretty.
Maybe I misunderstand. But if you are proposing this approach, I would
think you should have an idea of how it would look practically-speaking :)
Maybe this is the way to go, I just need to see what this looks like.
Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
semantic decorators attached previously. During rendering I would
expect
mainly those attachments to be of importance for the query creation.
Admittedly, that's all quite "high level", but so far it seems doable to
me in principle. It doesn't answer of course actual tree transformations
such as (x + 0) -> x. I am not sure whether there are cases like this.
Yes it is all extremely high-level. That is my concern. Principle and
practice are often 2 very different things.
I plan on spending some time taking my hibernate-antlr4-poc project and
expanding it specifically to try the "second grammar" approach and see what
practical difficulties that shakes out. Would you be willing to do the
same for this decorated approach? Then we'd have concrete stuff to compare
and base a decision on.
Also, `(x + 0) -> x` is actually a quite simple case. Ours is much more
complicated. In analyzing `c.headquarters.state.code` in the SELECT clause
we need a few things to happen in a few different parts of the tree. We
need:
1) `c.headquarters.state` to be transformed into 2 "implicit joins" in the
FROM clause
2) we need to replace `c.headquarters.state.code` as
`{implicit-alias}.code` in the SELECT
3) register `c.headquarters` and `c.headquarters.state` as implicit join
paths (additional implicit joins using these paths should re-use the same
joins).
3:11 p.m.
So today I spent some time cleaning up the basic HQL parser. Personally I
think it would be best if our 2 proof-of-concepts could share that first
grammar. IMO that would make the differences between the 2 approaches more
apparent. I will push those changes soon.
It is not complete yet. But it covers most cases.
On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve(a)hibernate.org> wrote:
On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling
<gunnar(a)hibernate.org>
wrote:
Yes, indeed I cheated here a bit. Probably it should be the following
> instead:
>
> [DOT] ---> AttributeReference("<gen:1>", "code")
> [DOT]
> [DOT]
> [IDENT, "c"]
> [IDENT, "headquarters"]
> [IDENT, "state"]
> [IDENT, "code"]
>
How do you identify one DOT as referring to something else versus any of
the other DOTs?
Or maybe something like:
>
> [SELECTION_PARTICLE] ---> AttributeReference("<gen:1>",
"code")
> [DOT]
> [DOT]
> [DOT]
> [IDENT, "c"]
> [IDENT, "headquarters"]
> [IDENT, "state"]
> [IDENT, "code"]
>
> Where SELECTION_PARTICLE would be an abstract representation of anything
> that can be selected (attribute ref, Java literal ref etc.) and the
> decorator element added in a later pass would specify its actual semantics
> based on the alias definitions etc. discovered before.
>
> Bottom line being, that decorators providing semantics are attached to
> the nodes of the parse tree based on information gathered in previous
> passes.
>
And what does that look like in real, practical terms? That's what
concerns me :) I don't know, and you are just speaking in generalities.
So what does that look like in practice?
Not into the tree itself, but we can encode that semantic resolution into
> decorators (node attachments).
>
Again, what do these "node attachments" look like in practice? I have
zero clue and based on my discussions with Antlr folks its not pretty.
Maybe I misunderstand. But if you are proposing this approach, I would
think you should have an idea of how it would look practically-speaking :)
Maybe this is the way to go, I just need to see what this looks like.
Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
> semantic decorators attached previously. During rendering I would expect
> mainly those attachments to be of importance for the query creation.
>
> Admittedly, that's all quite "high level", but so far it seems doable
to
> me in principle. It doesn't answer of course actual tree transformations
> such as (x + 0) -> x. I am not sure whether there are cases like this.
>
Yes it is all extremely high-level. That is my concern. Principle and
practice are often 2 very different things.
I plan on spending some time taking my hibernate-antlr4-poc project and
expanding it specifically to try the "second grammar" approach and see what
practical difficulties that shakes out. Would you be willing to do the
same for this decorated approach? Then we'd have concrete stuff to compare
and base a decision on.
Also, `(x + 0) -> x` is actually a quite simple case. Ours is much more
complicated. In analyzing `c.headquarters.state.code` in the SELECT clause
we need a few things to happen in a few different parts of the tree. We
need:
1) `c.headquarters.state` to be transformed into 2 "implicit joins" in
the FROM clause
2) we need to replace `c.headquarters.state.code` as
`{implicit-alias}.code` in the SELECT
3) register `c.headquarters` and `c.headquarters.state` as implicit join
paths (additional implicit joins using these paths should re-use the same
joins).
10:49 a.m.
2015-06-09 22:11 GMT+02:00 Steve Ebersole <steve(a)hibernate.org>:
So today I spent some time cleaning up the basic HQL parser.
Personally I
think it would be best if our 2 proof-of-concepts could share that first
grammar. IMO that would make the differences between the 2 approaches more
apparent. I will push those changes soon.
Ok, I can try and work on a PoC for the decorator-based approach. Not sure
what the outcome will be, as your's, my understanding of it is roughly
vague and high-level. But if it fails we can maybe settle for the Antlr3
approach with the better feeling of having investigated the alternative.
Can you let me know when you have pushed your stuff? What does it do,
render the query below as SQL?
It is not complete yet. But it covers most cases.
On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve(a)hibernate.org>
wrote:
> On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling <gunnar(a)hibernate.org>
> wrote:
>
> Yes, indeed I cheated here a bit. Probably it should be the following
>> instead:
>>
>> [DOT] ---> AttributeReference("<gen:1>",
"code")
>> [DOT]
>> [DOT]
>> [IDENT, "c"]
>> [IDENT, "headquarters"]
>> [IDENT, "state"]
>> [IDENT, "code"]
>>
>
> How do you identify one DOT as referring to something else versus any of
> the other DOTs?
>
>
> Or maybe something like:
>>
>> [SELECTION_PARTICLE] ---> AttributeReference("<gen:1>",
"code")
>> [DOT]
>> [DOT]
>> [DOT]
>> [IDENT, "c"]
>> [IDENT, "headquarters"]
>> [IDENT, "state"]
>> [IDENT, "code"]
>>
>> Where SELECTION_PARTICLE would be an abstract representation of anything
>> that can be selected (attribute ref, Java literal ref etc.) and the
>> decorator element added in a later pass would specify its actual semantics
>> based on the alias definitions etc. discovered before.
>>
>> Bottom line being, that decorators providing semantics are attached to
>> the nodes of the parse tree based on information gathered in previous
>> passes.
>>
>
> And what does that look like in real, practical terms? That's what
> concerns me :) I don't know, and you are just speaking in generalities.
> So what does that look like in practice?
>
>
> Not into the tree itself, but we can encode that semantic resolution into
>> decorators (node attachments).
>>
>
> Again, what do these "node attachments" look like in practice? I have
> zero clue and based on my discussions with Antlr folks its not pretty.
> Maybe I misunderstand. But if you are proposing this approach, I would
> think you should have an idea of how it would look practically-speaking :)
> Maybe this is the way to go, I just need to see what this looks like.
>
>
> Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
>> semantic decorators attached previously. During rendering I would expect
>> mainly those attachments to be of importance for the query creation.
>>
>> Admittedly, that's all quite "high level", but so far it seems
doable to
>> me in principle. It doesn't answer of course actual tree transformations
>> such as (x + 0) -> x. I am not sure whether there are cases like this.
>>
>
> Yes it is all extremely high-level. That is my concern. Principle and
> practice are often 2 very different things.
>
> I plan on spending some time taking my hibernate-antlr4-poc project and
> expanding it specifically to try the "second grammar" approach and see
what
> practical difficulties that shakes out. Would you be willing to do the
> same for this decorated approach? Then we'd have concrete stuff to compare
> and base a decision on.
>
> Also, `(x + 0) -> x` is actually a quite simple case. Ours is much more
> complicated. In analyzing `c.headquarters.state.code` in the SELECT clause
> we need a few things to happen in a few different parts of the tree. We
> need:
> 1) `c.headquarters.state` to be transformed into 2 "implicit joins" in
> the FROM clause
> 2) we need to replace `c.headquarters.state.code` as
> `{implicit-alias}.code` in the SELECT
> 3) register `c.headquarters` and `c.headquarters.state` as implicit join
> paths (additional implicit joins using these paths should re-use the same
> joins).
>
9:50 a.m.
In the re-write case we will have to decide between 2 courses.
The problem lies in the fact that the listeners and visitors expect a tree
specifically generated from the grammar that generated them. For HQL, say
we have one grammar named HqlParser (like in the poc). The listeners and
visitors built from HqlParser specifically expect the tree from HqlParser.
The trees are typed. Attempting to use trees from one grammar in the
listener/visitor from another grammar will not work.
As far as re-writing that effectively means 2 options.
If we want to have a second grammar for the "semantic query" we are going
to have to re-write the entire tree based on that second grammar. I had
thought we might be able to mix them. But that was based on my Antlr 2/3
understanding where the trees are de-typed. That approach will not work in
Antlr 4. Its not a huge deal, but worth mentioning.
The other option would be to encode the "semantically correct" rules into
the original grammar (HqlParser) as a higher precedence than their parse
tree corollary. This gets a little fugly. Consider again the `select
c.headquarters.state.code` fragment. During the parse phase we need to
accept any dotIdentifierPath as a selectable item. We simply do not know
during parse what that represents. So for the parse phase, a `selectItem`
rule (overly simplified) might look like:
selectItem : dotIdentifierPath;
In this approach we would re-write the tree "in place" during semantic
analysis. So at some point we know that the given dotIdentifierPath
represents a reference to a persistent attribute. So we'd alter that rule
to look contain alternatives for each semantic possibility:
selectItem : attributeReference | javaConstant | dotIdentifierPath;
The rules attributeReference and javaConstant would never match during the
parse phase.
Again, this is fugly imo.
On Wed, Jun 10, 2015 at 10:49 AM Gunnar Morling <gunnar(a)hibernate.org>
wrote:
2015-06-09 22:11 GMT+02:00 Steve Ebersole
<steve(a)hibernate.org>:
> So today I spent some time cleaning up the basic HQL parser. Personally
> I think it would be best if our 2 proof-of-concepts could share that first
> grammar. IMO that would make the differences between the 2 approaches more
> apparent. I will push those changes soon.
>
Ok, I can try and work on a PoC for the decorator-based approach. Not sure
what the outcome will be, as your's, my understanding of it is roughly
vague and high-level. But if it fails we can maybe settle for the Antlr3
approach with the better feeling of having investigated the alternative.
Can you let me know when you have pushed your stuff? What does it do,
render the query below as SQL?
It is not complete yet. But it covers most cases.
>
>
> On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve(a)hibernate.org>
> wrote:
>
>> On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling <gunnar(a)hibernate.org>
>> wrote:
>>
>> Yes, indeed I cheated here a bit. Probably it should be the following
>>> instead:
>>>
>>> [DOT] ---> AttributeReference("<gen:1>",
"code")
>>> [DOT]
>>> [DOT]
>>> [IDENT, "c"]
>>> [IDENT, "headquarters"]
>>> [IDENT, "state"]
>>> [IDENT, "code"]
>>>
>>
>> How do you identify one DOT as referring to something else versus any of
>> the other DOTs?
>>
>>
>> Or maybe something like:
>>>
>>> [SELECTION_PARTICLE] --->
AttributeReference("<gen:1>", "code")
>>> [DOT]
>>> [DOT]
>>> [DOT]
>>> [IDENT, "c"]
>>> [IDENT, "headquarters"]
>>> [IDENT, "state"]
>>> [IDENT, "code"]
>>>
>>> Where SELECTION_PARTICLE would be an abstract representation of
>>> anything that can be selected (attribute ref, Java literal ref etc.) and
>>> the decorator element added in a later pass would specify its actual
>>> semantics based on the alias definitions etc. discovered before.
>>>
>>> Bottom line being, that decorators providing semantics are attached to
>>> the nodes of the parse tree based on information gathered in previous
>>> passes.
>>>
>>
>> And what does that look like in real, practical terms? That's what
>> concerns me :) I don't know, and you are just speaking in generalities.
>> So what does that look like in practice?
>>
>>
>> Not into the tree itself, but we can encode that semantic resolution
>>> into decorators (node attachments).
>>>
>>
>> Again, what do these "node attachments" look like in practice? I
have
>> zero clue and based on my discussions with Antlr folks its not pretty.
>> Maybe I misunderstand. But if you are proposing this approach, I would
>> think you should have an idea of how it would look practically-speaking :)
>> Maybe this is the way to go, I just need to see what this looks like.
>>
>>
>> Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
>>> semantic decorators attached previously. During rendering I would expect
>>> mainly those attachments to be of importance for the query creation.
>>>
>>> Admittedly, that's all quite "high level", but so far it seems
doable
>>> to me in principle. It doesn't answer of course actual tree
transformations
>>> such as (x + 0) -> x. I am not sure whether there are cases like this.
>>>
>>
>> Yes it is all extremely high-level. That is my concern. Principle and
>> practice are often 2 very different things.
>>
>> I plan on spending some time taking my hibernate-antlr4-poc project and
>> expanding it specifically to try the "second grammar" approach and see
what
>> practical difficulties that shakes out. Would you be willing to do the
>> same for this decorated approach? Then we'd have concrete stuff to compare
>> and base a decision on.
>>
>> Also, `(x + 0) -> x` is actually a quite simple case. Ours is much
>> more complicated. In analyzing `c.headquarters.state.code` in the SELECT
>> clause we need a few things to happen in a few different parts of the
>> tree. We need:
>> 1) `c.headquarters.state` to be transformed into 2 "implicit joins" in
>> the FROM clause
>> 2) we need to replace `c.headquarters.state.code` as
>> `{implicit-alias}.code` in the SELECT
>> 3) register `c.headquarters` and `c.headquarters.state` as implicit
>> join paths (additional implicit joins using these paths should re-use the
>> same joins).
>>
>
5:30 p.m.
I just pushed my initial work on performing some indexing of explicit from
clauses. Essentially it takes the from clauses defined in the query and
begins the massage process. From here I will start working on handling
implicit from-clause-elements. Part of that however will require me being
able to know what is an entity and whether an IDENTIFIER in the query
represents an attribute in one (or more) of those from-clause-element
entities. IIRC this needs to be different between consumers of this
project as they do not always have persisters, etc. Previously we had
discussed an API that all consumers could provide. I had developed
org.hibernate.hql.ast.common.ParserContext
and friends as a means to that end. However, the parser project is
currently not using those. On the bright side that means we have a clean
slate for how to do this! :)
Part of the equation here is how we want certain things to work in terms of
an input/output paradigm, specifically what the "parser" should output.
Let's take some examples...
1) resolving possible unqualified attribute references. I have mentioned
this one before. Take a query like `select ssn from Person`. There are 2
main choices here when it comes to processing the `ssn` select expression.
It's a matter of precedence. The first option is to have
ATTRIBUTE_REFERENCE have the lowest precedence. Here we'd try other
possibilities first. Mainly that would entail trying it as various forms
of a constant. If all those attempts fail we would *assume* that the
expression is an ATTRIBUTE_REFERENCE. The assumption aspect is important.
It means that would not be validated here. This approach would not require
any form of API, but it has many downsides: it would require more expensive
resolution and it could potentially hide a ATTRIBUTE_REFERENCE. IMO
ATTRIBUTE_REFERENCE
should have the higher precedence. The other option is to have such an
API. This would allow the parser to ask the consumer whether the given
identifier (`ssn`) is a persistent attribute of any of the available
entities. If you can't tell, I am in big time favor of having such an API
:) But I am open to discussions as to the other side.
2) Understanding capabilities. The principle here is understanding what is
possible in different contexts based on the domain model being queried.
For example, if we see a query like `select c.addresses.city from Company
c` and if we know that `c.addresses` resolves to a persistent collection
then we know that the following de-reference is invalid. So here it is a
question of whether we want the parser to perform capability based
validations for us. Again, I'd argue we do as otherwise each consumer ends
up having to do these validations themselves. This is/was the intent of
the org.hibernate.hql.ast.TypeDescriptor stuff I had developed there
originally. So something like that.
3) We also need to decide how we want to handle polymorphic queries in this
parser. For a query like `from Object` what do we ultimately want
returned? Specifically how do we deal with the multi-valued
java.lang.Object reference in whatever we send back from the parser?
Because what we send out implies some things we need to send in (API).
Anyway, the from clause parser is looking nice so far.
On Thu, Jun 11, 2015 at 9:50 AM Steve Ebersole <steve(a)hibernate.org> wrote:
In the re-write case we will have to decide between 2 courses.
The problem lies in the fact that the listeners and visitors expect a tree
specifically generated from the grammar that generated them. For HQL, say
we have one grammar named HqlParser (like in the poc). The listeners and
visitors built from HqlParser specifically expect the tree from HqlParser.
The trees are typed. Attempting to use trees from one grammar in the
listener/visitor from another grammar will not work.
As far as re-writing that effectively means 2 options.
If we want to have a second grammar for the "semantic query" we are going
to have to re-write the entire tree based on that second grammar. I had
thought we might be able to mix them. But that was based on my Antlr 2/3
understanding where the trees are de-typed. That approach will not work in
Antlr 4. Its not a huge deal, but worth mentioning.
The other option would be to encode the "semantically correct" rules into
the original grammar (HqlParser) as a higher precedence than their parse
tree corollary. This gets a little fugly. Consider again the `select
c.headquarters.state.code` fragment. During the parse phase we need to
accept any dotIdentifierPath as a selectable item. We simply do not know
during parse what that represents. So for the parse phase, a `selectItem`
rule (overly simplified) might look like:
selectItem : dotIdentifierPath;
In this approach we would re-write the tree "in place" during semantic
analysis. So at some point we know that the given dotIdentifierPath
represents a reference to a persistent attribute. So we'd alter that rule
to look contain alternatives for each semantic possibility:
selectItem : attributeReference | javaConstant | dotIdentifierPath;
The rules attributeReference and javaConstant would never match during
the parse phase.
Again, this is fugly imo.
On Wed, Jun 10, 2015 at 10:49 AM Gunnar Morling <gunnar(a)hibernate.org>
wrote:
> 2015-06-09 22:11 GMT+02:00 Steve Ebersole <steve(a)hibernate.org>:
>
>> So today I spent some time cleaning up the basic HQL parser. Personally
>> I think it would be best if our 2 proof-of-concepts could share that first
>> grammar. IMO that would make the differences between the 2 approaches more
>> apparent. I will push those changes soon.
>>
>
> Ok, I can try and work on a PoC for the decorator-based approach. Not
> sure what the outcome will be, as your's, my understanding of it is roughly
> vague and high-level. But if it fails we can maybe settle for the Antlr3
> approach with the better feeling of having investigated the alternative.
>
> Can you let me know when you have pushed your stuff? What does it do,
> render the query below as SQL?
>
> It is not complete yet. But it covers most cases.
>>
>>
>> On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve(a)hibernate.org>
>> wrote:
>>
>>> On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling <gunnar(a)hibernate.org>
>>> wrote:
>>>
>>> Yes, indeed I cheated here a bit. Probably it should be the following
>>>> instead:
>>>>
>>>> [DOT] ---> AttributeReference("<gen:1>",
"code")
>>>> [DOT]
>>>> [DOT]
>>>> [IDENT, "c"]
>>>> [IDENT, "headquarters"]
>>>> [IDENT, "state"]
>>>> [IDENT, "code"]
>>>>
>>>
>>> How do you identify one DOT as referring to something else versus any
>>> of the other DOTs?
>>>
>>>
>>> Or maybe something like:
>>>>
>>>> [SELECTION_PARTICLE] --->
AttributeReference("<gen:1>", "code")
>>>> [DOT]
>>>> [DOT]
>>>> [DOT]
>>>> [IDENT, "c"]
>>>> [IDENT, "headquarters"]
>>>> [IDENT, "state"]
>>>> [IDENT, "code"]
>>>>
>>>> Where SELECTION_PARTICLE would be an abstract representation of
>>>> anything that can be selected (attribute ref, Java literal ref etc.) and
>>>> the decorator element added in a later pass would specify its actual
>>>> semantics based on the alias definitions etc. discovered before.
>>>>
>>>> Bottom line being, that decorators providing semantics are attached to
>>>> the nodes of the parse tree based on information gathered in previous
>>>> passes.
>>>>
>>>
>>> And what does that look like in real, practical terms? That's what
>>> concerns me :) I don't know, and you are just speaking in generalities.
>>> So what does that look like in practice?
>>>
>>>
>>> Not into the tree itself, but we can encode that semantic resolution
>>>> into decorators (node attachments).
>>>>
>>>
>>> Again, what do these "node attachments" look like in practice? I
have
>>> zero clue and based on my discussions with Antlr folks its not pretty.
>>> Maybe I misunderstand. But if you are proposing this approach, I would
>>> think you should have an idea of how it would look practically-speaking :)
>>> Maybe this is the way to go, I just need to see what this looks like.
>>>
>>>
>>> Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
>>>> semantic decorators attached previously. During rendering I would expect
>>>> mainly those attachments to be of importance for the query creation.
>>>>
>>>> Admittedly, that's all quite "high level", but so far it
seems doable
>>>> to me in principle. It doesn't answer of course actual tree
transformations
>>>> such as (x + 0) -> x. I am not sure whether there are cases like
this.
>>>>
>>>
>>> Yes it is all extremely high-level. That is my concern. Principle and
>>> practice are often 2 very different things.
>>>
>>> I plan on spending some time taking my hibernate-antlr4-poc project and
>>> expanding it specifically to try the "second grammar" approach and
see what
>>> practical difficulties that shakes out. Would you be willing to do the
>>> same for this decorated approach? Then we'd have concrete stuff to
compare
>>> and base a decision on.
>>>
>>> Also, `(x + 0) -> x` is actually a quite simple case. Ours is much
>>> more complicated. In analyzing `c.headquarters.state.code` in the SELECT
>>> clause we need a few things to happen in a few different parts of the
>>> tree. We need:
>>> 1) `c.headquarters.state` to be transformed into 2 "implicit joins"
in
>>> the FROM clause
>>> 2) we need to replace `c.headquarters.state.code` as
>>> `{implicit-alias}.code` in the SELECT
>>> 3) register `c.headquarters` and `c.headquarters.state` as implicit
>>> join paths (additional implicit joins using these paths should re-use the
>>> same joins).
>>>
>>
7:39 a.m.
Hi,
resolving possible unqualified attribute references [...]
The other option is to have such an API. This would allow the parser to
ask the
consumer whether the given identifier (`ssn`) is a persistent
attribute of any of the available entities.
Yes, having such extension point seems reasonable. OGM would probably use
the same implementation as ORM, but other users may plug in another impl
based on their own type of entity definitions. Would the scope of that
extension point be solely attribute resolution or also handling of other
things such as literals? I'd hope the latter could be done in a unified way
by the parser?
So here it is a question of whether we want the parser to perform
capability based validations for us. Again, I'd argue we do as otherwise
each consumer ends up having to do these validations themselves
+1
We also need to decide how we want to handle polymorphic queries in
this
parser.
I am not sure in terms of exact types to be returned, but it'd help if the
returned structure contained information about the actually affected
"tables" (or more generally, "structures" in the query backend), so
that
users don't each have to deal with resolving that information wrt. the
current mapping strategy. That need some extension point for specifying the
sub-types of given types. Again, OGM would probably share an impl. with ORM.
--Gunnar
2015-06-13 0:30 GMT+02:00 Steve Ebersole <steve(a)hibernate.org>:
I just pushed my initial work on performing some indexing of explicit
from
clauses. Essentially it takes the from clauses defined in the query and
begins the massage process. From here I will start working on handling
implicit from-clause-elements. Part of that however will require me being
able to know what is an entity and whether an IDENTIFIER in the query
represents an attribute in one (or more) of those from-clause-element
entities. IIRC this needs to be different between consumers of this
project as they do not always have persisters, etc. Previously we had
discussed an API that all consumers could provide. I had developed
org.hibernate.hql.ast.common.ParserContext
and friends as a means to that end. However, the parser project is
currently not using those. On the bright side that means we have a clean
slate for how to do this! :)
Part of the equation here is how we want certain things to work in terms
of an input/output paradigm, specifically what the "parser" should output.
Let's take some examples...
1) resolving possible unqualified attribute references. I have mentioned
this one before. Take a query like `select ssn from Person`. There are 2
main choices here when it comes to processing the `ssn` select
expression. It's a matter of precedence. The first option is to have
ATTRIBUTE_REFERENCE have the lowest precedence. Here we'd try other
possibilities first. Mainly that would entail trying it as various forms
of a constant. If all those attempts fail we would *assume* that the
expression is an ATTRIBUTE_REFERENCE. The assumption aspect is
important. It means that would not be validated here. This approach would
not require any form of API, but it has many downsides: it would require
more expensive resolution and it could potentially hide a ATTRIBUTE_REFERENCE.
IMO ATTRIBUTE_REFERENCE should have the higher precedence. The other
option is to have such an API. This would allow the parser to ask the
consumer whether the given identifier (`ssn`) is a persistent attribute of
any of the available entities. If you can't tell, I am in big time favor
of having such an API :) But I am open to discussions as to the other side.
2) Understanding capabilities. The principle here is understanding what
is possible in different contexts based on the domain model being queried.
For example, if we see a query like `select c.addresses.city from Company
c` and if we know that `c.addresses` resolves to a persistent collection
then we know that the following de-reference is invalid. So here it is a
question of whether we want the parser to perform capability based
validations for us. Again, I'd argue we do as otherwise each consumer ends
up having to do these validations themselves. This is/was the intent of
the org.hibernate.hql.ast.TypeDescriptor stuff I had developed there
originally. So something like that.
3) We also need to decide how we want to handle polymorphic queries in
this parser. For a query like `from Object` what do we ultimately want
returned? Specifically how do we deal with the multi-valued
java.lang.Object reference in whatever we send back from the parser?
Because what we send out implies some things we need to send in (API).
Anyway, the from clause parser is looking nice so far.
On Thu, Jun 11, 2015 at 9:50 AM Steve Ebersole <steve(a)hibernate.org>
wrote:
> In the re-write case we will have to decide between 2 courses.
>
> The problem lies in the fact that the listeners and visitors expect a
> tree specifically generated from the grammar that generated them. For HQL,
> say we have one grammar named HqlParser (like in the poc). The
> listeners and visitors built from HqlParser specifically expect the tree
> from HqlParser. The trees are typed. Attempting to use trees from one
> grammar in the listener/visitor from another grammar will not work.
>
> As far as re-writing that effectively means 2 options.
>
> If we want to have a second grammar for the "semantic query" we are going
> to have to re-write the entire tree based on that second grammar. I had
> thought we might be able to mix them. But that was based on my Antlr 2/3
> understanding where the trees are de-typed. That approach will not work in
> Antlr 4. Its not a huge deal, but worth mentioning.
>
> The other option would be to encode the "semantically correct" rules into
> the original grammar (HqlParser) as a higher precedence than their parse
> tree corollary. This gets a little fugly. Consider again the `select
> c.headquarters.state.code` fragment. During the parse phase we need to
> accept any dotIdentifierPath as a selectable item. We simply do not
> know during parse what that represents. So for the parse phase, a
> `selectItem` rule (overly simplified) might look like:
>
> selectItem : dotIdentifierPath;
>
> In this approach we would re-write the tree "in place" during semantic
> analysis. So at some point we know that the given dotIdentifierPath
> represents a reference to a persistent attribute. So we'd alter that rule
> to look contain alternatives for each semantic possibility:
>
> selectItem : attributeReference | javaConstant | dotIdentifierPath;
>
> The rules attributeReference and javaConstant would never match during
> the parse phase.
>
> Again, this is fugly imo.
>
>
> On Wed, Jun 10, 2015 at 10:49 AM Gunnar Morling <gunnar(a)hibernate.org>
> wrote:
>
>> 2015-06-09 22:11 GMT+02:00 Steve Ebersole <steve(a)hibernate.org>:
>>
>>> So today I spent some time cleaning up the basic HQL parser.
>>> Personally I think it would be best if our 2 proof-of-concepts could share
>>> that first grammar. IMO that would make the differences between the 2
>>> approaches more apparent. I will push those changes soon.
>>>
>>
>> Ok, I can try and work on a PoC for the decorator-based approach. Not
>> sure what the outcome will be, as your's, my understanding of it is roughly
>> vague and high-level. But if it fails we can maybe settle for the Antlr3
>> approach with the better feeling of having investigated the alternative.
>>
>> Can you let me know when you have pushed your stuff? What does it do,
>> render the query below as SQL?
>>
>> It is not complete yet. But it covers most cases.
>>>
>>>
>>> On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve(a)hibernate.org>
>>> wrote:
>>>
>>>> On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling
<gunnar(a)hibernate.org>
>>>> wrote:
>>>>
>>>> Yes, indeed I cheated here a bit. Probably it should be the following
>>>>> instead:
>>>>>
>>>>> [DOT] ---> AttributeReference("<gen:1>",
"code")
>>>>> [DOT]
>>>>> [DOT]
>>>>> [IDENT, "c"]
>>>>> [IDENT, "headquarters"]
>>>>> [IDENT, "state"]
>>>>> [IDENT, "code"]
>>>>>
>>>>
>>>> How do you identify one DOT as referring to something else versus any
>>>> of the other DOTs?
>>>>
>>>>
>>>> Or maybe something like:
>>>>>
>>>>> [SELECTION_PARTICLE] --->
AttributeReference("<gen:1>", "code")
>>>>> [DOT]
>>>>> [DOT]
>>>>> [DOT]
>>>>> [IDENT, "c"]
>>>>> [IDENT, "headquarters"]
>>>>> [IDENT, "state"]
>>>>> [IDENT, "code"]
>>>>>
>>>>> Where SELECTION_PARTICLE would be an abstract representation of
>>>>> anything that can be selected (attribute ref, Java literal ref etc.)
and
>>>>> the decorator element added in a later pass would specify its actual
>>>>> semantics based on the alias definitions etc. discovered before.
>>>>>
>>>>> Bottom line being, that decorators providing semantics are attached
>>>>> to the nodes of the parse tree based on information gathered in
previous
>>>>> passes.
>>>>>
>>>>
>>>> And what does that look like in real, practical terms? That's what
>>>> concerns me :) I don't know, and you are just speaking in
generalities.
>>>> So what does that look like in practice?
>>>>
>>>>
>>>> Not into the tree itself, but we can encode that semantic resolution
>>>>> into decorators (node attachments).
>>>>>
>>>>
>>>> Again, what do these "node attachments" look like in practice?
I
>>>> have zero clue and based on my discussions with Antlr folks its not
>>>> pretty. Maybe I misunderstand. But if you are proposing this approach,
I
>>>> would think you should have an idea of how it would look
>>>> practically-speaking :) Maybe this is the way to go, I just need to see
>>>> what this looks like.
>>>>
>>>>
>>>> Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
>>>>> semantic decorators attached previously. During rendering I would
expect
>>>>> mainly those attachments to be of importance for the query creation.
>>>>>
>>>>> Admittedly, that's all quite "high level", but so far
it seems doable
>>>>> to me in principle. It doesn't answer of course actual tree
transformations
>>>>> such as (x + 0) -> x. I am not sure whether there are cases like
this.
>>>>>
>>>>
>>>> Yes it is all extremely high-level. That is my concern. Principle
>>>> and practice are often 2 very different things.
>>>>
>>>> I plan on spending some time taking my hibernate-antlr4-poc project
>>>> and expanding it specifically to try the "second grammar"
approach and see
>>>> what practical difficulties that shakes out. Would you be willing to do
>>>> the same for this decorated approach? Then we'd have concrete stuff
to
>>>> compare and base a decision on.
>>>>
>>>> Also, `(x + 0) -> x` is actually a quite simple case. Ours is much
>>>> more complicated. In analyzing `c.headquarters.state.code` in the
SELECT
>>>> clause we need a few things to happen in a few different parts of the
>>>> tree. We need:
>>>> 1) `c.headquarters.state` to be transformed into 2 "implicit
joins"
>>>> in the FROM clause
>>>> 2) we need to replace `c.headquarters.state.code` as
>>>> `{implicit-alias}.code` in the SELECT
>>>> 3) register `c.headquarters` and `c.headquarters.state` as implicit
>>>> join paths (additional implicit joins using these paths should re-use
the
>>>> same joins).
>>>>
>>>
2:42 p.m.
On Wed, Jun 17, 2015 at 7:39 AM Gunnar Morling
Yes, having such extension point seems reasonable. OGM would probably
use
the same implementation as ORM, but other users may plug in another impl
based on their own type of entity definitions. Would the scope of that
extension point be solely attribute resolution or also handling of other
things such as literals? I'd hope the latter could be done in a unified way
by the parser?
That would be my preference as well. I think it makes the most sense to do
this stuff once in a single place. Some specific capabilities we would
need:
1) Ability to resolve "entity description" based on query token, including:
1.a) handling of "query imports" (aka, "MyEntity" ->
"com.acme.MyEntity")
1.b) polymorphic query references (i.e., "from java.lang.Object")
2) Access to "Attribute descriptors", ideally hooked into "entity/type
descriptors".
3) General classloading (Java/Enum constant/literal resolution)
These things combined would allow us to perform all the generic semantic
validations we need.
Much of this is in place already.
I am not sure in terms of exact types to be returned, but it'd help if the
returned structure contained information about the actually affected
"tables" (or more generally, "structures" in the query backend), so
that
users don't each have to deal with resolving that information wrt. the
current mapping strategy. That need some extension point for specifying the
sub-types of given types. Again, OGM would probably share an impl. with ORM.
At the moment this is handled
via org.hibernate.hql.parser.model.PolymorphicEntityTypeDescriptor which is
an extension to EntityTypeDescriptor. E.g., in the query "from
java.lang.Object" we'd report back PolymorphicEntityTypeDescriptor that
aggregates all root entities. The parser can always just deal with the
TypeDescriptor abstractions.
3453
days inactive
3673
days old
13 comments
3 participants
participants (3)
-
Gunnar Morling -
Sanne Grinovero -
Steve Ebersole