[hibernate-dev] Query handling : Antlr 3 versus Antlr 4

[Steve Ebersole]

I just pushed my initial work on performing some indexing of explicit from
clauses.  Essentially it takes the from clauses defined in the query and
begins the massage process.  From here I will start working on handling
implicit from-clause-elements.  Part of that however will require me being
able to know what is an entity and whether an IDENTIFIER in the query
represents an attribute in one (or more) of those from-clause-element
entities.  IIRC this needs to be different between consumers of this
project as they do not always have persisters, etc.  Previously we had
discussed an API that all consumers could provide.  I had developed
org.hibernate.hql.ast.common.ParserContext
and friends as a means to that end.  However, the parser project is
currently not using those.  On the bright side that means we have a clean
slate for how to do this! :)

Part of the equation here is how we want certain things to work in terms of
an input/output paradigm, specifically what the "parser" should output.
Let's take some examples...

1) resolving possible unqualified attribute references.  I have mentioned
this one before.  Take a query like `select ssn from Person`.  There are 2
main choices here when it comes to processing the `ssn` select expression.
It's a matter of precedence.  The first option is to have
ATTRIBUTE_REFERENCE have the lowest precedence.  Here we'd try other
possibilities first.  Mainly that would entail trying it as various forms
of a constant.  If all those attempts fail we would *assume* that the
expression is an ATTRIBUTE_REFERENCE.  The assumption aspect is important.
It means that would not be validated here.  This approach would not require
any form of API, but it has many downsides: it would require more expensive
resolution and it could potentially hide a ATTRIBUTE_REFERENCE.  IMO
ATTRIBUTE_REFERENCE
should have the higher precedence.  The other option is to have such an
API.  This would allow the parser to ask the consumer whether the given
identifier (`ssn`) is a persistent attribute of any of the available
entities.  If you can't tell, I am in big time favor of having such an API
:)  But I am open to discussions as to the other side.

2) Understanding capabilities.  The principle here is understanding what is
possible in different contexts based on the domain model being queried.
For example, if we see a query like `select c.addresses.city from Company
c` and if we know that `c.addresses` resolves to a persistent collection
then we know that the following de-reference is invalid.  So here it is a
question of whether we want the parser to perform capability based
validations for us.  Again, I'd argue we do as otherwise each consumer ends
up having to do these validations themselves.  This is/was the intent of
the org.hibernate.hql.ast.TypeDescriptor stuff I had developed there
originally.  So something like that.

3) We also need to decide how we want to handle polymorphic queries in this
parser.  For a query like `from Object` what do we ultimately want
returned?  Specifically how do we deal with the multi-valued
java.lang.Object reference in whatever we send back from the parser?
Because what we send out implies some things we need to send in (API).

Anyway, the from clause parser is looking nice so far.


On Thu, Jun 11, 2015 at 9:50 AM Steve Ebersole <steve at hibernate.org> wrote:

> In the re-write case we will have to decide between 2 courses.
>
> The problem lies in the fact that the listeners and visitors expect a tree
> specifically generated from the grammar that generated them.  For HQL, say
> we have one grammar named HqlParser (like in the poc).  The listeners and
> visitors built from HqlParser specifically expect the tree from HqlParser.
> The trees are typed.  Attempting to use trees from one grammar in the
> listener/visitor from another grammar will not work.
>
> As far as re-writing that effectively means 2 options.
>
> If we want to have a second grammar for the "semantic query" we are going
> to have to re-write the entire tree based on that second grammar.  I had
> thought we might be able to mix them. But that was based on my Antlr 2/3
> understanding where the trees are de-typed.  That approach will not work in
> Antlr 4.  Its not a huge deal, but worth mentioning.
>
> The other option would be to encode the "semantically correct" rules into
> the original grammar (HqlParser) as a higher precedence than their parse
> tree corollary.  This gets a little fugly.  Consider again the `select
> c.headquarters.state.code` fragment.  During the parse phase we need to
> accept any dotIdentifierPath as a selectable item.  We simply do not know
> during parse what that represents.  So for the parse phase, a `selectItem`
> rule (overly simplified) might look like:
>
> selectItem : dotIdentifierPath;
>
> In this approach we would re-write the tree "in place" during semantic
> analysis.  So at some point we know that the given dotIdentifierPath
> represents a reference to a persistent attribute.  So we'd alter that rule
> to look contain alternatives for each semantic possibility:
>
> selectItem : attributeReference | javaConstant | dotIdentifierPath;
>
> The rules attributeReference and javaConstant would never match during
> the parse phase.
>
> Again, this is fugly imo.
>
>
> On Wed, Jun 10, 2015 at 10:49 AM Gunnar Morling <gunnar at hibernate.org>
> wrote:
>
>> 2015-06-09 22:11 GMT+02:00 Steve Ebersole <steve at hibernate.org>:
>>
>>> So today I spent some time cleaning up the basic HQL parser.  Personally
>>> I think it would be best if our 2 proof-of-concepts could share that first
>>> grammar.  IMO that would make the differences between the 2 approaches more
>>> apparent.  I will push those changes soon.
>>>
>>
>> Ok, I can try and work on a PoC for the decorator-based approach. Not
>> sure what the outcome will be, as your's, my understanding of it is roughly
>> vague and high-level. But if it fails we can maybe settle for the Antlr3
>> approach with the better feeling of having investigated the alternative.
>>
>> Can you let me know when you have pushed your stuff? What does it do,
>> render the query below as SQL?
>>
>> It is not complete yet.  But it covers most cases.
>>>
>>>
>>> On Tue, Jun 9, 2015 at 10:47 AM Steve Ebersole <steve at hibernate.org>
>>> wrote:
>>>
>>>> On Tue, Jun 9, 2015 at 10:14 AM Gunnar Morling <gunnar at hibernate.org>
>>>> wrote:
>>>>
>>>> Yes, indeed I cheated here a bit. Probably it should be the following
>>>>> instead:
>>>>>
>>>>>       [DOT] ---> AttributeReference("<gen:1>", "code")
>>>>>         [DOT]
>>>>>           [DOT]
>>>>>             [IDENT, "c"]
>>>>>             [IDENT, "headquarters"]
>>>>>           [IDENT, "state"]
>>>>>         [IDENT, "code"]
>>>>>
>>>>
>>>> How do you identify one DOT as referring to something else versus any
>>>> of the other DOTs?
>>>>
>>>>
>>>> Or maybe something like:
>>>>>
>>>>>     [SELECTION_PARTICLE] ---> AttributeReference("<gen:1>", "code")
>>>>>       [DOT]
>>>>>         [DOT]
>>>>>           [DOT]
>>>>>             [IDENT, "c"]
>>>>>             [IDENT, "headquarters"]
>>>>>           [IDENT, "state"]
>>>>>         [IDENT, "code"]
>>>>>
>>>>> Where SELECTION_PARTICLE would be an abstract representation of
>>>>> anything that can be selected (attribute ref, Java literal ref etc.) and
>>>>> the decorator element added in a later pass would specify its actual
>>>>> semantics based on the alias definitions etc. discovered before.
>>>>>
>>>>> Bottom line being, that decorators providing semantics are attached to
>>>>> the nodes of the parse tree based on information gathered in previous
>>>>> passes.
>>>>>
>>>>
>>>> And what does that look like in real, practical terms?   That's what
>>>> concerns me :)  I don't know, and you are just speaking in generalities.
>>>> So what does that look like in practice?
>>>>
>>>>
>>>> Not into the tree itself, but we can encode that semantic resolution
>>>>> into decorators (node attachments).
>>>>>
>>>>
>>>> Again, what do these "node attachments" look like in  practice?  I have
>>>> zero clue and based on my discussions with Antlr folks its not pretty.
>>>> Maybe I misunderstand.  But if you are proposing this approach, I would
>>>> think you should have an idea of how it would look practically-speaking :)
>>>>  Maybe this is the way to go, I just need to see what this looks like.
>>>>
>>>>
>>>> Yes, they would deal with [[DOT][IDENT]] nodes but would benefit from
>>>>> semantic decorators attached previously. During rendering I would expect
>>>>> mainly those attachments to be of importance for the query creation.
>>>>>
>>>>> Admittedly, that's all quite "high level", but so far it seems doable
>>>>> to me in principle. It doesn't answer of course actual tree transformations
>>>>> such as (x + 0) -> x. I am not sure whether there are cases like this.
>>>>>
>>>>
>>>> Yes it is all extremely high-level.  That is my concern.  Principle and
>>>> practice are often 2 very different things.
>>>>
>>>> I plan on spending some time taking my hibernate-antlr4-poc project and
>>>> expanding it specifically to try the "second grammar" approach and see what
>>>> practical difficulties that shakes out.  Would you be willing to do the
>>>> same for this decorated approach?  Then we'd have concrete stuff to compare
>>>> and base a decision on.
>>>>
>>>> Also, `(x + 0) -> x` is actually a quite simple case.   Ours is much
>>>> more complicated.  In analyzing `c.headquarters.state.code` in the SELECT
>>>> clause we need a few things to happen in a few different parts of the
>>>> tree.  We need:
>>>> 1) `c.headquarters.state` to be transformed into 2 "implicit joins" in
>>>> the FROM clause
>>>> 2) we need to replace `c.headquarters.state.code` as
>>>> `{implicit-alias}.code` in the SELECT
>>>> 3) register `c.headquarters` and `c.headquarters.state` as implicit
>>>> join paths (additional implicit joins using these paths should re-use the
>>>> same joins).
>>>>
>>>