[teiid-designer-dev] [teiid-dev] [Teiid Designer] - Architecture for Access to multiple Teiid Runtimes

[phantomjinx]

>> Teiid has introduced a query syntax tree that it uses for
>> transformations. I think it difficult not
>> to make that available to clients wishing to create queries
>> consistent with teiid's syntax
>> (especially given that a partial api in the teiid-api library of the
>> syntax tree is already
>> provided). 

> The major hurdle in consolidating the two apis has been the Designer usage of the engine api.  

Would you mind elaborating on this as not sure as to what you are referring. As far as the designer
code is concerned, the prime use of the teiid engine has been:
* Syntax for transformation statements
* Visitor use, most notably for the SQL Editor, which appears to offer up all possible sytnax elements
* Validator use
* XML Storage mapping for the XML file translator

Should designer not be using these things?


> The secondary issue is the build-up of logic on the engine api that has nothing to do with the needs of translator developers and should
> not be exposed via a public api. 

Agreed, which is why interfaces are better to expose than implementation classes. The latter cannot
be exported at all then, ensuring such internal api is never exposed.


> Given Designer's willingness to massively change it's usage of the Teiid engine language objects we can revisit consolidating them on the Teiid side. 

Can you explain this further as I am not up to speed on discussions that may have preceded this?


> However I am not a fan of proliferating a lot of interfaces into Teiid at the language object level.

Interfaces offer a better method of exposing a standard api that allows internal logic of
implementing classes to be modified but ensures client contracts remain unaffected, in my
experience. As I mentioned previously, it would also be a good first step for uniting the two
different sytnax trees that exist in teiid, ie. the one in teiid-api and the one in teiid-engine.


> There is also the issue of the QueryMetadataInterface and the implementation of the metadata identifier object.  I would be absolutely unwilling to expose that paradigm at the 
> translator level which directly uses the metadata api instead.  You would have to promote yet another set of interfaces for the metadata objects.

So, in order to get everything working I had to address the QueryMetadataInterface problem. My
workaround was again a delegating class and heavily cutdown interface:

https://github.com/Teiid-Designer/teiid-designer/blob/master/plugins/org.teiid.designer.spi/src/org/teiid/designer/query/metadata/IQueryMetadataInterface.java

I can understand not wanting to expose everything and, as you can see my the commented-out code, it
is not a perfect workaround either. I agree, designer should not need to access this interface at
all but most of the teiid validators and resolver use it in their method signatures hence its
exposure. Could the validators and resolver be modified to take an object that could be
satisfactorily exposed, which could be 'unwrapped' to produce the QueryMetadataInterface internally,
eg. a proxy?


> For Designer to eventually support better comment/whitespace preservation, incremental parsing, etc. some split in our implementations may be needed - otherwise you'd have to push that logic to our side (so we'd have to update our
> language objects to generally include comments, line numbers, make parser modifications, etc.).

Could well be true but think that's for a different discussion. Being 'display' and 'formatting'
subjects, I would agree with you that there is no real place for those. However, provision for
callbacks in visitors would allow such things to be inserted into the syntax tree but still leverage
the teiid syntax visitors. To understand what I mean, see:

http://github.com/Teiid-Designer/teiid-designer/blob/master/plugins/org.teiid.designer.spi/src/org/teiid/designer/query/sql/ISQLStringVisitorCallback.java
http://github.com/Teiid-Designer/teiid-designer/blob/master/plugins/teiid/org.teiid.7-7-2/src/org/teiid772/sql/impl/visitor/CallbackSQLStringVisitorImpl.java


> An intermediate alternative step would be to move Designer off of our language objects and onto a more stable/lower level AST api (just a modified token chain out of JavaCC).  You'd be able to reuse our parser and we'd have to think > our way though the other usage - for example the major hurdle with a token approach assuming that you are reusing our parser/resolving logic is associating the resolved object reference / type info with the AST position.  

Doesn't that mean forking the resolver, validators, visitors (assuming they do not provide our
preferred logic) and require updating all designer logic relating to checking 'instanceofs',
'casting' etc... Ultimately, that would give us our own sytnax tree, which would allow an SQL
statement string to be delivered to teiid but we would lose quite a lot, wouldn't we?


> In any case I've never liked the interaction of placing a lot of designer/display concerns onto the Teiid side (such as requiring retaining parts of the parse string for display and the notion of modifying the display mode, etc.).  
> At the least for the runtime, there can be an unnecessary increase in the memory footprint of the language objects.

ok. Can open a fresh discussion about that.

>> If designer was to use another syntax tree, it would be
>> impossible to ensure consistency
>> between the two syntaxes, (kinda like designer creating ruby and
>> teiid processing it as python).

> That is not a good analogy.  The eclipse tooling already obviously has lots of support for dealing with many languages - without the requirement of having a language api provided to them (which is true for Java support itself since
> JDT uses an Eclipse specific AST api).  There is also quite a significant amount of code in the datatools area that attempts to provide Eclipse with SQL support.  I guess my point could be summed up as perhaps we can look for a
> better long term decoupling rather than simply refining/codifying the current dependencies.

My analogy sought to highlight that decoupling could easily render designer delivering to teiid a
dialect of SQL that is incompatible. If both used java then there would be no problem as java has a
standard syntax. However, as I understand it teiid is based on the SQL-92 standard. My worry is the
'based on' ... Can we be certain that using a different SQL library will produce SQL that teiid
likes and support or teiid SQL keywords? I really an not up to speed on this so your indulgence is
sought ;-)

My point is that we can only decouple if we can be sure that the SQL statement sent to teiid is the
same dialect as expected by teiid. For example, both Oracle and MySQL support SQL but have slightly
different dialects.


A final thought I had over the weekend, which may be really daft ... Designer currently included the
built teiid libraries in the client-runtime plugins. In may be better to include the teiid source,
which can then be patched with the current designer spi interfaces. The advantages of this are that
teiid does not have to do anything right now but can observe the introduction and impact of the
interfaces and designer gets to remove the delegating classes. Also, for designer's benefit, we can
track the *.*.x branches of the runtimes rather than have to update built jars.


Regards

PGR


----- Original Message -----
> Hi Ramesh,
> 
> Thought it better to continue this discussion on the mailing list ...
> 
> See answers in line.
> 
> > Do you know that these classes are Teiid internal classes and they
> > can change from version to
> > version? We have no intension of making these public as there is
> > little too much code.
> 
> Completely agree which is why I would suggest these classes implement
> interfaces, eg. LanguageObject
> implements ILanguageObject
> 
> The only inconsistency between teiid 7.7.1 and teiid 8.2 as far as
> the query syntax tree goes is
> that some classes have changed their hierarchy. The internals of the
> classes may have changed but
> that is uninteresting to client applications like designer. To
> illustrate I'll use the example of
> ElementSymbol:
> 
> * 275 references in the designer codebase of IElementSymbol
> 
> ** References in the teiid 8.2 plugin, providing the implementation,
> syntax factory implementation
> and class implementations of visitor interfaces
> ** References in the teiid 7.7.2 plugin, providing the
> implementation, syntax factory implementation
> and class implementations of visitor interfaces
> ** References as a method parameter, instanceof checks, casts from
> ILanguageObject or ISymbol, array
> declarations
> 
> ** References in the spi interface plugin, which would disappear if
> included in teiid
> *** References in the interface IUpdateValidator where
> implementations wrap the teiid
> UpdateValidator since the latter uses classes rather than interfaces
> *** Other spi interfaces of query syntax objects
> 
> ** Interface methods expected from IElementSymbol
> *** IGroupSymbol getGroupSymbol();
> *** void setGroupSymbol(IGroupSymbol groupSymbol);
> *** boolean isExternalReference();
> *** void setDisplayFullyQualified(boolean value);
> *** ESDisplayMode getDisplayMode();
> *** void setType(Class<?> targetType);
> *** Object getMetadataID();
> *** void setMetadataID(Object metadataID);
> 
> ** References in tests so can be discounted
> 
> Thus, designer expects an object that can be identified as an
> IElementSymbol, so it can be cast and
> provide a few methods. In a small number of cases a new
> IElementSymbol is expected to be created so
> a factory would be required.
> 
> Some of the less common syntax objects are referenced less
> frequently, in fact only in visitors. In
> this case, implementation delegate classes would not created but a
> LanguageObjectImpl was instead
> wrapping the delegate.
> 
> > When you say backport interfaces to Teiid, that amounts to same as
> > Teiid releasing the same as public api and
> > keep up with backwards compatibility. Teiid already defines another
> > set language objects in the public API, see the "teiid-api"
> > project,
> > however they are currently used only in the translator level. The
> > query engine and Designer use the
> > other internal copy. In long term, we hope to merge these two sets
> > of language objects into one that
> > are based on the "teiid-api" ones, but I do not expect that
> > happening any time real soon.
> 
> Even though the 'teiid-api' objects have not been unified with their
> engine counterparts, there is
> nevertheless an expectation that objects of this nature will be part
> of the public api due to their
> inclusion in the teiid-api library. However, it is quite right that
> such classes are subject to
> change but unlikely that their method signatures would be modified to
> the extent that a set of
> consistent interfaces could not be located in the teiid-api library.
> Both sets of current syntax
> classes (teiid-api and engine versions) could implement the
> interfaces thereby beginning the process
> of unifying them. Since the internals of these classes remain
> unimportant to designer, their
> existence as a set of interfaces and a factory is all that is
> required.
> 
> When I refer to 'backport interfaces to Teiid'. I mean that releases
> of teiid would be retrofitted
> with these interfaces to allow designer (and other applications that
> care) to have some access to
> the same query syntax tree that is being used by teiid. That way all
> client applications could
> understand and generate the same query language as the teiid server
> is expecting.
> 
> The changes in hierarchy remain an issue. I have worked around it by
> adding isFunction() and
> isExpression() methods to the ILanguageObject interface. Instead of
> doing instanceof checks, eg.
> object instanceof IExpression, in Designer, these method are called
> instead with their teiid 7.7.2
> and 8.2 implementations returning different answers.
> 
> > 
> > We have talked many times before about Designer forking these
> > necessary libraries or finding another
> > SQL language framework (DTP has one) to circumvent these issues.
> > So, in general we think this
> > feature  more complicated than creating delegates and I do not
> > believe that is the right approach.
> > 
> 
> Not sure why this feature is complex given the provision of
> interfaces. Would you mind clarifying
> this further for me?
> 
> Delegating classes is not a long term option but at this point, it
> was the only way of creating a
> class that implements a necessary interface but does not modify the
> teiid delegate class.
> 
> Teiid has introduced a query syntax tree that it uses for
> transformations. I think it difficult not
> to make that available to clients wishing to create queries
> consistent with teiid's syntax
> (especially given that a partial api in the teiid-api library of the
> syntax tree is already
> provided). If designer was to use another syntax tree, it would be
> impossible to ensure consistency
> between the two syntaxes, (kinda like designer creating ruby and
> teiid processing it as python).
> 
> 
> 
> As I am sure you know, it would be necessary for the teiid-api
> library with such interfaces to
> remain consistent and backward compatible between versions. Designer
> would only ever import the
> latest version of this api library, separately to the 7.7.2 and 8.2
> client-runtime plugins, both of
> which would then depend on it. In this way, most of the delegating
> classes would disappear.
> 
> Hopefully, that is not too opaque and sounds workable. I am sure
> further discussion is required and
> more issues need to be ironed out but at this point it seems that
> teiid only has to introduce these
> interfaces into the api and designer can remain compatible as well as
> support multiple versions of
> the client-runtime.
> 
> Your further thoughts always welcome.
> 
> PGR
> 
> --
> Paul Richardson
>