[teiid-dev] Translators in the "vdb.xml" file.

Barry Lafond blafond at redhat.com
Wed May 26 10:10:18 EDT 2010 

Thx for helping to clear more of this up for me.... just a couple of more points. 

============================================ 
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB? 

> The xml is not custom - the property sets may be (<property name="foo" value="bar"/> vs. <property name="foo1" value="bar"/>). I'm not sure what your perspective is with the worthless comment. Is a -ds.xml worthless without the corresponding .rar? 

BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server. 

>Yes they are both intended for AS. I'm not sure though if that has anything to do with your "worthless" observation? 

BML: For Translators to work, there has to be a Translator JAR available AND a corresponding -tl.xml file right? It seems you are suggesting that placing only the -tl.xml inside our VDB artifact will make it transportable across servers enviroments? What about the required JAR? Wouldn't that have to be included in the VDB also to make it transportable? 
============================================ 

7) 

> "its just not realistic to think of physical models not being intrinsically linked to their source system via their translator". 

BML: Well, as a result of our JDBC Import, we do persist DB-specific info in our models which can be viewed as a loose reference to a type of Translator. Because we are attempting to develop/maintain the notion of "design-time" and "run-time" features (WRT Teiid Server) for 7.0 the selection of a "Translator" will not be performed until a runtime operation is initiated (via Preview Data or Execute VDB). When that "does" happen and a "translator-name" is needed for either a VDB or the "Preview" VDB, then Designer will attempt to match one of the existing "translators" or ask the user to select one. We have no plans to link a relational model to a specific translator, other than through the "source" element's "translator-name" property of the vdb.xml. 

Barry 



----- Original Message ----- 
From: "Steven Hawkins" <shawkins at redhat.com> 
To: "Barry Lafond" <blafond at redhat.com> 
Sent: Tuesday, May 25, 2010 4:28:50 PM GMT -06:00 US/Canada Central 
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file. 

Some good old fashioned top posting: 

> BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server. 

Yes they are both intended for AS. I'm not sure though if that has anything to do with your "worthless" observation? 

> BML: I don't know what a translator-specific property set is. Last time we actually creating anything via Teiid AdminApi was with ConnectionFactory's. 

Based upon your other answers you seem to now know the answer, but just in case do you still need clarification? 

> BML: What I'm taking away from your proposal is to allow Designer to find a matching "Translator" for a given Source Model and place it's name in a VDB.XML. And... to optionaly embed that equivalent "translator" properties into the VDB.XML so translators don't have to be managed as global entities like Connection Factories and Data Sources (via JON/JOPR) ? 

That is correct. 

> BML: Your reponse to 5) below identified a number of properties dealing with (IMJ) very "non-Designer" translator properties... or properties that are not viewed from a modeling perspective and have NOT been exposed to users. Majority of these properties appear to define/change the more detailed behavior of a DB rather than the structure of the data. 

That is correct, the properties do not change the structure of the data (namely because the structure of teiid's data doesn't change at runtime). Only dynamic vdb import settings change the structure of the data from a translator perspective. Translator properties do typically change what data is being returned through. 

> BML: referring to my response to 4) above, these properties all seem to apply more to DB behavior than to Model Structure. We could add additional properties to our Relational Models, but at a cost. We could an extension model, but it's really not flexible enough (i.e. we would need to allow using multiple extension models to make it work). Adding properties to EMF is a bear. 

I'm not advocating that we do, just merely pointing out that we could (and at one point did start down that path with things like model support metadata). From my point of view there is not a hard line between structural metadata and behavioral metadata. 

> BML: I have looked at some of your Translator Factories and I see the methods for managing DB-specific functions, data type conversions, etc.... so I'm guessing this is what you are referring to as "metadata is tailored for the translator"?? 

Yes the Oracle/SQL Server will use nativetype to change behavior. Salesforce, LDAP, and XML/Relational will not work with the appropriate extension metadata. 

> BML: As a result of Import JDBC, we do convert datatypes (when needed) into our Metamodel (Built-in dtypes?) and reference a "nativeType". But we don't maintain any other info that references "What JDBC Flavor" a Relational Model is. If you are expecting Designer to do this, then we need to talk. Right now "free standing metadata is the intent of Teiid Designer" WRT Relational Models. The only caveate is the optional "JDBC Import info". 

I know that you don't maintain a direct reference to where the metadata came from, but if it did come from Oracle for example it can only really be expected to work with the Oracle translator. Just ask Warren how well pushdown works, even if you have extremely similar schemas, and change the source system from one database to the next. It's fine that you think of metadata as "free standing" from a "design-time" perspective, but from any path that includes execution its just not realistic to think of physical models not being intrinsically linked to their source system via their translator. 

----- Original Message ----- 
From: "Barry Lafond" <blafond at redhat.com> 
To: "Steven Hawkins" <shawkins at redhat.com> 
Sent: Tuesday, May 25, 2010 4:01:12 PM GMT -06:00 US/Canada Central 
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file. 



----- Original Message ----- 
From: "Barry Lafond" <blafond at redhat.com> 
To: "Steven Hawkins" <shawkins at redhat.com> 
Cc: teiid-dev at lists.jboss.org, "Van Halbert" <vhalbert at redhat.com> 
Sent: Tuesday, May 25, 2010 2:20:41 PM GMT -06:00 US/Canada Central 
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file. 


geesh......... here goes....some random thoughts and questions.... 

============================================ 
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB? 

> The xml is not custom - the property sets may be (<property name="foo" value="bar"/> vs. <property name="foo1" value="bar"/>). I'm not sure what your perspective is with the worthless comment. Is a -ds.xml worthless without the corresponding .rar? 

BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server. 
============================================ 
2) Designer is not intended to be a Translator creator/editor framework. 

> Nor are you being asked to be. You are already creating translator specific property sets, all we're saying is that there's a better place for that information. 

BML: I don't know what a translator-specific property set is. Last time we actually creating anything via Teiid AdminApi was with ConnectionFactory's. 

============================================ 

3) Isn't the CDK providing functionality for creating/editing Translators for custom connectors.....and isn't it the primary tool for maintaining all global "connector" entities specific to Teiid? 

> I'm not even sure what you are asking here, see the answer to 1) and 2) you may have the wrong idea about what's being proposed. 

BML: What I'm taking away from your proposal is to allow Designer to find a matching "Translator" for a given Source Model and place it's name in a VDB.XML. And... to optionaly embed that equivalent "translator" properties into the VDB.XML so translators don't have to be managed as global entities like Connection Factories and Data Sources (via JON/JOPR) ? 

============================================ 
4) From a Designer perspective, the user really only sees/knows about Data Source connection info like: driverClass, vendor, version, databaseName, URL, username and password. That's all the info they have, or required to have to import JDBC. 

> You may not require a translator as this point, but you are implicitly creating metadata that will function with the corresponding translator. 

BML: Your reponse to 5) below identified a number of properties dealing with (IMJ) very "non-Designer" translator properties... or properties that are not viewed from a modeling perspective and have NOT been exposed to users. Majority of these properties appear to define/change the more detailed behavior of a DB rather than the structure of the data. 

============================================ 
5) Not sure how Designer is moving down the path where "physical model metadata is tied directly to the translator used". We import JDBC metadata and "tweak" some datatypes to match our relational metamodel datatypes, but basically the Relational Source model is a structural representation (with properties) right? If the "translator" properties include items such as xa-capable, immutable, max-results-rows, and execution-factory-class, I don't really see much in the way physical model metadata. I did look at some of your ExecutionFactory classes and I see lots of Database-specific "translation" logic/constants, but none of this gets exposed to Designer, right? 

> Most definitely you have moved down that path. Any of the non-JDBC importers rely on extension metadata that is useful only to its corresponding translator. In some cases base metadata is overloaded, so for example an XML/Relational table may have the name in source of '/' - which conveys the XPath and is completely different than how name in source is used for JDBC. As for the base properties you mention, xa-capable will probably go away. There is only 1 case where it's even useful and that's to compensate for a bug in the mysql driver. Likewise for immutable, now that we don't care from a translator perspective whether we are transaction aware or not, immutable as a translator property isn't useful. max-results-rows could be done differently - it is typically only a development time aid and for most connectors doesn't actually prevent a runaway query. It just limits how much Teiid processes. 

As for what gets "exposed" to designer. Directly nothing, since you don't have any dependencies on that logic. However indirectly as stated in the response to 4) the metadata is tailored for the translator. 

The remaining translator properties fall into four camps: 
properties that should not change from environment to environment - text encodings, jdbc trim strings, jdbc databasetimezone, etc. These properties could have just as easily been seen as additional metadata on the model 
properties that are possibly misplaced - xml logRequestResponseDocs (could be based upon a specific log context) 
properties that are from legacy features - jdbc usecommentsinsourcequery 
properties that are related to performance - jdbc fetchSize, jdbc useBindVariables 

Only the last set have any expectation of being needed/changing between environments in such a way as vdb redelopment seems like overkill. In both of these JDBC cases we may be doing something wrong here. Most applications, see hibernate for example, will just always generate prepared statements. At the very least we should change the default to true and possiblely even drop the ability to set the value. Also, fetchSize is being applied to all statements regardless of the size of the resultset, which makes it extremely dangerous to use when there is variability across source queries. 

BML: referring to my response to 4) above, these properties all seem to apply more to DB behavior than to Model Structure. We could add additional properties to our Relational Models, but at a cost. We could an extension model, but it's really not flexible enough (i.e. we would need to allow using multiple extension models to make it work). Adding properties to EMF is a bear. 

BML: I have looked at some of your Translator Factories and I see the methods for managing DB-specific functions, data type conversions, etc.... so I'm guessing this is what you are referring to as "metadata is tailored for the translator"?? 

============================================ 
6) DELETED 
============================================ 
7) .. your statement "... native types, column names, etc., are not the same across all source types." Can you give me an example? ..............not sure what you are expecting Designer to do with these changes? I can see a "dynamic vdb" handling it, but our Relational Source models are instances or views of a specific DB schema (or other data source type). I don't think we have the notion of a "flexible" column name? 

> I'm not suggesting that you need a notion of a flexible column name / native type (although with non-homogeneous databases in a multisource mode we do have some problems). A good example would be that an Oracle native type of date is semantically different than a SQL Server date, which can be seen in the initial type mapping (Oracle will be to our timestamp). Having a model imported from Oracle implies that it should only be used with the Oracle translator. You may have thought of all this only from a modeling perspective - I've imported stuff and now I have a bunch of metadata. Indeed in that view of the world the metadata is completely detached from it's execution. However free standing metadata is not really the intent of Teiid Designer. 

BML: As a result of Import JDBC, we do convert datatypes (when needed) into our Metamodel (Built-in dtypes?) and reference a "nativeType". But we don't maintain any other info that references "What JDBC Flavor" a Relational Model is. If you are expecting Designer to do this, then we need to talk. Right now "free standing metadata is the intent of Teiid Designer" WRT Relational Models. The only caveate is the optional "JDBC Import info". 

============================================ 
Barry 

----- Original Message ----- 
From: "Steven Hawkins" <shawkins at redhat.com> 
To: "Van Halbert" <vhalbert at redhat.com> 
Cc: teiid-dev at lists.jboss.org 
Sent: Tuesday, May 25, 2010 1:31:39 PM GMT -06:00 US/Canada Central 
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file. 

This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity. 

In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory. 

If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs. 

I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries. 

So to state this a different way, change your translator == change you physical model. 

----- Original Message ----- 
From: "Van Halbert" <vhalbert at redhat.com> 
To: "John Verhaeg" <jverhaeg at redhat.com> 
Cc: teiid-dev at lists.jboss.org 
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central 
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file. 

I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release. 

I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it. 


On May 25, 2010, at 12:39 PM, John Verhaeg wrote: 

> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote: 
> 
>> What I forgot mention in the earlier is, if user chooses one of the 
>> default "translators", then there is not really any need to define the 
>> XML fragment for the Translator at all inside the vdb.xml. Which I 
>> suspect in majority of the cases. Only when user wants to alter the 
>> default behavior, Teiid would need the additional translator fragment. 
> 
> 
> So presumably there will be some property on the translators that indicates whether its a default translator? 
> 
> Thanks, 
> 
> JPAV 
> 
> 
> 
> 
> _______________________________________________ 
> teiid-dev mailing list 
> teiid-dev at lists.jboss.org 
> https://lists.jboss.org/mailman/listinfo/teiid-dev 

Van Halbert 
Principal Software Eng. 
Red Hat, Inc. 
------ 
vhalbert at redhat.com 






_______________________________________________ 
teiid-dev mailing list 
teiid-dev at lists.jboss.org 
https://lists.jboss.org/mailman/listinfo/teiid-dev 
_______________________________________________ 
teiid-dev mailing list 
teiid-dev at lists.jboss.org 
https://lists.jboss.org/mailman/listinfo/teiid-dev 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/teiid-dev/attachments/20100526/b21e0c9b/attachment-0001.html

More information about the teiid-dev mailing list