Re: [teiid-dev] Translators in the "vdb.xml" file.
by Barry Lafond
Thx for helping to clear more of this up for me.... just a couple of more points.
============================================
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB?
> The xml is not custom - the property sets may be (<property name="foo" value="bar"/> vs. <property name="foo1" value="bar"/>). I'm not sure what your perspective is with the worthless comment. Is a -ds.xml worthless without the corresponding .rar?
BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server.
>Yes they are both intended for AS. I'm not sure though if that has anything to do with your "worthless" observation?
BML: For Translators to work, there has to be a Translator JAR available AND a corresponding -tl.xml file right? It seems you are suggesting that placing only the -tl.xml inside our VDB artifact will make it transportable across servers enviroments? What about the required JAR? Wouldn't that have to be included in the VDB also to make it transportable?
============================================
7)
> "its just not realistic to think of physical models not being intrinsically linked to their source system via their translator".
BML: Well, as a result of our JDBC Import, we do persist DB-specific info in our models which can be viewed as a loose reference to a type of Translator. Because we are attempting to develop/maintain the notion of "design-time" and "run-time" features (WRT Teiid Server) for 7.0 the selection of a "Translator" will not be performed until a runtime operation is initiated (via Preview Data or Execute VDB). When that "does" happen and a "translator-name" is needed for either a VDB or the "Preview" VDB, then Designer will attempt to match one of the existing "translators" or ask the user to select one. We have no plans to link a relational model to a specific translator, other than through the "source" element's "translator-name" property of the vdb.xml.
Barry
----- Original Message -----
From: "Steven Hawkins" <shawkins(a)redhat.com>
To: "Barry Lafond" <blafond(a)redhat.com>
Sent: Tuesday, May 25, 2010 4:28:50 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
Some good old fashioned top posting:
> BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server.
Yes they are both intended for AS. I'm not sure though if that has anything to do with your "worthless" observation?
> BML: I don't know what a translator-specific property set is. Last time we actually creating anything via Teiid AdminApi was with ConnectionFactory's.
Based upon your other answers you seem to now know the answer, but just in case do you still need clarification?
> BML: What I'm taking away from your proposal is to allow Designer to find a matching "Translator" for a given Source Model and place it's name in a VDB.XML. And... to optionaly embed that equivalent "translator" properties into the VDB.XML so translators don't have to be managed as global entities like Connection Factories and Data Sources (via JON/JOPR) ?
That is correct.
> BML: Your reponse to 5) below identified a number of properties dealing with (IMJ) very "non-Designer" translator properties... or properties that are not viewed from a modeling perspective and have NOT been exposed to users. Majority of these properties appear to define/change the more detailed behavior of a DB rather than the structure of the data.
That is correct, the properties do not change the structure of the data (namely because the structure of teiid's data doesn't change at runtime). Only dynamic vdb import settings change the structure of the data from a translator perspective. Translator properties do typically change what data is being returned through.
> BML: referring to my response to 4) above, these properties all seem to apply more to DB behavior than to Model Structure. We could add additional properties to our Relational Models, but at a cost. We could an extension model, but it's really not flexible enough (i.e. we would need to allow using multiple extension models to make it work). Adding properties to EMF is a bear.
I'm not advocating that we do, just merely pointing out that we could (and at one point did start down that path with things like model support metadata). From my point of view there is not a hard line between structural metadata and behavioral metadata.
> BML: I have looked at some of your Translator Factories and I see the methods for managing DB-specific functions, data type conversions, etc.... so I'm guessing this is what you are referring to as "metadata is tailored for the translator"??
Yes the Oracle/SQL Server will use nativetype to change behavior. Salesforce, LDAP, and XML/Relational will not work with the appropriate extension metadata.
> BML: As a result of Import JDBC, we do convert datatypes (when needed) into our Metamodel (Built-in dtypes?) and reference a "nativeType". But we don't maintain any other info that references "What JDBC Flavor" a Relational Model is. If you are expecting Designer to do this, then we need to talk. Right now "free standing metadata is the intent of Teiid Designer" WRT Relational Models. The only caveate is the optional "JDBC Import info".
I know that you don't maintain a direct reference to where the metadata came from, but if it did come from Oracle for example it can only really be expected to work with the Oracle translator. Just ask Warren how well pushdown works, even if you have extremely similar schemas, and change the source system from one database to the next. It's fine that you think of metadata as "free standing" from a "design-time" perspective, but from any path that includes execution its just not realistic to think of physical models not being intrinsically linked to their source system via their translator.
----- Original Message -----
From: "Barry Lafond" <blafond(a)redhat.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Sent: Tuesday, May 25, 2010 4:01:12 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
----- Original Message -----
From: "Barry Lafond" <blafond(a)redhat.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org, "Van Halbert" <vhalbert(a)redhat.com>
Sent: Tuesday, May 25, 2010 2:20:41 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
geesh......... here goes....some random thoughts and questions....
============================================
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB?
> The xml is not custom - the property sets may be (<property name="foo" value="bar"/> vs. <property name="foo1" value="bar"/>). I'm not sure what your perspective is with the worthless comment. Is a -ds.xml worthless without the corresponding .rar?
BML: One difference is when we are using Teiid to create a data source, we aren't transporting it in the VDB. Both .rar & -ds.xml are ONLY on the server.
============================================
2) Designer is not intended to be a Translator creator/editor framework.
> Nor are you being asked to be. You are already creating translator specific property sets, all we're saying is that there's a better place for that information.
BML: I don't know what a translator-specific property set is. Last time we actually creating anything via Teiid AdminApi was with ConnectionFactory's.
============================================
3) Isn't the CDK providing functionality for creating/editing Translators for custom connectors.....and isn't it the primary tool for maintaining all global "connector" entities specific to Teiid?
> I'm not even sure what you are asking here, see the answer to 1) and 2) you may have the wrong idea about what's being proposed.
BML: What I'm taking away from your proposal is to allow Designer to find a matching "Translator" for a given Source Model and place it's name in a VDB.XML. And... to optionaly embed that equivalent "translator" properties into the VDB.XML so translators don't have to be managed as global entities like Connection Factories and Data Sources (via JON/JOPR) ?
============================================
4) From a Designer perspective, the user really only sees/knows about Data Source connection info like: driverClass, vendor, version, databaseName, URL, username and password. That's all the info they have, or required to have to import JDBC.
> You may not require a translator as this point, but you are implicitly creating metadata that will function with the corresponding translator.
BML: Your reponse to 5) below identified a number of properties dealing with (IMJ) very "non-Designer" translator properties... or properties that are not viewed from a modeling perspective and have NOT been exposed to users. Majority of these properties appear to define/change the more detailed behavior of a DB rather than the structure of the data.
============================================
5) Not sure how Designer is moving down the path where "physical model metadata is tied directly to the translator used". We import JDBC metadata and "tweak" some datatypes to match our relational metamodel datatypes, but basically the Relational Source model is a structural representation (with properties) right? If the "translator" properties include items such as xa-capable, immutable, max-results-rows, and execution-factory-class, I don't really see much in the way physical model metadata. I did look at some of your ExecutionFactory classes and I see lots of Database-specific "translation" logic/constants, but none of this gets exposed to Designer, right?
> Most definitely you have moved down that path. Any of the non-JDBC importers rely on extension metadata that is useful only to its corresponding translator. In some cases base metadata is overloaded, so for example an XML/Relational table may have the name in source of '/' - which conveys the XPath and is completely different than how name in source is used for JDBC. As for the base properties you mention, xa-capable will probably go away. There is only 1 case where it's even useful and that's to compensate for a bug in the mysql driver. Likewise for immutable, now that we don't care from a translator perspective whether we are transaction aware or not, immutable as a translator property isn't useful. max-results-rows could be done differently - it is typically only a development time aid and for most connectors doesn't actually prevent a runaway query. It just limits how much Teiid processes.
As for what gets "exposed" to designer. Directly nothing, since you don't have any dependencies on that logic. However indirectly as stated in the response to 4) the metadata is tailored for the translator.
The remaining translator properties fall into four camps:
properties that should not change from environment to environment - text encodings, jdbc trim strings, jdbc databasetimezone, etc. These properties could have just as easily been seen as additional metadata on the model
properties that are possibly misplaced - xml logRequestResponseDocs (could be based upon a specific log context)
properties that are from legacy features - jdbc usecommentsinsourcequery
properties that are related to performance - jdbc fetchSize, jdbc useBindVariables
Only the last set have any expectation of being needed/changing between environments in such a way as vdb redelopment seems like overkill. In both of these JDBC cases we may be doing something wrong here. Most applications, see hibernate for example, will just always generate prepared statements. At the very least we should change the default to true and possiblely even drop the ability to set the value. Also, fetchSize is being applied to all statements regardless of the size of the resultset, which makes it extremely dangerous to use when there is variability across source queries.
BML: referring to my response to 4) above, these properties all seem to apply more to DB behavior than to Model Structure. We could add additional properties to our Relational Models, but at a cost. We could an extension model, but it's really not flexible enough (i.e. we would need to allow using multiple extension models to make it work). Adding properties to EMF is a bear.
BML: I have looked at some of your Translator Factories and I see the methods for managing DB-specific functions, data type conversions, etc.... so I'm guessing this is what you are referring to as "metadata is tailored for the translator"??
============================================
6) DELETED
============================================
7) .. your statement "... native types, column names, etc., are not the same across all source types." Can you give me an example? ..............not sure what you are expecting Designer to do with these changes? I can see a "dynamic vdb" handling it, but our Relational Source models are instances or views of a specific DB schema (or other data source type). I don't think we have the notion of a "flexible" column name?
> I'm not suggesting that you need a notion of a flexible column name / native type (although with non-homogeneous databases in a multisource mode we do have some problems). A good example would be that an Oracle native type of date is semantically different than a SQL Server date, which can be seen in the initial type mapping (Oracle will be to our timestamp). Having a model imported from Oracle implies that it should only be used with the Oracle translator. You may have thought of all this only from a modeling perspective - I've imported stuff and now I have a bunch of metadata. Indeed in that view of the world the metadata is completely detached from it's execution. However free standing metadata is not really the intent of Teiid Designer.
BML: As a result of Import JDBC, we do convert datatypes (when needed) into our Metamodel (Built-in dtypes?) and reference a "nativeType". But we don't maintain any other info that references "What JDBC Flavor" a Relational Model is. If you are expecting Designer to do this, then we need to talk. Right now "free standing metadata is the intent of Teiid Designer" WRT Relational Models. The only caveate is the optional "JDBC Import info".
============================================
Barry
----- Original Message -----
From: "Steven Hawkins" <shawkins(a)redhat.com>
To: "Van Halbert" <vhalbert(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:31:39 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity.
In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory.
If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs.
I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries.
So to state this a different way, change your translator == change you physical model.
----- Original Message -----
From: "Van Halbert" <vhalbert(a)redhat.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release.
I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it.
On May 25, 2010, at 12:39 PM, John Verhaeg wrote:
> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote:
>
>> What I forgot mention in the earlier is, if user chooses one of the
>> default "translators", then there is not really any need to define the
>> XML fragment for the Translator at all inside the vdb.xml. Which I
>> suspect in majority of the cases. Only when user wants to alter the
>> default behavior, Teiid would need the additional translator fragment.
>
>
> So presumably there will be some property on the translators that indicates whether its a default translator?
>
> Thanks,
>
> JPAV
>
>
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Translators in the "vdb.xml" file.
by Ramesh Reddy
Taking a second look after functionality of the translator and
connections factories,
Translators are immutable across all the deployment environments. Once
defined they should never change from environment to environment.
Currently they are defined in global scope. i.e. a given translator
instance is global to the Teiid runtime instance, that is shared among
VDBs. This makes them to defined in their own deployment file.
However, if VDB need to be migrated to another environment, the user
will also need to migrate the corresponding translators. This adds
complexity in terms of migration. This can be avoided, by making the
translators VDB scoped, by defining them inside the "vdb.xml" file.
Then, they go along with VDB, so need to for migration of multiple
artifacts from environment to environment.
As per functionality this will not change any thing from what has been
discussed before. This adds minor additional changes to "vdb.xml", and
VDB, translator deployers. Translator functionality Designer would need
to designed such that they are written to vdb.xml, instead of calling
Admin API to create a translator. I will make changes to the vdb.xml's
schema file note the changes required.
Thanks. Let me know if you have any questions.
Ramesh..
16 years
Re: [teiid-dev] Translators in the "vdb.xml" file.
by Steven Hawkins
----- Original Message -----
From: "Barry Lafond" <blafond(a)redhat.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org, "Van Halbert" <vhalbert(a)redhat.com>
Sent: Tuesday, May 25, 2010 2:20:41 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
geesh......... here goes....some random thoughts and questions....
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB?
> The xml is not custom - the property sets may be (<property name="foo" value="bar"/> vs. <property name="foo1" value="bar"/>). I'm not sure what your perspective is with the worthless comment. Is a -ds.xml worthless without the corresponding .rar?
2) Designer is not intended to be a Translator creator/editor framework.
> Nor are you being asked to be. You are already creating translator specific property sets, all we're saying is that there's a better place for that information.
3) Isn't the CDK providing functionality for creating/editing Translators for custom connectors.....and isn't it the primary tool for maintaining all global "connector" entities specific to Teiid?
> I'm not even sure what you are asking here, see the answer to 1) and 2) you may have the wrong idea about what's being proposed.
4) From a Designer perspective, the user really only sees/knows about Data Source connection info like: driverClass, vendor, version, databaseName, URL, username and password. That's all the info they have, or required to have to import JDBC.
> You may not require a translator as this point, but you are implicitly creating metadata that will function with the corresponding translator.
5) Not sure how Designer is moving down the path where "physical model metadata is tied directly to the translator used". We import JDBC metadata and "tweak" some datatypes to match our relational metamodel datatypes, but basically the Relational Source model is a structural representation (with properties) right? If the "translator" properties include items such as xa-capable, immutable, max-results-rows, and execution-factory-class, I don't really see much in the way physical model metadata. I did look at some of your ExecutionFactory classes and I see lots of Database-specific "translation" logic/constants, but none of this gets exposed to Designer, right?
> Most definitely you have moved down that path. Any of the non-JDBC importers rely on extension metadata that is useful only to its corresponding translator. In some cases base metadata is overloaded, so for example an XML/Relational table may have the name in source of '/' - which conveys the XPath and is completely different than how name in source is used for JDBC. As for the base properties you mention, xa-capable will probably go away. There is only 1 case where it's even useful and that's to compensate for a bug in the mysql driver. Likewise for immutable, now that we don't care from a translator perspective whether we are transaction aware or not, immutable as a translator property isn't useful. max-results-rows could be done differently - it is typically only a development time aid and for most connectors doesn't actually prevent a runaway query. It just limits how much Teiid processes.
As for what gets "exposed" to designer. Directly nothing, since you don't have any dependencies on that logic. However indirectly as stated in the response to 4) the metadata is tailored for the translator.
The remaining translator properties fall into four camps:
properties that should not change from environment to environment - text encodings, jdbc trim strings, jdbc databasetimezone, etc. These properties could have just as easily been seen as additional metadata on the model
properties that are possibly misplaced - xml logRequestResponseDocs (could be based upon a specific log context)
properties that are from legacy features - jdbc usecommentsinsourcequery
properties that are related to performance - jdbc fetchSize, jdbc useBindVariables
Only the last set have any expectation of being needed/changing between environments in such a way as vdb redelopment seems like overkill. In both of these JDBC cases we may be doing something wrong here. Most applications, see hibernate for example, will just always generate prepared statements. At the very least we should change the default to true and possiblely even drop the ability to set the value. Also, fetchSize is being applied to all statements regardless of the size of the resultset, which makes it extremely dangerous to use when there is variability across source queries.
6) (see 5 above) Have no idea what "change your translator == change your physical model" means other than some cheer-leader reference :)
> Without trying to understand what's being presented to you, yes you could come to that conclusion.
7) .. your statement "... native types, column names, etc., are not the same across all source types." Can you give me an example? ..............not sure what you are expecting Designer to do with these changes? I can see a "dynamic vdb" handling it, but our Relational Source models are instances or views of a specific DB schema (or other data source type). I don't think we have the notion of a "flexible" column name?
> I'm not suggesting that you need a notion of a flexible column name / native type (although with non-homogeneous databases in a multisource mode we do have some problems). A good example would be that an Oracle native type of date is semantically different than a SQL Server date, which can be seen in the initial type mapping (Oracle will be to our timestamp). Having a model imported from Oracle implies that it should only be used with the Oracle translator. You may have thought of all this only from a modeling perspective - I've imported stuff and now I have a bunch of metadata. Indeed in that view of the world the metadata is completely detached from it's execution. However free standing metadata is not really the intent of Teiid Designer.
Barry
----- Original Message -----
From: "Steven Hawkins" <shawkins(a)redhat.com>
To: "Van Halbert" <vhalbert(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:31:39 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity.
In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory.
If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs.
I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries.
So to state this a different way, change your translator == change you physical model.
----- Original Message -----
From: "Van Halbert" <vhalbert(a)redhat.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release.
I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it.
On May 25, 2010, at 12:39 PM, John Verhaeg wrote:
> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote:
>
>> What I forgot mention in the earlier is, if user chooses one of the
>> default "translators", then there is not really any need to define the
>> XML fragment for the Translator at all inside the vdb.xml. Which I
>> suspect in majority of the cases. Only when user wants to alter the
>> default behavior, Teiid would need the additional translator fragment.
>
>
> So presumably there will be some property on the translators that indicates whether its a default translator?
>
> Thanks,
>
> JPAV
>
>
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Re: [teiid-dev] Translators in the "vdb.xml" file.
by Barry Lafond
geesh......... here goes....some random thoughts and questions....
1) If you are proposing embedding custom Translator XML into vdb.xml , wouldn't that info be worthless unless the JAR file came with it in the VDB?
2) Designer is not intended to be a Translator creator/editor framework.
3) Isn't the CDK providing functionality for creating/editing Translators for custom connectors.....and isn't it the primary tool for maintaining all global "connector" entities specific to Teiid?
4) From a Designer perspective, the user really only sees/knows about Data Source connection info like: driverClass, vendor, version, databaseName, URL, username and password. That's all the info they have, or required to have to import JDBC.
5) Not sure how Designer is moving down the path where "physical model metadata is tied directly to the translator used". We import JDBC metadata and "tweak" some datatypes to match our relational metamodel datatypes, but basically the Relational Source model is a structural representation (with properties) right? If the "translator" properties include items such as xa-capable, immutable, max-results-rows, and execution-factory-class, I don't really see much in the way physical model metadata. I did look at some of your ExecutionFactory classes and I see lots of Database-specific "translation" logic/constants, but none of this gets exposed to Designer, right?
6) (see 5 above) Have no idea what "change your translator == change your physical model" means other than some cheer-leader reference :)
7) .. your statement "... native types, column names, etc., are not the same across all source types." Can you give me an example? ..............not sure what you are expecting Designer to do with these changes? I can see a "dynamic vdb" handling it, but our Relational Source models are instances or views of a specific DB schema (or other data source type). I don't think we have the notion of a "flexible" column name?
Barry
----- Original Message -----
From: "Steven Hawkins" <shawkins(a)redhat.com>
To: "Van Halbert" <vhalbert(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:31:39 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity.
In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory.
If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs.
I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries.
So to state this a different way, change your translator == change you physical model.
----- Original Message -----
From: "Van Halbert" <vhalbert(a)redhat.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release.
I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it.
On May 25, 2010, at 12:39 PM, John Verhaeg wrote:
> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote:
>
>> What I forgot mention in the earlier is, if user chooses one of the
>> default "translators", then there is not really any need to define the
>> XML fragment for the Translator at all inside the vdb.xml. Which I
>> suspect in majority of the cases. Only when user wants to alter the
>> default behavior, Teiid would need the additional translator fragment.
>
>
> So presumably there will be some property on the translators that indicates whether its a default translator?
>
> Thanks,
>
> JPAV
>
>
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Re: [teiid-dev] Translators in the "vdb.xml" file.
by Steven Hawkins
> does this mean that each VDB will have its own Translators, with their own connection pools, and its own classloader, ensuring that VDBs will not share translator instances among each other?
Not quite. Translator instances will not be shared, but there is still a single clasloader for ExecutionFactories - this could be added later through OSGI support or hacked around with something like JarJar.
Translators are now typically stateless. They do not have their own connection pools - that is managed through JCA and the associated resource adapter. Pool sharing then is completely independent of the translator instance, all it is told is what jndi name to use to find its ConnectionFactory. If you want to share, then create only a single ConnectionFactory, if you don't then create multiple.
> Deploying a new version of a translator would not require ripping all previous versions out and rebuilding them from scratch, which is very tedious and requires downtime.
We'll have to validate whether just placing a new translator jar in the deploy directory will allow it to be used without a restart.
----- Original Message -----
From: "Michael Walker" <michael.walker(a)amentra.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:44:51 PM GMT -06:00 US/Canada Central
Subject: RE: [teiid-dev] Translators in the "vdb.xml" file.
Somewhat new to the translator nomenclature/architecture change...does this mean that each VDB will have its own Translators, with their own connection pools, and its own classloader, ensuring that VDBs will not share translator instances among each other?
If so, then this would be tremendously useful from a deployment perspective. You could finally deploy a VDB and not have to choose whether to use embedded binding properties or pre-existing properties created by a pre-existing VDB. Deploying a new version of a translator would not require ripping all previous versions out and rebuilding them from scratch, which is very tedious and requires downtime. You would have better guarantees about the level of service a connection pool would provide, since there's no possibility of two VDBs sharing a pool.
________________________________________
From: teiid-dev-bounces(a)lists.jboss.org [teiid-dev-bounces(a)lists.jboss.org] On Behalf Of Steven Hawkins [shawkins(a)redhat.com]
Sent: Tuesday, May 25, 2010 11:31 AM
To: Van Halbert
Cc: teiid-dev(a)lists.jboss.org
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity.
In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory.
If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs.
I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries.
So to state this a different way, change your translator == change you physical model.
----- Original Message -----
From: "Van Halbert" <vhalbert(a)redhat.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release.
I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it.
On May 25, 2010, at 12:39 PM, John Verhaeg wrote:
> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote:
>
>> What I forgot mention in the earlier is, if user chooses one of the
>> default "translators", then there is not really any need to define the
>> XML fragment for the Translator at all inside the vdb.xml. Which I
>> suspect in majority of the cases. Only when user wants to alter the
>> default behavior, Teiid would need the additional translator fragment.
>
>
> So presumably there will be some property on the translators that indicates whether its a default translator?
>
> Thanks,
>
> JPAV
>
>
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Re: [teiid-dev] Translators in the "vdb.xml" file.
by Steven Hawkins
This is a change toward simple and focused. With the environmental information held by a resource adapter connection factory the remaining properties of a translator are not expected to change. It simplifies deployment greatly to have the vdb be self contained than maintaining translators as another global entity.
In any event the deployment of a custom translator is also greatly simplified from before, and should now become as easy as placing the JAR with your ExecutionFactory in the deploy directory.
If a custom translator is changed (same name/class, but with different property definitions), then the expectation is that the developer either has considered the impact of the migration, such that the old translators property sets are compatible with the new. If a new name is given and the old custom ExecutionFactory is removed, then a forced migration would occur that requires the change of affected vdbs.
I don't necessarily see that as a limitation. We are steadily moving down the path where from dynamic vdb or even designer perspective the physical model metadata is tied directly to the translator used. This is especially true for LDAP and Salesforce, but is even true for JDBC as native types, column names, etc., are not the same across all source types. Only the loopback translator is a jack of all trades, but is of limited usefulness for anything other than basic queries.
So to state this a different way, change your translator == change you physical model.
----- Original Message -----
From: "Van Halbert" <vhalbert(a)redhat.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, May 25, 2010 1:11:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Translators in the "vdb.xml" file.
I vote the designer doesn't push the creation of a translator by adding additional translator properties to the vdb.xml file. If the user created a custom connector, they'll already be dealing with the task of deploying it to server(s). Also, if this information is stored in the vdb (call it A) and then if the original custom translator is changed (and replaced) and then vdb "A" is redeployed, don't we now have a usability issue that has to be resolved on whether to replace the existing translator properties or not. Carrying stale information in the vdb has caused us problems in the past, so I would hope we wouldn't head back down that road on the first release.
I think this first release should be simple and focused. If the usability in the next release drives it in this direction, then so be it.
On May 25, 2010, at 12:39 PM, John Verhaeg wrote:
> On May 25, 2010, at 11:21 AM, Ramesh Reddy wrote:
>
>> What I forgot mention in the earlier is, if user chooses one of the
>> default "translators", then there is not really any need to define the
>> XML fragment for the Translator at all inside the vdb.xml. Which I
>> suspect in majority of the cases. Only when user wants to alter the
>> default behavior, Teiid would need the additional translator fragment.
>
>
> So presumably there will be some property on the translators that indicates whether its a default translator?
>
> Thanks,
>
> JPAV
>
>
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Re: [teiid-dev] alternative to the text connector / translator
by Steven Hawkins
Yes, we would still support wild cards.
A more detailed text example:
There would still be a file resource adapter and an associated text translator exposing procedures, such as ResultSet(file clob) getTextFiles(string path) - note that this assumes an encoding.
To return all text files in the parent directory configured for the file resource adapter, you would execute:
call modelname.getTextFiles(*.txt)
By having the parent directory configured as a resource adapter, the translator/vdb do not need to know about absolute paths. However if you do not use the wildcard, you may still need to specify the relative file name.
Today I'm adding nested table functionality that is an alternative to our current procedural relational logic and that's more inline with features in oracle/db2. This feature allows for the use of correlated references to entries in the from clause proceeding the nested table.
For example:
select proc.* from t, table(call proc(t.col1 as param1)) proc where t.col2 = 'foo'
Note that t.col1 is being used as a named parameter to the procedure rather than our current approach, which would look something like:
select proc.* from t, proc where t.col2 = 'foo' and t.col1 = proc.param1
There are 3 major drawbacks to the second approach, first the use of defaults is not allowed (since the expectation is that values are being supplied through criteria), second null values cannot be "passed" through equality predicates, and lastly since we don't support tuple in clauses ( t.col1, t.col2 in (...) ) repeated executions are created off of a Cartesian product of the inputs. None of these issues exist with nested tables.
So what I propose adding for text extraction functionality is a texttable table function that can implicitly be a nested table with the use of the table keyword:
texttable(file COLUMNS column [, column]* [DELIMITER string] [ESCAPE string] [HEADER [int]] [SKIP int])
column := name datatype [ WIDTH int ]
If a width is specified for each column, then we'll use fixed width parsing and assume that the columns are listed positionally. Specifying a delimiter/escape/header would raise an error.
DELIMITER defaults to ','
ESCAPE defaults to '"'
If HEADER is specified, then it's argument (default of 1) indicates which line contains the column names. It is not necessary to specify SKIP lines if the header is the row preceding the data.
If SKIP is specified, then that number of rows will be skipped prior to parsing data. Should be greater than HEADER.
As a correlated nested table:
select t.* from (call modelname.getTextFiles(*.txt)) f, texttable(f.file COLUMNS string x, int y SKIP 1) t
This could be used as a user query, put in a transformation, etc.
It will process all files returned by the procedure and look for 2 columns separated by the default delimiter.
----- Original Message -----
From: "Ken Johnson" <kejohnso(a)redhat.com>
To: "Charles Mosher" <cmosher(a)redhat.com>, "teiid-dev" <teiid-dev(a)lists.jboss.org>
Sent: Monday, May 24, 2010 12:10:05 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] alternative to the text connector / translator
+1 on some kind of dynamic path/filename capability.
Charles Mosher wrote:
> And would this still support the multiple-files-in-a-directory use-case? I.e., can wild-cards be incorporated into these filenames (wherever they are stored)? Perhaps using a function as suggested would be better for this example, where logic could be used to determine the file/pathnames at query time.
>
> Best regards,
>
> Chuck
>
> ----- Original Message -----
> From: "John Doyle" <jdoyle(a)redhat.com>
> To: "Steven Hawkins" <shawkins(a)redhat.com>
> Cc: "teiid-dev" <teiid-dev(a)lists.jboss.org>
> Sent: Friday, May 21, 2010 11:11:43 AM GMT -05:00 US/Canada Eastern
> Subject: Re: [teiid-dev] alternative to the text connector / translator
>
>
> ----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
>
>
>> Assuming the use of a simple file translator/ resource adapter
>> procedure to get files, the downside to either approach is that
>> relative file names would now be stored on either the model or in a
>> function.
>>
>> I support the second, since it completely removes the need for a
>> heavy-weight text translator and is feasible before RC1.
>>
>>
>
> I like the second solution as well. It's consistent with the move made with XML. But why relative paths, why not absolute paths?
>
>
>> Steve
>>
>>
>> _______________________________________________
>> teiid-dev mailing list
>> teiid-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/teiid-dev
>>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
>
--
Ken Johnson
Sr. Product Manager
JBoss Middleware Business Unit
Red Hat, Inc
978.392.3917
ken.johnson(a)redhat.com
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
Re: [teiid-dev] alternative to the text connector / translator
by John Doyle
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
>
> Assuming the use of a simple file translator/ resource adapter
> procedure to get files, the downside to either approach is that
> relative file names would now be stored on either the model or in a
> function.
>
> I support the second, since it completely removes the need for a
> heavy-weight text translator and is feasible before RC1.
>
I like the second solution as well. It's consistent with the move made with XML. But why relative paths, why not absolute paths?
> Steve
>
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
16 years
alternative to the text connector / translator
by Steven Hawkins
Hello all,
In working through the kinks of the example with Paul in the latest milestone, it's clear that the text translator / file resource adapter functionality is as confusing as ever for a user. To simplify our approach to text we should at least remove the use of a separate descriptor file. There are two logical successors to that approach:
1. move the additional metadata onto the model as extension properties (HEADER_ROW, DELIMITER, etc.)
2. introduce a system function similar to Postgres' copy from (http://www.postgresql.org/docs/8.1/static/sql-copy.html) that would essentially reuse the connector logic, but not require a table based physical model of the files.
Assuming the use of a simple file translator/ resource adapter procedure to get files, the downside to either approach is that relative file names would now be stored on either the model or in a function.
I support the second, since it completely removes the need for a heavy-weight text translator and is feasible before RC1.
Steve
16 years
Teiid M4
by Steven Hawkins
Hello all,
Ramesh gave the all clear for tonight to be the M4 build. Not everything will be perfect with the new xml/ws logic, but it's good enough for now. I'll update the site tomorrow with the new milestone and docs.
Steve
16 years
