I have a proposal on how to import translator based medadata in the Teiid Designer instead of creating a custom importer each time we add a new source. To support dynamic VDBs, Teiid team creates metadata facilities for most sources, it is prudent to use same facility in the Designer for importing metadata rather than creating another facility. Currently it is determined that "DatabaseMetadata" object is not sufficient and Teiid does not expose any Admin API method for this. In 8.0, Teiid defined DDL based matadata importer for a translator, if we write a reverse of this, where one can export DDL from the imported metadata then Teiid can provide a Admin API method for this, thus allowing the Teiid Designer to hook into Teiid and retrieve the metadata. Any thoughts? Thank you. Ramesh.. _______________________________________________ teiid-dev mailing list teiid-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/teiid-dev

Importing into Designer means a couple of things (1) Import XXXX and create Source Relational Model * JDBC Importer * DDL Importer (though not really a "source" since it can't be queried) * Salesforce Importer (2) Import XXXX and create a View Model with Tables/Procedures that query Sources. - There ARE source models created, but they are only required to define the source procedures (getTextFiles(), invoke(), etc) - We also use the source models to persist and transport the source connection/translator info from the model in designer to the VDB (needed for Preview Data too) * Flat File Importer * XML File Importer * WSDL Importer So thinking that Translator-based Metadata would apply more directly to type (1) importers and not sure how applicable it would be to (2) since the "source" isn't really model, just the "views" of the data the user wishes to query. Barry ------------------------------------------------------------------------ *From: *"Steven Hawkins" <shawkins(a)redhat.com&gt; *To: *"teiid-designer-dev" <teiid-designer-dev(a)lists.jboss.org&gt; *Cc: *teiid-dev(a)lists.jboss.org *Sent: *Thursday, April 26, 2012 12:30:38 PM *Subject: *Re: [teiid-designer-dev] [teiid-dev] Translator based Metadata Moving over to designer dev. There are several integration options to get most of the importer code out of Designer: 1. Reuse translator libraries in Designer (but increases code dependency on Teiid). 2. Require a running server and use the dynamic vdb approach to capture a physical model. a. serialize the metadata objects for consumption by Designer (same as 1). b. as Ramesh suggests create a ddl dump for consumption by Designer. Either of the above approaches do provide a path toward a Designer integration of custom Translators that does not also involve necessarily coding a custom importer GUI. The hold ups for a redesign are that the import code, especially for JDBC, is quite expansive and intricately tied to the GUI. Also Teiid Translators to not advertise their import properties and Translators just provides a single getMetadata call that gets the full form or all metadata. Translators do not yet provide more interrogative methods that are better fits in a multi-step wizard - such as just providing schema/table/procedure names for multi-select filtering prior to full import. At one point it was also proposed to get rid of or hide physical models. I somewhat agree with that, however designer still needs to display the source metadata in some view and there are still situations that require hand-crafted metadata as a workaround (ldap, using sequences, etc.). Also for performance reasons the source metadata will need to be cached somewhere (which can also include being supplied to Teiid as an index file or DDL). A possible compromise is that VDBs in Teiid 8+ do have the ability to layer in metadata, so it is possible to let the bulk of the metadata be defined by "import foreign schema", but also to layer in additional metadata as needed. ----- Original Message ----- > I have a proposal on how to import translator based medadata in the > Teiid Designer instead of creating a custom importer each time we add > a > new source. > > To support dynamic VDBs, Teiid team creates metadata facilities for > most > sources, it is prudent to use same facility in the Designer for > importing metadata rather than creating another facility. Currently > it > is determined that "DatabaseMetadata" object is not sufficient and > Teiid > does not expose any Admin API method for this. > > In 8.0, Teiid defined DDL based matadata importer for a translator, > if > we write a reverse of this, where one can export DDL from the > imported > metadata then Teiid can provide a Admin API method for this, thus > allowing the Teiid Designer to hook into Teiid and retrieve the > metadata. Any thoughts? > > Thank you. > > Ramesh.. > > _______________________________________________ > teiid-dev mailing list > teiid-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/teiid-dev > _______________________________________________ teiid-designer-dev mailing list teiid-designer-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/teiid-designer-dev _______________________________________________ teiid-dev mailing list teiid-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/teiid-dev

Yes, my intent was to talk about "importers" that are backed by sources which can supply their metadata. If there is no source or there is no metadata, then the functionality is most likely some sort of hand exercise or GUI wizard to guide the user toward their goal. Even the "static" model sources (file/ws) benefit from the notion of a server based import since designer would not be responsible for holding a pre-defined version of their metadata - which may change over time. So the big questions are when do we want refine an importer framework in Teiid, and when will it be utilized by Designer. Are these appropriate to start in the 8 series, or are we better off coordinating in 9? ----- Original Message ----- > @Barry: yes and no. Technically both are source models, but the > translators in (1) provide more rich metadata, where as in (2) are > static models, are not that useful with out additional view based > metadata. > > Also, we do not need to replace what we have right away with new > ones, > but add the new translator based import and move towards that goal to > convert all as time permits. > > The basic idea is, if there is a running server, instead of Designer > making direct connection to the source we make use of server's > connection and get metadata. This can be true for jdbc, file, > web-service, salesforce etc. Designer already does this for preview, > we > need to push for metadata too. Once the metadata is retrieved, any > additional stuff like view generation (ex: xml stuff) can be done on > top. > > Using the translator-api in Designer does not bother me that much as > it > is public api. I know Barry was interested in hiding the physical > models > for atleast the translators in group (2), but not sure how we can > accomplish with so much legacy code and design habits. > > Ramesh.. > > On 04/26/2012 01:14 PM, Barry Lafond wrote: >> Importing into Designer means a couple of things >> >> (1) Import XXXX and create Source Relational Model >> >> * JDBC Importer >> * DDL Importer (though not really a "source" since it can't be >> queried) >> * Salesforce Importer >> >> >> (2) Import XXXX and create a View Model with Tables/Procedures that >> query Sources. >> - There ARE source models created, but they are only required to >> define >> the source procedures (getTextFiles(), invoke(), etc) >> - We also use the source models to persist and transport the source >> connection/translator info from the model in designer to the VDB >> (needed >> for Preview Data too) >> >> * Flat File Importer >> * XML File Importer >> * WSDL Importer >> >> So thinking that Translator-based Metadata would apply more >> directly to >> type (1) importers and not sure how applicable it would be to (2) >> since >> the "source" isn't really model, just the "views" of the data the >> user >> wishes to query. >> >> Barry >> >> ------------------------------------------------------------------------ >> >> *From: *"Steven Hawkins" <shawkins(a)redhat.com&gt; >> *To: *"teiid-designer-dev" <teiid-designer-dev(a)lists.jboss.org&gt; >> *Cc: *teiid-dev(a)lists.jboss.org >> *Sent: *Thursday, April 26, 2012 12:30:38 PM >> *Subject: *Re: [teiid-designer-dev] [teiid-dev] Translator >> based >> Metadata >> >> Moving over to designer dev. There are several integration >> options >> to get most of the importer code out of Designer: >> >> 1. Reuse translator libraries in Designer (but increases code >> dependency on Teiid). >> 2. Require a running server and use the dynamic vdb approach to >> capture a physical model. >> a. serialize the metadata objects for consumption by Designer >> (same as 1). >> b. as Ramesh suggests create a ddl dump for consumption by >> Designer. >> >> Either of the above approaches do provide a path toward a >> Designer >> integration of custom Translators that does not also involve >> necessarily coding a custom importer GUI. >> >> The hold ups for a redesign are that the import code, >> especially for >> JDBC, is quite expansive and intricately tied to the GUI. Also >> Teiid Translators to not advertise their import properties and >> Translators just provides a single getMetadata call that gets >> the >> full form or all metadata. Translators do not yet provide more >> interrogative methods that are better fits in a multi-step >> wizard - >> such as just providing schema/table/procedure names for >> multi-select >> filtering prior to full import. >> >> At one point it was also proposed to get rid of or hide >> physical >> models. I somewhat agree with that, however designer still >> needs to >> display the source metadata in some view and there are still >> situations that require hand-crafted metadata as a workaround >> (ldap, >> using sequences, etc.). Also for performance reasons the >> source >> metadata will need to be cached somewhere (which can also >> include >> being supplied to Teiid as an index file or DDL). A possible >> compromise is that VDBs in Teiid 8+ do have the ability to >> layer in >> metadata, so it is possible to let the bulk of the metadata be >> defined by "import foreign schema", but also to layer in >> additional >> metadata as needed. >> >> ----- Original Message ----- >> > I have a proposal on how to import translator based medadata >> > in the >> > Teiid Designer instead of creating a custom importer each >> > time we add >> > a >> > new source. >> > >> > To support dynamic VDBs, Teiid team creates metadata >> > facilities for >> > most >> > sources, it is prudent to use same facility in the Designer >> > for >> > importing metadata rather than creating another facility. >> > Currently >> > it >> > is determined that "DatabaseMetadata" object is not >> > sufficient and >> > Teiid >> > does not expose any Admin API method for this. >> > >> > In 8.0, Teiid defined DDL based matadata importer for a >> > translator, >> > if >> > we write a reverse of this, where one can export DDL from the >> > imported >> > metadata then Teiid can provide a Admin API method for this, >> > thus >> > allowing the Teiid Designer to hook into Teiid and retrieve >> > the >> > metadata. Any thoughts? >> > >> > Thank you. >> > >> > Ramesh.. >> > >> > _______________________________________________ >> > teiid-dev mailing list >> > teiid-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/teiid-dev >> > >> _______________________________________________ >> teiid-designer-dev mailing list >> teiid-designer-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/teiid-designer-dev >> >> >> >> >> _______________________________________________ >> teiid-dev mailing list >> teiid-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/teiid-dev > _______________________________________________ > teiid-dev mailing list > teiid-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/teiid-dev >