[teiid-designer-dev] [teiid-dev] Translator based Metadata

Wed May 2 12:08:52 EDT 2012

What changes in Teiid importer framework you are proposing, you mean
exposing through admin api?

I prefer we do this in 8, 9 seems to far away for product aspirations.

Ramesh..

On 05/01/2012 10:04 AM, Steven Hawkins wrote:
> Yes, my intent was to talk about "importers" that are backed by sources which can supply their metadata.  If there is no source or there is no metadata, then the functionality is most likely some sort of hand exercise or GUI wizard to guide the user toward their goal.  Even the "static" model sources (file/ws) benefit from the notion of a server based import since designer would not be responsible for holding a pre-defined version of their metadata - which may change over time.
> 
> So the big questions are when do we want refine an importer framework in Teiid, and when will it be utilized by Designer.  Are these appropriate to start in the 8 series, or are we better off coordinating in 9?
> 
> ----- Original Message -----
>> @Barry: yes and no. Technically both are source models, but the
>> translators in (1) provide more rich metadata, where as in (2) are
>> static models, are not that useful with out additional view based
>> metadata.
>>
>> Also, we do not need to replace what we have right away with new
>> ones,
>> but add the new translator based import and move towards that goal to
>> convert all as time permits.
>>
>> The basic idea is, if there is a running server, instead of Designer
>> making direct connection to the source we make use of server's
>> connection and get metadata. This can be true for jdbc, file,
>> web-service, salesforce etc. Designer already does this for preview,
>> we
>> need to push for metadata too. Once the metadata is retrieved, any
>> additional stuff like view generation (ex: xml stuff) can be done on
>> top.
>>
>> Using the translator-api in Designer does not bother me that much as
>> it
>> is public api. I know Barry was interested in hiding the physical
>> models
>> for atleast the translators in group (2), but not sure how we can
>> accomplish with so much legacy code and design habits.
>>
>> Ramesh..
>>
>> On 04/26/2012 01:14 PM, Barry Lafond wrote:
>>> Importing into Designer means a couple of things
>>>
>>> (1) Import XXXX and create Source Relational Model
>>>
>>>   * JDBC Importer
>>>   * DDL Importer (though not really a "source" since it can't be
>>>   queried)
>>>   * Salesforce Importer
>>>
>>>
>>> (2) Import XXXX and create a View Model with Tables/Procedures that
>>> query Sources.
>>> - There ARE source models created, but they are only required to
>>> define
>>> the source procedures (getTextFiles(), invoke(), etc)
>>> - We also use the source models to persist and transport the source
>>> connection/translator info from the model in designer to the VDB
>>> (needed
>>> for Preview Data too)
>>>
>>>   * Flat File Importer
>>>   * XML File Importer
>>>   * WSDL Importer
>>>
>>> So thinking that Translator-based Metadata would apply more
>>> directly to
>>> type (1) importers and not sure how applicable it would be to (2)
>>> since
>>> the "source" isn't really model, just the "views" of the data the
>>> user
>>> wishes to query.
>>>
>>> Barry
>>>
>>> ------------------------------------------------------------------------
>>>
>>>     *From: *"Steven Hawkins" <shawkins at redhat.com>
>>>     *To: *"teiid-designer-dev" <teiid-designer-dev at lists.jboss.org>
>>>     *Cc: *teiid-dev at lists.jboss.org
>>>     *Sent: *Thursday, April 26, 2012 12:30:38 PM
>>>     *Subject: *Re: [teiid-designer-dev] [teiid-dev] Translator
>>>     based
>>>     Metadata
>>>
>>>     Moving over to designer dev.  There are several integration
>>>     options
>>>     to get most of the importer code out of Designer:
>>>
>>>     1. Reuse translator libraries in Designer (but increases code
>>>     dependency on Teiid).
>>>     2. Require a running server and use the dynamic vdb approach to
>>>     capture a physical model.
>>>       a. serialize the metadata objects for consumption by Designer
>>>     (same as 1).
>>>       b. as Ramesh suggests create a ddl dump for consumption by
>>>       Designer.
>>>
>>>     Either of the above approaches do provide a path toward a
>>>     Designer
>>>     integration of custom Translators that does not also involve
>>>     necessarily coding a custom importer GUI.
>>>
>>>     The hold ups for a redesign are that the import code,
>>>     especially for
>>>     JDBC, is quite expansive and intricately tied to the GUI.  Also
>>>     Teiid Translators to not advertise their import properties and
>>>     Translators just provides a single getMetadata call that gets
>>>     the
>>>     full form or all metadata.  Translators do not yet provide more
>>>     interrogative methods that are better fits in a multi-step
>>>     wizard -
>>>     such as just providing schema/table/procedure names for
>>>     multi-select
>>>     filtering prior to full import.
>>>
>>>     At one point it was also proposed to get rid of or hide
>>>     physical
>>>     models.  I somewhat agree with that, however designer still
>>>     needs to
>>>     display the source metadata in some view and there are still
>>>     situations that require hand-crafted metadata as a workaround
>>>     (ldap,
>>>     using sequences, etc.).  Also for performance reasons the
>>>     source
>>>     metadata will need to be cached somewhere (which can also
>>>     include
>>>     being supplied to Teiid as an index file or DDL).  A possible
>>>     compromise is that VDBs in Teiid 8+ do have the ability to
>>>     layer in
>>>     metadata, so it is possible to let the bulk of the metadata be
>>>     defined by "import foreign schema", but also to layer in
>>>     additional
>>>     metadata as needed.
>>>
>>>     ----- Original Message -----
>>>     > I have a proposal on how to import translator based medadata
>>>     > in the
>>>     > Teiid Designer instead of creating a custom importer each
>>>     > time we add
>>>     > a
>>>     > new source.
>>>     >
>>>     > To support dynamic VDBs, Teiid team creates metadata
>>>     > facilities for
>>>     > most
>>>     > sources, it is prudent to use same facility in the Designer
>>>     > for
>>>     > importing metadata rather than creating another facility.
>>>     > Currently
>>>     > it
>>>     > is determined that "DatabaseMetadata" object is not
>>>     > sufficient and
>>>     > Teiid
>>>     > does not expose any Admin API method for this.
>>>     >
>>>     > In 8.0, Teiid defined DDL based matadata importer for a
>>>     > translator,
>>>     > if
>>>     > we write a reverse of this, where one can export DDL from the
>>>     > imported
>>>     > metadata then Teiid can provide a Admin API method for this,
>>>     > thus
>>>     > allowing the Teiid Designer to hook into Teiid and retrieve
>>>     > the
>>>     > metadata. Any thoughts?
>>>     >
>>>     > Thank you.
>>>     >
>>>     > Ramesh..
>>>     >
>>>     > _______________________________________________
>>>     > teiid-dev mailing list
>>>     > teiid-dev at lists.jboss.org
>>>     > https://lists.jboss.org/mailman/listinfo/teiid-dev
>>>     >
>>>     _______________________________________________
>>>     teiid-designer-dev mailing list
>>>     teiid-designer-dev at lists.jboss.org
>>>     https://lists.jboss.org/mailman/listinfo/teiid-designer-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> teiid-dev mailing list
>>> teiid-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/teiid-dev
>> _______________________________________________
>> teiid-dev mailing list
>> teiid-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/teiid-dev
>>