[teiid-dev] Connector Based Metadata

[Steven Hawkins]

> Do migrating users of old implementations with their own connectors have to implement this new API, or would we make it optional because they don't use VDB free mode?

Like all of the other advanced connector scenarios, this would be an optional api.  

> How many ways are we going to allow the creation of metadata, and how many is too many?  Can we rationalize?

We have exactly 1 if you are referring to persistent metadata.  If you are referring to the inputs to persistent metadata, then we have 1 for each of the importer types.  Adding a facility for connectors to expose metadata is not attempting to replace our persistent form, it's more analogous to standardizing importers.  Another conceptual model for this work is to think about the connector api as a feature full toolkit for implementing JDBC - with full database processing constructs - that dove tails into a our robust and high-performance integration story, rather than seeing connectors as being something solely for data access.

> Does this really satisfy the needs of the majority of developers?  

For developers who just need simple integration and drop in usage of select non-releational sources, this is the easiest and best path that we can present.

> Is this really an easy entry point that leads to fuller adoption?

>From the rest of the dev's perspective all that matters is that it leads to adoption.  Whether people need full Teiid is up to them. 

----- Original Message -----
From: "John Doyle" <jdoyle at redhat.com>
To: "Ramesh Reddy" <rareddy at redhat.com>
Cc: "teiid-dev" <teiid-dev at lists.jboss.org>
Sent: Wednesday, June 17, 2009 1:19:13 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Connector Based Metadata


----- "Ramesh Reddy" <rareddy at redhat.com> wrote:

> On Wed, 2009-06-17 at 11:47 -0400, John Doyle wrote:
> > If I understand the proposal, the connector API would be expanded
> so
> > the connector can provide the metadata of the source system via an
> > object representation akin to JDBC metadata.  This would allow a
> user
> > of Teiid embedded to create bindings to source systems and query
> those
> > systems without any model or VDB.  Users would be able to submit
> > queries to Teiid that joined across the source systems much like
> they
> > would write in a transformation for a view model.  Is all of this
> > correct?
> > 
> Yes, that is my understand as well.
> 
> > Unless I'm missing some additional utility of this approach, I
> don't
> > really see much value in this approach.  On the contrary, I think
> it
> > undermines the value of Teiid by offering federation without much
> > abstraction.  I don't see how this leads to application code that
> > significantly less brittle than what users would write without us,
> and
> > I think it leads users down a dead end. Much of the code they
> create
> > around this approach would likely be thrown away were they to adopt
> > full usage of Teiid.
> > 
> Can you explain why you think there is additional utility? Are you
> thinking for example taking text connector, we need a utility to
> convert
> the def file into a metadata format? or like DDL for JDBC connector
> to
> metadata format?
> 
> What Teiid does is Federation at its core, abstraction is layer above
> that which provides us to define much more manageable schema
> independent
> of sources. You are correct on that respect. But, I would disagree,
> that
> there is no value. The code is no more brittle than what they would
> have
> written themselves, however the they would not have federation. That
> is
> what we fill in, with very easy processing model.
> 
> Whether they throw away or not, my hunch is this will satisfy the
> needs
> of majority of the developers, and it is quick way somebody can see
> the
> benefits of Teiid based data integration.

I didn't say there was no value, I do see how this satisfies a simple use case.  I just think we need to consider all of the implications of this and balance the value of the feature against the trade-offs.  

Do migrating users of old implementations with their own connectors have to implement this new API, or would we make it optional because they don't use VDB free mode?

How many ways are we going to allow the creation of metadata, and how many is too many?  Can we rationalize?

Does this really satisfy the needs of the majority of developers?  

Is this really an easy entry point that leads to fuller adoption?
  
> 
> > I also think that by doing this now, we are potentially missing an
> > opportunity to improve our integration story for community members
> who
> > might want to integrate their products with Teiid.  There has been
> a
> > longstanding desire to reduce the effort required to create an
> > importer, and there is also a requirement for alternative tooling.
> > There are a lot of balls up in the air right now and I think we
> should
> > take this opportunity to look at the all of APIs that we are going
> to
> > expose and rationalize them as much as possible.
> > 
> 
> This is not about tooling, this about alternative ways to defining
> the
> metadata to be used by Teiid, independent of any tooling.

A subset of the metadata, right?

> 
> > My preferred approach would be to answer the big question first. 
> What
> > is our canonical representation of metadata?  Is it DDL, or
> something
> > else?  Once we've done that, then we should define the APIs and
> tools
> > to create the metadata, and the hooks to provide it. 
> These two are separate concerns, granted they will compliment each
> other. We are committed for defining this, there is already
> discussion
> happening in the community as to how we want to approach this.
> 
> What we want to do now is define the runtime model we need for the
> engine, then whatever the persistent form may that be DDL or SQL/MED
> or
> XML will defined in coming releases, that produces this runtime
> model.

I shouldn't have referred to DDL, I was thinking about the runtime model.  But don't we have one already, is redefining it part of this?  Won't the resolutions to these two concerns compliment each other better if they are resolved together?   
> 
> _______________________________________________
> teiid-dev mailing list
> teiid-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev