[teiid-dev] VDB metadata consumed by Teiid

[Steven Hawkins]

----- Original Message -----
From: "Alex Miller" <alexdmiller at yahoo.com>
To: "Steven Hawkins" <shawkins at redhat.com>
Cc: teiid-dev at lists.jboss.org, "John Verhaeg" <jverhaeg at redhat.com>
Sent: Wednesday, June 10, 2009 3:48:15 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] VDB metadata consumed by Teiid




> Yes, the dqp shell was killed off - may it rest in peace with QueryBuilder.

Ah, QB (single tear rolls down my cheek). :)

> The current thinking is that Teiid is moving closer to the data layer with as 
> much standardization as possible.  As you mention below with view creation, 
> almost everything we do maps directly to SQL standards - including most of how 
> of source/external metadata is represented http://en.wikipedia.org/wiki/SQL/MED 

maybe things have changed in the last couple years but I never got the impression that anyone other than IBM care much about SQL/MED.  

No, there's still not a lot of interest in SQL/MED.  In the open source world Postgres has some actual designs toward consolidating existing integration work under SQL/MED and there's a Farrago project that has actual support, but that's about it.  Using the same ddl as SQL/MED still does make sense given how closely it fits our usage.  For example, http://farrago.sourceforge.net/design/sqlmed.html shows how it can even address the same concepts as connector types and connector bindings.

> (the processing model of SQL/MED is another discussion).  We'd be introducing 
> new, but fairly trivial, syntax for things like access patterns, update 
> procedures, and external update options.  Additional or extension metadata 
> needed by the source at runtime can be treated as a separate issue.  The 
> connector itself could be required to capture additional metadata in whatever 
> format it wants (which is currently done by the text connector), or the ddl 
> syntax could be massaged to accept attributes.

So, if you want to stick to DDL-like stuff (plus extensions), seems like that would work.

> XML document models are a different story, but I'm not sure there is much 
> interest in moving that feature forward.

Another tear... ;)

Yep, we can cry too.  The document model processing logic had already been rewritten to accommodate more mapping class transformations, criteria, and a couple of optimizations, but it's definitely past its prime.  

> 
> ----- Original Message -----
> From: "Alex Miller" 
> To: "John Verhaeg" 
> Cc: teiid-dev at lists.jboss.org
> Sent: Tuesday, June 9, 2009 1:52:14 PM GMT -06:00 US/Canada Central
> Subject: Re: [teiid-dev] VDB metadata consumed by Teiid
> 
> 
> 
> 
> > > Did you guys kill off the command-line vdb tool?  Seems like that was one 
> > possible direction for importing physical metadata and potentially to add 
> views 
> > within a lightweight tool.
> > 
> > We could still go this direction, except for a) it's not really that 
> > lightweight, b) it does nothing to allow for integration with metadata from 
> > other products, and c) I don't believe it even supported views, just access to 
> 
> > multiple physical sources.
> 
> a) true (although it was lighter weight than the designer), 
> b) true although it would be theoretically possible to create an infrastructure 
> to import metadata through some sort of combined connector/metadata importer 
> infrastructure that we always talked about 
> c) correct, although a relational view can almost entirely be defined by the 
> view sql statement.  view definition was in the original design work.  other 
> views (like xml) are far more complicated and not amenable to a command-like 
> interface, imho.  a dsl benefits from being able to be command-like for simple 
> stuff and a declarative definition for hairy stuff so it can scale better 
> (although it's easy to do dsls badly as well).
> 
> I'm not really arguing for or against any of these, just brainstorming.  The 
> benefits of a command driven tool are that it frees you from committing to 
> anything other than the commands (and not vdb formats or definition languages).
> 
> > > Another possibly crazy idea that I would seriously consider these days is 
> > using an embedded dsl / builder in a language like Groovy to define vdb 
> > metadata.  You could take the groovy script itself as the vdb definition or 
> even 
> > run it to produce a compiled vdb.  If you were clever enough, the dsl could be 
> 
> > extensible to allow metadata creation with new metamodels.  But this is 
> probably 
> > only a valid solution if you expect a developer or other suitably precise user 
> 
> > to be the target.
> > 
> > 
> > I believe the developer audience you mentioned is a serious concern, and as 
> with 
> > your first suggestion, this also provides no type of standard with which to 
> > leverage any sort of interoperability with other products.
> 
> Extensible dsls would allow you or others to build plugins that could take DDL 
> or whatever format is appropriate.
> 
> One way to think about this problem is to what you want to nail down and live 
> with and what you want to allow to change in the future.  Depending on your 
> roadmap, you would make different decisions.  Some things you might want to lock 
> down:
> - commands in a tool
> - vdb format -> currently this is locked down binary archive and is highly 
> structured and opaque
> - a language describing the contents of a vdb - "language" could be SQL or some 
> other more flexible DSL 
> - an api 
> 
> I suspect that to foster a community, whatever choice you make would be served 
> by having a pluggable way to add new metadata.  Seems like that's where you 
> started from in the original email.  
> 
> _______________________________________________
> teiid-dev mailing list
> teiid-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev