10:47 a.m.
I'd like to kick off a discussion concerning one of the obstacles to
getting users to embrace Teiid: you currently can't provide the
metadata that Teiid requires to integrate data from multiple data
sources without the aid of Teiid Designer. The Designer is a large
footprint client application that requires particular versions of
Java, Eclipse, and several Eclipse-based libraries, and has a non-
trivial learning curve. The metadata that is produced by the Designer
and consumed by Teiid Embedded (hereafter referred to simply as the
"server", although the full Teiid server is not yet available) is
represented in a non-standard, binary format, which means:
1) Unlike most other JBoss solutions, the server cannot expose new
features that require additional metadata without first waiting for
its client tooling to support the creation and manipulation of that
metadata.
2) There is no easy path to convert the models and metadata produced
by complimentary data modeling tools (such as for ERwin or some ETL
product) to the format required by Teiid without manual reproduction
by the user or the use of proprietary, non-OSS products.
One of the suggested solutions to these issues involves changing the
format of our metadata files (i.e., index files) within VDBs to be
represented by DDL. Thus, at a minimum, a valid VDB could be created
using nothing more than a simple text editor and possibly an archive
utility. But is this really feasible? I'm sure there's a plethora of
things to consider, but here are some of the questions that
immediately come to mind:
1) Would the DDL "flavor" need to be Teiid-specific (especially for
what we currently call virtual models), and thus still require tooling
to convert metadata/DDL exported from other products to the Teiid
flavor?
2) How would we store all of the metadata we currently support (such
as references to external sources, UUIDs, and the numerous properties
we see in Designer for each model element) that isn't part of any DDL
grammar?
Thoughts?
JPAV
11:49 a.m.
Did you guys kill off the command-line vdb tool? Seems like that was one possible
direction for importing physical metadata and potentially to add views within a
lightweight tool.
Another possibly crazy idea that I would seriously consider these days is using an
embedded dsl / builder in a language like Groovy to define vdb metadata. You could take
the groovy script itself as the vdb definition or even run it to produce a compiled vdb.
If you were clever enough, the dsl could be extensible to allow metadata creation with new
metamodels. But this is probably only a valid solution if you expect a developer or other
suitably precise user to be the target.
----- Original Message ----
From: John Verhaeg <jverhaeg(a)redhat.com>
To: teiid-dev(a)lists.jboss.org
Sent: Tuesday, June 9, 2009 10:47:26 AM
Subject: [teiid-dev] VDB metadata consumed by Teiid
I'd like to kick off a discussion concerning one of the obstacles to getting
users to embrace Teiid: you currently can't provide the metadata that Teiid
requires to integrate data from multiple data sources without the aid of Teiid
Designer. The Designer is a large footprint client application that requires
particular versions of Java, Eclipse, and several Eclipse-based libraries, and
has a non-trivial learning curve. The metadata that is produced by the Designer
and consumed by Teiid Embedded (hereafter referred to simply as the "server",
although the full Teiid server is not yet available) is represented in a
non-standard, binary format, which means:
1) Unlike most other JBoss solutions, the server cannot expose new features that
require additional metadata without first waiting for its client tooling to
support the creation and manipulation of that metadata.
2) There is no easy path to convert the models and metadata produced by
complimentary data modeling tools (such as for ERwin or some ETL product) to the
format required by Teiid without manual reproduction by the user or the use of
proprietary, non-OSS products.
One of the suggested solutions to these issues involves changing the format of
our metadata files (i.e., index files) within VDBs to be represented by DDL.
Thus, at a minimum, a valid VDB could be created using nothing more than a
simple text editor and possibly an archive utility. But is this really
feasible? I'm sure there's a plethora of things to consider, but here are some
of the questions that immediately come to mind:
1) Would the DDL "flavor" need to be Teiid-specific (especially for what we
currently call virtual models), and thus still require tooling to convert
metadata/DDL exported from other products to the Teiid flavor?
2) How would we store all of the metadata we currently support (such as
references to external sources, UUIDs, and the numerous properties we see in
Designer for each model element) that isn't part of any DDL grammar?
Thoughts?
JPAV
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
1:18 p.m.
On Jun 9, 2009, at 11:49 AM, Alex Miller wrote:
Did you guys kill off the command-line vdb tool? Seems like that
was one possible direction for importing physical metadata and
potentially to add views within a lightweight tool.
We could still go this direction, except for a) it's not really that
lightweight, b) it does nothing to allow for integration with metadata
from other products, and c) I don't believe it even supported views,
just access to multiple physical sources.
Another possibly crazy idea that I would seriously consider these
days is using an embedded dsl / builder in a language like Groovy to
define vdb metadata. You could take the groovy script itself as the
vdb definition or even run it to produce a compiled vdb. If you
were clever enough, the dsl could be extensible to allow metadata
creation with new metamodels. But this is probably only a valid
solution if you expect a developer or other suitably precise user to
be the target.
I believe the developer audience you mentioned is a serious concern,
and as with your first suggestion, this also provides no type of
standard with which to leverage any sort of interoperability with
other products.
1:52 p.m.
> Did you guys kill off the command-line vdb tool? Seems like
that was one
possible direction for importing physical metadata and potentially to add views
within a lightweight tool.
We could still go this direction, except for a) it's not really that
lightweight, b) it does nothing to allow for integration with metadata from
other products, and c) I don't believe it even supported views, just access to
multiple physical sources.
a) true (although it was lighter weight than the designer),
b) true although it would be theoretically possible to create an infrastructure to import
metadata through some sort of combined connector/metadata importer infrastructure that we
always talked about
c) correct, although a relational view can almost entirely be defined by the view sql
statement. view definition was in the original design work. other views (like xml) are
far more complicated and not amenable to a command-like interface, imho. a dsl benefits
from being able to be command-like for simple stuff and a declarative definition for hairy
stuff so it can scale better (although it's easy to do dsls badly as well).
I'm not really arguing for or against any of these, just brainstorming. The benefits
of a command driven tool are that it frees you from committing to anything other than the
commands (and not vdb formats or definition languages).
> Another possibly crazy idea that I would seriously consider
these days is
using an embedded dsl / builder in a language like Groovy to define vdb
metadata. You could take the groovy script itself as the vdb definition or even
run it to produce a compiled vdb. If you were clever enough, the dsl could be
extensible to allow metadata creation with new metamodels. But this is probably
only a valid solution if you expect a developer or other suitably precise user
to be the target.
I believe the developer audience you mentioned is a serious concern, and as with
your first suggestion, this also provides no type of standard with which to
leverage any sort of interoperability with other products.
Extensible dsls would allow you or others to build plugins that could take DDL or whatever
format is appropriate.
One way to think about this problem is to what you want to nail down and live with and
what you want to allow to change in the future. Depending on your roadmap, you would make
different decisions. Some things you might want to lock down:
- commands in a tool
- vdb format -> currently this is locked down binary archive and is highly structured
and opaque
- a language describing the contents of a vdb - "language" could be SQL or some
other more flexible DSL
- an api
I suspect that to foster a community, whatever choice you make would be served by having a
pluggable way to add new metadata. Seems like that's where you started from in the
original email.
1:54 p.m.
New subject: Strange Loop submission?
By the way, I'm hoping someone will submit a Teiid talk for the Strange Loop
conference in October. Red Hat is a top sponsor of the conference and I think it's a
great opportunity for a talk.
http://thestrangeloop.com
5487
days inactive
5487
days old
4 comments
2 participants
participants (2)
-
Alex Miller -
John Verhaeg