Re: [teiid-dev] 6.2 planning
by Steven Hawkins
----- Original Message -----
From: "John Doyle" <jdoyle(a)redhat.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: "teiid-dev" <teiid-dev(a)lists.jboss.org>
Sent: Wednesday, June 17, 2009 10:47:31 AM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] 6.2 planning
I want to discuss the proposal to modify the connector api to produce runtime metadata objects from
source metadata a bit; firstly to confirm my correct understanding and ensure that I'm understanding the full value.
If I understand the proposal, the connector API would be expanded so the connector can provide the metadata of the source system via an object representation akin to JDBC metadata. This would allow a user of Teiid embedded to create bindings to source systems and query those systems without any model or VDB. Users would be able to submit queries to Teiid that joined across the source systems much like they would write in a transformation for a view model. Is all of this correct?
- Yes that is correct.
Unless I'm missing some additional utility of this approach, I don't really see much value in this approach. On the contrary, I think it undermines the value of Teiid by offering federation without much abstraction. I don't see how this leads to application code that significantly less brittle than what users would write without us, and I think it leads users down a dead end. Much of the code they create around this approach would likely be thrown away were they to adopt full usage of Teiid.
- So you're arguing against a feature because it would lead users to adopt the full usage of Teiid? I'm not sure if I follow how that undermines Teiid. Allowing users to choose they their usage model based upon their integration needs seems more compelling than dictating a specific approach. I would also say that you're overstating the need for our abstraction. From a Java developers perspective JDO/JPA already provides a layer of abstraction between them and JDBC. To require yet another layer to perform integration is not desirable for simple integration scenarios. And it's not as if our abstraction comes for free. The most interesting scenario, an integrated updatable view, requires the use of update procedures (which have a whole host of limitations).
I also think that by doing this now, we are potentially missing an opportunity to improve our integration story for community members who might want to integrate their products with Teiid. There has been a longstanding desire to reduce the effort required to create an importer, and there is also a requirement for alternative tooling. There are a lot of balls up in the air right now and I think we should take this opportunity to look at the all of APIs that we are going to expose and rationalize them as much as possible.
My preferred approach would be to answer the big question first. What is our canonical representation of metadata? Is it DDL, or something else? Once we've done that, then we should define the APIs and tools to create the metadata, and the hooks to provide it.
- I cannot stress enough that our persistent/command form of metadata is only indirectly related to how we would want to expose it from connectors. Also if you had a JDBC DatabaseMetadata like facility available on connectors that wouldn't that address your concern about how people can standardize importers.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
> There are three main engineering goals for the 6.2 release cycle (~12
> weeks):
> To provide legacy server features to the embedded deployment, to
> provide a simple client/server runtime mode, and to provide metadata
> through connectors for a simple query mode.
>
> At a high level our requirements are:
>
> 1. Port the Membership, Authentication, Session, and Tracking
> (CommandLogging) services for use in embedded. This is
> straight-forward except for the authentication service, which will
> require treating the only source for authorization information as the
> vdb itself in the form of an associated xml file (note that we are not
> doing something new here yet, the schema for defining policies was
> introduced in 5.5.1). This is covered by TEIID-663 (with subtasks
> TEIID-664 - 667) and TEIID-668.
>
> 2. Provide a Java main that will launch the "server" with a socket
> transport. - Needs a JIRA
>
> 3. Ensure adminapi/admin shell coverage of all exposed functionality -
> the console should not be necessary for this release. There is not a
> a JIRA for this as such, but needs to be addressed as service changes
> are made.
>
> 4. Modify the connector api to produce runtime metadata objects from
> source metadata. This metadata facility will probably have a
> superficial resemblance to JDBC DatadataMetadata. The QMI and other
> engine logic will need to be modified to support this "modelless"
> query mode. - Needs a JIRA
>
> The assumption with all these is that the current server mechanisms
> for configuration and clustering are disregarded. We are specifically
> not attempting to provide a clustered server that is feature/api
> compatible with legacy MetaMatrix. This work should also have minimal
> to no impact on Designer's usage of Teiid embedded. Any new work in
> designer would be a stretch goal only.
>
> The engineering stretch goals for this release are:
>
> 1. Provide JOPR plugins for monitoring and additional adminapi
> functionality. Van has initially indicated that he would like for
> these to remain out of Teiid proper until they seem usable wrt the
> release at some point much later in the cycle.
>
> 2. Have definitive plans around Teiid owned metadata in a persistent
> form, which has already been on introduced as a topic on the
> dev-list.
>
> 3. Re-implement file and database based artifact access through JCR
> (DNA) - with an eye toward using JCR as a means to provide "cluster"
> features, but that would still be api compatible with the embedded
> implementation (JCR backed by the filesystem).
>
> 4. Refine service loading/wiring/configuration in MC or other
> consistent container strategy.
>
> 5. Allow for pass-through ddl in both the single source and
> multi-source scenarios - TEIID-669
>
> In this bucket as well is a task to provide a doc set around our
> legacy SOAP services and some proposed REST services in terms of how
> they can be implemented with JBoss projects with a loose coupling to
> Teiid. This will be covered under its own JIRA with multiple subtasks
> covering prospective services use cases.
>
> A 6.3 bucket has been created out of pushed 6.2 items and select
> issues in 6.x. This will serve as the place holder for the further
> refinement of the new simplified server approach At the end of 6.3 we
> should have re-established our clustering story, a good story around
> configuration changes, fully incorporated JOPR, and carried-forward
> any lingering server features.
>
> On the non-engineering side of things we have the requirements of:
>
> 1. Updating the server extension guide to cover things from an
> embedded/simplified perspective given the changes to command logging,
> membership domains, etc.
>
> 2. Update the connector developers guide - TEIID-104
>
> 3. Add the jdbc developers guide - TEIID-315
>
> There are stretch goals of:
>
> 1. Adding additional screen casts covering a broader set of common
> uses.
>
> 2. Add more examples to the kit, especially highlighting use of
> non-relational sources.
>
> 3. Get Teiid as a recognized source in other products/projects -
> squirrel, dtp, hibernate, etc.
>
> Note there are missing JIRAs for a lot of these items and the priority
> may not yet be assigned correctly for items identified as
> requirements. Devs and leads should feel free to help fill in gaps.
> Let me know if there is anything that has been left out or what we
> should dive into a little more detail on.
>
> Steve
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
15 years, 6 months
6.2 planning
by Steven Hawkins
There are three main engineering goals for the 6.2 release cycle (~12 weeks):
To provide legacy server features to the embedded deployment, to provide a simple client/server runtime mode, and to provide metadata through connectors for a simple query mode.
At a high level our requirements are:
1. Port the Membership, Authentication, Session, and Tracking (CommandLogging) services for use in embedded. This is straight-forward except for the authentication service, which will require treating the only source for authorization information as the vdb itself in the form of an associated xml file (note that we are not doing something new here yet, the schema for defining policies was introduced in 5.5.1). This is covered by TEIID-663 (with subtasks TEIID-664 - 667) and TEIID-668.
2. Provide a Java main that will launch the "server" with a socket transport. - Needs a JIRA
3. Ensure adminapi/admin shell coverage of all exposed functionality - the console should not be necessary for this release. There is not a a JIRA for this as such, but needs to be addressed as service changes are made.
4. Modify the connector api to produce runtime metadata objects from source metadata. This metadata facility will probably have a superficial resemblance to JDBC DatadataMetadata. The QMI and other engine logic will need to be modified to support this "modelless" query mode. - Needs a JIRA
The assumption with all these is that the current server mechanisms for configuration and clustering are disregarded. We are specifically not attempting to provide a clustered server that is feature/api compatible with legacy MetaMatrix. This work should also have minimal to no impact on Designer's usage of Teiid embedded. Any new work in designer would be a stretch goal only.
The engineering stretch goals for this release are:
1. Provide JOPR plugins for monitoring and additional adminapi functionality. Van has initially indicated that he would like for these to remain out of Teiid proper until they seem usable wrt the release at some point much later in the cycle.
2. Have definitive plans around Teiid owned metadata in a persistent form, which has already been on introduced as a topic on the dev-list.
3. Re-implement file and database based artifact access through JCR (DNA) - with an eye toward using JCR as a means to provide "cluster" features, but that would still be api compatible with the embedded implementation (JCR backed by the filesystem).
4. Refine service loading/wiring/configuration in MC or other consistent container strategy.
5. Allow for pass-through ddl in both the single source and multi-source scenarios - TEIID-669
In this bucket as well is a task to provide a doc set around our legacy SOAP services and some proposed REST services in terms of how they can be implemented with JBoss projects with a loose coupling to Teiid. This will be covered under its own JIRA with multiple subtasks covering prospective services use cases.
A 6.3 bucket has been created out of pushed 6.2 items and select issues in 6.x. This will serve as the place holder for the further refinement of the new simplified server approach At the end of 6.3 we should have re-established our clustering story, a good story around configuration changes, fully incorporated JOPR, and carried-forward any lingering server features.
On the non-engineering side of things we have the requirements of:
1. Updating the server extension guide to cover things from an embedded/simplified perspective given the changes to command logging, membership domains, etc.
2. Update the connector developers guide - TEIID-104
3. Add the jdbc developers guide - TEIID-315
There are stretch goals of:
1. Adding additional screen casts covering a broader set of common uses.
2. Add more examples to the kit, especially highlighting use of non-relational sources.
3. Get Teiid as a recognized source in other products/projects - squirrel, dtp, hibernate, etc.
Note there are missing JIRAs for a lot of these items and the priority may not yet be assigned correctly for items identified as requirements. Devs and leads should feel free to help fill in gaps. Let me know if there is anything that has been left out or what we should dive into a little more detail on.
Steve
15 years, 6 months
Re: [teiid-dev] VDB metadata consumed by Teiid
by Steven Hawkins
----- Original Message -----
From: "Alex Miller" <alexdmiller(a)yahoo.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org, "John Verhaeg" <jverhaeg(a)redhat.com>
Sent: Wednesday, June 10, 2009 3:48:15 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] VDB metadata consumed by Teiid
> Yes, the dqp shell was killed off - may it rest in peace with QueryBuilder.
Ah, QB (single tear rolls down my cheek). :)
> The current thinking is that Teiid is moving closer to the data layer with as
> much standardization as possible. As you mention below with view creation,
> almost everything we do maps directly to SQL standards - including most of how
> of source/external metadata is represented http://en.wikipedia.org/wiki/SQL/MED
maybe things have changed in the last couple years but I never got the impression that anyone other than IBM care much about SQL/MED.
No, there's still not a lot of interest in SQL/MED. In the open source world Postgres has some actual designs toward consolidating existing integration work under SQL/MED and there's a Farrago project that has actual support, but that's about it. Using the same ddl as SQL/MED still does make sense given how closely it fits our usage. For example, http://farrago.sourceforge.net/design/sqlmed.html shows how it can even address the same concepts as connector types and connector bindings.
> (the processing model of SQL/MED is another discussion). We'd be introducing
> new, but fairly trivial, syntax for things like access patterns, update
> procedures, and external update options. Additional or extension metadata
> needed by the source at runtime can be treated as a separate issue. The
> connector itself could be required to capture additional metadata in whatever
> format it wants (which is currently done by the text connector), or the ddl
> syntax could be massaged to accept attributes.
So, if you want to stick to DDL-like stuff (plus extensions), seems like that would work.
> XML document models are a different story, but I'm not sure there is much
> interest in moving that feature forward.
Another tear... ;)
Yep, we can cry too. The document model processing logic had already been rewritten to accommodate more mapping class transformations, criteria, and a couple of optimizations, but it's definitely past its prime.
>
> ----- Original Message -----
> From: "Alex Miller"
> To: "John Verhaeg"
> Cc: teiid-dev(a)lists.jboss.org
> Sent: Tuesday, June 9, 2009 1:52:14 PM GMT -06:00 US/Canada Central
> Subject: Re: [teiid-dev] VDB metadata consumed by Teiid
>
>
>
>
> > > Did you guys kill off the command-line vdb tool? Seems like that was one
> > possible direction for importing physical metadata and potentially to add
> views
> > within a lightweight tool.
> >
> > We could still go this direction, except for a) it's not really that
> > lightweight, b) it does nothing to allow for integration with metadata from
> > other products, and c) I don't believe it even supported views, just access to
>
> > multiple physical sources.
>
> a) true (although it was lighter weight than the designer),
> b) true although it would be theoretically possible to create an infrastructure
> to import metadata through some sort of combined connector/metadata importer
> infrastructure that we always talked about
> c) correct, although a relational view can almost entirely be defined by the
> view sql statement. view definition was in the original design work. other
> views (like xml) are far more complicated and not amenable to a command-like
> interface, imho. a dsl benefits from being able to be command-like for simple
> stuff and a declarative definition for hairy stuff so it can scale better
> (although it's easy to do dsls badly as well).
>
> I'm not really arguing for or against any of these, just brainstorming. The
> benefits of a command driven tool are that it frees you from committing to
> anything other than the commands (and not vdb formats or definition languages).
>
> > > Another possibly crazy idea that I would seriously consider these days is
> > using an embedded dsl / builder in a language like Groovy to define vdb
> > metadata. You could take the groovy script itself as the vdb definition or
> even
> > run it to produce a compiled vdb. If you were clever enough, the dsl could be
>
> > extensible to allow metadata creation with new metamodels. But this is
> probably
> > only a valid solution if you expect a developer or other suitably precise user
>
> > to be the target.
> >
> >
> > I believe the developer audience you mentioned is a serious concern, and as
> with
> > your first suggestion, this also provides no type of standard with which to
> > leverage any sort of interoperability with other products.
>
> Extensible dsls would allow you or others to build plugins that could take DDL
> or whatever format is appropriate.
>
> One way to think about this problem is to what you want to nail down and live
> with and what you want to allow to change in the future. Depending on your
> roadmap, you would make different decisions. Some things you might want to lock
> down:
> - commands in a tool
> - vdb format -> currently this is locked down binary archive and is highly
> structured and opaque
> - a language describing the contents of a vdb - "language" could be SQL or some
> other more flexible DSL
> - an api
>
> I suspect that to foster a community, whatever choice you make would be served
> by having a pluggable way to add new metadata. Seems like that's where you
> started from in the original email.
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
15 years, 6 months
Re: [teiid-dev] VDB metadata consumed by Teiid
by Steven Hawkins
Yes, the dqp shell was killed off - may it rest in peace with QueryBuilder.
The current thinking is that Teiid is moving closer to the data layer with as much standardization as possible. As you mention below with view creation, almost everything we do maps directly to SQL standards - including most of how of source/external metadata is represented http://en.wikipedia.org/wiki/SQL/MED (the processing model of SQL/MED is another discussion). We'd be introducing new, but fairly trivial, syntax for things like access patterns, update procedures, and external update options. Additional or extension metadata needed by the source at runtime can be treated as a separate issue. The connector itself could be required to capture additional metadata in whatever format it wants (which is currently done by the text connector), or the ddl syntax could be massaged to accept attributes.
XML document models are a different story, but I'm not sure there is much interest in moving that feature forward.
----- Original Message -----
From: "Alex Miller" <alexdmiller(a)yahoo.com>
To: "John Verhaeg" <jverhaeg(a)redhat.com>
Cc: teiid-dev(a)lists.jboss.org
Sent: Tuesday, June 9, 2009 1:52:14 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] VDB metadata consumed by Teiid
> > Did you guys kill off the command-line vdb tool? Seems like that was one
> possible direction for importing physical metadata and potentially to add views
> within a lightweight tool.
>
> We could still go this direction, except for a) it's not really that
> lightweight, b) it does nothing to allow for integration with metadata from
> other products, and c) I don't believe it even supported views, just access to
> multiple physical sources.
a) true (although it was lighter weight than the designer),
b) true although it would be theoretically possible to create an infrastructure to import metadata through some sort of combined connector/metadata importer infrastructure that we always talked about
c) correct, although a relational view can almost entirely be defined by the view sql statement. view definition was in the original design work. other views (like xml) are far more complicated and not amenable to a command-like interface, imho. a dsl benefits from being able to be command-like for simple stuff and a declarative definition for hairy stuff so it can scale better (although it's easy to do dsls badly as well).
I'm not really arguing for or against any of these, just brainstorming. The benefits of a command driven tool are that it frees you from committing to anything other than the commands (and not vdb formats or definition languages).
> > Another possibly crazy idea that I would seriously consider these days is
> using an embedded dsl / builder in a language like Groovy to define vdb
> metadata. You could take the groovy script itself as the vdb definition or even
> run it to produce a compiled vdb. If you were clever enough, the dsl could be
> extensible to allow metadata creation with new metamodels. But this is probably
> only a valid solution if you expect a developer or other suitably precise user
> to be the target.
>
>
> I believe the developer audience you mentioned is a serious concern, and as with
> your first suggestion, this also provides no type of standard with which to
> leverage any sort of interoperability with other products.
Extensible dsls would allow you or others to build plugins that could take DDL or whatever format is appropriate.
One way to think about this problem is to what you want to nail down and live with and what you want to allow to change in the future. Depending on your roadmap, you would make different decisions. Some things you might want to lock down:
- commands in a tool
- vdb format -> currently this is locked down binary archive and is highly structured and opaque
- a language describing the contents of a vdb - "language" could be SQL or some other more flexible DSL
- an api
I suspect that to foster a community, whatever choice you make would be served by having a pluggable way to add new metadata. Seems like that's where you started from in the original email.
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
15 years, 6 months
6.1 update
by Steven Hawkins
We will be performing additional testing on the 6.1 release today and if everything goes well the release will happen tomorrow.
As for the next releases, the 6.1.1 release will be incorporated with the 6.2 release. The 6.x bucket will also become the 6.3 release and will absorb anything that needs pushed from 6.2. Be sure to vote for any issues that need more immediate attention.
The Teiid Leads
15 years, 6 months
VDB metadata consumed by Teiid
by John Verhaeg
I'd like to kick off a discussion concerning one of the obstacles to
getting users to embrace Teiid: you currently can't provide the
metadata that Teiid requires to integrate data from multiple data
sources without the aid of Teiid Designer. The Designer is a large
footprint client application that requires particular versions of
Java, Eclipse, and several Eclipse-based libraries, and has a non-
trivial learning curve. The metadata that is produced by the Designer
and consumed by Teiid Embedded (hereafter referred to simply as the
"server", although the full Teiid server is not yet available) is
represented in a non-standard, binary format, which means:
1) Unlike most other JBoss solutions, the server cannot expose new
features that require additional metadata without first waiting for
its client tooling to support the creation and manipulation of that
metadata.
2) There is no easy path to convert the models and metadata produced
by complimentary data modeling tools (such as for ERwin or some ETL
product) to the format required by Teiid without manual reproduction
by the user or the use of proprietary, non-OSS products.
One of the suggested solutions to these issues involves changing the
format of our metadata files (i.e., index files) within VDBs to be
represented by DDL. Thus, at a minimum, a valid VDB could be created
using nothing more than a simple text editor and possibly an archive
utility. But is this really feasible? I'm sure there's a plethora of
things to consider, but here are some of the questions that
immediately come to mind:
1) Would the DDL "flavor" need to be Teiid-specific (especially for
what we currently call virtual models), and thus still require tooling
to convert metadata/DDL exported from other products to the Teiid
flavor?
2) How would we store all of the metadata we currently support (such
as references to external sources, UUIDs, and the numerous properties
we see in Designer for each model element) that isn't part of any DDL
grammar?
Thoughts?
JPAV
15 years, 6 months
trunk is now 6.2
by Steven Hawkins
Hello all,
Work on 6.2 can begin on trunk. There will also be a reassessment of 6.1.1 defects as to what belongs in 6.2. Any remaining 6.1.0 work should be done in the 6.1.x branch, which will ideally be finished today.
Thanks for all your efforts on 6.1,
Steve
15 years, 6 months
Re: [teiid-dev] Outstanding issues
by Steven Hawkins
What embedded deployment documentation are you referencing? Only the classpath profile indicates that jars should be included from the Teiid lib/extensions in a higher level classloader.
----- Original Message -----
From: "John Doyle" <jdoyle(a)redhat.com>
To: "Steven Hawkins" <shawkins(a)redhat.com>
Cc: "teiid-dev" <teiid-dev(a)lists.jboss.org>
Sent: Friday, June 5, 2009 3:08:22 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Outstanding issues
I do. I believe that the documentation on the Teiid site is incorrect about what jars you need to include in your classpath (the cause of the constraint violation). I sent an inquiry about that earlier today but haven't gotten a response.
Also, because of this SAX issue, I haven't successfully executed any queries with the XML connector in the M4 build. I would like to validate those changes more fully.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
> 643 has been resolved and I believe 644 is an issue with your JRE (see
> my last comment on the JIRA). I would like to still proceed with the
> release unless you know of any other issues.
>
> ----- Original Message -----
> From: "John Doyle" <jdoyle(a)redhat.com>
> To: "teiid-dev" <teiid-dev(a)lists.jboss.org>
> Sent: Friday, June 5, 2009 2:36:29 PM GMT -06:00 US/Canada Central
> Subject: [teiid-dev] Outstanding issues
>
> In light of the outstanding issues I'm working (644 and 643), I wanted
> to suggest that we delay releasing and sync up early next week.
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
15 years, 6 months
Re: [teiid-dev] Outstanding issues
by John Doyle
I do. I believe that the documentation on the Teiid site is incorrect about what jars you need to include in your classpath (the cause of the constraint violation). I sent an inquiry about that earlier today but haven't gotten a response.
Also, because of this SAX issue, I haven't successfully executed any queries with the XML connector in the M4 build. I would like to validate those changes more fully.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
> 643 has been resolved and I believe 644 is an issue with your JRE (see
> my last comment on the JIRA). I would like to still proceed with the
> release unless you know of any other issues.
>
> ----- Original Message -----
> From: "John Doyle" <jdoyle(a)redhat.com>
> To: "teiid-dev" <teiid-dev(a)lists.jboss.org>
> Sent: Friday, June 5, 2009 2:36:29 PM GMT -06:00 US/Canada Central
> Subject: [teiid-dev] Outstanding issues
>
> In light of the outstanding issues I'm working (644 and 643), I wanted
> to suggest that we delay releasing and sync up early next week.
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
15 years, 6 months
