5:47 p.m.
I want to discuss the proposal to modify the connector api to produce runtime metadata
objects from
source metadata a bit; firstly to confirm my correct understanding and ensure that I'm
understanding the full value.
If I understand the proposal, the connector API would be expanded so the connector can
provide the metadata of the source system via an object representation akin to JDBC
metadata. This would allow a user of Teiid embedded to create bindings to source systems
and query those systems without any model or VDB. Users would be able to submit queries
to Teiid that joined across the source systems much like they would write in a
transformation for a view model. Is all of this correct?
Unless I'm missing some additional utility of this approach, I don't really see
much value in this approach. On the contrary, I think it undermines the value of Teiid by
offering federation without much abstraction. I don't see how this leads to
application code that significantly less brittle than what users would write without us,
and I think it leads users down a dead end. Much of the code they create around this
approach would likely be thrown away were they to adopt full usage of Teiid.
I also think that by doing this now, we are potentially missing an opportunity to improve
our integration story for community members who might want to integrate their products
with Teiid. There has been a longstanding desire to reduce the effort required to create
an importer, and there is also a requirement for alternative tooling. There are a lot of
balls up in the air right now and I think we should take this opportunity to look at the
all of APIs that we are going to expose and rationalize them as much as possible.
My preferred approach would be to answer the big question first. What is our canonical
representation of metadata? Is it DDL, or something else? Once we've done that, then
we should define the APIs and tools to create the metadata, and the hooks to provide it.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
There are three main engineering goals for the 6.2 release cycle
(~12
weeks):
To provide legacy server features to the embedded deployment, to
provide a simple client/server runtime mode, and to provide metadata
through connectors for a simple query mode.
At a high level our requirements are:
1. Port the Membership, Authentication, Session, and Tracking
(CommandLogging) services for use in embedded. This is
straight-forward except for the authentication service, which will
require treating the only source for authorization information as the
vdb itself in the form of an associated xml file (note that we are not
doing something new here yet, the schema for defining policies was
introduced in 5.5.1). This is covered by TEIID-663 (with subtasks
TEIID-664 - 667) and TEIID-668.
2. Provide a Java main that will launch the "server" with a socket
transport. - Needs a JIRA
3. Ensure adminapi/admin shell coverage of all exposed functionality -
the console should not be necessary for this release. There is not a
a JIRA for this as such, but needs to be addressed as service changes
are made.
4. Modify the connector api to produce runtime metadata objects from
source metadata. This metadata facility will probably have a
superficial resemblance to JDBC DatadataMetadata. The QMI and other
engine logic will need to be modified to support this "modelless"
query mode. - Needs a JIRA
The assumption with all these is that the current server mechanisms
for configuration and clustering are disregarded. We are specifically
not attempting to provide a clustered server that is feature/api
compatible with legacy MetaMatrix. This work should also have minimal
to no impact on Designer's usage of Teiid embedded. Any new work in
designer would be a stretch goal only.
The engineering stretch goals for this release are:
1. Provide JOPR plugins for monitoring and additional adminapi
functionality. Van has initially indicated that he would like for
these to remain out of Teiid proper until they seem usable wrt the
release at some point much later in the cycle.
2. Have definitive plans around Teiid owned metadata in a persistent
form, which has already been on introduced as a topic on the
dev-list.
3. Re-implement file and database based artifact access through JCR
(DNA) - with an eye toward using JCR as a means to provide "cluster"
features, but that would still be api compatible with the embedded
implementation (JCR backed by the filesystem).
4. Refine service loading/wiring/configuration in MC or other
consistent container strategy.
5. Allow for pass-through ddl in both the single source and
multi-source scenarios - TEIID-669
In this bucket as well is a task to provide a doc set around our
legacy SOAP services and some proposed REST services in terms of how
they can be implemented with JBoss projects with a loose coupling to
Teiid. This will be covered under its own JIRA with multiple subtasks
covering prospective services use cases.
A 6.3 bucket has been created out of pushed 6.2 items and select
issues in 6.x. This will serve as the place holder for the further
refinement of the new simplified server approach At the end of 6.3 we
should have re-established our clustering story, a good story around
configuration changes, fully incorporated JOPR, and carried-forward
any lingering server features.
On the non-engineering side of things we have the requirements of:
1. Updating the server extension guide to cover things from an
embedded/simplified perspective given the changes to command logging,
membership domains, etc.
2. Update the connector developers guide - TEIID-104
3. Add the jdbc developers guide - TEIID-315
There are stretch goals of:
1. Adding additional screen casts covering a broader set of common
uses.
2. Add more examples to the kit, especially highlighting use of
non-relational sources.
3. Get Teiid as a recognized source in other products/projects -
squirrel, dtp, hibernate, etc.
Note there are missing JIRAs for a lot of these items and the priority
may not yet be assigned correctly for items identified as
requirements. Devs and leads should feel free to help fill in gaps.
Let me know if there is anything that has been left out or what we
should dive into a little more detail on.
Steve
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
6:29 p.m.
New subject: 6.2 planning
I think this is of great value.
For starters, this was one of the hits against us in one of our
competitive analysis.
Also, this has been asked for for years. It was originally
implemented for one of our integrations, long time ago. And with
the advent of mmquery, which utilizes this feature, it was used at a
military site.
The value lies in the ability of a user to get going quickly in using
Teiid. Once seeing the integration value, they can proceed to its
more complex uses.
As for reusable code, it's all reusable. Any sql created and
executed to our driver should be able to be pasted into the Designer
and used. This goes especially for any transformations. Any sql
the user creates to query multiple sources should be able to be
used. The only thing the user has to do to get started is import
from the sources (which should present the same metadata the connector
did), then paste their transformation.
Additionally, this was also brought up in the R&D demo yesterday in
the context of integrating with other tools (i.e., hibernate).
Van
On Jun 17, 2009, at 10:47 AM, John Doyle wrote:
I want to discuss the proposal to modify the connector api to
produce runtime metadata objects from
source metadata a bit; firstly to confirm my correct understanding
and ensure that I'm understanding the full value.
If I understand the proposal, the connector API would be expanded so
the connector can provide the metadata of the source system via an
object representation akin to JDBC metadata. This would allow a
user of Teiid embedded to create bindings to source systems and
query those systems without any model or VDB. Users would be able
to submit queries to Teiid that joined across the source systems
much like they would write in a transformation for a view model. Is
all of this correct?
Unless I'm missing some additional utility of this approach, I don't
really see much value in this approach. On the contrary, I think it
undermines the value of Teiid by offering federation without much
abstraction. I don't see how this leads to application code that
significantly less brittle than what users would write without us,
and I think it leads users down a dead end. Much of the code they
create around this approach would likely be thrown away were they to
adopt full usage of Teiid.
I also think that by doing this now, we are potentially missing an
opportunity to improve our integration story for community members
who might want to integrate their products with Teiid. There has
been a longstanding desire to reduce the effort required to create
an importer, and there is also a requirement for alternative
tooling. There are a lot of balls up in the air right now and I
think we should take this opportunity to look at the all of APIs
that we are going to expose and rationalize them as much as possible.
My preferred approach would be to answer the big question first.
What is our canonical representation of metadata? Is it DDL, or
something else? Once we've done that, then we should define the
APIs and tools to create the metadata, and the hooks to provide it.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
> There are three main engineering goals for the 6.2 release cycle (~12
> weeks):
> To provide legacy server features to the embedded deployment, to
> provide a simple client/server runtime mode, and to provide metadata
> through connectors for a simple query mode.
>
> At a high level our requirements are:
>
> 1. Port the Membership, Authentication, Session, and Tracking
> (CommandLogging) services for use in embedded. This is
> straight-forward except for the authentication service, which will
> require treating the only source for authorization information as the
> vdb itself in the form of an associated xml file (note that we are
> not
> doing something new here yet, the schema for defining policies was
> introduced in 5.5.1). This is covered by TEIID-663 (with subtasks
> TEIID-664 - 667) and TEIID-668.
>
> 2. Provide a Java main that will launch the "server" with a socket
> transport. - Needs a JIRA
>
> 3. Ensure adminapi/admin shell coverage of all exposed
> functionality -
> the console should not be necessary for this release. There is not a
> a JIRA for this as such, but needs to be addressed as service changes
> are made.
>
> 4. Modify the connector api to produce runtime metadata objects from
> source metadata. This metadata facility will probably have a
> superficial resemblance to JDBC DatadataMetadata. The QMI and other
> engine logic will need to be modified to support this "modelless"
> query mode. - Needs a JIRA
>
> The assumption with all these is that the current server mechanisms
> for configuration and clustering are disregarded. We are
> specifically
> not attempting to provide a clustered server that is feature/api
> compatible with legacy MetaMatrix. This work should also have
> minimal
> to no impact on Designer's usage of Teiid embedded. Any new work in
> designer would be a stretch goal only.
>
> The engineering stretch goals for this release are:
>
> 1. Provide JOPR plugins for monitoring and additional adminapi
> functionality. Van has initially indicated that he would like for
> these to remain out of Teiid proper until they seem usable wrt the
> release at some point much later in the cycle.
>
> 2. Have definitive plans around Teiid owned metadata in a persistent
> form, which has already been on introduced as a topic on the
> dev-list.
>
> 3. Re-implement file and database based artifact access through JCR
> (DNA) - with an eye toward using JCR as a means to provide "cluster"
> features, but that would still be api compatible with the embedded
> implementation (JCR backed by the filesystem).
>
> 4. Refine service loading/wiring/configuration in MC or other
> consistent container strategy.
>
> 5. Allow for pass-through ddl in both the single source and
> multi-source scenarios - TEIID-669
>
> In this bucket as well is a task to provide a doc set around our
> legacy SOAP services and some proposed REST services in terms of how
> they can be implemented with JBoss projects with a loose coupling to
> Teiid. This will be covered under its own JIRA with multiple
> subtasks
> covering prospective services use cases.
>
> A 6.3 bucket has been created out of pushed 6.2 items and select
> issues in 6.x. This will serve as the place holder for the further
> refinement of the new simplified server approach At the end of 6.3 we
> should have re-established our clustering story, a good story around
> configuration changes, fully incorporated JOPR, and carried-forward
> any lingering server features.
>
> On the non-engineering side of things we have the requirements of:
>
> 1. Updating the server extension guide to cover things from an
> embedded/simplified perspective given the changes to command logging,
> membership domains, etc.
>
> 2. Update the connector developers guide - TEIID-104
>
> 3. Add the jdbc developers guide - TEIID-315
>
> There are stretch goals of:
>
> 1. Adding additional screen casts covering a broader set of common
> uses.
>
> 2. Add more examples to the kit, especially highlighting use of
> non-relational sources.
>
> 3. Get Teiid as a recognized source in other products/projects -
> squirrel, dtp, hibernate, etc.
>
> Note there are missing JIRAs for a lot of these items and the
> priority
> may not yet be assigned correctly for items identified as
> requirements. Devs and leads should feel free to help fill in gaps.
> Let me know if there is anything that has been left out or what we
> should dive into a little more detail on.
>
> Steve
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
6:32 p.m.
New subject: Connector Based Metadata
On Wed, 2009-06-17 at 11:47 -0400, John Doyle wrote:
If I understand the proposal, the connector API would be expanded so
the connector can provide the metadata of the source system via an
object representation akin to JDBC metadata. This would allow a user
of Teiid embedded to create bindings to source systems and query those
systems without any model or VDB. Users would be able to submit
queries to Teiid that joined across the source systems much like they
would write in a transformation for a view model. Is all of this
correct?
Yes, that is my understand as well.
Unless I'm missing some additional utility of this approach, I
don't
really see much value in this approach. On the contrary, I think it
undermines the value of Teiid by offering federation without much
abstraction. I don't see how this leads to application code that
significantly less brittle than what users would write without us, and
I think it leads users down a dead end. Much of the code they create
around this approach would likely be thrown away were they to adopt
full usage of Teiid.
Can you explain why you think there is additional utility? Are you
thinking for example taking text connector, we need a utility to convert
the def file into a metadata format? or like DDL for JDBC connector to
metadata format?
What Teiid does is Federation at its core, abstraction is layer above
that which provides us to define much more manageable schema independent
of sources. You are correct on that respect. But, I would disagree, that
there is no value. The code is no more brittle than what they would have
written themselves, however the they would not have federation. That is
what we fill in, with very easy processing model.
Whether they throw away or not, my hunch is this will satisfy the needs
of majority of the developers, and it is quick way somebody can see the
benefits of Teiid based data integration.
I also think that by doing this now, we are potentially missing an
opportunity to improve our integration story for community members who
might want to integrate their products with Teiid. There has been a
longstanding desire to reduce the effort required to create an
importer, and there is also a requirement for alternative tooling.
There are a lot of balls up in the air right now and I think we should
take this opportunity to look at the all of APIs that we are going to
expose and rationalize them as much as possible.
This is not about tooling, this about alternative ways to defining the
metadata to be used by Teiid, independent of any tooling.
My preferred approach would be to answer the big question first.
What
is our canonical representation of metadata? Is it DDL, or something
else? Once we've done that, then we should define the APIs and tools
to create the metadata, and the hooks to provide it.
These two are separate
concerns, granted they will compliment each
other. We are committed for defining this, there is already discussion
happening in the community as to how we want to approach this.
What we want to do now is define the runtime model we need for the
engine, then whatever the persistent form may that be DDL or SQL/MED or
XML will defined in coming releases, that produces this runtime model.
7:42 p.m.
New subject: Connector Based Metadata
Ramesh Reddy wrote:
On Wed, 2009-06-17 at 11:47 -0400, John Doyle wrote:
> If I understand the proposal, the connector API would be expanded so
> the connector can provide the metadata of the source system via an
> object representation akin to JDBC metadata. This would allow a user
> of Teiid embedded to create bindings to source systems and query those
> systems without any model or VDB. Users would be able to submit
> queries to Teiid that joined across the source systems much like they
> would write in a transformation for a view model. Is all of this
> correct?
>
>
Yes, that is my understand as well.
> Unless I'm missing some additional utility of this approach, I don't
> really see much value in this approach. On the contrary, I think it
> undermines the value of Teiid by offering federation without much
> abstraction. I don't see how this leads to application code that
> significantly less brittle than what users would write without us, and
> I think it leads users down a dead end. Much of the code they create
> around this approach would likely be thrown away were they to adopt
> full usage of Teiid.
>
>
Can you explain why you think there is additional utility? Are you
thinking for example taking text connector, we need a utility to convert
the def file into a metadata format? or like DDL for JDBC connector to
metadata format?
What Teiid does is Federation at its core, abstraction is layer above
that which provides us to define much more manageable schema independent
of sources. You are correct on that respect. But, I would disagree, that
there is no value. The code is no more brittle than what they would have
written themselves, however the they would not have federation. That is
what we fill in, with very easy processing model.
True that the code written in the usage model is no more brittle than if
the user was to code directly to a database. However it is more brittle
(at least from the application perspective) than if they wrote to an
abstract view. So this is valuable to a developer that's comfortable
with tight coupling but lacks federation functionality. This capability
does not directly provide value to the user interested in data source
virtualization and loose coupling. That's fine as long as the project
is clear that these modes of use are distinct and each have their
positives/negatives.
Whether they throw away or not, my hunch is this will satisfy the
needs
of majority of the developers, and it is quick way somebody can see the
benefits of Teiid based data integration.
It's a quick way to demonstrate/achieve *federation* in it's simplest
form. However Teiid provides substantial value above federation. That
should not get lost nor compromised by this effort.
> I also think that by doing this now, we are potentially missing
an
> opportunity to improve our integration story for community members who
> might want to integrate their products with Teiid. There has been a
> longstanding desire to reduce the effort required to create an
> importer, and there is also a requirement for alternative tooling.
> There are a lot of balls up in the air right now and I think we should
> take this opportunity to look at the all of APIs that we are going to
> expose and rationalize them as much as possible.
>
>
This is not about tooling, this about alternative ways to defining the
metadata to be used by Teiid, independent of any tooling.
> My preferred approach would be to answer the big question first. What
> is our canonical representation of metadata? Is it DDL, or something
> else? Once we've done that, then we should define the APIs and tools
> to create the metadata, and the hooks to provide it.
>
These two are separate concerns, granted they will compliment each
other. We are committed for defining this, there is already discussion
happening in the community as to how we want to approach this.
What we want to do now is define the runtime model we need for the
engine, then whatever the persistent form may that be DDL or SQL/MED or
XML will defined in coming releases, that produces this runtime model.
I'm a bit unclear here. There's a need to define the runtime model we
need for the engine?
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
--
Ken Johnson
Sr. Product Manager
Data Services/MetaMatrix
JBoss, a division of Red Hat
978.392.3917
ken.johnson(a)redhat.com
9:32 p.m.
New subject: Connector Based Metadata
I completely agree with your other points. Yes, these both modes are
different and provide different levels integration to the user. We have
to pursue both modes.
On Wed, 2009-06-17 at 13:42 -0400, Ken Johnson wrote:
> What we want to do now is define the runtime model we need for
the
> engine, then whatever the persistent form may that be DDL or SQL/MED
or
> XML will defined in coming releases, that produces this runtime
model.
>
I'm a bit unclear here. There's a need to define the runtime model
we
need for the engine?
The current metadata language object definitions we have is
the contract
between the index/xmi/vdb files to Teiid Engine. Currently we do not
define a metadata model that can be used by the connector (I am talking
about metadata consumption by the engine from connector, not how engine
communicates with connector).
We need some APIs in the Connector API that we need to either define or
elevate from the metadata module, so that connectors can define their
metadata in a consistent manner across all connectors. Then it is up to
Connector developer to define its persistent from metadata of whether it
is ad-hoc program or DDL or XMI files or Custom Script (ex:text
connector).
The example SteveH keep using us akin to JDBC Metadata, where you can
inquire JDBC source about its metadata data, where it provides the
metadata in resultset, here I am talking about a object model, that is
the difference.
Ramesh..
5420
days inactive
5420
days old
4 comments
4 participants
participants (4)
-
John Doyle -
Ken Johnson -
Ramesh Reddy -
Van Halbert