Re: [teiid-dev] Connector Development Tooling (CDK)
by John Doyle
A wizard to create a project would be helpful and a nice addition, but I think enhancements in testing would prove extremely valuable.
I think it would be valuable to enable a developer to test a connector directly, like ConnectorHost in the existing CDK does, but also to easily test a connector executing in Teiid. My personal experience is that when I started developing connectors, I was surprised by the SQL that the engine would pass down to my connector, and often found defects in the connector this way, after I had completed my direct testing of the connector. In addition, joins cannot be tested with the existing CDK for connectors that do not support joins, putting Teiid or some parts of Teiid in to execute the test seems the right solution.
An idea that MWalker and I had discussed some time ago, based upon a tool he had seen at a client site, was a test driver that generated a bunch of SQL based upon JDBC metadata. Client use of this tool improved the SF connector while it was undergoing BETA testing at the client. A capability like that would alleviate the some of the testing burden on the developer, and also be helpful in exposing gaps in the developer's understanding of the SQL that they will receive based upon the capabilities they select.
~jd
----- "Ramesh Reddy" <rareddy(a)redhat.com> wrote:
> This email solicitation for contributors and to discuss about tooling
> requirement for developing custom/user developed Connectors in Teiid.
> This is email contains contents from email that Van sent out
> internally
> last week.
>
> Teiid currently provides command line tool called CDK, which can be
> used
> to test Custom connectors, however this tool is inadequate, does not
> provide development time support, it only helps during testing. It
> would
> be ideal if there was a UI tool that quickly helped with setting up a
> java project for Connector development and helped with testing.
>
> Some basic requirements are
> - eclipse based plug-in, and support java development
> - using connector api as the template to stub out the classes
> required
> to start writing the custom connector
> - provide the ability to test the connector by executing a query.
> Much
> like SQL Explorer functionality Teiid Designer has
> - Manage the connector type properties
> - Provide ability to package the connector
>
> Since Connectors may be the very first extensions to Teiid system that
> a
> user might extend/contribute to integrate data from custom data
> sources
> making that experience less painful would be IMO a good goal.
>
> Please comment if you think this tool is
> 1) valuable to pursue,
> 2) what other features you would wish to see in this this tool to do
> in
> terms of connector development
>
> Thanks
>
> Ramesh..
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
17 years
New JDBC Driver in 6.2
by Ramesh Reddy
Teiid has on its road map to introduce Server platform in 6.2 release.
Client applications can access Teiid using a new JDBC driver called
"org.teiid.jdbc.TeiidDriver". At the same time we are
retiring/deprecating the legacy "com.metamatrix.jdbc.MMDriver" that is
based on old legacy Server from MetaMatrix days. For backward
compatibility purposes, this new driver will still support the old URL
format.
Ramesh..
17 years
Integration Testing
by Van Halbert
A new integration testing project will be created to house the teiid
integration tests. I'm starting this thread to open this topic
up for discussion so that this process can begin to move forward.
The following are some initial goals of this new project:
- A new test integration project is going to be created, so that we
start fresh, create a manageable project that is organized in a manner
that we can make some sense what coverage we have. The goal of this
new project is that it will be pushed to open source.
- The new project will use the existing junit testing framework used
on the product side. This framework enables the developers to
create their own complex junit test (i.e., complex transaction test),
but at the same time, allow bulk test sets to be run.
- The new testing process will support the create/drop of database
schemas and data refresh. So if a contributor can supply a database
user with the right admin rights, the process will create a schema and
all appropriate tables in order to run the test. Note: 1 schema
will be created that will hold all necessary tables (i.e., bqt,
mmfinancials, etc) required to support the tests.
- To facilitate data loading/refreshing before a test, an extension to
junit, DBUnit, will be used.
- The existing test on the product side will be brought over, but in
an organized fashion. This includes bulk test sets (mainly
validating select type statements) and existing product junit tests.
Additional Discussion points:
- how best to organize tests so that we have some notion of what
coverage we have
- managing the project, ensuring additions by contributors, GSS,
developers, etc. are done in a manner that ensures consistency
- reducing the number of data source schemas we support (i..e, books,
partssupplier, mmfinancials, bqt, and there are others). There's no
reason to have that many different schemas to manage across all our
different database types.
- In order for the vdb artifacts to be usable in the open, the
connector properties need to be cleaned out. The adminapi will be
used to update the connector properties before the tests are run.
This will require some manner of mapping the connector bindings to a
specific datasource.
--------------------------------------------------
Here's the order of the tasks:
- implement framework, have it ready for developers to start adding
their junit tests
- add the bulk set tests (baseline validation)
- move over the existing junit tests, reworking those were the schema
is no longer supported,
As for any timeline on this project, your guess is as good as mine.
This is one of those projects that doesn't get high priority, but it's
really needed. And you go as fast as you can when you have the time.
Please respond with your comments or suggestions.
Van Halbert
Principal Software Eng.
Red Hat, Inc.
------
vhalbert(a)redhat.com
17 years
Teiid Server Kit
by Ramesh Reddy
As part of 6.2 release Teiid is will be providing a "Server" kit for
community. A initial version of this kit is available now!
The build scripts are modified such that current "embedded" kit can now
be used in multi-purpose mode, where it can support either "embedded"
mode or standalone server mode.
The way user invokes the environment defines how the run time is
started.
1) If user specifies embedded URL on the connection string, the Teiid
runtime is loaded into host JVM, and user has Teiid Embedded
2) If server is manually started, and then if user uses the Server
specific URL as the connection string, the client program can connect to
Teiid Server instance over the socket.
Startup scripts for running the server are provided in the "bin" folder.
This kit is still under progress and you can monitor it under
https://jira.jboss.org/jira/browse/TEIID-259. Try it out, give us your
feedback.
Now, that we have single kit for both Embedded and Server, it is
confusing to call it "embedded" kit. So, we need to re-name this kit to
something else. Teiid leads would like call this just "teiid-runtime",
but are open for other suggestions. So, if you have other suggestions
for name let's hear um?
Thanks
Ramesh..
17 years
6.2 milestone 1
by Steven Hawkins
Hi all,
Now that we have the basic form of the changes for server mode and connector metadata we should be looking at M1 by the end of the week. At that time we should also ensure that the remaining defects targeted to 6.2 better reflect what can be done of the remaining weeks (9/25 is still the target date). Ideally we should have additional milestones every two weeks after this one as features are completed.
There's still plenty of documentation work to be done if anyone wants to pick up one of the out-of-date or yet to be ported guides. I'm on the hook to add some wiki documentation and blog about Teiid usage without Designer. Ramesh will be adding docs around server mode.
Steve
17 years
Connector Development Tooling (CDK)
by Ramesh Reddy
This email solicitation for contributors and to discuss about tooling
requirement for developing custom/user developed Connectors in Teiid.
This is email contains contents from email that Van sent out internally
last week.
Teiid currently provides command line tool called CDK, which can be used
to test Custom connectors, however this tool is inadequate, does not
provide development time support, it only helps during testing. It would
be ideal if there was a UI tool that quickly helped with setting up a
java project for Connector development and helped with testing.
Some basic requirements are
- eclipse based plug-in, and support java development
- using connector api as the template to stub out the classes required
to start writing the custom connector
- provide the ability to test the connector by executing a query. Much
like SQL Explorer functionality Teiid Designer has
- Manage the connector type properties
- Provide ability to package the connector
Since Connectors may be the very first extensions to Teiid system that a
user might extend/contribute to integrate data from custom data sources
making that experience less painful would be IMO a good goal.
Please comment if you think this tool is
1) valuable to pursue,
2) what other features you would wish to see in this this tool to do in
terms of connector development
Thanks
Ramesh..
17 years
Re: [teiid-dev] Connector supplied metadata
by Steven Hawkins
Here's my understanding of the timeline. An initial internal revision of the 6.2 plan was sent out for comment from TDM and PM. That plan already was shaped by input from the TDM/PM post internal discussions focused on the next 6 months. After some updates it was sent to the dev list, which kicked off an internal call. Based upon that call the connector metadata feature was elevated in priority - with the larger metadata roadmap goal still in place. After the updated plan was sent out, there was a dev list discussion about that priority change as well as a conversation with the TDM. The development consensus at large (see emails by Van and others) was that "this is a useful and desirable feature" and since it was determined to not be negatively impactful on the rest of the 6.2 schedule, I was inclined to move ahead.
Additional comments:
Yes the connector metadata feature purposely "narrowly scoped". If it were not, it should/could not have been done in the 6.2 timeframe.
It does not detract from a long term metadata story around alternative persistent forms of metadata. We are merely elevating a previously opaque metadata api for programatic use - rather than just loading through indexes. I know that is not exiting or grandiose, but it doesn't need to be. Without dramatically changing our system table/metadata connector logic, or breaking designer integration in a deep manner, this is the best that we can do. If anyone has issues with the code, please comment on the JIRAs.
I don't want to see this conversation take negative tones. There's no blame shifting, and complaining is welcome. The major qualms as I understand it is that John primarily believed that this feature negatively impacted the "value proposition" of Teiid and that he wanted it more clearly tied to a larger metadata roadmap. Even though I thought I was clear in that the latter was not an issue, for John it still was. To be even more clear however, we do now have the beginnings of a proper object representation of metadata that should eventually be used as the master representation of metadata, whether it's built programtically, parsed from ddl, or loaded from an index file. I'm not sure how to address the concerns as to the value of this feature. It gives us a story of "integrate your databases (and text files) in 5 minutes or less", which allows Teiid to finally be usable as a standalone download. And there's absolutely nothing about this approach that prevents "up sale" to the designer. The connector metadata can be obtained through an importer and will be (nearly) identical.
----- Original Message -----
From: "Steve Jacobs" <sjacobs(a)redhat.com>
To: "Ramesh Reddy" <rareddy(a)redhat.com>
Cc: "teiid-dev" <teiid-dev(a)lists.jboss.org>
Sent: Friday, July 10, 2009 1:23:00 PM GMT -06:00 US/Canada Central
Subject: Re: [teiid-dev] Connector supplied metadata
Ramesh Reddy wrote:
> Also, 6.2 list is based on the product discussions with TDM and PM.
>
Sorry, but that statement is not correct with regards to this feature.
The 6.2 list that was made available to TDM and PM included a "stretch
goal" to develop "a definitive plan around Teiid owned metadata - both
for SAP-like functionality (Connector exposed metadata) and for our
persistent form, which has already been on introduced as a topic on the
dev-list."
Ken and I were as surprised as anyone to see that change to a required
feature in 6.2. If the R&D call made it apparent that this goal needed
to be pushed forward, that's great, but no blame shifting please. When
I saw the item, developing a plan sounded like exactly the community
process that John is expressing his concerns about.
Steve
> I know, we need to get better at this, it's not late. So let's hear
> technical discussions around the feature rather than complaining.
>
> Ramesh..
>
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
>
_______________________________________________
teiid-dev mailing list
teiid-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/teiid-dev
17 years
Re: [teiid-dev] Connector supplied metadata
by John Doyle
I think our public discussion on this feature started after this was communicated as a requirement for 6.2, which was one day after the R&D call. Please do correct me if I'm wrong about that. I don't think that's the way this community should work. I won't speak for TDM and PM, but from my discussions with them, I didn't get the impression that they pushed this feature.
Do we really want to follow a process where we make a feature a requirement of a release, before it's discussed publicly, and then say that you can only have 'technical discussions' of the feature? That's not open. This feature is not the problem, it's an instance of the problem. Technical discussion of the feature is beside the point.
On a related point, you cannot tell me that I'm confusing one goal with another. Anybody who wants to participate in this community can define goals however they want. Those goals can be rejected or accepted, but this has to happen in the open. I've been advocating for bridging the metadata language goal and this feature, you can't tell me that's invalid. If the consensus is that that's a bad idea, then that's the consensus, and I'm fine with that, but you can't tell me it's invalid or that I'm confused.
~jd
----- "Ramesh Reddy" <rareddy(a)redhat.com> wrote:
> On Thu, 2009-07-09 at 19:03 -0400, John Doyle wrote:
> > I've looked at the work checked in on the 7th, and given that it's
> the
> > first check-in I think it's very good. I have some technical
> comments
> > that I will put in the JIRAs, but what's there is good.
> >
> Let's hear them on this list before we make JIRA, as they may be
> already
> being worked on?
>
> > My main point is about process. We discussed the connector
> metadata
> > feature before we were open source, and I thought the consensus was
> > going the other way, it's hard to say from my remote perspective.
> The
> > next time I see it it's on the short list for 6.2.
> We have had discussions about the feature in public, that is when we
> had
> our differences. May be not discussions as to how to implement it,
> but
> as I said before there was not much to re-invent. This feature took
> precedence from the R&D call we had, where overwhelmingly most of the
> questions were why we did not have similar feature. Also, 6.2 list is
> based on the product discussions with TDM and PM.
>
> > But I read your responses below and I still don't know where this
> > thing is going. Is the plan to expose the "metadata objects" more
> > broadly as THE way to specify metadata?
> > We need to hash out what a feature will be in the forums before we
> > commit to it, so that everybody gets to see and pitch in if they
> > choose, and some consensus can be built. Everybody can see the
> > implications so that people can react/plan.
> I think you confusing with our another broader goal in defining the
> metadata language. That is being discussed on thread [VDB metadata
> consumed by Teiid]. That is discussion how we define a metadata
> language
> in DDL or XMI or Groovy etc. Please do contribute on that thread with
> any suggestions. The intent is that language is the persistent form,
> and
> the metadata objects being defined now are the runtime graph that
> represent the language.
>
> > It's not clear to me that the Designer team is signed up for any
> work
> > associated with this.
> We have not solicited them for any tooling around this feature. That
> is
> the main point of this feature too. Teiid to work (minimally only
> federation not any abstraction) with out any tooling around it.
>
> > I don't know if I'm going to try and adapt the SF connector to this
> > new API because I don't know what's going to be there.
> Again, let us know what you need to make this happen. Documentation
> is
> also coming up before release of 6.2 on Connector development.
>
> > It looks like Extensions are not in the offing for this release?
> Is
> > that right, or wrong?
> I think they are part of next set of commits, Steve?
>
> > I think we're on the same page here technologically, I just don't
> have
> > anyway to tell because the vision has not been elaborated.
> The metadata vision is make it available to the Teiid in tool
> independent way and make it easy for the developer to define and use.
> 1)
> Connector Metadata 2) Metadata Language 3) ??
>
> I know, we need to get better at this, it's not late. So let's hear
> technical discussions around the feature rather than complaining.
>
> Ramesh..
17 years
Re: [teiid-dev] Connector supplied metadata
by John Doyle
I think I've made my position on this feature clear in earlier threads. This feature is to narrowly conceived.
Maybe the problem is that it has not been sufficiently explained, at least not in a public forum. I don't have any problem with implementing a new vision over several versions, which I suspect is what you're doing, but what's the vision. As far as I can tell from the public discussions and JIRAs, we are implementation a new API to be used in connector scope, and have an expectation that it will be grown into a full blown metadata API. That's backwards in my opinion. We should be defining the best API we can based upon the universe of potential environments/clients that we envision for Teiid. We can then implement over successive versions in different components, right away in connectors if that's the priority.
I don't know enough to respond to the issue you raise. Seems to me that it should have been resolved before this feature got on the roadmap and implemented.
~jd
----- "Steven Hawkins" <shawkins(a)redhat.com> wrote:
> https://jira.jboss.org/jira/browse/TEIID-684 covers allowing
> connectors to supply metadata. An initial check-in of this feature
> was added last night. For those that are interested have a look at
> TestVDBLessExecution in the test-integration module.
>
> Through just a ConfigurationInfo.def file:
>
> <VDB>
> <VDBInfo>
> <Property Name="Name" Value="VDBLess" />
> <Property Name="Version" Value="4" />
> <Property Name="UseConnectorMetadata" Value="true" />
> </VDBInfo>
> <Model>
> <Property Name="Name" Value="SummitData" />
> <ConnectorBindings>
> <Connector Name="Text Connector" />
> </ConnectorBindings>
> </Model>
> <ConnectorBindings>
> <Connector Name="Text Connector" ComponentType="Text File
> Connector">
> <Properties>
> <Property Name="Immutable">true</Property>
> <Property
> Name="DescriptorFile">src/test/resources/vdbless/TextDescriptor.txt</Property>
> </Properties>
> </Connector>
> </ConnectorBindings>
> </VDB>
>
> The text connector supplies the metadata for the SummitData model
> (through the TextDescriptor file). I'll have the wiring added for the
> JDBC connector shortly along with additional metadata features, such
> as keys and procedures.
>
> Note that the changes have heavily refactored the metadata module.
> The intention is for 6.2 Designer to specifically fork the the 6.1
> version of the metadata code to provide their own implementation of
> the QueryMetadataInterface with their legacy usage of MetadataRecords.
> This does not require any api level changes on their part or changes
> to the current preview integration mechanism. There are currently no
> plans to incorporate this work into Designer as this is currently
> envisioned for ease of use of stand-alone Teiid.
>
> An issue that is open for discussion is that the connector api now
> exposes the MetadataRecord objects as well as it the MetadataObject
> interfaces (Group, Element, etc). A case could be made for expanding
> the MetadataObject interfaces to include getters and setters for all
> relevant fields on MetadataRecords (there are currently many fields
> that the user doesn't need to modify that are only used by index logic
> or are handled by the MetadataFactory). We still however would need
> to have separate implementations of the interfaces since the Designer
> preview feature would still provide metadata through its QMI in the
> from of legacy MetadataRecords. If we do expand the interfaces in
> this way, we would also probably want to add logic to ensure that no
> modifications can be performed when the records are supplied as
> RuntimeMetadata.
>
> One warning that should be added is that since the metadata is
> non-persistent direct usage of uuid values in system tables should be
> discouraged.
>
> Steve
> _______________________________________________
> teiid-dev mailing list
> teiid-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/teiid-dev
17 years
Connector supplied metadata
by Steven Hawkins
https://jira.jboss.org/jira/browse/TEIID-684 covers allowing connectors to supply metadata. An initial check-in of this feature was added last night. For those that are interested have a look at TestVDBLessExecution in the test-integration module.
Through just a ConfigurationInfo.def file:
<VDB>
<VDBInfo>
<Property Name="Name" Value="VDBLess" />
<Property Name="Version" Value="4" />
<Property Name="UseConnectorMetadata" Value="true" />
</VDBInfo>
<Model>
<Property Name="Name" Value="SummitData" />
<ConnectorBindings>
<Connector Name="Text Connector" />
</ConnectorBindings>
</Model>
<ConnectorBindings>
<Connector Name="Text Connector" ComponentType="Text File Connector">
<Properties>
<Property Name="Immutable">true</Property>
<Property Name="DescriptorFile">src/test/resources/vdbless/TextDescriptor.txt</Property>
</Properties>
</Connector>
</ConnectorBindings>
</VDB>
The text connector supplies the metadata for the SummitData model (through the TextDescriptor file). I'll have the wiring added for the JDBC connector shortly along with additional metadata features, such as keys and procedures.
Note that the changes have heavily refactored the metadata module. The intention is for 6.2 Designer to specifically fork the the 6.1 version of the metadata code to provide their own implementation of the QueryMetadataInterface with their legacy usage of MetadataRecords. This does not require any api level changes on their part or changes to the current preview integration mechanism. There are currently no plans to incorporate this work into Designer as this is currently envisioned for ease of use of stand-alone Teiid.
An issue that is open for discussion is that the connector api now exposes the MetadataRecord objects as well as it the MetadataObject interfaces (Group, Element, etc). A case could be made for expanding the MetadataObject interfaces to include getters and setters for all relevant fields on MetadataRecords (there are currently many fields that the user doesn't need to modify that are only used by index logic or are handled by the MetadataFactory). We still however would need to have separate implementations of the interfaces since the Designer preview feature would still provide metadata through its QMI in the from of legacy MetadataRecords. If we do expand the interfaces in this way, we would also probably want to add logic to ensure that no modifications can be performed when the records are supplied as RuntimeMetadata.
One warning that should be added is that since the metadata is non-persistent direct usage of uuid values in system tables should be discouraged.
Steve
17 years