Per Ramesh's suggestion, posting the conversation we had on this topic...<br><br><br><br><br>Sridhar wrote...<br><br>Hi Ramesh<br><br>Great blog! Great concept! However, I have few questions.<br><br>I am really curious about the performances. If you think about any data integration project, we run into bottle necks very frequently due to the volume of data. I'd like to know how Teiid address this area, let me know if you have any blog on this topic?<br>
<br>One other question is....assume that there are 2 Oracle database servers on which data resides and needs to be integrated. What additional value does Teiid provide when compared to using Oracle dblinks? I understand, the biggest value add of Teiid server is to integrate disparate data sources (Oracle, MySQL, SQLServer...etc). But, for this argument sake, let's just take 2 Oracle databases servers for integrating.<br>
<br>Please let me know your thoughts.<br><br><br>cheers<br>Sridhar<br><br><br>Ramesh wrote...<br><br>> I am really curious about the performances. If you think about any<br>> data integration project, we run into bottle necks very frequently due<br>
> to the volume of data. I'd like to know how Teiid address this area,<br>> let me know if you have any blog on this topic?<br>><br>During integration Teiid uses "batching" of tuple sources. What it means<br>
is it breaks down the results in to multiple sets of manageable size<br>such that it can load/unload variable size of the results in and out of<br>memory easily to handle very large sets.<br><br>I will not say that performance is as good as query going to the native<br>
system, but Teiid goes to great lengths to minimize the latency<br>introduced by its processing. Like the batching above, prepared<br>statements, caching, join optimizations etc. Yes, it is great blog<br>topic, we will put something out soon. Mean while I encourage to test it<br>
out.<br><br>> One other question is....assume that there are 2 Oracle database<br>> servers on which data resides and needs to be integrated. What<br>> additional value does Teiid provide when compared to using Oracle<br>
> dblinks? I understand, the biggest value add of Teiid server is to<br>> integrate disparate data sources (Oracle, MySQL, SQLServer...etc).<br>> But, for this argument sake, let's just take 2 Oracle databases<br>
> servers for integrating.<br>><br>I do not have any experience with Oracle dblinks, so I do not know.<br><br>Thanks<br><br>Ramesh..<br><br><br>Sridhar wrote...<br><br>Ramesh<br><br>Thanks for the response. When you say Teiid uses 'batching', do you recommend Teiid for real-time applications, where a query is fired requesting Teiid to fetch data across different databases?<br>
<br>Please keep me updated of your blogs.<br><br>cheers<br>Sridhar<br><br><br>Ramesh wrote...<br><br>Absolutely. Teiid is designed for the real-time solutions. If your<br>application does not need real-time then you need to look into ETL<br>
tools.<br><br>"batching" does *not* mean batch processing. Batching in Teiid's lingo<br>means working with a *set* of results. Let's say a Table has 1M rows, we<br>do not load all the rows into memory<br>
<br>1) Teiid trys to get only rows that are needed by query by pushing the<br>criteria into source system to get as few rows as needed.<br><br>2) Only work with about say ~2000 rows at time in the memory, so that<br>there are no out of memory issues. Then it will swap these results for<br>
next set of rows, so it only works on a "batch" of results at a time.<br>The rest of results are either stored to disk temporarily or thrown out<br>if they did not meet the criteria.<br><br>Hope this helps.<br><br>
Ramesh..<br><br>PS: if you do not mind from next time can you post to the user forums,<br>so that if any other user has similar questions, it would help them out<br>and have more people can provide input. Thanks.<br><br><br>
<br><br><br><div class="gmail_quote">On Wed, Sep 2, 2009 at 9:56 AM, Ramesh Reddy <span dir="ltr"><<a href="mailto:rareddy@redhat.com">rareddy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
If you are not subscribed to Teiid's blog check out a new posting at<br>
<br>
<a href="http://teiid.blogspot.com/" target="_blank">http://teiid.blogspot.com/</a><br>
<br>
Thanks.<br>
<br>
Ramesh..<br>
<br>
_______________________________________________<br>
teiid-users mailing list<br>
<a href="mailto:teiid-users@lists.jboss.org">teiid-users@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/teiid-users" target="_blank">https://lists.jboss.org/mailman/listinfo/teiid-users</a><br>
</blockquote></div><br>