<html>
<head>
<base href="https://docs.jboss.org/author">
<link rel="stylesheet" href="/author/s/en/2172/19/5/_/styles/combined.css?spaceKey=TEIID&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://docs.jboss.org/author/display/TEIID/Multi-Source+Models">Multi-Source Models</a></h2>
<h4>Page <b>edited</b> by <a href="https://docs.jboss.org/author/display/~shawkins">Steven Hawkins</a>
</h4>
<br/>
<h4>Changes (33)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Multi-source models can be used to quickly access data in multiple sources with homogeneous metadata. When you have multiple instances of data that are using identical schema (horizontal sharding), Teiid can help you aggregate data across all the instances, using "multi-source" models. In this scenario, instead of creating/importing a model for every data source, user needs to define one source model that represents the schema and configure multiple data "sources" underneath it. During runtime, when a query issued against this model, the query engine analyzes the information and gathers the required data from all the sources configured and aggregates the results and provides in a single result set. Since all sources utilize the same physical metadata, this feature is most appropriate for accessing the same source type with multiple instances. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Multi-source models can be used to quickly access data in multiple sources with homogeneous metadata. When you have multiple instances using identical schema (horizontal sharding), Teiid can help you gather data across all the instances, using "multi-source" models. In this scenario, instead of creating/importing a model for every data source, one source model is defined to represents the schema and is configured with multiple data "sources" underneath it. During runtime when a query issued against this model, the query engine analyzes the information and gathers the required data from all sources configured and gathers the results and provides in a single result. Since all sources utilize the same physical metadata, this feature is most appropriate for accessing the same source type with multiple instances. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">To mark a model as multi-source, the user needs to supply property called _supports-multi-source_\-bindings, in the "vdb.xml" file. Also, the user needs to define multiple sources. Here is code example showing dynamic vdb with single model with multiple sources defined. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Configuration <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">To mark a model as multi-source, the model property _supports-multi-source_\-bindings needs set to true in the "vdb.xml" file. The model is then allowed to define more than one source. Here is a code example showing dynamic vdb with single model with multiple sources defined. <br> <br></td></tr>
<tr><td class="diff-unchanged" >{code:lang=XML}<vdb name="vdbname" version="1"> <br> <model visible="true" type="PHYSICAL" name="Customers" path="/Test/Customers.xmi"> <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" ></vdb>{code} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">In the above example, the VDB defined has single model called {{Customers}}, that has multiple sources ({{chicago}}, {{newyork}}, and {{la}}) that define different instances of data. Every time a model is marked as "multi-source", the runtime engine adds a additional column called "SOURCE_NAME" to every table in that model. This column maps to the source's name from the XML. In the above XML code that would be {{chicago}}, {{la}}, {{newyork}}. This allows queries like the following: <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">{note}Currently the tooling support for managing the multi-source feature is limited, so if you need to use this feature build the VDB as usual in the Teiid Designer and then edit the "vdb.xml" file in the VDB archive using a Text editor to add the additional sources as defined above. You must deploy a separate data source for each source defined in the xml file. <br>{note} <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">In the above example, the VDB has a single model called {{Customers}}, that has multiple sources ({{chicago}}, {{newyork}}, and {{la}}) that define different instances of data. <br> <br>h3. The Multi-source Column <br> <br>When a model is marked as multi-source, the engine will add or use an existing column on each table to represent the source name values. In the above vdb.xml the column would return {{chicago}}, {{la}}, {{newyork}} for each of the respective sources. The name of the column defaults to SOURCE_NAME, but is configurable by setting the vdb property multisource.columnName.&nbsp; If a column already exists on the table with the same name, the engine will assume that it should represent the multi-source column and it will not be used to retrieve physical data. If the multi-source column is added, it will be treated as a pseudo column which is not selectable via wildcards (\* nor tbl.*). <br> <br>This allows queries like the following: <br> <br></td></tr>
<tr><td class="diff-unchanged" >{code:lang=SQL} <br>select * from table where SOURCE_NAME = 'newyork' <br></td></tr>
<tr><td class="diff-changed-lines" >update table column=value <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;"> </span> where SOURCE_NAME='chicago' <br></td></tr>
<tr><td class="diff-unchanged" >delete from table where column = x and SOURCE_NAME='la'
insert into table (column, SOURCE_NAME) VALUES ('value', 'newyork') <br></td></tr>
<tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Note that when user do not supply the "SOURCE_NAME" in the criteria, the command applies to all the sources. If SOURCE_NAME supplied, the query is executed only aginst the source specified. Another useful feature along with this feature is "partial results" to skip unavailable sources if they are down. <br> <br>More complex partitioning scenarios, such as heterogeneous sources or list partitioning will require the use of a [Federated Optimizations#Partitioned Union]. <br> <br>{note}Currently the tooling support for managing the multi-source feature is limited, so if you need to use this feature build the VDB as usual in the Teiid Designer and then edit the "vdb.xml" file in the VDB archive using a Text editor to add the additional sources as defined above. You must deploy a separate data source for each source defined in the xml file. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">{note}If you would like to use "SOURCE_NAME" in your transformations to control which sources are accessed or updated, you would manually need to add this extra column on your view table in the Designer. This column will not be automatically added on the source table, when you import the medata from source. A column or IN procedure parameter named source_name in multi-source mode will always be treated as the explicit form of the multi-source source_name column and will no longer be treated as an actual physical column or procedure parameter. <br></td></tr>
<tr><td class="diff-unchanged" >{note} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">{note}If you would like to use "SOURCE_NAME" in your transformations to control which sources are accessed or updated, you would manually need to add this extra column on your view table in the Designer. This column will not be automatically added on the source table, when you import the medata from source. It is important to understand that a column or IN procedure parameter named source_name in multi-source mode will always be treated as the explicit form of the multi-source source_name column and will no longer be treated as an actual physical column or procedure parameter. <br>{note} <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">h1. Multi-source SELECTs <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h2. Planning and Execution <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">A multi-source SELECT may use the source_name column anywhere a column reference is allowed. As a final stage of planning, a source query will be generated against each source and each instance of the source_name column replaced by the appropriate value. If the resulting query still needs executed, it is sent to the source. If the WHERE clause evaluates to always false, then the query is pruned from the result. All results are then unioned together and returned as the full result. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">The planner logically treats a multi-source table as if it were a view containing the union all of the respective source tables.&nbsp; More complex partitioning scenarios, such as heterogeneous sources or list partitioning will require the use of a [Federated Optimizations#Partitioned Union]. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Most of the federated optimizations available over unions are still applicable in multi-source mode. This includes aggregation pushdown/decomposition, limit pushdown, join partitioning, etc. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">h1. Multi-source INSERTs <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h3. SELECTs, UPDATEs, DELETEs <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">A multi-source INSERT may use the source_name column as an insert target column to specify which source should be targeted by the INSERT. Only a INSERT using the VALUES clause is supported and the source_name column value must be a literal. If the source_name column is not part of the INSERT column, then the INSERT will be issued against every source. The sum of the update counts will be returned as the resultant update count. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">- A multi-source query against a SELECT/UPDATE/DELETE may affect any subset of the sources based upon the evaluation of the WHERE clause. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">- The multi-source column may not be targeted in an update change set. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">h1. Multi-source UPDATEs <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">- The sum of the update counts for UPDATEs/DELETEs will be returned as the resultant update count. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">A multi-source delete functions just like SELECT, however it is not possible to use the source_name column as a target column in the change set. Any other usage of the source_name column will be the appropriate value for each source. If the WHERE clause evaluates to always false, then no update will be issued to the source. The sum of the update counts will be returned as the resultant update count. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">- When running under a transaction in a mode that detects the need for a transaction and multiple updates may performed or a transactional read is required and multiple sources may be read from, a transaction will be started to enlist each source. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h3. INSERTs <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">h1. Multi-source DELETEs <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">- A multi-source INSERT must use the source_name column as an insert column to specify which source should be targeted by the INSERT. Only an INSERT using the VALUES clause is supported. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">A multi-source delete functions just like SELECT. Any usage of the source_name column will be the appropriate value for each source. If the WHERE clause evaluates to always false, then no delete will be issued to the source. The sum of the update counts will be returned as the resultant update count. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h3. Stored Procedures <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> <br>h1. Multi-source Stored Procedures <br> <br></td></tr>
<tr><td class="diff-unchanged" >A physical stored procedures requires the manual addition of a string source_name parameter to allow for specifying which source the procedure is executed on. If the source_name parameter is not added or if named parameters are used and the source_name parameter is allowed to default to a null value, then the procedure will be executed on each source and the results unioned together. <br> <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >EXEC PROC('x', 'y') <br>{code} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> <br>{note}You are currently limited to only literal source_name values, you cannot for example pass the source name via a procedural relational join.{note} <br> <br> <br>h1. Additional Concerns <br> <br>When running under a transaction of in a mode that detects the need for a transaction and multiple updates are performed, an attempt will be made to enlist each source in the same XA transaction. <br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<p>Multi-source models can be used to quickly access data in multiple sources with homogeneous metadata. When you have multiple instances using identical schema (horizontal sharding), Teiid can help you gather data across all the instances, using "multi-source" models. In this scenario, instead of creating/importing a model for every data source, one source model is defined to represents the schema and is configured with multiple data "sources" underneath it. During runtime when a query issued against this model, the query engine analyzes the information and gathers the required data from all sources configured and gathers the results and provides in a single result. Since all sources utilize the same physical metadata, this feature is most appropriate for accessing the same source type with multiple instances.</p>
<h2><a name="Multi-SourceModels-Configuration"></a>Configuration</h2>
<p>To mark a model as multi-source, the model property <em>supports-multi-source</em>-bindings needs set to true in the "vdb.xml" file. The model is then allowed to define more than one source. Here is a code example showing dynamic vdb with single model with multiple sources defined.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: xml; gutter: false" style="font-size:12px; font-family: ConfluenceInstalledFont,monospace;"><vdb name="vdbname" version="1">
<model visible="true" type="PHYSICAL" name="Customers" path="/Test/Customers.xmi">
<property name="supports-multi-source-bindings" value="true"/>
<source name="chicago"
translator-name="oracle" connection-jndi-name="chicago-customers"/>
<source name="newyork"
translator-name="oracle" connection-jndi-name="newyork-customers"/>
<source name="la"
translator-name="oracle" connection-jndi-name="la-customers"/>
</model>
</vdb></pre>
</div></div>
<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/author/images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Currently the tooling support for managing the multi-source feature is limited, so if you need to use this feature build the VDB as usual in the Teiid Designer and then edit the "vdb.xml" file in the VDB archive using a Text editor to add the additional sources as defined above. You must deploy a separate data source for each source defined in the xml file.</td></tr></table></div>
<p>In the above example, the VDB has a single model called <tt>Customers</tt>, that has multiple sources (<tt>chicago</tt>, <tt>newyork</tt>, and <tt>la</tt>) that define different instances of data.</p>
<h3><a name="Multi-SourceModels-TheMultisourceColumn"></a>The Multi-source Column</h3>
<p>When a model is marked as multi-source, the engine will add or use an existing column on each table to represent the source name values. In the above vdb.xml the column would return <tt>chicago</tt>, <tt>la</tt>, <tt>newyork</tt> for each of the respective sources. The name of the column defaults to SOURCE_NAME, but is configurable by setting the vdb property multisource.columnName. If a column already exists on the table with the same name, the engine will assume that it should represent the multi-source column and it will not be used to retrieve physical data. If the multi-source column is added, it will be treated as a pseudo column which is not selectable via wildcards (* nor tbl.*).</p>
<p>This allows queries like the following:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: sql; gutter: false" style="font-size:12px; font-family: ConfluenceInstalledFont,monospace;">
select * from table where SOURCE_NAME = 'newyork'
update table column=value where SOURCE_NAME='chicago'
delete from table where column = x and SOURCE_NAME='la'
insert into table (column, SOURCE_NAME) VALUES ('value', 'newyork')
</pre>
</div></div>
<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/author/images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>If you would like to use "SOURCE_NAME" in your transformations to control which sources are accessed or updated, you would manually need to add this extra column on your view table in the Designer. This column will not be automatically added on the source table, when you import the medata from source. A column or IN procedure parameter named source_name in multi-source mode will always be treated as the explicit form of the multi-source source_name column and will no longer be treated as an actual physical column or procedure parameter.</td></tr></table></div>
<h2><a name="Multi-SourceModels-PlanningandExecution"></a>Planning and Execution</h2>
<p>The planner logically treats a multi-source table as if it were a view containing the union all of the respective source tables. More complex partitioning scenarios, such as heterogeneous sources or list partitioning will require the use of a <a href="/author/display/TEIID/Federated+Optimizations#FederatedOptimizations-PartitionedUnion">Federated Optimizations#Partitioned Union</a>.</p>
<p>Most of the federated optimizations available over unions are still applicable in multi-source mode. This includes aggregation pushdown/decomposition, limit pushdown, join partitioning, etc.</p>
<h3><a name="Multi-SourceModels-SELECTs%2CUPDATEs%2CDELETEs"></a>SELECTs, UPDATEs, DELETEs</h3>
<ul class="alternate" type="square">
<li>A multi-source query against a SELECT/UPDATE/DELETE may affect any subset of the sources based upon the evaluation of the WHERE clause.</li>
</ul>
<ul class="alternate" type="square">
<li>The multi-source column may not be targeted in an update change set.</li>
</ul>
<ul class="alternate" type="square">
<li>The sum of the update counts for UPDATEs/DELETEs will be returned as the resultant update count.</li>
</ul>
<ul class="alternate" type="square">
<li>When running under a transaction in a mode that detects the need for a transaction and multiple updates may performed or a transactional read is required and multiple sources may be read from, a transaction will be started to enlist each source.</li>
</ul>
<h3><a name="Multi-SourceModels-INSERTs"></a>INSERTs</h3>
<ul class="alternate" type="square">
<li>A multi-source INSERT must use the source_name column as an insert column to specify which source should be targeted by the INSERT. Only an INSERT using the VALUES clause is supported.</li>
</ul>
<h3><a name="Multi-SourceModels-StoredProcedures"></a>Stored Procedures</h3>
<p>A physical stored procedures requires the manual addition of a string source_name parameter to allow for specifying which source the procedure is executed on. If the source_name parameter is not added or if named parameters are used and the source_name parameter is allowed to default to a null value, then the procedure will be executed on each source and the results unioned together.</p>
<p>It is not possible to execute procedures that required to return IN/OUT, OUT, or RETURN parameters values on more than 1 source at a time.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>Example DDL</b></div><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: sql; gutter: false" style="font-size:12px; font-family: ConfluenceInstalledFont,monospace;">
CREATE FOREIGN PROCEDURE PROC (arg1 IN STRING NOT NULL, arg2 IN STRING, SOURCE_NAME IN STRING)
</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>Example Calls Against A Single Source</b></div><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: sql; gutter: false" style="font-size:12px; font-family: ConfluenceInstalledFont,monospace;">
CALL PROC(arg1=>'x', SOURCE_NAME=>'sourceA')
EXEC PROC('x', 'y', 'sourceB')
</pre>
</div></div>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>Example Calls Against All Sources</b></div><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: sql; gutter: false" style="font-size:12px; font-family: ConfluenceInstalledFont,monospace;">
CALL PROC(arg1=>'x')
EXEC PROC('x', 'y')
</pre>
</div></div>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://docs.jboss.org/author/users/removespacenotification.action?spaceKey=TEIID">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://docs.jboss.org/author/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://docs.jboss.org/author/display/TEIID/Multi-Source+Models">View Online</a>
|
<a href="https://docs.jboss.org/author/pages/diffpagesbyversion.action?pageId=18646287&revisedVersion=5&originalVersion=4">View Changes</a>
|
<a href="https://docs.jboss.org/author/display/TEIID/Multi-Source+Models?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>