[teiid-commits] teiid SVN: r2968 - in trunk: engine/src/main/java/org/teiid/query/optimizer/relational/rules and 1 other directories.
teiid-commits at lists.jboss.org
teiid-commits at lists.jboss.org
Fri Mar 4 16:26:05 EST 2011
Author: shawkins
Date: 2011-03-04 16:26:05 -0500 (Fri, 04 Mar 2011)
New Revision: 2968
Modified:
trunk/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleAssignOutputElements.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRemoveOptionalJoins.java
trunk/engine/src/test/java/org/teiid/query/optimizer/TestOptionalJoins.java
Log:
TEIID-1494 expanding the optional join logic to work more broadly
Modified: trunk/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml
===================================================================
--- trunk/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml 2011-03-04 18:03:55 UTC (rev 2967)
+++ trunk/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml 2011-03-04 21:26:05 UTC (rev 2968)
@@ -217,32 +217,54 @@
<section id="optional_join">
<title>Optional Join</title>
<para>The optional join hint indicates to the
- optimizer that a join clause should be omitted if none of its
- columns are used in either user criteria or output columns in
- the result. This hint is typically only used in view layers
+ optimizer that a joined table should be omitted if none of its
+ columns are used by the output of the user query or in a meaningful way to construct the results of the user query. This hint is typically only used in view layers
containing multi-source joins.</para>
<para>
- The optional join hint is applied as a comment on a join clause.
+ The optional join hint is applied as a comment on a join clause. It can be applied in both ANSI and non-ANSI joins. With non-ANSI joins an entire joined table may be marked as optional.
<example>
<title>Example Optional Join Hint</title>
- <programlisting>select a.column1, b.column2 from a inner join /*+ optional */ b on a.key = b.key</programlisting>
+ <programlisting language="SQL">select a.column1, b.column2 from a, /*+ optional */ b WHERE a.key = b.key</programlisting>
+ <para>Suppose this example defines a view layer X.
+ If X is queried in such a way as to not need b.column2, then the
+ optional join hint will cause b to be omitted from the query
+ plan. The result would be the same as if X were defined as:
+ </para>
+ <programlisting language="SQL">select a.column1 from a</programlisting>
</example>
+ <example>
+ <title>Example ANSI Optional Join Hint</title>
+ <programlisting>select a.column1, b.column2, c.column3 from /*+ optional */ (a inner join b ON a.key = b.key) INNER JOIN c ON a.key = c.key</programlisting>
+ <para>In this example the ANSI join syntax allows for the join of a and b to be marked as optional. Suppose this example defines a view layer X. Only if both column a.column1 and b.column2 are not needed, e.g. "SELECT column3 FROM X" will the join be removed.</para>
+ </example>
</para>
- <para> Suppose that the preceding example defined a view layer X.
- If X is queried in such a way as to not need b.column2, then the
- optional join hint will cause b to be omitted from the query
- plan. The result would be the same as if X were defined as:
+ <para>
+ The optional join hint will not remove a bridging table that is still required.
+ <example>
+ <title>Example Briding Table</title>
+ <programlisting language="SQL">select a.column1, b.column2, c.column3 from /*+ optional */ a, b, c WHERE ON a.key = b.key AND a.key = c.key</programlisting>
+ <para>Suppose this example defines a view layer X. If b.column2 or c.column3 are solely required by a query to X, then the join on a be removed.
+ However if a.column1 or both b.column2 and c.column3 are needed, then the optional join hint will not take effect.</para>
+ </example>
</para>
- <programlisting>select a.column1 from a</programlisting>
<tip>
- <para>When a join clause is omitted via the optional join hint, the relevant join criteria
+ <para>When a join clause is omitted via the optional join hint, the relevant criteria
is not applied. Thus it is possible that the query results may
not have the same cardinality or even the same row values as
when the join is fully applied.</para>
<para>Left/right outer joins where the inner side values are not used
and whose rows under go a distinct operation will automatically be
- treated as an optional join and does not require a hint.</para>
+ treated as an optional join and do not require a hint.
+ <example>
+ <title>Example Unnecessary Optional Join Hint</title>
+ <programlisting language="SQL">select a.column1, b.column2 from a LEFT OUTER JOIN /*+optional*/ b ON a.key = b.key</programlisting>
+ </example>
+ </para>
</tip>
+ <warning><para>
+ A simple SELECT COUNT(*) ... against a view where all join tables are marked as optional will not return a meaningful result.
+ </para>
+ </warning>
</section>
<section id="partitioned_union">
<title>Partitioned Union</title>
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleAssignOutputElements.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleAssignOutputElements.java 2011-03-04 18:03:55 UTC (rev 2967)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleAssignOutputElements.java 2011-03-04 21:26:05 UTC (rev 2968)
@@ -36,7 +36,6 @@
import org.teiid.core.TeiidComponentException;
import org.teiid.query.analysis.AnalysisRecord;
import org.teiid.query.metadata.QueryMetadataInterface;
-import org.teiid.query.metadata.SupportConstants;
import org.teiid.query.optimizer.capabilities.CapabilitiesFinder;
import org.teiid.query.optimizer.capabilities.SourceCapabilities.Capability;
import org.teiid.query.optimizer.relational.OptimizerRule;
@@ -44,7 +43,6 @@
import org.teiid.query.optimizer.relational.plantree.NodeConstants;
import org.teiid.query.optimizer.relational.plantree.NodeEditor;
import org.teiid.query.optimizer.relational.plantree.PlanNode;
-import org.teiid.query.resolver.util.ResolverUtil;
import org.teiid.query.sql.lang.Command;
import org.teiid.query.sql.lang.Criteria;
import org.teiid.query.sql.lang.OrderBy;
@@ -509,7 +507,6 @@
}
// Gather elements from correlated subquery references;
- // currently only for SELECT or PROJECT nodes
for (SymbolMap refs : node.getAllReferences()) {
for (Expression expr : refs.asMap().values()) {
AggregateSymbolCollectorVisitor.getAggregates(expr, requiredSymbols, requiredSymbols);
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRemoveOptionalJoins.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRemoveOptionalJoins.java 2011-03-04 18:03:55 UTC (rev 2967)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRemoveOptionalJoins.java 2011-03-04 21:26:05 UTC (rev 2968)
@@ -25,6 +25,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Set;
@@ -40,6 +41,7 @@
import org.teiid.query.optimizer.relational.plantree.NodeEditor;
import org.teiid.query.optimizer.relational.plantree.PlanNode;
import org.teiid.query.sql.LanguageObject;
+import org.teiid.query.sql.lang.Criteria;
import org.teiid.query.sql.lang.JoinType;
import org.teiid.query.sql.symbol.AggregateSymbol;
import org.teiid.query.sql.symbol.GroupSymbol;
@@ -62,26 +64,51 @@
CommandContext context) throws QueryPlannerException,
QueryMetadataException,
TeiidComponentException {
-
- List<PlanNode> joinNodes = NodeEditor.findAllNodes(plan, NodeConstants.Types.JOIN);
- HashSet<PlanNode> removedNodes = new HashSet<PlanNode>();
- for (PlanNode planNode : joinNodes) {
- if (removedNodes.contains(planNode)) {
+ List<PlanNode> projectNodes = NodeEditor.findAllNodes(plan, NodeConstants.Types.PROJECT);
+ HashSet<PlanNode> skipNodes = new HashSet<PlanNode>();
+ for (PlanNode projectNode : projectNodes) {
+ if (projectNode.getChildCount() == 0 || projectNode.getProperty(NodeConstants.Info.INTO_GROUP) != null) {
continue;
}
- Set<GroupSymbol> groups = GroupsUsedByElementsVisitor.getGroups((Collection<? extends LanguageObject>)planNode.getProperty(NodeConstants.Info.OUTPUT_COLS));
- List<PlanNode> removed = removeJoin(groups, planNode, planNode.getFirstChild());
- if (removed != null) {
- removedNodes.addAll(removed);
- continue;
+ PlanNode groupNode = NodeEditor.findNodePreOrder(projectNode, NodeConstants.Types.GROUP, NodeConstants.Types.SOURCE | NodeConstants.Types.JOIN);
+ if (groupNode != null) {
+ projectNode = groupNode;
}
- removed = removeJoin(groups, planNode, planNode.getLastChild());
- if (removed != null) {
- removedNodes.addAll(removed);
+ Set<GroupSymbol> requiredForOptional = getRequiredGroupSymbols(projectNode.getFirstChild());
+ boolean done = false;
+ while (!done) {
+ done = true;
+ List<PlanNode> joinNodes = NodeEditor.findAllNodes(projectNode, NodeConstants.Types.JOIN, NodeConstants.Types.SOURCE);
+ for (PlanNode planNode : joinNodes) {
+ if (skipNodes.contains(planNode)) {
+ continue;
+ }
+ if (!planNode.getExportedCorrelatedReferences().isEmpty()) {
+ skipNodes.add(planNode);
+ continue;
+ }
+ Set<GroupSymbol> required = getRequiredGroupSymbols(planNode);
+
+ List<PlanNode> removed = removeJoin(required, requiredForOptional, planNode, planNode.getFirstChild(), analysisRecord);
+ if (removed != null) {
+ skipNodes.addAll(removed);
+ done = false;
+ continue;
+ }
+ removed = removeJoin(required, requiredForOptional, planNode, planNode.getLastChild(), analysisRecord);
+ if (removed != null) {
+ skipNodes.addAll(removed);
+ done = false;
+ }
+ }
}
- }
+ }
return plan;
}
+
+ private Set<GroupSymbol> getRequiredGroupSymbols(PlanNode planNode) {
+ return GroupsUsedByElementsVisitor.getGroups((Collection<? extends LanguageObject>)planNode.getProperty(NodeConstants.Info.OUTPUT_COLS));
+ }
/**
* remove the optional node if possible
@@ -89,14 +116,42 @@
* @throws TeiidComponentException
* @throws QueryMetadataException
*/
- private List<PlanNode> removeJoin(Set<GroupSymbol> groups, PlanNode joinNode, PlanNode optionalNode) throws QueryPlannerException, QueryMetadataException, TeiidComponentException {
- if (!Collections.disjoint(optionalNode.getGroups(), groups)) {
+ private List<PlanNode> removeJoin(Set<GroupSymbol> required, Set<GroupSymbol> requiredForOptional, PlanNode joinNode, PlanNode optionalNode, AnalysisRecord record) throws QueryPlannerException, QueryMetadataException, TeiidComponentException {
+ boolean correctFrame = false;
+ boolean isOptional = optionalNode.hasBooleanProperty(NodeConstants.Info.IS_OPTIONAL);
+ if (isOptional) {
+ required = requiredForOptional;
+ correctFrame = true;
+ //prevent bridge table removal
+ HashSet<GroupSymbol> joinGroups = new HashSet<GroupSymbol>();
+ PlanNode parentNode = joinNode;
+ while (parentNode.getType() != NodeConstants.Types.PROJECT) {
+ PlanNode current = parentNode;
+ parentNode = parentNode.getParent();
+ if (current.getType() != NodeConstants.Types.SELECT && current.getType() != NodeConstants.Types.JOIN) {
+ continue;
+ }
+ Set<GroupSymbol> currentGroups = current.getGroups();
+ if (current.getType() == NodeConstants.Types.JOIN) {
+ currentGroups = GroupsUsedByElementsVisitor.getGroups((List<Criteria>)current.getProperty(NodeConstants.Info.JOIN_CRITERIA));
+ }
+ if (!Collections.disjoint(currentGroups, optionalNode.getGroups()) && !optionalNode.getGroups().containsAll(currentGroups)) {
+ //we're performing a join
+ boolean wasEmpty = joinGroups.isEmpty();
+ boolean modified = joinGroups.addAll(current.getGroups());
+ if (!wasEmpty && modified) {
+ return null;
+ }
+ }
+ }
+ }
+ if (!Collections.disjoint(optionalNode.getGroups(), required)) {
return null;
}
JoinType jt = (JoinType)joinNode.getProperty(NodeConstants.Info.JOIN_TYPE);
- if (!optionalNode.hasBooleanProperty(NodeConstants.Info.IS_OPTIONAL) &&
+ if (!isOptional &&
(jt != JoinType.JOIN_LEFT_OUTER || optionalNode != joinNode.getLastChild() || useNonDistinctRows(joinNode.getParent()))) {
return null;
}
@@ -105,7 +160,46 @@
joinNode.removeChild(optionalNode);
joinNode.getFirstChild().setProperty(NodeConstants.Info.OUTPUT_COLS, joinNode.getProperty(NodeConstants.Info.OUTPUT_COLS));
NodeEditor.removeChildNode(parentNode, joinNode);
+ if (record != null && record.recordDebug()) {
+ record.println("Removing join node since " + (isOptional?"it was marked as optional ":"it will not affect the results") + joinNode); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
+ }
+ while (parentNode.getType() != NodeConstants.Types.PROJECT) {
+ PlanNode current = parentNode;
+ parentNode = parentNode.getParent();
+ if (correctFrame) {
+ if (current.getType() == NodeConstants.Types.SELECT) {
+ if (!Collections.disjoint(current.getGroups(), optionalNode.getGroups())) {
+ current.getFirstChild().setProperty(NodeConstants.Info.OUTPUT_COLS, current.getProperty(NodeConstants.Info.OUTPUT_COLS));
+ NodeEditor.removeChildNode(parentNode, current);
+ }
+ } else if (current.getType() == NodeConstants.Types.JOIN) {
+ if (!Collections.disjoint(current.getGroups(), optionalNode.getGroups())) {
+ List<Criteria> crits = (List<Criteria>) current.getProperty(NodeConstants.Info.JOIN_CRITERIA);
+ if (crits != null && !crits.isEmpty()) {
+ for (Iterator<Criteria> iterator = crits.iterator(); iterator.hasNext();) {
+ Criteria criteria = iterator.next();
+ if (!Collections.disjoint(GroupsUsedByElementsVisitor.getGroups(criteria), optionalNode.getGroups())) {
+ iterator.remove();
+ }
+ }
+ if (crits.isEmpty()) {
+ JoinType joinType = (JoinType) current.getProperty(NodeConstants.Info.JOIN_TYPE);
+ if (joinType == JoinType.JOIN_INNER) {
+ current.setProperty(NodeConstants.Info.JOIN_TYPE, JoinType.JOIN_CROSS);
+ }
+ }
+ }
+ }
+ }
+ } else if (current.getType() != NodeConstants.Types.JOIN) {
+ break;
+ }
+ if (current.getType() == NodeConstants.Types.JOIN) {
+ current.getGroups().removeAll(optionalNode.getGroups());
+ }
+ }
+
return NodeEditor.findAllNodes(optionalNode, NodeConstants.Types.JOIN);
}
Modified: trunk/engine/src/test/java/org/teiid/query/optimizer/TestOptionalJoins.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/optimizer/TestOptionalJoins.java 2011-03-04 18:03:55 UTC (rev 2967)
+++ trunk/engine/src/test/java/org/teiid/query/optimizer/TestOptionalJoins.java 2011-03-04 21:26:05 UTC (rev 2968)
@@ -27,7 +27,7 @@
import org.teiid.query.processor.ProcessorPlan;
import org.teiid.query.unittest.FakeMetadataFactory;
-
+ at SuppressWarnings("nls")
public class TestOptionalJoins {
@Test public void testOptionalJoinNode1() {
@@ -37,6 +37,42 @@
TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
}
+ @Test public void testOptionalJoinNode1WithPredicate() {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT pm1.g1.e1 FROM pm1.g1, /* optional */ pm1.g2 WHERE pm1.g1.e1 = pm1.g2.e1", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT pm1.g1.e1 FROM pm1.g1"} ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
+ }
+
+ @Test public void testOptionalJoinNode1WithJoinCriteria() {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT pm1.g3.e1 FROM (pm1.g1 CROSS JOIN /* optional */ pm1.g2) INNER JOIN pm1.g3 ON pm1.g1.e1 = pm1.g3.e1 AND pm1.g2.e1 = pm1.g3.e1", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT g_1.e1 FROM pm1.g1 AS g_0, pm1.g3 AS g_1 WHERE g_0.e1 = g_1.e1"} ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
+ }
+
+ @Test public void testOptionalJoinNodeNonAnsiWithHaving() {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT e1 FROM (SELECT pm1.g1.e1, max(pm1.g2.e2) FROM pm1.g1, /* optional */ pm1.g2 WHERE pm1.g1.e1 = pm1.g2.e1 GROUP BY pm1.g1.e1) x", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT pm1.g1.e1 FROM pm1.g1"} ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, new int[] {
+ 1, // Access
+ 0, // DependentAccess
+ 0, // DependentSelect
+ 0, // DependentProject
+ 0, // DupRemove
+ 1, // Grouping
+ 0, // NestedLoopJoinStrategy
+ 0, // MergeJoinStrategy
+ 0, // Null
+ 0, // PlanExecution
+ 1, // Project
+ 0, // Select
+ 0, // Sort
+ 0 // UnionAll
+ });
+ }
+
@Test public void testOptionalJoinNode1_1() {
ProcessorPlan plan = TestOptimizer.helpPlan("SELECT pm1.g1.e1,pm2.g2.e1 FROM pm1.g1, /* optional */ pm2.g2", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
new String[] {"SELECT pm1.g1.e1 FROM pm1.g1", "SELECT pm2.g2.e1 FROM pm2.g2"} ); //$NON-NLS-1$//$NON-NLS-2$
@@ -507,4 +543,40 @@
TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
}
+ @Test public void testOptionalJoinNodeStarTransitiveAnsi() throws Exception {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT g3.e1 FROM ( /* optional */ pm1.g1 as g1 makedep INNER JOIN /* optional */ pm2.g2 ON g1.e1 = pm2.g2.e1) makedep INNER JOIN /* optional */ pm2.g3 ON g1.e1 = pm2.g3.e1", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT g_0.e1 FROM pm2.g3 AS g_0"}, ComparisonMode.EXACT_COMMAND_STRING ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
+ }
+
+ @Test public void testOptionalJoinNodeStarNonAnsi() throws Exception {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT g3.e1 FROM /* optional */ pm1.g1 as g1, /* optional */ pm2.g2, /* optional */ pm2.g3 WHERE g1.e1 = pm2.g2.e1 AND g1.e1 = pm2.g3.e1", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT g_0.e1 FROM pm2.g3 AS g_0"}, ComparisonMode.EXACT_COMMAND_STRING ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, TestOptimizer.FULL_PUSHDOWN);
+ }
+
+ @Test public void testOptionalJoinNodeBridgeNonAnsi() throws Exception {
+ ProcessorPlan plan = TestOptimizer.helpPlan("SELECT g3.e1 FROM /* optional */ pm1.g1 as g1 makedep, pm2.g2, /* optional */ pm2.g3 WHERE g1.e1 = pm2.g2.e1 AND g1.e1 = pm2.g3.e1", FakeMetadataFactory.example1Cached(), //$NON-NLS-1$
+ new String[] {"SELECT g_1.e1 AS c_0, g_0.e1 AS c_1 FROM pm2.g2 AS g_0, pm2.g3 AS g_1 WHERE g_1.e1 = g_0.e1 ORDER BY c_0, c_1", "SELECT g_0.e1 AS c_0 FROM pm1.g1 AS g_0 WHERE (g_0.e1 IN (<dependent values>)) AND (g_0.e1 IN (<dependent values>)) ORDER BY c_0"}, ComparisonMode.EXACT_COMMAND_STRING ); //$NON-NLS-1$
+
+ TestOptimizer.checkNodeTypes(plan, new int[] {
+ 1, // Access
+ 1, // DependentAccess
+ 0, // DependentSelect
+ 0, // DependentProject
+ 0, // DupRemove
+ 0, // Grouping
+ 0, // Join
+ 1, // MergeJoin
+ 0, // Null
+ 0, // PlanExecution
+ 1, // Project
+ 0, // Select
+ 0, // Sort
+ 0 // UnionAll
+ });
+ }
+
}
More information about the teiid-commits
mailing list