[teiid-commits] teiid SVN: r2156 - in trunk: engine/src/main/java/org/teiid/query/metadata and 20 other directories.
teiid-commits at lists.jboss.org
teiid-commits at lists.jboss.org
Thu May 27 09:45:45 EDT 2010
Author: shawkins
Date: 2010-05-27 09:45:40 -0400 (Thu, 27 May 2010)
New Revision: 2156
Added:
trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java
trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java
trunk/engine/src/test/resources/text/
trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv
trunk/engine/src/test/resources/text/cdm_dos.txt
trunk/engine/src/test/resources/text/test-file.txt.csv
Modified:
trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml
trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java
trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java
trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java
trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java
trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java
trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java
trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java
trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java
trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java
trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java
trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj
trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties
trunk/engine/src/main/resources/org/teiid/query/i18n.properties
trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java
trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java
trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java
trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java
Log:
TEIID-1102 adding a replacement for the text connector with a built in texttable table function
Modified: trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml
===================================================================
--- trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2010-05-27 13:45:40 UTC (rev 2156)
@@ -55,7 +55,7 @@
ways, some prepending catalog or schema or user information,
Teiid allows table specification to be a dot-delimited construct.
<note>
- <para> When a table specification contains a dot resolving will allow for the match of
+ <para>When a table specification contains a dot resolving will allow for the match of
a partial name against any number of the end segments in the name. e.g. a table with the
fully-qualified name vdbname."sourcescema.sourcetable" would match the partial name
sourcetable.
@@ -569,7 +569,8 @@
input of the parameter.</para>
</listitem>
<listitem>
- <para>Input values are passed via criteria. Values can be passed by '=','is null', or 'in' predicates. Disjuncts however are not allowed.
+ <para>Input values are passed via criteria. Values can be passed by '=','is null', or 'in' predicates. Disjuncts are not allowed.
+ It is also not possible to pass the value of a non-comparable column through an equality predicate.
</para>
</listitem>
<listitem>
@@ -704,34 +705,121 @@
<listitem><para>FROM table1 [INNER|LEFT OUTER|RIGHT OUTER|FULL OUTER] JOIN table2 ON join-criteria</para></listitem>
<listitem><para>FROM table1 CROSS JOIN table2</para></listitem>
<listitem><para>FROM (subquery) [AS] alias</para></listitem>
- <listitem><para>FROM <link linkend="nested_table">table(subquery)</link> [AS] alias</para></listitem>
+ <listitem><para>FROM <link linkend="nested_table">TABLE(subquery)</link> [AS] alias</para></listitem>
<listitem><para>FROM table1 JOIN table2 MAKEDEP ON join-criteria</para></listitem>
<listitem><para>FROM table1 JOIN table2 MAKENOTDEP ON join-criteria</para></listitem>
<listitem><para>FROM table1 left outer join <link linkend="optional_join">/* optional */</link> table2 ON join-criteria</para></listitem>
+ <listitem><para>FROM <link linkend="texttable">TEXTTABLE...</link></para></listitem>
</itemizedlist>
<note>
<title>DEP Hints</title>
<para>
MAKEDEP and MAKENOTDEP are hints used to control
- <link linkend="dependent_join">dependent join</link>
+ <link linkend="dependent_joins">dependent join</link>
behavior. They should only be used in situations where the optimizer
does not choose the most optimal plan based upon query structure,
metadata, and costing information.
</para>
</note>
<sect3 id="nested_table">
- <title>Nested Table Reference</title>
- <para>Nested tables may appear in the FROM clause with the TABLE keyword. They are an alternative to using a view with normal join semantics.
+ <title>Nested Table Reference</title>
+ <para>Nested tables may appear in the FROM clause with the TABLE
+ keyword. They are an alternative to using a view with normal join
+ semantics. The columns projected from the command contained in the nested table
+ may be used just as any of the other FROM clause projected columns in join criteria, the where clause, etc.
</para>
- <para>A nested table may have correlated references to preceeding FROM clause
- column references as long as INNER and LEFT OUTER joins are used. This is especially useful in cases where then nested expression is a procedure or function call.</para>
- <para>Valid Example: select * from t1, TABLE(call proc(t1.x)) t2</para>
- <para>Invalid Example: select * from TABLE(call proc(t1.x)) t2, t1</para>
- <note>
- <title>Multiple Execution</title>
- <para>The usage of a correlated nested table may result in multiple executions of the table expression - once for each correlated row.</para>
- </note>
- </sect3>
+ <para>A nested table may have correlated references to preceeding FROM
+ clause column references as long as INNER and LEFT OUTER joins are used. This is
+ especially useful in cases where then nested expression is a
+ procedure or function call.</para>
+ <para>Valid example:
+ <programlisting>select * from t1, TABLE(call proc(t1.x)) t2</programlisting>
+ </para>
+ <para>Invalid example, since t1 appears after the nested table in the from clause:
+ <programlisting>select * from TABLE(call proc(t1.x)) t2, t1</programlisting>
+ </para>
+ <note>
+ <title>Multiple Execution</title>
+ <para>The usage of a correlated nested table may result in multiple
+ executions of the table expression - once for each correlated row.
+ </para>
+ </note>
+ </sect3>
+ <sect3 id="texttable">
+ <title>TEXTTABLE</title>
+ <para>The TEXTTABLE funciton processes character input to produce tabular ouptut. It supports both fixed and delimited file format parsing.
+ The function itself defines what columns it projects.
+ The TEXTTABLE function is implicitly a nested table and may be correlated to preceeding FROM clause entries.
+ </para>
+ <para>
+ Usage:
+ <synopsis label="Usage">TEXTTABLE(expression COLUMNS column, ... [DELIMITER char] [(QUOTE|ESCAPE) char] [HEADER [integer]] [SKIP integer]) AS identifier</synopsis>
+ <synopsis label="Usage">column := identifier datatype [WIDTH integer]</synopsis>
+ </para>
+ <itemizedlist>
+ <para>Syntax Rules:
+ </para>
+ <listitem>
+ <para>If width is specified for one column it must be specified for all columns.
+ </para>
+ </listitem>
+ <listitem>
+ <para>If width is specified, then fixed width parsing is used and ESCAPE, QUOTE, and HEADER should not be specified.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <itemizedlist>
+ <para>Parameters</para>
+ <listitem>
+ <para>expression - the text content to process, which should be convertable to CLOB.
+ </para>
+ </listitem>
+ <listitem>
+ <para>DELIMITER sets the field delimiter character to use. Defaults to ','.
+ </para>
+ </listitem>
+ <listitem>
+ <para>QUOTE sets the quote, or qualifier, character used to wrap field values. Defaults to '"'.
+ </para>
+ </listitem>
+ <listitem>
+ <para>ESCAPE sets the escape character to use if no quoting character is in use.
+ This is used in situations where the delimiter or new line characters are escaped with a preceding character, e.g. \,
+ </para>
+ </listitem>
+ <listitem>
+ <para>HEADER specifies the text line number (counting every new line) on which the column names occur. All lines prior to the header will be skipped.
+ If HEADER is specified, then the header line will be used to determine the TEXTTABLE column position by case-insensitive name matching. This is especially useful in situations where only a subset of the columns are needed.
+ If the HEADER value is not specified, it defaults to 1.
+ If HEADER is not specified, then columns are expected to match positionally with the text contents.
+ </para>
+ </listitem>
+ <listitem>
+ <para>SKIP specifies the number of text lines (counting every new line) to skip before parsing the contents. HEADER may still be specified with SKP.
+ </para>
+ </listitem>
+ </itemizedlist>
+ <itemizedlist>
+ <para>Examples</para>
+ <listitem>
+ <para>Use of the HEADER parameter, returns 1 row ['b']: <programlisting>select * from texttable('col1,col2,col3\na,b,c' COLUMNS col2 string HEADER) x</programlisting>
+ </para>
+ </listitem>
+ <listitem>
+ <para>Use of fixed width, returns 1 row ['a', 'b', 'c']: <programlisting>select * from texttable('abc' COLUMNS col1 string width 1, col2 string width 1, col3 string width 1) x</programlisting>
+ </para>
+ </listitem>
+ <listitem>
+ <para>Use of ESCAPE parameter, returns 1 row ['a,', 'b']: <programlisting>select * from texttable('a:,,b' COLUMNS col1 string, col2 string ESCAPE ':') x</programlisting>
+ </para>
+ </listitem>
+ <listitem>
+ <para>As a nested table: <programlisting>select x.* from t, texttable(t.clobcolumn COLUMNS first string, second date SKIP 1) x</programlisting>
+ </para>
+ </listitem>
+
+ </itemizedlist>
+ </sect3>
</sect2>
<sect2 id="where_clause">
<title>WHERE Clause</title>
Modified: trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -234,7 +234,7 @@
* @param tempElement Short element name
* @return Metadata ID or null if not found
*/
- public List getTempElementElementIDs(String tempGroup) {
+ public List<TempMetadataID> getTempElementElementIDs(String tempGroup) {
TempMetadataID groupID = tempGroups.get(tempGroup.toUpperCase());
if(groupID != null) {
return groupID.getElements();
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -67,6 +67,7 @@
import org.teiid.query.processor.relational.RelationalPlan;
import org.teiid.query.processor.relational.SelectNode;
import org.teiid.query.processor.relational.SortNode;
+import org.teiid.query.processor.relational.TextTableNode;
import org.teiid.query.processor.relational.UnionAllNode;
import org.teiid.query.processor.relational.JoinNode.JoinStrategyType;
import org.teiid.query.processor.relational.MergeJoinStrategy.SortOption;
@@ -79,6 +80,7 @@
import org.teiid.query.sql.lang.OrderBy;
import org.teiid.query.sql.lang.Query;
import org.teiid.query.sql.lang.StoredProcedure;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.SetQuery.Operation;
import org.teiid.query.sql.symbol.ElementSymbol;
import org.teiid.query.sql.symbol.Expression;
@@ -229,14 +231,10 @@
} else if (stype == JoinStrategyType.NESTED_TABLE) {
NestedTableJoinStrategy ntjStrategy = new NestedTableJoinStrategy();
jnode.setJoinStrategy(ntjStrategy);
- Command command = (Command)FrameUtil.findJoinSourceNode(node.getFirstChild()).getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null) {
- ntjStrategy.setLeftMap(command.getCorrelatedReferences());
- }
- command = (Command)FrameUtil.findJoinSourceNode(node.getLastChild()).getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null) {
- ntjStrategy.setRightMap(command.getCorrelatedReferences());
- }
+ SymbolMap references = (SymbolMap)FrameUtil.findJoinSourceNode(node.getFirstChild()).getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ ntjStrategy.setLeftMap(references);
+ references = (SymbolMap)FrameUtil.findJoinSourceNode(node.getLastChild()).getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ ntjStrategy.setRightMap(references);
} else {
NestedLoopJoinStrategy nljStrategy = new NestedLoopJoinStrategy();
jnode.setJoinStrategy(nljStrategy);
@@ -368,9 +366,14 @@
child.setProperty(NodeConstants.Info.OUTPUT_COLS, node.getProperty(NodeConstants.Info.OUTPUT_COLS));
}
}
-
- return null;
-
+ Object source = node.getProperty(NodeConstants.Info.TABLE_FUNCTION);
+ if (!(source instanceof TextTable)) {
+ return null;
+ }
+ TextTableNode ttn = new TextTableNode(getID());
+ ttn.setTable((TextTable)source);
+ processNode = ttn;
+ break;
case NodeConstants.Types.SET_OP:
Operation setOp = (Operation) node.getProperty(NodeConstants.Info.SET_OPERATION);
boolean useAll = ((Boolean) node.getProperty(NodeConstants.Info.USE_ALL)).booleanValue();
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -66,6 +66,7 @@
import org.teiid.query.resolver.QueryResolver;
import org.teiid.query.resolver.util.BindVariableVisitor;
import org.teiid.query.rewriter.QueryRewriter;
+import org.teiid.query.sql.LanguageObject;
import org.teiid.query.sql.LanguageObject.Util;
import org.teiid.query.sql.lang.Command;
import org.teiid.query.sql.lang.Criteria;
@@ -86,6 +87,7 @@
import org.teiid.query.sql.lang.StoredProcedure;
import org.teiid.query.sql.lang.SubqueryContainer;
import org.teiid.query.sql.lang.SubqueryFromClause;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.proc.CreateUpdateProcedureCommand;
import org.teiid.query.sql.symbol.GroupSymbol;
@@ -672,25 +674,20 @@
Command nestedCommand = sfc.getCommand();
node = NodeFactory.getNewNode(NodeConstants.Types.SOURCE);
if (sfc.isTable()) {
- PlanNode rootJoin = parent;
- while (rootJoin.getParent() != null && rootJoin.getParent().getType() == NodeConstants.Types.JOIN) {
- rootJoin = rootJoin.getParent();
- }
- List<Reference> correlatedReferences = new ArrayList<Reference>();
- CorrelatedReferenceCollectorVisitor.collectReferences(sfc, rootJoin.getGroups(), correlatedReferences);
-
- if (!correlatedReferences.isEmpty()) {
- SymbolMap map = new SymbolMap();
- for (Reference reference : correlatedReferences) {
- map.addMapping(reference.getExpression(), reference.getExpression());
- }
- sfc.getCommand().setCorrelatedReferences(map);
- }
+ sfc.getCommand().setCorrelatedReferences(getCorrelatedReferences(parent, node, sfc));
}
node.addGroup(group);
addNestedCommand(node, group, nestedCommand, nestedCommand, true);
hints.hasVirtualGroups = true;
parent.addLastChild(node);
+ } else if (clause instanceof TextTable) {
+ TextTable tt = (TextTable)clause;
+ GroupSymbol group = tt.getGroupSymbol();
+ node = NodeFactory.getNewNode(NodeConstants.Types.SOURCE);
+ node.setProperty(NodeConstants.Info.TABLE_FUNCTION, tt);
+ tt.setCorrelatedReferences(getCorrelatedReferences(parent, node, tt));
+ node.addGroup(group);
+ parent.addLastChild(node);
}
if (clause.isOptional()) {
@@ -705,6 +702,26 @@
}
}
+ private SymbolMap getCorrelatedReferences(PlanNode parent, PlanNode node,
+ LanguageObject lo) {
+ PlanNode rootJoin = parent;
+ while (rootJoin.getParent() != null && rootJoin.getParent().getType() == NodeConstants.Types.JOIN) {
+ rootJoin = rootJoin.getParent();
+ }
+ List<Reference> correlatedReferences = new ArrayList<Reference>();
+ CorrelatedReferenceCollectorVisitor.collectReferences(lo, rootJoin.getGroups(), correlatedReferences);
+
+ if (correlatedReferences.isEmpty()) {
+ return null;
+ }
+ SymbolMap map = new SymbolMap();
+ for (Reference reference : correlatedReferences) {
+ map.addMapping(reference.getExpression(), reference.getExpression());
+ }
+ node.setProperty(NodeConstants.Info.CORRELATED_REFERENCES, map);
+ return map;
+ }
+
private void addNestedCommand(PlanNode node,
GroupSymbol group, Command nestedCommand, Command toPlan, boolean merge) throws TeiidComponentException, QueryMetadataException, TeiidProcessingException {
if (nestedCommand instanceof QueryCommand) {
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -118,6 +118,8 @@
MAKE_DEP, // ??? List of Groups ???
PROCESSOR_PLAN, // ProcessorPlan for non-relational sub plan
NESTED_COMMAND, // Command for nested processor plan
+ TABLE_FUNCTION, // Table Function
+ CORRELATED_REFERENCES, // SymbolMap
MAKE_NOT_DEP, // Source should not be dependent
INLINE_VIEW, // If the source node represents an inline view
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -35,7 +35,6 @@
import java.util.Set;
import org.teiid.query.sql.LanguageObject;
-import org.teiid.query.sql.lang.Command;
import org.teiid.query.sql.lang.Criteria;
import org.teiid.query.sql.lang.SubqueryContainer;
import org.teiid.query.sql.symbol.ElementSymbol;
@@ -325,16 +324,16 @@
}
LinkedList<SymbolMap> result = new LinkedList<SymbolMap>();
for (PlanNode child : NodeEditor.findAllNodes(this, NodeConstants.Types.SOURCE, NodeConstants.Types.ACCESS)) {
- Command command = (Command)child.getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command == null || command.getCorrelatedReferences() == null) {
+ SymbolMap references = (SymbolMap)child.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ if (references == null) {
continue;
}
- Set<GroupSymbol> correlationGroups = GroupsUsedByElementsVisitor.getGroups(command.getCorrelatedReferences().getValues());
+ Set<GroupSymbol> correlationGroups = GroupsUsedByElementsVisitor.getGroups(references.getValues());
PlanNode joinNode = NodeEditor.findParent(child, NodeConstants.Types.JOIN, NodeConstants.Types.SOURCE);
while (joinNode != null) {
if (joinNode.getGroups().containsAll(correlationGroups)) {
if (joinNode == this) {
- result.add(command.getCorrelatedReferences());
+ result.add(references);
}
break;
}
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -345,9 +345,9 @@
float cost = UNKNOWN_VALUE;
if(node.getChildCount() > 0) {
- Command command = (Command)node.getProperty(NodeConstants.Info.NESTED_COMMAND);
+ SymbolMap references = (SymbolMap)node.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
//only cost non-correlated TODO: a better estimate for correlated
- if (command == null || command.getCorrelatedReferences() == null) {
+ if (references == null) {
PlanNode child = node.getFirstChild();
Float childCostFloat = (Float)child.getProperty(NodeConstants.Info.EST_CARDINALITY);
if (childCostFloat != null) {
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -47,6 +47,7 @@
import org.teiid.query.sql.lang.OrderBy;
import org.teiid.query.sql.symbol.GroupSymbol;
import org.teiid.query.sql.symbol.SingleElementSymbol;
+import org.teiid.query.sql.util.SymbolMap;
import org.teiid.query.sql.visitor.GroupsUsedByElementsVisitor;
import org.teiid.query.util.CommandContext;
@@ -69,9 +70,9 @@
TeiidComponentException {
for (PlanNode sourceNode : NodeEditor.findAllNodes(plan, NodeConstants.Types.SOURCE, NodeConstants.Types.ACCESS)) {
- Command command = (Command)sourceNode.getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null && command.getCorrelatedReferences() != null) {
- Set<GroupSymbol> groups = GroupsUsedByElementsVisitor.getGroups(command.getCorrelatedReferences().getValues());
+ SymbolMap references = (SymbolMap)sourceNode.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ if (references != null) {
+ Set<GroupSymbol> groups = GroupsUsedByElementsVisitor.getGroups(references.getValues());
PlanNode joinNode = NodeEditor.findParent(sourceNode, NodeConstants.Types.JOIN, NodeConstants.Types.SOURCE);
while (joinNode != null) {
joinNode.setProperty(NodeConstants.Info.JOIN_STRATEGY, JoinStrategyType.NESTED_TABLE);
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -90,8 +90,8 @@
return root;
}
- Command command = (Command)frame.getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null && command.getCorrelatedReferences() != null) {
+ SymbolMap references = (SymbolMap)frame.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ if (references != null) {
return root; //correlated nested table commands should not be merged
}
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -44,6 +44,7 @@
import org.teiid.query.optimizer.relational.plantree.PlanNode;
import org.teiid.query.resolver.util.ResolverUtil;
import org.teiid.query.sql.lang.Insert;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.symbol.ElementSymbol;
import org.teiid.query.sql.symbol.Expression;
import org.teiid.query.sql.symbol.GroupSymbol;
@@ -107,6 +108,10 @@
if (req == null) {
req = sourceNode.getProperty(NodeConstants.Info.NESTED_COMMAND);
}
+
+ if (sourceNode.getProperty(NodeConstants.Info.TABLE_FUNCTION) != null) {
+ return;
+ }
if (req instanceof Insert) {
isInsert = true;
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -47,7 +47,6 @@
import org.teiid.query.optimizer.relational.plantree.PlanNode;
import org.teiid.query.processor.relational.JoinNode.JoinStrategyType;
import org.teiid.query.resolver.util.AccessPattern;
-import org.teiid.query.sql.lang.Command;
import org.teiid.query.sql.lang.Criteria;
import org.teiid.query.sql.lang.JoinType;
import org.teiid.query.sql.symbol.ElementSymbol;
@@ -128,14 +127,11 @@
//account for nested table correlations
for (PlanNode joinSource : joinRegion.getJoinSourceNodes().keySet()) {
- Command command = (Command)joinSource.getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null) {
- SymbolMap map = command.getCorrelatedReferences();
- if (map !=null) {
- joinSource.setProperty(NodeConstants.Info.REQUIRED_ACCESS_PATTERN_GROUPS, GroupsUsedByElementsVisitor.getGroups(map.getValues()));
- joinRegion.setContainsNestedTable(true);
- }
- }
+ SymbolMap map = (SymbolMap)joinSource.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ if (map !=null) {
+ joinSource.setProperty(NodeConstants.Info.REQUIRED_ACCESS_PATTERN_GROUPS, GroupsUsedByElementsVisitor.getGroups(map.getValues()));
+ joinRegion.setContainsNestedTable(true);
+ }
}
//check for unsatisfied dependencies
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -162,7 +162,7 @@
List<SingleElementSymbol> groupingExpressions, PlanNode setOp)
throws TeiidComponentException, QueryMetadataException,
QueryPlannerException, QueryResolverException {
- if (setOp.getType() != NodeConstants.Types.SET_OP || setOp.getProperty(NodeConstants.Info.SET_OPERATION) != Operation.UNION) {
+ if (setOp == null || setOp.getType() != NodeConstants.Types.SET_OP || setOp.getProperty(NodeConstants.Info.SET_OPERATION) != Operation.UNION) {
return; //must not be a union
}
LinkedHashSet<AggregateSymbol> aggregates = collectAggregates(groupNode);
Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -178,8 +178,8 @@
return null;
}
- Command command = (Command)parentNode.getProperty(NodeConstants.Info.NESTED_COMMAND);
- if (command != null && command.getCorrelatedReferences() != null) {
+ SymbolMap references = (SymbolMap)parentNode.getProperty(NodeConstants.Info.CORRELATED_REFERENCES);
+ if (references != null) {
return null;
}
Modified: trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -146,7 +146,7 @@
} else {
rightSource.close();
for (Map.Entry<ElementSymbol, Expression> entry : rightMap.asMap().entrySet()) {
- joinNode.getContext().getVariableContext().setValue(entry.getKey(), null);
+ joinNode.getContext().getVariableContext().remove(entry.getKey());
}
}
@@ -157,7 +157,7 @@
if (leftMap != null) {
leftSource.close();
for (Map.Entry<ElementSymbol, Expression> entry : leftMap.asMap().entrySet()) {
- joinNode.getContext().getVariableContext().setValue(entry.getKey(), null);
+ joinNode.getContext().getVariableContext().remove(entry.getKey());
}
}
}
@@ -167,7 +167,11 @@
for (int i = 0; i < elements.size(); i++) {
SingleElementSymbol element = elements.get(i);
if (element instanceof ElementSymbol) {
- joinNode.getContext().getVariableContext().setValue((ElementSymbol)element, tuple==null?null:tuple.get(i));
+ if (tuple == null) {
+ joinNode.getContext().getVariableContext().remove((ElementSymbol)element);
+ } else {
+ joinNode.getContext().getVariableContext().setValue((ElementSymbol)element, tuple.get(i));
+ }
}
}
}
Added: trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java (rev 0)
+++ trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,391 @@
+/*
+ * JBoss, Home of Professional Open Source.
+ * See the COPYRIGHT.txt file distributed with this work for information
+ * regarding copyright ownership. Some portions may be licensed
+ * to Red Hat, Inc. under one or more contributor license agreements.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301 USA.
+ */
+
+package org.teiid.query.processor.relational;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.teiid.api.exception.query.ExpressionEvaluationException;
+import org.teiid.common.buffer.BlockedException;
+import org.teiid.common.buffer.BufferManager;
+import org.teiid.common.buffer.TupleBatch;
+import org.teiid.core.TeiidComponentException;
+import org.teiid.core.TeiidProcessingException;
+import org.teiid.core.types.ClobType;
+import org.teiid.core.types.DataTypeManager;
+import org.teiid.core.types.TransformationException;
+import org.teiid.query.execution.QueryExecPlugin;
+import org.teiid.query.processor.ProcessorDataManager;
+import org.teiid.query.sql.lang.TextTable;
+import org.teiid.query.sql.lang.TextTable.TextColumn;
+import org.teiid.query.sql.symbol.ElementSymbol;
+import org.teiid.query.sql.symbol.Expression;
+import org.teiid.query.util.CommandContext;
+
+/**
+ * Handles text file processing.
+ *
+ * TODO: unix style escape handling \t \n, etc.
+ * TODO: parse using something more memory safe than strings
+ * TODO: allow for escaping with fixed parsing
+ * TODO: allow for fixed parsing without new lines
+ * TODO: allow for a configurable line terminator
+ */
+public class TextTableNode extends SubqueryAwareRelationalNode {
+
+ private TextTable table;
+
+ //initialized state
+ private int skip = 0;
+ private int header = -1;
+ private boolean noQuote;
+ private char quote;
+ private char delimiter;
+ private int lineWidth;
+ private List<Class<?>> types;
+ private Map elementMap;
+ private int[] projectionIndexes;
+
+ //per file state
+ private BufferedReader reader;
+ private int textLine = 0;
+ private Map<String, Integer> nameIndexes;
+
+ public TextTableNode(int nodeID) {
+ super(nodeID);
+ }
+
+ @Override
+ public void initialize(CommandContext context, BufferManager bufferManager,
+ ProcessorDataManager dataMgr) {
+ super.initialize(context, bufferManager, dataMgr);
+ if (types != null) {
+ return;
+ }
+ if (table.getSkip() != null) {
+ skip = table.getSkip();
+ }
+ if (table.getHeader() != null) {
+ skip = Math.max(table.getHeader(), skip);
+ header = table.getHeader() - 1;
+ }
+ if (table.isFixedWidth()) {
+ for (TextColumn col : table.getColumns()) {
+ lineWidth += col.getWidth();
+ }
+ } else {
+ if (table.getDelimiter() == null) {
+ delimiter = ',';
+ } else {
+ delimiter = table.getDelimiter();
+ }
+ if (table.getQuote() == null) {
+ quote = '"';
+ } else {
+ noQuote = table.isEscape();
+ quote = table.getQuote();
+ }
+ }
+ types = new ArrayList<Class<?>>();
+ //TODO: if all string types, then we can skip this step
+ for (TextColumn col : table.getColumns()) {
+ types.add(DataTypeManager.getDataTypeClass(col.getType()));
+ }
+ this.elementMap = createLookupMap(table.getProjectedSymbols());
+ this.projectionIndexes = getProjectionIndexes(elementMap, getElements());
+ }
+
+ @Override
+ public void closeDirect() {
+ super.closeDirect();
+ reset();
+ }
+
+ @Override
+ public void reset() {
+ super.reset();
+ if (this.reader != null) {
+ try {
+ this.reader.close();
+ } catch (IOException e) {
+ }
+ this.reader = null;
+ }
+ this.nameIndexes = null;
+ this.textLine = 0;
+ }
+
+ public void setTable(TextTable table) {
+ this.table = table;
+ }
+
+ @Override
+ public TextTableNode clone() {
+ TextTableNode clone = new TextTableNode(getID());
+ this.copy(this, clone);
+ clone.setTable(table);
+ return clone;
+ }
+
+ @Override
+ protected TupleBatch nextBatchDirect() throws BlockedException,
+ TeiidComponentException, TeiidProcessingException {
+
+ if (reader == null) {
+ initReader();
+ }
+
+ if (reader == null) {
+ terminateBatches();
+ return pullBatch();
+ }
+
+ while (!isBatchFull()) {
+ String line = readLine();
+
+ if (line == null) {
+ terminateBatches();
+ break;
+ }
+
+ List<String> vals = parseLine(line);
+
+ List<Object> tuple = new ArrayList<Object>(projectionIndexes.length);
+ for (int output : projectionIndexes) {
+ TextColumn col = table.getColumns().get(output);
+ String val = null;
+ int index = output;
+ if (nameIndexes != null) {
+ index = nameIndexes.get(col.getName());
+ }
+ if (index >= vals.size()) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.no_value", col.getName(), textLine)); //$NON-NLS-1$
+ }
+ val = vals.get(index);
+ try {
+ tuple.add(DataTypeManager.transformValue(val, types.get(output)));
+ } catch (TransformationException e) {
+ throw new TeiidProcessingException(e, QueryExecPlugin.Util.getString("TextTableNode.conversion_error", col.getName(), textLine)); //$NON-NLS-1$
+ }
+ }
+ addBatchRow(tuple);
+ }
+
+ return pullBatch();
+ }
+
+ private String readLine() throws TeiidProcessingException {
+ while (true) {
+ try {
+ String line = reader.readLine();
+ if (line != null) {
+ textLine++;
+ if (line.length() == 0) {
+ continue;
+ }
+ }
+ return line;
+ } catch (IOException e) {
+ throw new TeiidProcessingException(e);
+ }
+ }
+ }
+
+ private void initReader() throws ExpressionEvaluationException,
+ BlockedException, TeiidComponentException, TeiidProcessingException {
+
+ if (table.getCorrelatedReferences() != null) {
+ for (Map.Entry<ElementSymbol, Expression> entry : table.getCorrelatedReferences().asMap().entrySet()) {
+ getContext().getVariableContext().setValue(entry.getKey(), getEvaluator(Collections.emptyMap()).evaluate(entry.getValue(), null));
+ }
+ }
+ ClobType file = (ClobType)getEvaluator(Collections.emptyMap()).evaluate(table.getFile(), null);
+
+ if (file == null) {
+ return;
+ }
+
+ //get the reader
+ try {
+ Reader r = file.getCharacterStream();
+ if (!(r instanceof BufferedReader)) {
+ reader = new BufferedReader(r);
+ } else {
+ reader = (BufferedReader)r;
+ }
+ } catch (SQLException e) {
+ throw new TeiidProcessingException(e);
+ }
+
+ //process the skip field
+ if (skip <= 0) {
+ return;
+ }
+ while (textLine < skip) {
+ boolean isHeader = textLine == header;
+ String line = readLine();
+ if (line == null) { //just return an empty batch
+ reset();
+ return;
+ }
+ if (isHeader) {
+ processHeader(parseLine(line));
+ }
+ }
+ }
+
+ private void processHeader(List<String> line) throws TeiidProcessingException {
+ nameIndexes = new HashMap<String, Integer>();
+ for (String string : line) {
+ if (string == null) {
+ continue;
+ }
+ nameIndexes.put(string.toUpperCase(), nameIndexes.size());
+ }
+ for (TextColumn col : table.getColumns()) {
+ Integer index = nameIndexes.get(col.getName().toUpperCase());
+ if (index == null) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.header_missing", col.getName())); //$NON-NLS-1$
+ }
+ nameIndexes.put(col.getName(), index);
+ }
+ }
+
+ private List<String> parseLine(String line) throws TeiidProcessingException {
+ if (table.isFixedWidth()) {
+ return parseFixedWidth(line);
+ }
+ return parseDelimitedLine(line);
+ }
+
+ private List<String> parseDelimitedLine(String line) throws TeiidProcessingException {
+ ArrayList<String> result = new ArrayList<String>();
+ StringBuilder builder = new StringBuilder();
+ boolean escaped = false;
+ boolean wasQualified = false;
+ boolean qualified = false;
+ while (true) {
+ if (line == null) {
+ if (escaped) {
+ builder.append('\n'); //allow for escaped new lines
+ escaped = false;
+ line = readLine();
+ continue;
+ }
+ if (!qualified) {
+ //close the last entry
+ addValue(result, wasQualified, builder.toString());
+ return result;
+ }
+ line = readLine();
+ if (line == null) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.unclosed")); //$NON-NLS-1$
+ }
+ }
+ char[] chars = line.toCharArray();
+ for (int i = 0; i < chars.length; i++) {
+ char chr = chars[i];
+ if (chr == delimiter) {
+ if (escaped || qualified) {
+ builder.append(chr);
+ escaped = false;
+ } else {
+ addValue(result, wasQualified, builder.toString());
+ wasQualified = false;
+ builder = new StringBuilder(); //next entry
+ }
+ } else if (chr == quote) {
+ if (noQuote) { //it's the escape char
+ if (escaped) {
+ builder.append(quote);
+ }
+ escaped = !escaped;
+ } else {
+ if (qualified) {
+ qualified = false;
+ } else {
+ if (wasQualified) {
+ qualified = true;
+ builder.append(chr);
+ } else {
+ if (builder.toString().trim().length() != 0) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.character_not_allowed", textLine)); //$NON-NLS-1$
+ }
+ qualified = true;
+ builder = new StringBuilder(); //start the entry over
+ wasQualified = true;
+ }
+ }
+ }
+ } else {
+ if (escaped) {
+ //don't understand other escape sequences yet
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.unknown_escape", chr, textLine)); //$NON-NLS-1$
+ }
+ if (wasQualified && !qualified) {
+ if (!Character.isWhitespace(chr)) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.character_not_allowed", textLine)); //$NON-NLS-1$
+ }
+ //else just ignore
+ } else {
+ builder.append(chr);
+ }
+ }
+ }
+ line = null;
+ }
+ }
+
+ private void addValue(ArrayList<String> result, boolean wasQualified, String val) {
+ if (!wasQualified) {
+ val = val.trim();
+ if (val.length() == 0) {
+ val = null;
+ }
+ }
+ result.add(val);
+ }
+
+ private List<String> parseFixedWidth(String line)
+ throws TeiidProcessingException {
+ if (line.length() < lineWidth) {
+ throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.invalid_width", line.length(), lineWidth, textLine)); //$NON-NLS-1$
+ }
+ ArrayList<String> result = new ArrayList<String>();
+ int beginIndex = 0;
+ for (TextColumn col : table.getColumns()) {
+ String val = line.substring(beginIndex, beginIndex + col.getWidth());
+ addValue(result, false, val);
+ beginIndex += col.getWidth();
+ }
+ return result;
+ }
+
+}
Property changes on: trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java
___________________________________________________________________
Name: svn:mime-type
+ text/plain
Modified: trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -40,6 +40,7 @@
import org.teiid.core.TeiidComponentException;
import org.teiid.core.TeiidException;
import org.teiid.core.TeiidRuntimeException;
+import org.teiid.core.types.DataTypeManager;
import org.teiid.query.QueryPlugin;
import org.teiid.query.analysis.AnalysisRecord;
import org.teiid.query.metadata.QueryMetadataInterface;
@@ -67,6 +68,7 @@
import org.teiid.query.sql.lang.SubqueryContainer;
import org.teiid.query.sql.lang.SubqueryFromClause;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.navigator.PostOrderNavigator;
import org.teiid.query.sql.symbol.AliasSymbol;
@@ -78,6 +80,7 @@
import org.teiid.query.sql.symbol.Reference;
import org.teiid.query.sql.symbol.ScalarSubquery;
import org.teiid.query.sql.symbol.SingleElementSymbol;
+import org.teiid.query.sql.visitor.ElementCollectorVisitor;
import org.teiid.query.util.ErrorMessageKeys;
@@ -147,7 +150,7 @@
public static class QueryResolverVisitor extends PostOrderNavigator {
private LinkedHashSet<GroupSymbol> currentGroups = new LinkedHashSet<GroupSymbol>();
- private List<GroupSymbol> discoveredGroups = new LinkedList<GroupSymbol>();
+ private LinkedList<GroupSymbol> discoveredGroups = new LinkedList<GroupSymbol>();
private List<GroupSymbol> implicitGroups = new LinkedList<GroupSymbol>();
private TempMetadataAdapter metadata;
private Query query;
@@ -277,6 +280,60 @@
postVisitVisitor(obj);
}
+ /**
+ * TextTables have similar resolving semantics to a nested table.
+ */
+ @Override
+ public void visit(TextTable obj) {
+ LinkedHashSet<GroupSymbol> saved = new LinkedHashSet<GroupSymbol>(this.currentGroups);
+ if (allowImplicit) {
+ currentGroups.addAll(this.implicitGroups);
+ }
+ this.visitNode(obj.getFile());
+ try {
+ obj.setFile(ResolverUtil.convertExpression(obj.getFile(), DataTypeManager.DefaultDataTypes.CLOB, metadata));
+ } catch (QueryResolverException e) {
+ throw new TeiidRuntimeException(e);
+ }
+ //we didn't create a true external context, so we manually mark external
+ for (ElementSymbol symbol : ElementCollectorVisitor.getElements(obj, false)) {
+ if (symbol.isExternalReference()) {
+ continue;
+ }
+ if (implicitGroups.contains(symbol.getGroupSymbol())) {
+ symbol.setIsExternalReference(true);
+ }
+ }
+ if (allowImplicit) {
+ this.currentGroups.clear();
+ this.currentGroups.addAll(saved);
+ }
+ discoveredGroup(obj.getGroupSymbol());
+ try {
+ ResolverUtil.addTempGroup(metadata, obj.getGroupSymbol(), obj.getProjectedSymbols(), false);
+ } catch (QueryResolverException err) {
+ throw new TeiidRuntimeException(err);
+ }
+ obj.getGroupSymbol().setMetadataID(metadata.getMetadataStore().getTempGroupID(obj.getGroupSymbol().getName()));
+ //now resolve the projected symbols
+ Set<GroupSymbol> groups = new HashSet<GroupSymbol>();
+ groups.add(obj.getGroupSymbol());
+ for (ElementSymbol symbol : obj.getProjectedSymbols()) {
+ try {
+ ResolverVisitor.resolveLanguageObject(symbol, groups, null, metadata);
+ } catch (TeiidException e) {
+ throw new TeiidRuntimeException(e);
+ }
+ }
+ //set to fixed width if any column has width specified
+ for (TextTable.TextColumn col : obj.getColumns()) {
+ if (col.getWidth() != null) {
+ obj.setFixedWidth(true);
+ break;
+ }
+ }
+ }
+
public void visit(SubqueryFromClause obj) {
Collection<GroupSymbol> externalGroups = this.currentGroups;
if (obj.isTable() && allowImplicit) {
Modified: trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -106,6 +106,7 @@
import org.teiid.query.sql.lang.SubqueryContainer;
import org.teiid.query.sql.lang.SubqueryFromClause;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.TranslatableProcedureContainer;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.lang.Update;
@@ -917,6 +918,9 @@
return rewriteJoinPredicate(parent, (JoinPredicate) clause);
} else if (clause instanceof SubqueryFromClause) {
rewriteSubqueryContainer((SubqueryFromClause)clause, true);
+ } else if (clause instanceof TextTable) {
+ TextTable tt = (TextTable)clause;
+ tt.setFile(rewriteExpressionDirect(tt.getFile()));
}
return clause;
}
Modified: trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -130,4 +130,5 @@
public void visit(XMLAttributes obj) {}
public void visit(XMLForest obj) {}
public void visit(XMLNamespaces obj) {}
+ public void visit(TextTable obj) {}
}
Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -25,11 +25,10 @@
import java.util.ArrayList;
import java.util.List;
-import org.teiid.core.TeiidComponentException;
import org.teiid.core.util.EquivalenceUtil;
import org.teiid.core.util.HashCodeUtil;
-import org.teiid.query.metadata.QueryMetadataInterface;
import org.teiid.query.sql.LanguageVisitor;
+import org.teiid.query.sql.symbol.ElementSymbol;
import org.teiid.query.sql.symbol.GroupSymbol;
import org.teiid.query.sql.visitor.SQLStringVisitor;
@@ -41,7 +40,7 @@
/** Identifies the table to be created. */
private GroupSymbol table;
- private List columns = new ArrayList();
+ private List<ElementSymbol> columns = new ArrayList<ElementSymbol>();
public GroupSymbol getTable() {
return table;
@@ -51,7 +50,7 @@
this.table = table;
}
- public List getColumns() {
+ public List<ElementSymbol> getColumns() {
return columns;
}
@@ -100,7 +99,7 @@
visitor.visit(this);
}
- public void setColumns(List columns) {
+ public void setColumns(List<ElementSymbol> columns) {
this.columns = columns;
}
Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -26,6 +26,7 @@
import org.teiid.query.sql.LanguageObject;
import org.teiid.query.sql.LanguageVisitor;
+import org.teiid.query.sql.visitor.SQLStringVisitor;
/**
@@ -87,4 +88,9 @@
&& other.isMakeDep() == this.isMakeDep()
&& other.isMakeNotDep() == this.isMakeNotDep();
}
+
+ @Override
+ public String toString() {
+ return SQLStringVisitor.getSQLString(this);
+ }
}
Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -22,12 +22,14 @@
package org.teiid.query.sql.lang;
-import java.util.*;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
import org.teiid.core.util.EquivalenceUtil;
import org.teiid.core.util.HashCodeUtil;
-import org.teiid.query.sql.*;
-import org.teiid.query.sql.visitor.SQLStringVisitor;
+import org.teiid.query.sql.LanguageVisitor;
/**
@@ -249,12 +251,4 @@
return clonedJoinPredicate;
}
- /**
- * Returns a string representation of an instance of this class.
- * @return String representation of object
- */
- public String toString() {
- return SQLStringVisitor.getSQLString(this);
- }
-
}
Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -26,7 +26,6 @@
import org.teiid.query.sql.LanguageVisitor;
import org.teiid.query.sql.symbol.GroupSymbol;
-import org.teiid.query.sql.visitor.SQLStringVisitor;
/**
@@ -174,12 +173,4 @@
return clause;
}
- /**
- * Returns a string representation of an instance of this class.
- * @return String representation of object
- */
- public String toString() {
- return SQLStringVisitor.getSQLString(this);
- }
-
}
Added: trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java (rev 0)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,263 @@
+package org.teiid.query.sql.lang;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import org.teiid.core.types.DataTypeManager;
+import org.teiid.core.util.EquivalenceUtil;
+import org.teiid.query.sql.LanguageObject;
+import org.teiid.query.sql.LanguageVisitor;
+import org.teiid.query.sql.symbol.ElementSymbol;
+import org.teiid.query.sql.symbol.Expression;
+import org.teiid.query.sql.symbol.GroupSymbol;
+import org.teiid.query.sql.util.SymbolMap;
+
+public class TextTable extends FromClause {
+
+ public static class TextColumn {
+ private String name;
+ private String type;
+ private Integer width;
+
+ public TextColumn(String name, String type, Integer width) {
+ this.name = name;
+ this.type = type;
+ this.width = width;
+ }
+
+ public void setName(String name) {
+ this.name = name;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public String getType() {
+ return type;
+ }
+
+ public void setType(String type) {
+ this.type = type;
+ }
+
+ public Integer getWidth() {
+ return width;
+ }
+
+ public void setWidth(Integer width) {
+ this.width = width;
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (!(obj instanceof TextColumn)) {
+ return false;
+ }
+ TextColumn other = (TextColumn)obj;
+ return this.name.equalsIgnoreCase(other.name) && this.type.equalsIgnoreCase(other.type)
+ && EquivalenceUtil.areEqual(width, other.width);
+ }
+
+ public int hashCode() {
+ return name.hashCode();
+ }
+
+ @Override
+ public TextColumn clone() {
+ return new TextColumn(name, type, width);
+ }
+ }
+
+ private GroupSymbol symbol;
+ private List<ElementSymbol> projectedSymbols;
+ private SymbolMap correlatedReferences;
+
+ private Expression file;
+ private List<TextColumn> columns = new ArrayList<TextColumn>();
+ private Character delimiter;
+ private Character quote;
+ private boolean escape;
+ private Integer header;
+ private Integer skip;
+
+ private boolean fixedWidth;
+
+ public Character getQuote() {
+ return quote;
+ }
+
+ public void setQuote(Character quote) {
+ this.quote = quote;
+ }
+
+ public SymbolMap getCorrelatedReferences() {
+ return correlatedReferences;
+ }
+
+ public void setCorrelatedReferences(SymbolMap correlatedReferences) {
+ this.correlatedReferences = correlatedReferences;
+ }
+
+ public boolean isEscape() {
+ return escape;
+ }
+
+ public void setEscape(boolean escape) {
+ this.escape = escape;
+ }
+
+ public boolean isFixedWidth() {
+ return fixedWidth;
+ }
+
+ public void setFixedWidth(boolean fixedWidth) {
+ this.fixedWidth = fixedWidth;
+ }
+
+ public List<TextColumn> getColumns() {
+ return columns;
+ }
+
+ public void setColumns(List<TextColumn> columns) {
+ this.columns = columns;
+ }
+
+ public Character getDelimiter() {
+ return delimiter;
+ }
+
+ public void setDelimiter(Character delimiter) {
+ this.delimiter = delimiter;
+ }
+
+ public Integer getHeader() {
+ return header;
+ }
+
+ public void setHeader(Integer header) {
+ this.header = header;
+ }
+
+ public Integer getSkip() {
+ return skip;
+ }
+
+ public void setSkip(Integer skip) {
+ this.skip = skip;
+ }
+
+ public Expression getFile() {
+ return file;
+ }
+
+ public void setFile(Expression file) {
+ this.file = file;
+ }
+
+ @Override
+ public void acceptVisitor(LanguageVisitor visitor) {
+ visitor.visit(this);
+ }
+
+ @Override
+ public TextTable clone() {
+ TextTable clone = new TextTable();
+ clone.setDelimiter(this.delimiter);
+ clone.setFile((Expression)this.file.clone());
+ clone.setHeader(this.header);
+ clone.setSkip(this.skip);
+ clone.setQuote(this.quote);
+ clone.escape = this.escape;
+ clone.symbol = (GroupSymbol)this.symbol.clone();
+ for (TextColumn column : columns) {
+ clone.getColumns().add(column.clone());
+ }
+ if (projectedSymbols != null) {
+ clone.projectedSymbols = LanguageObject.Util.deepClone(this.projectedSymbols, ElementSymbol.class);
+ }
+ if (correlatedReferences != null) {
+ clone.correlatedReferences = correlatedReferences.clone();
+ }
+ clone.fixedWidth = this.fixedWidth;
+ return clone;
+ }
+
+ @Override
+ public void collectGroups(Collection groups) {
+ groups.add(getGroupSymbol());
+ }
+
+ /**
+ * Get name of this clause.
+ * @return Name of clause
+ */
+ public String getName() {
+ return this.symbol.getName();
+ }
+
+ public String getOutputName() {
+ return this.symbol.getOutputName();
+ }
+
+ /**
+ * Get GroupSymbol representing the named subquery
+ * @return GroupSymbol representing the subquery
+ */
+ public GroupSymbol getGroupSymbol() {
+ return this.symbol;
+ }
+
+ /**
+ * Reset the alias for this subquery from clause and it's pseudo-GroupSymbol.
+ * WARNING: this will modify the hashCode and equals semantics and will cause this object
+ * to be lost if currently in a HashMap or HashSet.
+ * @param name New name
+ * @since 4.3
+ */
+ public void setName(String name) {
+ this.symbol = new GroupSymbol(name);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj == this) {
+ return true;
+ }
+ if (!(obj instanceof TextTable)) {
+ return false;
+ }
+ TextTable other = (TextTable)obj;
+ return this.columns.equals(other.columns)
+ && EquivalenceUtil.areEqual(file, other.file)
+ && EquivalenceUtil.areEqual(symbol, other.symbol)
+ && EquivalenceUtil.areEqual(delimiter, other.delimiter)
+ && EquivalenceUtil.areEqual(escape, other.escape)
+ && EquivalenceUtil.areEqual(quote, other.quote)
+ && EquivalenceUtil.areEqual(header, other.header)
+ && EquivalenceUtil.areEqual(skip, other.skip);
+
+ }
+
+ @Override
+ public int hashCode() {
+ return this.symbol.hashCode();
+ }
+
+ public List<ElementSymbol> getProjectedSymbols() {
+ if (projectedSymbols == null) {
+ projectedSymbols = new ArrayList<ElementSymbol>(columns.size());
+ for (TextColumn column : columns) {
+ ElementSymbol elementSymbol = new ElementSymbol(column.getName());
+ elementSymbol.setType(DataTypeManager.getDataTypeClass(column.getType()));
+ projectedSymbols.add(elementSymbol);
+ }
+ }
+ return projectedSymbols;
+ }
+
+}
Property changes on: trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java
___________________________________________________________________
Name: svn:mime-type
+ text/plain
Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -25,9 +25,8 @@
import java.util.Collection;
import org.teiid.core.util.EquivalenceUtil;
-import org.teiid.query.sql.*;
+import org.teiid.query.sql.LanguageVisitor;
import org.teiid.query.sql.symbol.GroupSymbol;
-import org.teiid.query.sql.visitor.SQLStringVisitor;
/**
@@ -133,15 +132,6 @@
return clonedUnaryFromClause;
}
-
- /**
- * Returns a string representation of an instance of this class.
- * @return String representation of object
- */
- public String toString() {
- return SQLStringVisitor.getSQLString(this);
- }
-
/**
* @return Returns the expandedCommand.
*/
Modified: trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -61,6 +61,7 @@
import org.teiid.query.sql.lang.SubqueryCompareCriteria;
import org.teiid.query.sql.lang.SubqueryFromClause;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.lang.Update;
import org.teiid.query.sql.lang.XQuery;
@@ -551,6 +552,14 @@
postVisitVisitor(obj);
}
+ @Override
+ public void visit(TextTable obj) {
+ preVisitVisitor(obj);
+ visitNode(obj.getFile());
+ visitNode(obj.getGroupSymbol());
+ postVisitVisitor(obj);
+ }
+
public static void doVisit(LanguageObject object, LanguageVisitor visitor, boolean order) {
PreOrPostOrderNavigator nav = new PreOrPostOrderNavigator(visitor, order);
object.acceptVisitor(nav);
Modified: trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -75,9 +75,11 @@
import org.teiid.query.sql.lang.SubqueryCompareCriteria;
import org.teiid.query.sql.lang.SubqueryFromClause;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.lang.Update;
import org.teiid.query.sql.lang.XQuery;
+import org.teiid.query.sql.lang.TextTable.TextColumn;
import org.teiid.query.sql.proc.AssignmentStatement;
import org.teiid.query.sql.proc.Block;
import org.teiid.query.sql.proc.BreakStatement;
@@ -427,11 +429,11 @@
parts.add(SPACE);
// Columns clause
- List columns = obj.getColumns();
+ List<ElementSymbol> columns = obj.getColumns();
parts.add("("); //$NON-NLS-1$
- Iterator iter = columns.iterator();
+ Iterator<ElementSymbol> iter = columns.iterator();
while(iter.hasNext()) {
- ElementSymbol element = (ElementSymbol) iter.next();
+ ElementSymbol element = iter.next();
element.setDisplayMode(ElementSymbol.DisplayMode.SHORT_OUTPUT_NAME);
parts.add(registerNode(element));
parts.add(SPACE);
@@ -1604,13 +1606,12 @@
parts.add("NO DEFAULT"); //$NON-NLS-1$
} else {
parts.add("DEFAULT '"); //$NON-NLS-1$
- parts.add(escapeStringValue(item.getUri(), "'")); //$NON-NLS-1$
+ parts.add(registerNode(new Constant(item.getUri())));
parts.add("'"); //$NON-NLS-1$
}
} else {
- parts.add("'"); //$NON-NLS-1$
- parts.add(escapeStringValue(item.getUri(), "'")); //$NON-NLS-1$
- parts.add("' AS "); //$NON-NLS-1$
+ parts.add(registerNode(new Constant(item.getUri())));
+ parts.add(" AS "); //$NON-NLS-1$
outputDisplayName(item.getPrefix());
}
if (items.hasNext()) {
@@ -1630,6 +1631,66 @@
parts.add(SPACE);
parts.add(registerNode(obj.getRowLimit()));
}
+
+ @Override
+ public void visit(TextTable obj) {
+ parts.add("TEXTTABLE("); //$NON-NLS-1$
+ parts.add(registerNode(obj.getFile()));
+ parts.add(SPACE);
+ parts.add("COLUMNS"); //$NON-NLS-1$
+
+ for (Iterator<TextColumn> cols = obj.getColumns().iterator(); cols.hasNext();) {
+ TextColumn col = cols.next();
+ parts.add(SPACE);
+ outputDisplayName(col.getName());
+ parts.add(SPACE);
+ parts.add(col.getType());
+ if (col.getWidth() != null) {
+ parts.add(SPACE);
+ parts.add("WIDTH"); //$NON-NLS-1$
+ parts.add(SPACE);
+ parts.add(col.getWidth());
+ }
+ if (cols.hasNext()) {
+ parts.add(","); //$NON-NLS-1$
+ }
+ }
+ if (obj.getDelimiter() != null) {
+ parts.add(SPACE);
+ parts.add("DELIMITER"); //$NON-NLS-1$
+ parts.add(SPACE);
+ parts.add(registerNode(new Constant(obj.getDelimiter())));
+ }
+ if (obj.getQuote() != null) {
+ parts.add(SPACE);
+ if (obj.isEscape()) {
+ parts.add(SQLReservedWords.ESCAPE);
+ } else {
+ parts.add("QUOTE"); //$NON-NLS-1$
+ }
+ parts.add(SPACE);
+ parts.add(registerNode(new Constant(obj.getQuote())));
+ }
+ if (obj.getHeader() != null) {
+ parts.add(SPACE);
+ parts.add("HEADER"); //$NON-NLS-1$
+ if (1 != obj.getHeader()) {
+ parts.add(SPACE);
+ parts.add(obj.getHeader());
+ }
+ }
+ if (obj.getSkip() != null) {
+ parts.add(SPACE);
+ parts.add("SKIP"); //$NON-NLS-1$
+ parts.add(SPACE);
+ parts.add(obj.getSkip());
+ }
+ parts.add(")");//$NON-NLS-1$
+ parts.add(SPACE);
+ parts.add(SQLReservedWords.AS);
+ parts.add(SPACE);
+ outputDisplayName(obj.getName());
+ }
public static String escapeSinglePart(String part) {
if(isReservedWord(part)) {
Modified: trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java
===================================================================
--- trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -40,6 +40,7 @@
import org.teiid.core.TeiidComponentException;
import org.teiid.core.TeiidProcessingException;
import org.teiid.core.types.DataTypeManager;
+import org.teiid.core.util.EquivalenceUtil;
import org.teiid.query.QueryPlugin;
import org.teiid.query.eval.Evaluator;
import org.teiid.query.function.FunctionLibrary;
@@ -78,6 +79,7 @@
import org.teiid.query.sql.lang.SetQuery;
import org.teiid.query.sql.lang.SubqueryCompareCriteria;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.Update;
import org.teiid.query.sql.lang.SetQuery.Operation;
import org.teiid.query.sql.navigator.PreOrderNavigator;
@@ -114,7 +116,6 @@
import org.teiid.query.util.ErrorMessageKeys;
import org.teiid.translator.SourceSystemFunctions;
-
public class ValidationVisitor extends AbstractValidationVisitor {
private final class PositiveIntegerConstraint implements
@@ -1157,5 +1158,51 @@
}
}
}
+
+ @Override
+ public void visit(TextTable obj) {
+ boolean widthSet = false;
+ Character delimiter = null;
+ Character quote = null;
+ for (TextTable.TextColumn column : obj.getColumns()) {
+ if (column.getWidth() != null) {
+ widthSet = true;
+ if (column.getWidth() < 0) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$
+ }
+ } else if (widthSet) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_invalid_width"), obj); //$NON-NLS-1$
+ }
+ }
+ if (widthSet) {
+ if (obj.getDelimiter() != null || obj.getHeader() != null || obj.getQuote() != null) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_width"), obj); //$NON-NLS-1$
+ }
+ } else {
+ delimiter = obj.getDelimiter();
+ if (delimiter == null) {
+ delimiter = ',';
+ }
+ }
+ if (obj.getSkip() != null && obj.getSkip() < 0) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$
+ }
+ if (!widthSet) {
+ if (obj.getHeader() != null && obj.getHeader() < 0) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$
+ }
+ quote = obj.getQuote();
+ if (quote == null) {
+ quote = '"';
+ }
+ if (EquivalenceUtil.areEqual(quote, delimiter)) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_delimiter"), obj); //$NON-NLS-1$
+ }
+ if (EquivalenceUtil.areEqual(quote, '\n')
+ || EquivalenceUtil.areEqual(delimiter, '\n')) {
+ handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_newline"), obj); //$NON-NLS-1$
+ }
+ }
+ }
}
Modified: trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj
===================================================================
--- trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj 2010-05-27 13:45:40 UTC (rev 2156)
@@ -1880,7 +1880,9 @@
Token lparen = null;
}
{
- ( clause = unaryFromClause(info)
+ ( LOOKAHEAD(<ID> <LPAREN>, { "texttable".equalsIgnoreCase(getToken(1).image) }) clause = textTable(info)
+ |
+ clause = unaryFromClause(info)
|
LOOKAHEAD(subqueryFromClause(info)) clause = subqueryFromClause(info)
|
@@ -1889,7 +1891,7 @@
setFromClauseOptions(lparen, clause);
}
<RPAREN>
- )
+ )
)
[ (<MAKEDEP> { clause.setMakeDep(true); })
| (<MAKENOTDEP> { clause.setMakeNotDep(true); } )
@@ -1899,6 +1901,103 @@
}
}
+TextTable textTable(ParseInfo info) :
+{
+ Expression file = null;
+ String name = null;
+ Constant datatype = null;
+ List<TextTable.TextColumn> columns = new ArrayList<TextTable.TextColumn>();
+ Integer width = null;
+ Character delimiter = null;
+ boolean escape = false;
+ Character quote = null;
+ Integer header = null;
+ Integer skip = null;
+ Token columnsToken = null;
+ String aliasID = null;
+}
+{
+ <ID> <LPAREN> file = expression(info)
+ columnsToken = <ID> { if (!"columns".equalsIgnoreCase(columnsToken.image)) throw new ParseException(QueryPlugin.Util.getString("SQLParser.expected_columns")); }
+ name = id()
+ datatype = dataType()
+ [
+ LOOKAHEAD(<ID>, { "width".equalsIgnoreCase(getToken(1).image) }) <ID>
+ width = intVal()
+ ]
+ {
+ columns.add(new TextTable.TextColumn(validateElementName(name), (String)datatype.getValue(), width));
+ width = null;
+ }
+ (<COMMA>
+ name = id()
+ datatype = dataType()
+ [
+ LOOKAHEAD(<ID>, { "width".equalsIgnoreCase(getToken(1).image) }) <ID>
+ width = intVal()
+ ]
+ {
+ columns.add(new TextTable.TextColumn(validateElementName(name), (String)datatype.getValue(), width));
+ width = null;
+ }
+ )*
+ [
+ LOOKAHEAD(<ID>, { "delimiter".equalsIgnoreCase(getToken(1).image) }) <ID>
+ delimiter = charVal(info, "DELMITER")
+ ]
+ [
+ (<ESCAPE> quote = charVal(info, "ESCAPE") ) { escape = true; }
+ |
+ LOOKAHEAD(<ID>, { "quote".equalsIgnoreCase(getToken(1).image) })
+ ( <ID> quote = charVal(info, "QUOTE") )
+ ]
+ [
+ LOOKAHEAD(<ID>, { "header".equalsIgnoreCase(getToken(1).image) }) <ID>
+ [
+ header = intVal()
+ ]
+ {
+ if (header == null) {
+ header = 1;
+ }
+ }
+ ]
+ [
+ LOOKAHEAD(<ID>, { "skip".equalsIgnoreCase(getToken(1).image) }) <ID>
+ skip = intVal()
+ ]
+ <RPAREN>
+ [<AS>] aliasID=id()
+ {
+ TextTable result = new TextTable();
+ result.setFile(file);
+ result.setColumns(columns);
+ result.setDelimiter(delimiter);
+ result.setEscape(escape);
+ result.setHeader(header);
+ result.setSkip(skip);
+ result.setName(validateAlias(aliasID));
+ result.setQuote(quote);
+ return result;
+ }
+}
+
+int intVal() :
+{
+ Token t;
+}
+{
+ t = <INTEGERVAL>
+ {
+ try {
+ return Integer.valueOf(t.image);
+ } catch(NumberFormatException e1) {
+ throw new ParseException(QueryPlugin.Util.getString("SQLParser.Integer_parse", t.image)); //$NON-NLS-1$
+ }
+ }
+}
+
+
/**
* Parse a unary from clause, which is just a single group name and an optional alias.
* @return Unary from clause containing the group
@@ -2228,8 +2327,8 @@
[<NOT> {negated = true;}]
<LIKE>
value=expression(info)
- [ esc = escapeChar(info) |
- (<LBRACE> esc = escapeChar(info) <RBRACE>)
+ [ <ESCAPE> esc = charVal(info, "LIKE ESCAPE") |
+ (<LBRACE> <ESCAPE> esc = charVal(info, "LIKE ESCAPE") <RBRACE>)
]
)
{
@@ -2242,15 +2341,15 @@
}
}
-Character escapeChar(ParseInfo info) :
+Character charVal(ParseInfo info, String type) :
{
String escStr = null;
}
{
- <ESCAPE> escStr=stringVal()
+ escStr=stringVal()
{
if (escStr.length() != 1) {
- throw new ParseException(QueryPlugin.Util.getString("SQLParser.Invalid_escape_char", escStr)); //$NON-NLS-1$
+ throw new ParseException(QueryPlugin.Util.getString("SQLParser.Invalid_char", type, escStr)); //$NON-NLS-1$
}
return Character.valueOf(escStr.charAt(0));
}
Modified: trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties
===================================================================
--- trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties 2010-05-27 13:45:40 UTC (rev 2156)
@@ -208,3 +208,11 @@
FileStorageManager.error_getting_free_disk_space=Error while seeking free disk space for directory: {0}
FileStorageManager.can_not_save_lobs=Can not save *Reference Lobs* into persistent disk storage.
FileStorageManager.batch_error=Error while converting batched data to file storage.
+
+TextTableNode.no_value=No value found for column \"{0}\" in the row ending on text line {1}.
+TextTableNode.conversion_error=Could not convert value for column \"{0}\" in the row ending on text line {1}.
+TextTableNode.header_missing=HEADER entry missing for column name \"{0}\".
+TextTableNode.unclosed=Text parse error: Unclosed qualifier at end of text.
+TextTableNode.character_not_allowed=Text parse error: Non-whitespace character found between the qualifier and the delimiter in text line {0}.
+TextTableNode.unknown_escape=Text parse error: Unknown escape sequence \\{0} in text line {1}.
+TextTableNode.invalid_width=Text parse error: Fixed width line width {0} is smaller than the expected {1} on text line {2}
Modified: trunk/engine/src/main/resources/org/teiid/query/i18n.properties
===================================================================
--- trunk/engine/src/main/resources/org/teiid/query/i18n.properties 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/main/resources/org/teiid/query/i18n.properties 2010-05-27 13:45:40 UTC (rev 2156)
@@ -422,8 +422,8 @@
SQLParser.Invalid_id=Invalid metadata ID format: [{0}]
SQLParser.Invalid_alias=Invalid alias format: [{0}]
SQLParser.Invalid_short_name=Invalid simple identifier format: [{0}]
-SQLParser.call_expected=Call keyword expected in callable statement
-SQLParser.Invalid_escape_char=Like escape value must be a single character.
+SQLParser.Invalid_char={0} value must be a single character: [{1}].
+SQLParser.expected_columns=Columns non-reserved word expected in TEXTTABLE.
SystemSource.Add_desc=Add two numbers
SystemSource.Add_result_desc=Left operand + right operand
SystemSource.Subtract_desc=Subtract two numbers
@@ -884,6 +884,11 @@
ValidationVisitor.drop_of_nontemptable=Cannot drop a non temporary table "{0}".
ValidationVisitor.unrelated_orderby_xml=XML queries cannot order by an unrelated order by item.
ValidationVisitor.orderby_expression_xml=XML queries cannot order by an expression.
+ValidationVisitor.text_table_invalid_width=For a fixed width text table, all columns must have width set.
+ValidationVisitor.text_table_width=Fixed width text tables should not have DELIMITER, QUOTE, ESCAPE, or HEADER specified.
+ValidationVisitor.text_table_negative=Text table WIDTH, HEADER, or SKIP values must not be negative.
+ValidationVisitor.text_table_delimiter=Text table DELIMITER cannot be the same as the QUOTE or ESCAPE characters.
+ValidationVisitor.text_table_newline=Text table DELIMITER, QUOTE, and ESCAPE characters cannot be the new line character.
UpdateProcedureResolver.only_variables=Element symbol "{0}" cannot be assigned a value. Only declared VARIABLES can be assigned values.
wrong_result_type=No results found; or non-XML result object has been produced as a result of the execution of XQuery expression. Please note that only XML type results are supported.
MappingLoader.unknown_node_type=Unknown Node Type "{0}" being loaded by the XML mapping document.
Modified: trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -71,9 +71,11 @@
import org.teiid.query.sql.lang.SubqueryCompareCriteria;
import org.teiid.query.sql.lang.SubqueryFromClause;
import org.teiid.query.sql.lang.SubquerySetCriteria;
+import org.teiid.query.sql.lang.TextTable;
import org.teiid.query.sql.lang.UnaryFromClause;
import org.teiid.query.sql.lang.Update;
import org.teiid.query.sql.lang.SetQuery.Operation;
+import org.teiid.query.sql.lang.TextTable.TextColumn;
import org.teiid.query.sql.proc.AssignmentStatement;
import org.teiid.query.sql.proc.Block;
import org.teiid.query.sql.proc.BreakStatement;
@@ -2800,7 +2802,7 @@
}
@Test public void testLikeWithEscapeException(){
- helpException("SELECT a from db.g where b like '#String' escape '#1'", "Parsing error: Like escape value must be a single character."); //$NON-NLS-1$ //$NON-NLS-2$
+ helpException("SELECT a from db.g where b like '#String' escape '#1'", "Parsing error: LIKE ESCAPE value must be a single character: [#1]."); //$NON-NLS-1$ //$NON-NLS-2$
}
/** SELECT "date"."time" from db.g */
@@ -6754,5 +6756,35 @@
query.setFrom(new From(Arrays.asList(sfc)));
helpTest(sql, "SELECT * FROM TABLE(EXEC foo()) AS x", query);
}
+
+ @Test public void testTextTable() throws Exception {
+ String sql = "SELECT * from texttable(file columns x string WIDTH 1, y date width 10 skip 10) as x"; //$NON-NLS-1$
+ Query query = new Query();
+ query.setSelect(new Select(Arrays.asList(new AllSymbol())));
+ TextTable tt = new TextTable();
+ tt.setFile(new ElementSymbol("file"));
+ List<TextTable.TextColumn> columns = new ArrayList<TextTable.TextColumn>();
+ columns.add(new TextTable.TextColumn("x", "string", 1));
+ columns.add(new TextTable.TextColumn("y", "date", 10));
+ tt.setColumns(columns);
+ tt.setSkip(10);
+ tt.setName("x");
+ query.setFrom(new From(Arrays.asList(tt)));
+ helpTest(sql, "SELECT * FROM TEXTTABLE(file COLUMNS x string WIDTH 1, y date WIDTH 10 SKIP 10) AS x", query);
+
+ sql = "SELECT * from texttable(file columns x string, y date delimiter ',' escape '\"' header skip 10) as x"; //$NON-NLS-1$
+ tt.setDelimiter(',');
+ tt.setQuote('"');
+ tt.setEscape(true);
+ tt.setHeader(1);
+ for (TextColumn textColumn : columns) {
+ textColumn.setWidth(null);
+ }
+ helpTest(sql, "SELECT * FROM TEXTTABLE(file COLUMNS x string, y date DELIMITER ',' ESCAPE '\"' HEADER SKIP 10) AS x", query);
+ }
+
+ @Test public void testTextTableColumns() throws Exception {
+ helpException("SELECT * from texttable(foo x string)", "Parsing error: Columns non-reserved word expected in TEXTTABLE.");
+ }
}
Modified: trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -93,7 +93,7 @@
import org.teiid.query.validator.Validator;
import org.teiid.query.validator.ValidatorReport;
-
+ at SuppressWarnings("nls")
public class TestProcessor {
// ################################## TEST HELPERS ################################
@@ -5254,21 +5254,28 @@
FunctionLibrary funcLibrary = new FunctionLibrary(SystemFunctionManager.getSystemFunctions(), new FunctionTree(new UDFSource(new FakeFunctionMetadataSource().getFunctionMethods())));
FakeMetadataFacade metadata = new FakeMetadataFacade(FakeMetadataFactory.example1Cached().getStore(), funcLibrary);
-
- Command command = helpParse(sql);
+ processPreparedStatement(sql, expected, dataManager, capFinder,
+ metadata, Arrays.asList("a "));
+ }
+
+ static void processPreparedStatement(String sql, List[] expected,
+ ProcessorDataManager dataManager, CapabilitiesFinder capFinder,
+ QueryMetadataInterface metadata, List<?> values) throws Exception {
+ Command command = helpParse(sql);
CommandContext context = createCommandContext();
context.setMetadata(metadata);
ProcessorPlan plan = helpGetPlan(command, metadata, capFinder, context);
// Collect reference, set value
- Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next();
VariableContext vc = new VariableContext();
- vc.setGlobalValue(ref.getContextSymbol(), "a "); //$NON-NLS-1$
+ Iterator<?> valIter = values.iterator();
+ for (Reference ref : ReferenceCollectorVisitor.getReferences(command)) {
+ vc.setGlobalValue(ref.getContextSymbol(), valIter.next()); //$NON-NLS-1$
+ }
context.setVariableContext(vc);
// Run query
- helpProcess(plan, context, dataManager, expected);
-
- }
+ helpProcess(plan, context, dataManager, expected);
+ }
/** defect 15348
* @throws Exception */
@@ -5295,21 +5302,8 @@
FunctionLibrary funcLibrary = new FunctionLibrary(SystemFunctionManager.getSystemFunctions(), new FunctionTree(new UDFSource(new FakeFunctionMetadataSource().getFunctionMethods())));
FakeMetadataFacade metadata = new FakeMetadataFacade(FakeMetadataFactory.example1Cached().getStore(), funcLibrary);
-
- Command command = helpParse(sql);
- CommandContext context = createCommandContext();
- context.setMetadata(metadata);
- ProcessorPlan plan = helpGetPlan(command, metadata, capFinder, context);
-
- // Collect reference, set value
- Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next();
- VariableContext vc = new VariableContext();
- vc.setGlobalValue(ref.getContextSymbol(), "a"); //$NON-NLS-1$
- context.setVariableContext(vc);
-
- // Run query
- helpProcess(plan, context, dataManager, expected);
-
+ processPreparedStatement(sql, expected, dataManager, capFinder,
+ metadata, Arrays.asList("a"));
}
@Test public void testSourceDoesntSupportGroupAlias() {
@@ -7297,15 +7291,6 @@
FakeMetadataFacade metadata = FakeMetadataFactory.example1Cached();
- Command command = helpParse("select ?, e1 from pm1.g1"); //$NON-NLS-1$
- ProcessorPlan plan = helpGetPlan(command, metadata, capFinder);
-
- Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next();
- VariableContext vc = new VariableContext();
- vc.setGlobalValue(ref.getContextSymbol(), "a"); //$NON-NLS-1$
- CommandContext context = createCommandContext();
- context.setVariableContext(vc);
-
List[] expected = new List[] {
Arrays.asList("a", "b"), //$NON-NLS-1$ //$NON-NLS-2$
Arrays.asList("a", "c") //$NON-NLS-1$ //$NON-NLS-2$
@@ -7314,7 +7299,8 @@
HardcodedDataManager manager = new HardcodedDataManager();
manager.addData("SELECT 'a', pm1.g1.e1 FROM pm1.g1", expected); //$NON-NLS-1$
- helpProcess(plan, context, manager, expected);
+ processPreparedStatement("select ?, e1 from pm1.g1", expected, manager, capFinder,
+ metadata, Arrays.asList("a"));
}
@Test public void testCase6486() {
Added: trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java (rev 0)
+++ trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,189 @@
+/*
+ * JBoss, Home of Professional Open Source.
+ * See the COPYRIGHT.txt file distributed with this work for information
+ * regarding copyright ownership. Some portions may be licensed
+ * to Red Hat, Inc. under one or more contributor license agreements.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+ * 02110-1301 USA.
+ */
+
+package org.teiid.query.processor;
+
+import static org.teiid.query.processor.TestProcessor.*;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.List;
+
+import org.junit.Test;
+import org.teiid.core.TeiidProcessingException;
+import org.teiid.core.types.ClobImpl;
+import org.teiid.core.types.ClobType;
+import org.teiid.core.types.InputStreamFactory;
+import org.teiid.core.util.UnitTestUtil;
+import org.teiid.query.optimizer.capabilities.DefaultCapabilitiesFinder;
+import org.teiid.query.unittest.FakeMetadataFactory;
+
+ at SuppressWarnings({"unchecked", "nls"})
+public class TestTextTable {
+
+ @Test public void testCorrelatedTextTable() throws Exception {
+ String sql = "select x.* from pm1.g1, texttable(e1 || ',' || e2 COLUMNS x string, y integer) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList("a", 0),
+ Arrays.asList("a", 3),
+ Arrays.asList("c", 1),
+ Arrays.asList("b", 2),
+ Arrays.asList("a", 0),
+ };
+
+ process(sql, expected);
+ }
+
+ @Test public void testTextTableFixed() throws Exception {
+ String sql = "select count(*) from texttable(? COLUMNS compkey string width 77, CDM_ID string width 14, CURRENCY string width 9, \"START\" string width 31, MATURITY string width 38, AMOUNT double width 13, RECORDSOURCE string width 13, SUMMIT_ID string width 25, RATE double width 21, SPREAD double width 9, DESK string width 13) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList(52),
+ };
+
+ FakeDataManager dataManager = new FakeDataManager();
+ sampleData1(dataManager);
+
+ processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/cdm_dos.txt")));
+ }
+
+ @Test public void testNamedMultilineHeader() throws Exception {
+ String sql = "SELECT * from texttable(? COLUMNS Col3Head string HEADER) x";
+
+ List[] expected = new List[] {
+ Arrays.asList("VAL2"),
+ Arrays.asList("VAL4"),
+ Arrays.asList("abc"),
+ Arrays.asList("VAL9"),
+ };
+
+ FakeDataManager dataManager = new FakeDataManager();
+ sampleData1(dataManager);
+
+ processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/test-file.txt.csv")));
+ }
+
+ @Test public void testHeaderWithSkip() throws Exception {
+ String sql = "select count(*) from texttable(? COLUMNS PARTNAME string HEADER 3 SKIP 5) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList(21),
+ };
+
+ FakeDataManager dataManager = new FakeDataManager();
+ sampleData1(dataManager);
+
+ processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/TextParts_HeaderRow2.csv")));
+ }
+
+ @Test public void testEscape() throws Exception {
+ String sql = "select * from texttable('a\\,b,c\\\na\na,b\\\\' COLUMNS c1 string, c2 string ESCAPE '\\') x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList("a,b", "c\na"),
+ Arrays.asList("a", "b\\"),
+ };
+
+ process(sql, expected);
+ }
+
+ @Test(expected=TeiidProcessingException.class) public void testEscapeError() throws Exception {
+ String sql = "select * from texttable('axq' COLUMNS c1 string ESCAPE 'x') x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {};
+
+ process(sql, expected);
+ }
+
+ @Test public void testDelimiter() throws Exception {
+ String sql = "select * from texttable('\na\\,b,c' COLUMNS c1 string, c2 string DELIMITER 'b') x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList("a\\,", ",c"),
+ };
+
+ process(sql, expected);
+ }
+
+ @Test public void testNoRows() throws Exception {
+ String sql = "select * from texttable('' COLUMNS c1 string, c2 string SKIP 3) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {};
+
+ process(sql, expected);
+ }
+
+ @Test public void testQuote() throws Exception {
+ String sql = "select * from texttable(' \" a\", \" \"\" \"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {
+ Arrays.asList(" a", " \" ")
+ };
+
+ process(sql, expected);
+ }
+
+ @Test(expected=TeiidProcessingException.class) public void testUnclosedQuoteError() throws Exception {
+ String sql = "select * from texttable(' \" a\", \" \"\"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {};
+
+ process(sql, expected);
+ }
+
+ @Test(expected=TeiidProcessingException.class) public void testQuoteError() throws Exception {
+ String sql = "select * from texttable(' \" a\", x\" \"\" \"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {};
+
+ process(sql, expected);
+ }
+
+ @Test(expected=TeiidProcessingException.class) public void testHeaderError() throws Exception {
+ String sql = "select * from texttable('notc1,notc2' COLUMNS c1 string, c2 string HEADER) x"; //$NON-NLS-1$
+
+ List[] expected = new List[] {};
+
+ process(sql, expected);
+ }
+
+ public static void process(String sql, List[] expectedResults) throws Exception {
+ FakeDataManager dataManager = new FakeDataManager();
+ sampleData1(dataManager);
+ ProcessorPlan plan = helpGetPlan(helpParse(sql), FakeMetadataFactory.example1Cached());
+ helpProcess(plan, createCommandContext(), dataManager, expectedResults);
+ }
+
+ public ClobType clobFromFile(final String file) {
+ return new ClobType(new ClobImpl(new InputStreamFactory() {
+
+ @Override
+ public InputStream getInputStream() throws IOException {
+ return new FileInputStream(UnitTestUtil.getTestDataFile(file));
+ }
+ }, -1));
+ }
+
+}
Property changes on: trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java
___________________________________________________________________
Name: svn:mime-type
+ text/plain
Modified: trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -2941,4 +2941,9 @@
helpResolveException("select pm1.g1.e1 from pm1.g1, (exec pm1.sq2(pm1.g1.e2)) x"); //$NON-NLS-1$
}
+ @Test public void testCorrelatedTextTable() {
+ Command command = helpResolve("select x.* from pm1.g1, texttable(e1 COLUMNS x string) x"); //$NON-NLS-1$
+ assertEquals(1, command.getProjectedSymbols().size());
+ }
+
}
\ No newline at end of file
Modified: trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java
===================================================================
--- trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java 2010-05-25 19:02:53 UTC (rev 2155)
+++ trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java 2010-05-27 13:45:40 UTC (rev 2156)
@@ -63,6 +63,7 @@
import org.teiid.query.unittest.FakeMetadataFactory;
import org.teiid.query.unittest.FakeMetadataObject;
import org.teiid.query.unittest.FakeMetadataStore;
+import org.teiid.query.unittest.RealMetadataFactory;
import org.teiid.query.validator.Validator;
import org.teiid.query.validator.ValidatorReport;
@@ -1983,6 +1984,17 @@
String userSql = "SELECT xpathValue('<?xml version=\"1.0\" encoding=\"utf-8\" ?><a><b><c>test</c></b></a>', '//*[local-name()=\"bookName\"]', 'xmlns==')"; //$NON-NLS-1$
helpValidate(userSql, new String[] {"xpathValue('<?xml version=\"1.0\" encoding=\"utf-8\" ?><a><b><c>test</c></b></a>', '//*[local-name()=\"bookName\"]', 'xmlns==')"}, FakeMetadataFactory.exampleBQTCached());
}
+
+ @Test public void testTextTableNegativeWidth() {
+ helpValidate("SELECT * from texttable(null columns x string width -1) as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH -1) AS x"}, RealMetadataFactory.exampleBQTCached());
+ }
+
+ @Test public void testTextTableNoWidth() {
+ helpValidate("SELECT * from texttable(null columns x string width 1, y integer) as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH 1, y integer) AS x"}, RealMetadataFactory.exampleBQTCached());
+ }
-
+ @Test public void testTextTableInvalidDelimiter() {
+ helpValidate("SELECT * from texttable(null columns x string width 1 DELIMITER 'z') as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH 1 DELIMITER 'z') AS x"}, RealMetadataFactory.exampleBQTCached());
+ }
+
}
Added: trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv
===================================================================
--- trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv (rev 0)
+++ trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,26 @@
+.
+.
+PARTID,PARTNAME,PARTCOLOR,PARTWEIGHT
+.
+.
+P301,Frame Base,Bisque,100.9
+P302,Tower,Honeydew,99.9
+P303,Axle Housing,Peach,98.9
+P304,Trough,Thistle,97.9
+P305,Outriggers,Lavender,96.9
+P306,Axle,Coral,95.9
+P307,Throwing Arm,Crimson,94.9
+P308,Sling and Pouch,Grey,93.9
+P309,Release Pin,Brick,92.9
+P310,Trigger,Rose,91.9
+P311,Counter-weight (fixed),Mauve,90.9
+P312,Counter-weight (hinged),Chartreuse,89.9
+P313,Dog ring,Salmon,88.9
+P314,Rack,Beige,87.9
+P315,Pinion,Red,86.9
+P316,Dzus fastener,Sepia,85.9
+P317,Half-shaft,Sienna,84.9
+P318,Input shaft,Carrot,83.9
+P319,Fuel port,Khaki,82.9
+P320,Caliper,Goldenrod,81.9
+P321,Splitter,Gold,80.9
Added: trunk/engine/src/test/resources/text/cdm_dos.txt
===================================================================
--- trunk/engine/src/test/resources/text/cdm_dos.txt (rev 0)
+++ trunk/engine/src/test/resources/text/cdm_dos.txt 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,52 @@
+ 000183USD07/26/200107/26/2004 000183 USD 07/26/2001 07/26/2004 10000000.00 Summit 195205NY 0.005450 0.000000 MCREDCLN
+ 000184USD07/26/200107/26/2004 000184 USD 07/26/2001 07/26/2004 10000000.00 Summit 195219NY 0.005450 0.000000 NYBDUMMY
+ 000185USD07/26/200107/26/2004 000185 USD 07/26/2001 07/26/2004 10000000.00 Summit 195233NY 0.009700 0.000000 MCREDCLN
+ 000186USD07/26/200107/26/2004 000186 USD 07/26/2001 07/26/2004 10000000.00 Summit 195234NY 0.009700 0.000000 NYBDUMMY
+ 000258AUSD12/12/200209/15/2005 000258 USD 12/12/2002 09/15/2005 10000000.00 Summit 247299N 0.011500 0.000000 DCC2
+ 000259AUSD01/17/200303/20/2005 000259 USD 01/17/2003 03/20/2005 10000000.00 Summit 247300N 0.011000 0.000000 DCC2
+ 000273USD04/25/200307/07/2006 000273 USD 04/25/2003 07/07/2006 5000000.00 Summit 258123NY 0.008000 0.000000 LTINV
+ 000276AUSD05/10/200306/20/2008 000276 USD 05/10/2003 06/20/2008 10000000.00 Summit 260100N 0.033000 0.000000 DCC2
+ 000277AUSD05/10/200306/20/2008 000277 USD 05/10/2003 06/20/2008 5000000.00 Summit 260101N 0.006500 0.000000 DCC2
+ 000278AUSD05/13/200306/20/2008 000278 USD 05/13/2003 06/20/2008 5000000.00 Summit 260102N 0.033000 0.000000 DCC2
+ 000279AUSD05/14/200306/20/2008 000279 USD 05/14/2003 06/20/2008 5000000.00 Summit 260131N 0.033000 0.000000 DCC2
+ 000280USD06/11/200307/07/2006 000280 USD 06/11/2003 07/07/2006 5000000.00 Summit 264602NY 0.008900 0.000000 LTINV
+ 000289AUSD07/03/200309/20/2008 000289 USD 07/03/2003 09/20/2008 5000000.00 Summit 265496NY 0.004400 0.000000 LTINV
+ 000292USD07/19/200309/20/2008 000292 USD 07/19/2003 09/20/2008 5000000.00 Summit 271005NY 0.005000 0.000000 LTINV
+ 000350USD01/28/200403/20/2009 000350 USD 01/28/2004 03/20/2009 10000000.00 Summit 314996N 0.005200 0.000000 DCC2
+ 000354USD03/10/200403/10/2007 000354 USD 03/10/2004 03/10/2007 9324082.77 Summit 318622NY 0.037500 0.000000 LTTRSCDS
+ 000357USD03/16/200403/16/2007 000357 USD 03/16/2004 03/16/2007 9924050.62 Summit 318638NY 0.030000 0.000000 LTTRSCDS
+ 000359USD03/09/200403/09/2007 000359 USD 03/09/2004 03/09/2007 9329349.59 Summit 318641NY 0.027500 0.000000 LTTRSCDS
+ 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 232935.47 Summit 318642NY 0.032500 0.000000 LTTRSCDS
+ 000352USD04/08/200404/08/2007 000352 USD 04/08/2004 04/08/2007 9788013.59 Summit 318643NY 0.032500 0.000000 LTTRSCDS
+ 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 809831.58 Summit 318644NY 0.032500 0.000000 LTTRSCDS
+ 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 705882.35 Summit 325357NY 0.032500 0.000000 LTTRSCDS
+ 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 156623.38 Summit 325358NY 0.032500 0.000000 LTTRSCDS
+ 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 278643.02 Summit 325363NY 0.032500 0.000000 LTTRSCDS
+ 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 711809.48 Summit 325364NY 0.032500 0.000000 LTTRSCDS
+ 000359USD03/09/200403/09/2007 000359 USD 03/09/2004 03/09/2007 76890.25 Summit 325366NY 0.027500 0.000000 LTTRSCDS
+ 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1049811.00 Summit 329001NY 0.032500 0.000000 LTTRSCDS
+ 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 414407.12 Summit 329002NY 0.032500 0.000000 LTTRSCDS
+ 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1204408.30 Summit 329003NY 0.032500 0.000000 LTTRSCDS
+ 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1058626.60 Summit 329004NY 0.032500 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 194934.82 Summit 331567NY 0.040000 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 131062.23 Summit 335678NY 0.040000 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 1094136.66 Summit 335679NY 0.040000 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2785075.14 Summit 335680NY 0.040000 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2785075.14 Summit 335681NY 0.040000 0.000000 LTTRSCDS
+ 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2984009.08 Summit 335682NY 0.040000 0.000000 LTTRSCDS
+ 000363USD05/03/200405/03/2007 000363 USD 05/03/2004 05/03/2007 10000000.00 Summit 338242NY 0.025000 0.000000 LTTRSCDS
+ 000364USD05/17/200405/17/2007 000364 USD 05/17/2004 05/17/2007 5000000.00 Summit 338820NY 0.032500 0.000000 LTTRSCDS
+ 000365USD05/07/200406/20/2009 000365 USD 05/07/2004 06/20/2009 5000000.00 Summit 340411N 0.004800 0.000000 DCC2
+ 000099USD12/06/199912/06/2004 000099 USD 12/06/1999 12/06/2004 200000000.00 Summit 340805NY 0.000000 0.000000 PCTRSCDS
+ 000129USD04/03/200010/01/2008 000129 USD 04/03/2000 10/01/2008 200000000.00 Summit 340834NY 0.000000 0.000000 PCTRSCDS
+ 000148USD09/12/200012/19/2005 000148 USD 09/12/2000 12/19/2005 573667.12 Summit 340844NY 0.000000 0.000000 PCTRSCDS
+ 000149USD09/12/200012/19/2005 000149 USD 09/12/2000 12/19/2005 2118155.52 Summit 340847NY 0.000000 0.000000 PCTRSCDS
+ 000150USD09/12/200012/19/2005 000150 USD 09/12/2000 12/19/2005 4986491.88 Summit 340848NY 0.000000 0.000000 PCTRSCDS
+ 000366USD05/06/200405/06/2007 000366 USD 05/06/2004 05/06/2007 3058823.53 Summit 341369NY 0.030000 0.000000 LTTRSCDS
+ 000367USD05/06/200405/06/2007 000367 USD 05/06/2004 05/06/2007 6941176.47 Summit 341377NY 0.030000 0.000000 LTTRSCDS
+ 000213USD02/11/200202/11/2007 000213 USD 02/11/2002 02/11/2007 8000000.00 Summit 343117NY 0.056500 0.000000 CNA2
+ 000272USD03/05/200301/31/2005 000272 USD 03/05/2003 01/31/2005 28000000.00 Summit 343123NY 0.006000 0.000000 CNA2
+ 000333USD11/13/200311/20/2004 000333 USD 11/13/2003 11/20/2004 25000000.00 Summit 343151NY 0.003800 0.000000 CNA2
+ 000368USD05/24/200405/24/2007 000368 USD 05/24/2004 05/24/2007 10000000.00 Summit 343169NY 0.025000 0.000000 LTTRSCDS
+ 000168USD12/18/200001/31/2006 000168 USD 12/18/2000 01/31/2006 672096851.46 Summit 344846NY 0.000000 0.000000 ASGCGD
+ 000369USD05/20/200405/20/2007 000369 USD 05/20/2004 05/20/2007 10000000.00 Summit 344851NY 0.032500 0.000000 LTTRSCDS
Property changes on: trunk/engine/src/test/resources/text/cdm_dos.txt
___________________________________________________________________
Name: svn:mime-type
+ text/plain
Added: trunk/engine/src/test/resources/text/test-file.txt.csv
===================================================================
--- trunk/engine/src/test/resources/text/test-file.txt.csv (rev 0)
+++ trunk/engine/src/test/resources/text/test-file.txt.csv 2010-05-27 13:45:40 UTC (rev 2156)
@@ -0,0 +1,7 @@
+Col1Header,"Col2
+Header on newline",Col3Head
+VAL1,,VAL2
+VAL3,,VAL4
+VAL5,Test,abc
+VAL6,"VAL7
+VAL8 on a newline",VAL9
More information about the teiid-commits
mailing list