[teiid-commits] teiid SVN: r2156 - in trunk: engine/src/main/java/org/teiid/query/metadata and 20 other directories.

Author: shawkins Date: 2010-05-27 09:45:40 -0400 (Thu, 27 May 2010) New Revision: 2156 Added: trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java trunk/engine/src/test/resources/text/ trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv trunk/engine/src/test/resources/text/cdm_dos.txt trunk/engine/src/test/resources/text/test-file.txt.csv Modified: trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties trunk/engine/src/main/resources/org/teiid/query/i18n.properties trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java Log: TEIID-1102 adding a replacement for the text connector with a built in texttable table function Modified: trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml =================================================================== --- trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2010-05-27 13:45:40 UTC (rev 2156) @@ -55,7 +55,7 @@ ways, some prepending catalog or schema or user information, Teiid allows table specification to be a dot-delimited construct. <note> - <para> When a table specification contains a dot resolving will allow for the match of + <para>When a table specification contains a dot resolving will allow for the match of a partial name against any number of the end segments in the name. e.g. a table with the fully-qualified name vdbname."sourcescema.sourcetable" would match the partial name sourcetable. @@ -569,7 +569,8 @@ input of the parameter.</para> </listitem> <listitem> - <para>Input values are passed via criteria. Values can be passed by '=','is null', or 'in' predicates. Disjuncts however are not allowed. + <para>Input values are passed via criteria. Values can be passed by '=','is null', or 'in' predicates. Disjuncts are not allowed. + It is also not possible to pass the value of a non-comparable column through an equality predicate. </para> </listitem> <listitem> @@ -704,34 +705,121 @@ <listitem><para>FROM table1 [INNER|LEFT OUTER|RIGHT OUTER|FULL OUTER] JOIN table2 ON join-criteria</para></listitem> <listitem><para>FROM table1 CROSS JOIN table2</para></listitem> <listitem><para>FROM (subquery) [AS] alias</para></listitem> - <listitem><para>FROM <link linkend="nested_table">table(subquery)</link> [AS] alias</para></listitem> + <listitem><para>FROM <link linkend="nested_table">TABLE(subquery)</link> [AS] alias</para></listitem> <listitem><para>FROM table1 JOIN table2 MAKEDEP ON join-criteria</para></listitem> <listitem><para>FROM table1 JOIN table2 MAKENOTDEP ON join-criteria</para></listitem> <listitem><para>FROM table1 left outer join <link linkend="optional_join">/* optional */</link> table2 ON join-criteria</para></listitem> + <listitem><para>FROM <link linkend="texttable">TEXTTABLE...</link></para></listitem> </itemizedlist> <note> <title>DEP Hints</title> <para> MAKEDEP and MAKENOTDEP are hints used to control - <link linkend="dependent_join">dependent join</link> + <link linkend="dependent_joins">dependent join</link> behavior. They should only be used in situations where the optimizer does not choose the most optimal plan based upon query structure, metadata, and costing information. </para> </note> <sect3 id="nested_table"> - <title>Nested Table Reference</title> - <para>Nested tables may appear in the FROM clause with the TABLE keyword. They are an alternative to using a view with normal join semantics. + <title>Nested Table Reference</title> + <para>Nested tables may appear in the FROM clause with the TABLE + keyword. They are an alternative to using a view with normal join + semantics. The columns projected from the command contained in the nested table + may be used just as any of the other FROM clause projected columns in join criteria, the where clause, etc. </para> - <para>A nested table may have correlated references to preceeding FROM clause - column references as long as INNER and LEFT OUTER joins are used. This is especially useful in cases where then nested expression is a procedure or function call.</para> - <para>Valid Example: select * from t1, TABLE(call proc(t1.x)) t2</para> - <para>Invalid Example: select * from TABLE(call proc(t1.x)) t2, t1</para> - <note> - <title>Multiple Execution</title> - <para>The usage of a correlated nested table may result in multiple executions of the table expression - once for each correlated row.</para> - </note> - </sect3> + <para>A nested table may have correlated references to preceeding FROM + clause column references as long as INNER and LEFT OUTER joins are used. This is + especially useful in cases where then nested expression is a + procedure or function call.</para> + <para>Valid example: + <programlisting>select * from t1, TABLE(call proc(t1.x)) t2</programlisting> + </para> + <para>Invalid example, since t1 appears after the nested table in the from clause: + <programlisting>select * from TABLE(call proc(t1.x)) t2, t1</programlisting> + </para> + <note> + <title>Multiple Execution</title> + <para>The usage of a correlated nested table may result in multiple + executions of the table expression - once for each correlated row. + </para> + </note> + </sect3> + <sect3 id="texttable"> + <title>TEXTTABLE</title> + <para>The TEXTTABLE funciton processes character input to produce tabular ouptut. It supports both fixed and delimited file format parsing. + The function itself defines what columns it projects. + The TEXTTABLE function is implicitly a nested table and may be correlated to preceeding FROM clause entries. + </para> + <para> + Usage: + <synopsis label="Usage">TEXTTABLE(expression COLUMNS column, ... [DELIMITER char] [(QUOTE|ESCAPE) char] [HEADER [integer]] [SKIP integer]) AS identifier</synopsis> + <synopsis label="Usage">column := identifier datatype [WIDTH integer]</synopsis> + </para> + <itemizedlist> + <para>Syntax Rules: + </para> + <listitem> + <para>If width is specified for one column it must be specified for all columns. + </para> + </listitem> + <listitem> + <para>If width is specified, then fixed width parsing is used and ESCAPE, QUOTE, and HEADER should not be specified. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <para>Parameters</para> + <listitem> + <para>expression - the text content to process, which should be convertable to CLOB. + </para> + </listitem> + <listitem> + <para>DELIMITER sets the field delimiter character to use. Defaults to ','. + </para> + </listitem> + <listitem> + <para>QUOTE sets the quote, or qualifier, character used to wrap field values. Defaults to '"'. + </para> + </listitem> + <listitem> + <para>ESCAPE sets the escape character to use if no quoting character is in use. + This is used in situations where the delimiter or new line characters are escaped with a preceding character, e.g. \, + </para> + </listitem> + <listitem> + <para>HEADER specifies the text line number (counting every new line) on which the column names occur. All lines prior to the header will be skipped. + If HEADER is specified, then the header line will be used to determine the TEXTTABLE column position by case-insensitive name matching. This is especially useful in situations where only a subset of the columns are needed. + If the HEADER value is not specified, it defaults to 1. + If HEADER is not specified, then columns are expected to match positionally with the text contents. + </para> + </listitem> + <listitem> + <para>SKIP specifies the number of text lines (counting every new line) to skip before parsing the contents. HEADER may still be specified with SKP. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <para>Examples</para> + <listitem> + <para>Use of the HEADER parameter, returns 1 row ['b']: <programlisting>select * from texttable('col1,col2,col3\na,b,c' COLUMNS col2 string HEADER) x</programlisting> + </para> + </listitem> + <listitem> + <para>Use of fixed width, returns 1 row ['a', 'b', 'c']: <programlisting>select * from texttable('abc' COLUMNS col1 string width 1, col2 string width 1, col3 string width 1) x</programlisting> + </para> + </listitem> + <listitem> + <para>Use of ESCAPE parameter, returns 1 row ['a,', 'b']: <programlisting>select * from texttable('a:,,b' COLUMNS col1 string, col2 string ESCAPE ':') x</programlisting> + </para> + </listitem> + <listitem> + <para>As a nested table: <programlisting>select x.* from t, texttable(t.clobcolumn COLUMNS first string, second date SKIP 1) x</programlisting> + </para> + </listitem> + + </itemizedlist> + </sect3> </sect2> <sect2 id="where_clause"> <title>WHERE Clause</title> Modified: trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/metadata/TempMetadataStore.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -234,7 +234,7 @@ * @param tempElement Short element name * @return Metadata ID or null if not found */ - public List getTempElementElementIDs(String tempGroup) { + public List<TempMetadataID> getTempElementElementIDs(String tempGroup) { TempMetadataID groupID = tempGroups.get(tempGroup.toUpperCase()); if(groupID != null) { return groupID.getElements(); Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/PlanToProcessConverter.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -67,6 +67,7 @@ import org.teiid.query.processor.relational.RelationalPlan; import org.teiid.query.processor.relational.SelectNode; import org.teiid.query.processor.relational.SortNode; +import org.teiid.query.processor.relational.TextTableNode; import org.teiid.query.processor.relational.UnionAllNode; import org.teiid.query.processor.relational.JoinNode.JoinStrategyType; import org.teiid.query.processor.relational.MergeJoinStrategy.SortOption; @@ -79,6 +80,7 @@ import org.teiid.query.sql.lang.OrderBy; import org.teiid.query.sql.lang.Query; import org.teiid.query.sql.lang.StoredProcedure; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.SetQuery.Operation; import org.teiid.query.sql.symbol.ElementSymbol; import org.teiid.query.sql.symbol.Expression; @@ -229,14 +231,10 @@ } else if (stype == JoinStrategyType.NESTED_TABLE) { NestedTableJoinStrategy ntjStrategy = new NestedTableJoinStrategy(); jnode.setJoinStrategy(ntjStrategy); - Command command = (Command)FrameUtil.findJoinSourceNode(node.getFirstChild()).getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null) { - ntjStrategy.setLeftMap(command.getCorrelatedReferences()); - } - command = (Command)FrameUtil.findJoinSourceNode(node.getLastChild()).getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null) { - ntjStrategy.setRightMap(command.getCorrelatedReferences()); - } + SymbolMap references = (SymbolMap)FrameUtil.findJoinSourceNode(node.getFirstChild()).getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + ntjStrategy.setLeftMap(references); + references = (SymbolMap)FrameUtil.findJoinSourceNode(node.getLastChild()).getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + ntjStrategy.setRightMap(references); } else { NestedLoopJoinStrategy nljStrategy = new NestedLoopJoinStrategy(); jnode.setJoinStrategy(nljStrategy); @@ -368,9 +366,14 @@ child.setProperty(NodeConstants.Info.OUTPUT_COLS, node.getProperty(NodeConstants.Info.OUTPUT_COLS)); } } - - return null; - + Object source = node.getProperty(NodeConstants.Info.TABLE_FUNCTION); + if (!(source instanceof TextTable)) { + return null; + } + TextTableNode ttn = new TextTableNode(getID()); + ttn.setTable((TextTable)source); + processNode = ttn; + break; case NodeConstants.Types.SET_OP: Operation setOp = (Operation) node.getProperty(NodeConstants.Info.SET_OPERATION); boolean useAll = ((Boolean) node.getProperty(NodeConstants.Info.USE_ALL)).booleanValue(); Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/RelationalPlanner.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -66,6 +66,7 @@ import org.teiid.query.resolver.QueryResolver; import org.teiid.query.resolver.util.BindVariableVisitor; import org.teiid.query.rewriter.QueryRewriter; +import org.teiid.query.sql.LanguageObject; import org.teiid.query.sql.LanguageObject.Util; import org.teiid.query.sql.lang.Command; import org.teiid.query.sql.lang.Criteria; @@ -86,6 +87,7 @@ import org.teiid.query.sql.lang.StoredProcedure; import org.teiid.query.sql.lang.SubqueryContainer; import org.teiid.query.sql.lang.SubqueryFromClause; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.proc.CreateUpdateProcedureCommand; import org.teiid.query.sql.symbol.GroupSymbol; @@ -672,25 +674,20 @@ Command nestedCommand = sfc.getCommand(); node = NodeFactory.getNewNode(NodeConstants.Types.SOURCE); if (sfc.isTable()) { - PlanNode rootJoin = parent; - while (rootJoin.getParent() != null && rootJoin.getParent().getType() == NodeConstants.Types.JOIN) { - rootJoin = rootJoin.getParent(); - } - List<Reference> correlatedReferences = new ArrayList<Reference>(); - CorrelatedReferenceCollectorVisitor.collectReferences(sfc, rootJoin.getGroups(), correlatedReferences); - - if (!correlatedReferences.isEmpty()) { - SymbolMap map = new SymbolMap(); - for (Reference reference : correlatedReferences) { - map.addMapping(reference.getExpression(), reference.getExpression()); - } - sfc.getCommand().setCorrelatedReferences(map); - } + sfc.getCommand().setCorrelatedReferences(getCorrelatedReferences(parent, node, sfc)); } node.addGroup(group); addNestedCommand(node, group, nestedCommand, nestedCommand, true); hints.hasVirtualGroups = true; parent.addLastChild(node); + } else if (clause instanceof TextTable) { + TextTable tt = (TextTable)clause; + GroupSymbol group = tt.getGroupSymbol(); + node = NodeFactory.getNewNode(NodeConstants.Types.SOURCE); + node.setProperty(NodeConstants.Info.TABLE_FUNCTION, tt); + tt.setCorrelatedReferences(getCorrelatedReferences(parent, node, tt)); + node.addGroup(group); + parent.addLastChild(node); } if (clause.isOptional()) { @@ -705,6 +702,26 @@ } } + private SymbolMap getCorrelatedReferences(PlanNode parent, PlanNode node, + LanguageObject lo) { + PlanNode rootJoin = parent; + while (rootJoin.getParent() != null && rootJoin.getParent().getType() == NodeConstants.Types.JOIN) { + rootJoin = rootJoin.getParent(); + } + List<Reference> correlatedReferences = new ArrayList<Reference>(); + CorrelatedReferenceCollectorVisitor.collectReferences(lo, rootJoin.getGroups(), correlatedReferences); + + if (correlatedReferences.isEmpty()) { + return null; + } + SymbolMap map = new SymbolMap(); + for (Reference reference : correlatedReferences) { + map.addMapping(reference.getExpression(), reference.getExpression()); + } + node.setProperty(NodeConstants.Info.CORRELATED_REFERENCES, map); + return map; + } + private void addNestedCommand(PlanNode node, GroupSymbol group, Command nestedCommand, Command toPlan, boolean merge) throws TeiidComponentException, QueryMetadataException, TeiidProcessingException { if (nestedCommand instanceof QueryCommand) { Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/NodeConstants.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -118,6 +118,8 @@ MAKE_DEP, // ??? List of Groups ??? PROCESSOR_PLAN, // ProcessorPlan for non-relational sub plan NESTED_COMMAND, // Command for nested processor plan + TABLE_FUNCTION, // Table Function + CORRELATED_REFERENCES, // SymbolMap MAKE_NOT_DEP, // Source should not be dependent INLINE_VIEW, // If the source node represents an inline view Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/plantree/PlanNode.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -35,7 +35,6 @@ import java.util.Set; import org.teiid.query.sql.LanguageObject; -import org.teiid.query.sql.lang.Command; import org.teiid.query.sql.lang.Criteria; import org.teiid.query.sql.lang.SubqueryContainer; import org.teiid.query.sql.symbol.ElementSymbol; @@ -325,16 +324,16 @@ } LinkedList<SymbolMap> result = new LinkedList<SymbolMap>(); for (PlanNode child : NodeEditor.findAllNodes(this, NodeConstants.Types.SOURCE, NodeConstants.Types.ACCESS)) { - Command command = (Command)child.getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command == null || command.getCorrelatedReferences() == null) { + SymbolMap references = (SymbolMap)child.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + if (references == null) { continue; } - Set<GroupSymbol> correlationGroups = GroupsUsedByElementsVisitor.getGroups(command.getCorrelatedReferences().getValues()); + Set<GroupSymbol> correlationGroups = GroupsUsedByElementsVisitor.getGroups(references.getValues()); PlanNode joinNode = NodeEditor.findParent(child, NodeConstants.Types.JOIN, NodeConstants.Types.SOURCE); while (joinNode != null) { if (joinNode.getGroups().containsAll(correlationGroups)) { if (joinNode == this) { - result.add(command.getCorrelatedReferences()); + result.add(references); } break; } Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/NewCalculateCostUtil.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -345,9 +345,9 @@ float cost = UNKNOWN_VALUE; if(node.getChildCount() > 0) { - Command command = (Command)node.getProperty(NodeConstants.Info.NESTED_COMMAND); + SymbolMap references = (SymbolMap)node.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); //only cost non-correlated TODO: a better estimate for correlated - if (command == null || command.getCorrelatedReferences() == null) { + if (references == null) { PlanNode child = node.getFirstChild(); Float childCostFloat = (Float)child.getProperty(NodeConstants.Info.EST_CARDINALITY); if (childCostFloat != null) { Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleImplementJoinStrategy.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -47,6 +47,7 @@ import org.teiid.query.sql.lang.OrderBy; import org.teiid.query.sql.symbol.GroupSymbol; import org.teiid.query.sql.symbol.SingleElementSymbol; +import org.teiid.query.sql.util.SymbolMap; import org.teiid.query.sql.visitor.GroupsUsedByElementsVisitor; import org.teiid.query.util.CommandContext; @@ -69,9 +70,9 @@ TeiidComponentException { for (PlanNode sourceNode : NodeEditor.findAllNodes(plan, NodeConstants.Types.SOURCE, NodeConstants.Types.ACCESS)) { - Command command = (Command)sourceNode.getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null && command.getCorrelatedReferences() != null) { - Set<GroupSymbol> groups = GroupsUsedByElementsVisitor.getGroups(command.getCorrelatedReferences().getValues()); + SymbolMap references = (SymbolMap)sourceNode.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + if (references != null) { + Set<GroupSymbol> groups = GroupsUsedByElementsVisitor.getGroups(references.getValues()); PlanNode joinNode = NodeEditor.findParent(sourceNode, NodeConstants.Types.JOIN, NodeConstants.Types.SOURCE); while (joinNode != null) { joinNode.setProperty(NodeConstants.Info.JOIN_STRATEGY, JoinStrategyType.NESTED_TABLE); Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleMergeVirtual.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -90,8 +90,8 @@ return root; } - Command command = (Command)frame.getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null && command.getCorrelatedReferences() != null) { + SymbolMap references = (SymbolMap)frame.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + if (references != null) { return root; //correlated nested table commands should not be merged } Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlaceAccess.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -44,6 +44,7 @@ import org.teiid.query.optimizer.relational.plantree.PlanNode; import org.teiid.query.resolver.util.ResolverUtil; import org.teiid.query.sql.lang.Insert; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.symbol.ElementSymbol; import org.teiid.query.sql.symbol.Expression; import org.teiid.query.sql.symbol.GroupSymbol; @@ -107,6 +108,10 @@ if (req == null) { req = sourceNode.getProperty(NodeConstants.Info.NESTED_COMMAND); } + + if (sourceNode.getProperty(NodeConstants.Info.TABLE_FUNCTION) != null) { + return; + } if (req instanceof Insert) { isInsert = true; Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePlanJoins.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -47,7 +47,6 @@ import org.teiid.query.optimizer.relational.plantree.PlanNode; import org.teiid.query.processor.relational.JoinNode.JoinStrategyType; import org.teiid.query.resolver.util.AccessPattern; -import org.teiid.query.sql.lang.Command; import org.teiid.query.sql.lang.Criteria; import org.teiid.query.sql.lang.JoinType; import org.teiid.query.sql.symbol.ElementSymbol; @@ -128,14 +127,11 @@ //account for nested table correlations for (PlanNode joinSource : joinRegion.getJoinSourceNodes().keySet()) { - Command command = (Command)joinSource.getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null) { - SymbolMap map = command.getCorrelatedReferences(); - if (map !=null) { - joinSource.setProperty(NodeConstants.Info.REQUIRED_ACCESS_PATTERN_GROUPS, GroupsUsedByElementsVisitor.getGroups(map.getValues())); - joinRegion.setContainsNestedTable(true); - } - } + SymbolMap map = (SymbolMap)joinSource.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + if (map !=null) { + joinSource.setProperty(NodeConstants.Info.REQUIRED_ACCESS_PATTERN_GROUPS, GroupsUsedByElementsVisitor.getGroups(map.getValues())); + joinRegion.setContainsNestedTable(true); + } } //check for unsatisfied dependencies Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RulePushAggregates.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -162,7 +162,7 @@ List<SingleElementSymbol> groupingExpressions, PlanNode setOp) throws TeiidComponentException, QueryMetadataException, QueryPlannerException, QueryResolverException { - if (setOp.getType() != NodeConstants.Types.SET_OP || setOp.getProperty(NodeConstants.Info.SET_OPERATION) != Operation.UNION) { + if (setOp == null || setOp.getType() != NodeConstants.Types.SET_OP || setOp.getProperty(NodeConstants.Info.SET_OPERATION) != Operation.UNION) { return; //must not be a union } LinkedHashSet<AggregateSymbol> aggregates = collectAggregates(groupNode); Modified: trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/optimizer/relational/rules/RuleRaiseAccess.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -178,8 +178,8 @@ return null; } - Command command = (Command)parentNode.getProperty(NodeConstants.Info.NESTED_COMMAND); - if (command != null && command.getCorrelatedReferences() != null) { + SymbolMap references = (SymbolMap)parentNode.getProperty(NodeConstants.Info.CORRELATED_REFERENCES); + if (references != null) { return null; } Modified: trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/processor/relational/NestedTableJoinStrategy.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -146,7 +146,7 @@ } else { rightSource.close(); for (Map.Entry<ElementSymbol, Expression> entry : rightMap.asMap().entrySet()) { - joinNode.getContext().getVariableContext().setValue(entry.getKey(), null); + joinNode.getContext().getVariableContext().remove(entry.getKey()); } } @@ -157,7 +157,7 @@ if (leftMap != null) { leftSource.close(); for (Map.Entry<ElementSymbol, Expression> entry : leftMap.asMap().entrySet()) { - joinNode.getContext().getVariableContext().setValue(entry.getKey(), null); + joinNode.getContext().getVariableContext().remove(entry.getKey()); } } } @@ -167,7 +167,11 @@ for (int i = 0; i < elements.size(); i++) { SingleElementSymbol element = elements.get(i); if (element instanceof ElementSymbol) { - joinNode.getContext().getVariableContext().setValue((ElementSymbol)element, tuple==null?null:tuple.get(i)); + if (tuple == null) { + joinNode.getContext().getVariableContext().remove((ElementSymbol)element); + } else { + joinNode.getContext().getVariableContext().setValue((ElementSymbol)element, tuple.get(i)); + } } } } Added: trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java (rev 0) +++ trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,391 @@ +/* + * JBoss, Home of Professional Open Source. + * See the COPYRIGHT.txt file distributed with this work for information + * regarding copyright ownership. Some portions may be licensed + * to Red Hat, Inc. under one or more contributor license agreements. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA + * 02110-1301 USA. + */ + +package org.teiid.query.processor.relational; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.Reader; +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.teiid.api.exception.query.ExpressionEvaluationException; +import org.teiid.common.buffer.BlockedException; +import org.teiid.common.buffer.BufferManager; +import org.teiid.common.buffer.TupleBatch; +import org.teiid.core.TeiidComponentException; +import org.teiid.core.TeiidProcessingException; +import org.teiid.core.types.ClobType; +import org.teiid.core.types.DataTypeManager; +import org.teiid.core.types.TransformationException; +import org.teiid.query.execution.QueryExecPlugin; +import org.teiid.query.processor.ProcessorDataManager; +import org.teiid.query.sql.lang.TextTable; +import org.teiid.query.sql.lang.TextTable.TextColumn; +import org.teiid.query.sql.symbol.ElementSymbol; +import org.teiid.query.sql.symbol.Expression; +import org.teiid.query.util.CommandContext; + +/** + * Handles text file processing. + * + * TODO: unix style escape handling \t \n, etc. + * TODO: parse using something more memory safe than strings + * TODO: allow for escaping with fixed parsing + * TODO: allow for fixed parsing without new lines + * TODO: allow for a configurable line terminator + */ +public class TextTableNode extends SubqueryAwareRelationalNode { + + private TextTable table; + + //initialized state + private int skip = 0; + private int header = -1; + private boolean noQuote; + private char quote; + private char delimiter; + private int lineWidth; + private List<Class<?>> types; + private Map elementMap; + private int[] projectionIndexes; + + //per file state + private BufferedReader reader; + private int textLine = 0; + private Map<String, Integer> nameIndexes; + + public TextTableNode(int nodeID) { + super(nodeID); + } + + @Override + public void initialize(CommandContext context, BufferManager bufferManager, + ProcessorDataManager dataMgr) { + super.initialize(context, bufferManager, dataMgr); + if (types != null) { + return; + } + if (table.getSkip() != null) { + skip = table.getSkip(); + } + if (table.getHeader() != null) { + skip = Math.max(table.getHeader(), skip); + header = table.getHeader() - 1; + } + if (table.isFixedWidth()) { + for (TextColumn col : table.getColumns()) { + lineWidth += col.getWidth(); + } + } else { + if (table.getDelimiter() == null) { + delimiter = ','; + } else { + delimiter = table.getDelimiter(); + } + if (table.getQuote() == null) { + quote = '"'; + } else { + noQuote = table.isEscape(); + quote = table.getQuote(); + } + } + types = new ArrayList<Class<?>>(); + //TODO: if all string types, then we can skip this step + for (TextColumn col : table.getColumns()) { + types.add(DataTypeManager.getDataTypeClass(col.getType())); + } + this.elementMap = createLookupMap(table.getProjectedSymbols()); + this.projectionIndexes = getProjectionIndexes(elementMap, getElements()); + } + + @Override + public void closeDirect() { + super.closeDirect(); + reset(); + } + + @Override + public void reset() { + super.reset(); + if (this.reader != null) { + try { + this.reader.close(); + } catch (IOException e) { + } + this.reader = null; + } + this.nameIndexes = null; + this.textLine = 0; + } + + public void setTable(TextTable table) { + this.table = table; + } + + @Override + public TextTableNode clone() { + TextTableNode clone = new TextTableNode(getID()); + this.copy(this, clone); + clone.setTable(table); + return clone; + } + + @Override + protected TupleBatch nextBatchDirect() throws BlockedException, + TeiidComponentException, TeiidProcessingException { + + if (reader == null) { + initReader(); + } + + if (reader == null) { + terminateBatches(); + return pullBatch(); + } + + while (!isBatchFull()) { + String line = readLine(); + + if (line == null) { + terminateBatches(); + break; + } + + List<String> vals = parseLine(line); + + List<Object> tuple = new ArrayList<Object>(projectionIndexes.length); + for (int output : projectionIndexes) { + TextColumn col = table.getColumns().get(output); + String val = null; + int index = output; + if (nameIndexes != null) { + index = nameIndexes.get(col.getName()); + } + if (index >= vals.size()) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.no_value", col.getName(), textLine)); //$NON-NLS-1$ + } + val = vals.get(index); + try { + tuple.add(DataTypeManager.transformValue(val, types.get(output))); + } catch (TransformationException e) { + throw new TeiidProcessingException(e, QueryExecPlugin.Util.getString("TextTableNode.conversion_error", col.getName(), textLine)); //$NON-NLS-1$ + } + } + addBatchRow(tuple); + } + + return pullBatch(); + } + + private String readLine() throws TeiidProcessingException { + while (true) { + try { + String line = reader.readLine(); + if (line != null) { + textLine++; + if (line.length() == 0) { + continue; + } + } + return line; + } catch (IOException e) { + throw new TeiidProcessingException(e); + } + } + } + + private void initReader() throws ExpressionEvaluationException, + BlockedException, TeiidComponentException, TeiidProcessingException { + + if (table.getCorrelatedReferences() != null) { + for (Map.Entry<ElementSymbol, Expression> entry : table.getCorrelatedReferences().asMap().entrySet()) { + getContext().getVariableContext().setValue(entry.getKey(), getEvaluator(Collections.emptyMap()).evaluate(entry.getValue(), null)); + } + } + ClobType file = (ClobType)getEvaluator(Collections.emptyMap()).evaluate(table.getFile(), null); + + if (file == null) { + return; + } + + //get the reader + try { + Reader r = file.getCharacterStream(); + if (!(r instanceof BufferedReader)) { + reader = new BufferedReader(r); + } else { + reader = (BufferedReader)r; + } + } catch (SQLException e) { + throw new TeiidProcessingException(e); + } + + //process the skip field + if (skip <= 0) { + return; + } + while (textLine < skip) { + boolean isHeader = textLine == header; + String line = readLine(); + if (line == null) { //just return an empty batch + reset(); + return; + } + if (isHeader) { + processHeader(parseLine(line)); + } + } + } + + private void processHeader(List<String> line) throws TeiidProcessingException { + nameIndexes = new HashMap<String, Integer>(); + for (String string : line) { + if (string == null) { + continue; + } + nameIndexes.put(string.toUpperCase(), nameIndexes.size()); + } + for (TextColumn col : table.getColumns()) { + Integer index = nameIndexes.get(col.getName().toUpperCase()); + if (index == null) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.header_missing", col.getName())); //$NON-NLS-1$ + } + nameIndexes.put(col.getName(), index); + } + } + + private List<String> parseLine(String line) throws TeiidProcessingException { + if (table.isFixedWidth()) { + return parseFixedWidth(line); + } + return parseDelimitedLine(line); + } + + private List<String> parseDelimitedLine(String line) throws TeiidProcessingException { + ArrayList<String> result = new ArrayList<String>(); + StringBuilder builder = new StringBuilder(); + boolean escaped = false; + boolean wasQualified = false; + boolean qualified = false; + while (true) { + if (line == null) { + if (escaped) { + builder.append('\n'); //allow for escaped new lines + escaped = false; + line = readLine(); + continue; + } + if (!qualified) { + //close the last entry + addValue(result, wasQualified, builder.toString()); + return result; + } + line = readLine(); + if (line == null) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.unclosed")); //$NON-NLS-1$ + } + } + char[] chars = line.toCharArray(); + for (int i = 0; i < chars.length; i++) { + char chr = chars[i]; + if (chr == delimiter) { + if (escaped || qualified) { + builder.append(chr); + escaped = false; + } else { + addValue(result, wasQualified, builder.toString()); + wasQualified = false; + builder = new StringBuilder(); //next entry + } + } else if (chr == quote) { + if (noQuote) { //it's the escape char + if (escaped) { + builder.append(quote); + } + escaped = !escaped; + } else { + if (qualified) { + qualified = false; + } else { + if (wasQualified) { + qualified = true; + builder.append(chr); + } else { + if (builder.toString().trim().length() != 0) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.character_not_allowed", textLine)); //$NON-NLS-1$ + } + qualified = true; + builder = new StringBuilder(); //start the entry over + wasQualified = true; + } + } + } + } else { + if (escaped) { + //don't understand other escape sequences yet + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.unknown_escape", chr, textLine)); //$NON-NLS-1$ + } + if (wasQualified && !qualified) { + if (!Character.isWhitespace(chr)) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.character_not_allowed", textLine)); //$NON-NLS-1$ + } + //else just ignore + } else { + builder.append(chr); + } + } + } + line = null; + } + } + + private void addValue(ArrayList<String> result, boolean wasQualified, String val) { + if (!wasQualified) { + val = val.trim(); + if (val.length() == 0) { + val = null; + } + } + result.add(val); + } + + private List<String> parseFixedWidth(String line) + throws TeiidProcessingException { + if (line.length() < lineWidth) { + throw new TeiidProcessingException(QueryExecPlugin.Util.getString("TextTableNode.invalid_width", line.length(), lineWidth, textLine)); //$NON-NLS-1$ + } + ArrayList<String> result = new ArrayList<String>(); + int beginIndex = 0; + for (TextColumn col : table.getColumns()) { + String val = line.substring(beginIndex, beginIndex + col.getWidth()); + addValue(result, false, val); + beginIndex += col.getWidth(); + } + return result; + } + +} Property changes on: trunk/engine/src/main/java/org/teiid/query/processor/relational/TextTableNode.java ___________________________________________________________________ Name: svn:mime-type + text/plain Modified: trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/resolver/command/SimpleQueryResolver.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -40,6 +40,7 @@ import org.teiid.core.TeiidComponentException; import org.teiid.core.TeiidException; import org.teiid.core.TeiidRuntimeException; +import org.teiid.core.types.DataTypeManager; import org.teiid.query.QueryPlugin; import org.teiid.query.analysis.AnalysisRecord; import org.teiid.query.metadata.QueryMetadataInterface; @@ -67,6 +68,7 @@ import org.teiid.query.sql.lang.SubqueryContainer; import org.teiid.query.sql.lang.SubqueryFromClause; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.navigator.PostOrderNavigator; import org.teiid.query.sql.symbol.AliasSymbol; @@ -78,6 +80,7 @@ import org.teiid.query.sql.symbol.Reference; import org.teiid.query.sql.symbol.ScalarSubquery; import org.teiid.query.sql.symbol.SingleElementSymbol; +import org.teiid.query.sql.visitor.ElementCollectorVisitor; import org.teiid.query.util.ErrorMessageKeys; @@ -147,7 +150,7 @@ public static class QueryResolverVisitor extends PostOrderNavigator { private LinkedHashSet<GroupSymbol> currentGroups = new LinkedHashSet<GroupSymbol>(); - private List<GroupSymbol> discoveredGroups = new LinkedList<GroupSymbol>(); + private LinkedList<GroupSymbol> discoveredGroups = new LinkedList<GroupSymbol>(); private List<GroupSymbol> implicitGroups = new LinkedList<GroupSymbol>(); private TempMetadataAdapter metadata; private Query query; @@ -277,6 +280,60 @@ postVisitVisitor(obj); } + /** + * TextTables have similar resolving semantics to a nested table. + */ + @Override + public void visit(TextTable obj) { + LinkedHashSet<GroupSymbol> saved = new LinkedHashSet<GroupSymbol>(this.currentGroups); + if (allowImplicit) { + currentGroups.addAll(this.implicitGroups); + } + this.visitNode(obj.getFile()); + try { + obj.setFile(ResolverUtil.convertExpression(obj.getFile(), DataTypeManager.DefaultDataTypes.CLOB, metadata)); + } catch (QueryResolverException e) { + throw new TeiidRuntimeException(e); + } + //we didn't create a true external context, so we manually mark external + for (ElementSymbol symbol : ElementCollectorVisitor.getElements(obj, false)) { + if (symbol.isExternalReference()) { + continue; + } + if (implicitGroups.contains(symbol.getGroupSymbol())) { + symbol.setIsExternalReference(true); + } + } + if (allowImplicit) { + this.currentGroups.clear(); + this.currentGroups.addAll(saved); + } + discoveredGroup(obj.getGroupSymbol()); + try { + ResolverUtil.addTempGroup(metadata, obj.getGroupSymbol(), obj.getProjectedSymbols(), false); + } catch (QueryResolverException err) { + throw new TeiidRuntimeException(err); + } + obj.getGroupSymbol().setMetadataID(metadata.getMetadataStore().getTempGroupID(obj.getGroupSymbol().getName())); + //now resolve the projected symbols + Set<GroupSymbol> groups = new HashSet<GroupSymbol>(); + groups.add(obj.getGroupSymbol()); + for (ElementSymbol symbol : obj.getProjectedSymbols()) { + try { + ResolverVisitor.resolveLanguageObject(symbol, groups, null, metadata); + } catch (TeiidException e) { + throw new TeiidRuntimeException(e); + } + } + //set to fixed width if any column has width specified + for (TextTable.TextColumn col : obj.getColumns()) { + if (col.getWidth() != null) { + obj.setFixedWidth(true); + break; + } + } + } + public void visit(SubqueryFromClause obj) { Collection<GroupSymbol> externalGroups = this.currentGroups; if (obj.isTable() && allowImplicit) { Modified: trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/rewriter/QueryRewriter.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -106,6 +106,7 @@ import org.teiid.query.sql.lang.SubqueryContainer; import org.teiid.query.sql.lang.SubqueryFromClause; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.TranslatableProcedureContainer; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.lang.Update; @@ -917,6 +918,9 @@ return rewriteJoinPredicate(parent, (JoinPredicate) clause); } else if (clause instanceof SubqueryFromClause) { rewriteSubqueryContainer((SubqueryFromClause)clause, true); + } else if (clause instanceof TextTable) { + TextTable tt = (TextTable)clause; + tt.setFile(rewriteExpressionDirect(tt.getFile())); } return clause; } Modified: trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/LanguageVisitor.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -130,4 +130,5 @@ public void visit(XMLAttributes obj) {} public void visit(XMLForest obj) {} public void visit(XMLNamespaces obj) {} + public void visit(TextTable obj) {} } Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/Create.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -25,11 +25,10 @@ import java.util.ArrayList; import java.util.List; -import org.teiid.core.TeiidComponentException; import org.teiid.core.util.EquivalenceUtil; import org.teiid.core.util.HashCodeUtil; -import org.teiid.query.metadata.QueryMetadataInterface; import org.teiid.query.sql.LanguageVisitor; +import org.teiid.query.sql.symbol.ElementSymbol; import org.teiid.query.sql.symbol.GroupSymbol; import org.teiid.query.sql.visitor.SQLStringVisitor; @@ -41,7 +40,7 @@ /** Identifies the table to be created. */ private GroupSymbol table; - private List columns = new ArrayList(); + private List<ElementSymbol> columns = new ArrayList<ElementSymbol>(); public GroupSymbol getTable() { return table; @@ -51,7 +50,7 @@ this.table = table; } - public List getColumns() { + public List<ElementSymbol> getColumns() { return columns; } @@ -100,7 +99,7 @@ visitor.visit(this); } - public void setColumns(List columns) { + public void setColumns(List<ElementSymbol> columns) { this.columns = columns; } Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/FromClause.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -26,6 +26,7 @@ import org.teiid.query.sql.LanguageObject; import org.teiid.query.sql.LanguageVisitor; +import org.teiid.query.sql.visitor.SQLStringVisitor; /** @@ -87,4 +88,9 @@ && other.isMakeDep() == this.isMakeDep() && other.isMakeNotDep() == this.isMakeNotDep(); } + + @Override + public String toString() { + return SQLStringVisitor.getSQLString(this); + } } Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/JoinPredicate.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -22,12 +22,14 @@ package org.teiid.query.sql.lang; -import java.util.*; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Iterator; +import java.util.List; import org.teiid.core.util.EquivalenceUtil; import org.teiid.core.util.HashCodeUtil; -import org.teiid.query.sql.*; -import org.teiid.query.sql.visitor.SQLStringVisitor; +import org.teiid.query.sql.LanguageVisitor; /** @@ -249,12 +251,4 @@ return clonedJoinPredicate; } - /** - * Returns a string representation of an instance of this class. - * @return String representation of object - */ - public String toString() { - return SQLStringVisitor.getSQLString(this); - } - } Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/SubqueryFromClause.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -26,7 +26,6 @@ import org.teiid.query.sql.LanguageVisitor; import org.teiid.query.sql.symbol.GroupSymbol; -import org.teiid.query.sql.visitor.SQLStringVisitor; /** @@ -174,12 +173,4 @@ return clause; } - /** - * Returns a string representation of an instance of this class. - * @return String representation of object - */ - public String toString() { - return SQLStringVisitor.getSQLString(this); - } - } Added: trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java (rev 0) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,263 @@ +package org.teiid.query.sql.lang; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +import org.teiid.core.types.DataTypeManager; +import org.teiid.core.util.EquivalenceUtil; +import org.teiid.query.sql.LanguageObject; +import org.teiid.query.sql.LanguageVisitor; +import org.teiid.query.sql.symbol.ElementSymbol; +import org.teiid.query.sql.symbol.Expression; +import org.teiid.query.sql.symbol.GroupSymbol; +import org.teiid.query.sql.util.SymbolMap; + +public class TextTable extends FromClause { + + public static class TextColumn { + private String name; + private String type; + private Integer width; + + public TextColumn(String name, String type, Integer width) { + this.name = name; + this.type = type; + this.width = width; + } + + public void setName(String name) { + this.name = name; + } + + public String getName() { + return name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public Integer getWidth() { + return width; + } + + public void setWidth(Integer width) { + this.width = width; + } + + @Override + public boolean equals(Object obj) { + if (obj == this) { + return true; + } + if (!(obj instanceof TextColumn)) { + return false; + } + TextColumn other = (TextColumn)obj; + return this.name.equalsIgnoreCase(other.name) && this.type.equalsIgnoreCase(other.type) + && EquivalenceUtil.areEqual(width, other.width); + } + + public int hashCode() { + return name.hashCode(); + } + + @Override + public TextColumn clone() { + return new TextColumn(name, type, width); + } + } + + private GroupSymbol symbol; + private List<ElementSymbol> projectedSymbols; + private SymbolMap correlatedReferences; + + private Expression file; + private List<TextColumn> columns = new ArrayList<TextColumn>(); + private Character delimiter; + private Character quote; + private boolean escape; + private Integer header; + private Integer skip; + + private boolean fixedWidth; + + public Character getQuote() { + return quote; + } + + public void setQuote(Character quote) { + this.quote = quote; + } + + public SymbolMap getCorrelatedReferences() { + return correlatedReferences; + } + + public void setCorrelatedReferences(SymbolMap correlatedReferences) { + this.correlatedReferences = correlatedReferences; + } + + public boolean isEscape() { + return escape; + } + + public void setEscape(boolean escape) { + this.escape = escape; + } + + public boolean isFixedWidth() { + return fixedWidth; + } + + public void setFixedWidth(boolean fixedWidth) { + this.fixedWidth = fixedWidth; + } + + public List<TextColumn> getColumns() { + return columns; + } + + public void setColumns(List<TextColumn> columns) { + this.columns = columns; + } + + public Character getDelimiter() { + return delimiter; + } + + public void setDelimiter(Character delimiter) { + this.delimiter = delimiter; + } + + public Integer getHeader() { + return header; + } + + public void setHeader(Integer header) { + this.header = header; + } + + public Integer getSkip() { + return skip; + } + + public void setSkip(Integer skip) { + this.skip = skip; + } + + public Expression getFile() { + return file; + } + + public void setFile(Expression file) { + this.file = file; + } + + @Override + public void acceptVisitor(LanguageVisitor visitor) { + visitor.visit(this); + } + + @Override + public TextTable clone() { + TextTable clone = new TextTable(); + clone.setDelimiter(this.delimiter); + clone.setFile((Expression)this.file.clone()); + clone.setHeader(this.header); + clone.setSkip(this.skip); + clone.setQuote(this.quote); + clone.escape = this.escape; + clone.symbol = (GroupSymbol)this.symbol.clone(); + for (TextColumn column : columns) { + clone.getColumns().add(column.clone()); + } + if (projectedSymbols != null) { + clone.projectedSymbols = LanguageObject.Util.deepClone(this.projectedSymbols, ElementSymbol.class); + } + if (correlatedReferences != null) { + clone.correlatedReferences = correlatedReferences.clone(); + } + clone.fixedWidth = this.fixedWidth; + return clone; + } + + @Override + public void collectGroups(Collection groups) { + groups.add(getGroupSymbol()); + } + + /** + * Get name of this clause. + * @return Name of clause + */ + public String getName() { + return this.symbol.getName(); + } + + public String getOutputName() { + return this.symbol.getOutputName(); + } + + /** + * Get GroupSymbol representing the named subquery + * @return GroupSymbol representing the subquery + */ + public GroupSymbol getGroupSymbol() { + return this.symbol; + } + + /** + * Reset the alias for this subquery from clause and it's pseudo-GroupSymbol. + * WARNING: this will modify the hashCode and equals semantics and will cause this object + * to be lost if currently in a HashMap or HashSet. + * @param name New name + * @since 4.3 + */ + public void setName(String name) { + this.symbol = new GroupSymbol(name); + } + + @Override + public boolean equals(Object obj) { + if (obj == this) { + return true; + } + if (!(obj instanceof TextTable)) { + return false; + } + TextTable other = (TextTable)obj; + return this.columns.equals(other.columns) + && EquivalenceUtil.areEqual(file, other.file) + && EquivalenceUtil.areEqual(symbol, other.symbol) + && EquivalenceUtil.areEqual(delimiter, other.delimiter) + && EquivalenceUtil.areEqual(escape, other.escape) + && EquivalenceUtil.areEqual(quote, other.quote) + && EquivalenceUtil.areEqual(header, other.header) + && EquivalenceUtil.areEqual(skip, other.skip); + + } + + @Override + public int hashCode() { + return this.symbol.hashCode(); + } + + public List<ElementSymbol> getProjectedSymbols() { + if (projectedSymbols == null) { + projectedSymbols = new ArrayList<ElementSymbol>(columns.size()); + for (TextColumn column : columns) { + ElementSymbol elementSymbol = new ElementSymbol(column.getName()); + elementSymbol.setType(DataTypeManager.getDataTypeClass(column.getType())); + projectedSymbols.add(elementSymbol); + } + } + return projectedSymbols; + } + +} Property changes on: trunk/engine/src/main/java/org/teiid/query/sql/lang/TextTable.java ___________________________________________________________________ Name: svn:mime-type + text/plain Modified: trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/lang/UnaryFromClause.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -25,9 +25,8 @@ import java.util.Collection; import org.teiid.core.util.EquivalenceUtil; -import org.teiid.query.sql.*; +import org.teiid.query.sql.LanguageVisitor; import org.teiid.query.sql.symbol.GroupSymbol; -import org.teiid.query.sql.visitor.SQLStringVisitor; /** @@ -133,15 +132,6 @@ return clonedUnaryFromClause; } - - /** - * Returns a string representation of an instance of this class. - * @return String representation of object - */ - public String toString() { - return SQLStringVisitor.getSQLString(this); - } - /** * @return Returns the expandedCommand. */ Modified: trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/navigator/PreOrPostOrderNavigator.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -61,6 +61,7 @@ import org.teiid.query.sql.lang.SubqueryCompareCriteria; import org.teiid.query.sql.lang.SubqueryFromClause; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.lang.Update; import org.teiid.query.sql.lang.XQuery; @@ -551,6 +552,14 @@ postVisitVisitor(obj); } + @Override + public void visit(TextTable obj) { + preVisitVisitor(obj); + visitNode(obj.getFile()); + visitNode(obj.getGroupSymbol()); + postVisitVisitor(obj); + } + public static void doVisit(LanguageObject object, LanguageVisitor visitor, boolean order) { PreOrPostOrderNavigator nav = new PreOrPostOrderNavigator(visitor, order); object.acceptVisitor(nav); Modified: trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/sql/visitor/SQLStringVisitor.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -75,9 +75,11 @@ import org.teiid.query.sql.lang.SubqueryCompareCriteria; import org.teiid.query.sql.lang.SubqueryFromClause; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.lang.Update; import org.teiid.query.sql.lang.XQuery; +import org.teiid.query.sql.lang.TextTable.TextColumn; import org.teiid.query.sql.proc.AssignmentStatement; import org.teiid.query.sql.proc.Block; import org.teiid.query.sql.proc.BreakStatement; @@ -427,11 +429,11 @@ parts.add(SPACE); // Columns clause - List columns = obj.getColumns(); + List<ElementSymbol> columns = obj.getColumns(); parts.add("("); //$NON-NLS-1$ - Iterator iter = columns.iterator(); + Iterator<ElementSymbol> iter = columns.iterator(); while(iter.hasNext()) { - ElementSymbol element = (ElementSymbol) iter.next(); + ElementSymbol element = iter.next(); element.setDisplayMode(ElementSymbol.DisplayMode.SHORT_OUTPUT_NAME); parts.add(registerNode(element)); parts.add(SPACE); @@ -1604,13 +1606,12 @@ parts.add("NO DEFAULT"); //$NON-NLS-1$ } else { parts.add("DEFAULT '"); //$NON-NLS-1$ - parts.add(escapeStringValue(item.getUri(), "'")); //$NON-NLS-1$ + parts.add(registerNode(new Constant(item.getUri()))); parts.add("'"); //$NON-NLS-1$ } } else { - parts.add("'"); //$NON-NLS-1$ - parts.add(escapeStringValue(item.getUri(), "'")); //$NON-NLS-1$ - parts.add("' AS "); //$NON-NLS-1$ + parts.add(registerNode(new Constant(item.getUri()))); + parts.add(" AS "); //$NON-NLS-1$ outputDisplayName(item.getPrefix()); } if (items.hasNext()) { @@ -1630,6 +1631,66 @@ parts.add(SPACE); parts.add(registerNode(obj.getRowLimit())); } + + @Override + public void visit(TextTable obj) { + parts.add("TEXTTABLE("); //$NON-NLS-1$ + parts.add(registerNode(obj.getFile())); + parts.add(SPACE); + parts.add("COLUMNS"); //$NON-NLS-1$ + + for (Iterator<TextColumn> cols = obj.getColumns().iterator(); cols.hasNext();) { + TextColumn col = cols.next(); + parts.add(SPACE); + outputDisplayName(col.getName()); + parts.add(SPACE); + parts.add(col.getType()); + if (col.getWidth() != null) { + parts.add(SPACE); + parts.add("WIDTH"); //$NON-NLS-1$ + parts.add(SPACE); + parts.add(col.getWidth()); + } + if (cols.hasNext()) { + parts.add(","); //$NON-NLS-1$ + } + } + if (obj.getDelimiter() != null) { + parts.add(SPACE); + parts.add("DELIMITER"); //$NON-NLS-1$ + parts.add(SPACE); + parts.add(registerNode(new Constant(obj.getDelimiter()))); + } + if (obj.getQuote() != null) { + parts.add(SPACE); + if (obj.isEscape()) { + parts.add(SQLReservedWords.ESCAPE); + } else { + parts.add("QUOTE"); //$NON-NLS-1$ + } + parts.add(SPACE); + parts.add(registerNode(new Constant(obj.getQuote()))); + } + if (obj.getHeader() != null) { + parts.add(SPACE); + parts.add("HEADER"); //$NON-NLS-1$ + if (1 != obj.getHeader()) { + parts.add(SPACE); + parts.add(obj.getHeader()); + } + } + if (obj.getSkip() != null) { + parts.add(SPACE); + parts.add("SKIP"); //$NON-NLS-1$ + parts.add(SPACE); + parts.add(obj.getSkip()); + } + parts.add(")");//$NON-NLS-1$ + parts.add(SPACE); + parts.add(SQLReservedWords.AS); + parts.add(SPACE); + outputDisplayName(obj.getName()); + } public static String escapeSinglePart(String part) { if(isReservedWord(part)) { Modified: trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java =================================================================== --- trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -40,6 +40,7 @@ import org.teiid.core.TeiidComponentException; import org.teiid.core.TeiidProcessingException; import org.teiid.core.types.DataTypeManager; +import org.teiid.core.util.EquivalenceUtil; import org.teiid.query.QueryPlugin; import org.teiid.query.eval.Evaluator; import org.teiid.query.function.FunctionLibrary; @@ -78,6 +79,7 @@ import org.teiid.query.sql.lang.SetQuery; import org.teiid.query.sql.lang.SubqueryCompareCriteria; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.Update; import org.teiid.query.sql.lang.SetQuery.Operation; import org.teiid.query.sql.navigator.PreOrderNavigator; @@ -114,7 +116,6 @@ import org.teiid.query.util.ErrorMessageKeys; import org.teiid.translator.SourceSystemFunctions; - public class ValidationVisitor extends AbstractValidationVisitor { private final class PositiveIntegerConstraint implements @@ -1157,5 +1158,51 @@ } } } + + @Override + public void visit(TextTable obj) { + boolean widthSet = false; + Character delimiter = null; + Character quote = null; + for (TextTable.TextColumn column : obj.getColumns()) { + if (column.getWidth() != null) { + widthSet = true; + if (column.getWidth() < 0) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$ + } + } else if (widthSet) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_invalid_width"), obj); //$NON-NLS-1$ + } + } + if (widthSet) { + if (obj.getDelimiter() != null || obj.getHeader() != null || obj.getQuote() != null) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_width"), obj); //$NON-NLS-1$ + } + } else { + delimiter = obj.getDelimiter(); + if (delimiter == null) { + delimiter = ','; + } + } + if (obj.getSkip() != null && obj.getSkip() < 0) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$ + } + if (!widthSet) { + if (obj.getHeader() != null && obj.getHeader() < 0) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_negative"), obj); //$NON-NLS-1$ + } + quote = obj.getQuote(); + if (quote == null) { + quote = '"'; + } + if (EquivalenceUtil.areEqual(quote, delimiter)) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_delimiter"), obj); //$NON-NLS-1$ + } + if (EquivalenceUtil.areEqual(quote, '\n') + || EquivalenceUtil.areEqual(delimiter, '\n')) { + handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.text_table_newline"), obj); //$NON-NLS-1$ + } + } + } } Modified: trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj =================================================================== --- trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/javacc/org/teiid/query/parser/SQLParser.jj 2010-05-27 13:45:40 UTC (rev 2156) @@ -1880,7 +1880,9 @@ Token lparen = null; } { - ( clause = unaryFromClause(info) + ( LOOKAHEAD(<ID> <LPAREN>, { "texttable".equalsIgnoreCase(getToken(1).image) }) clause = textTable(info) + | + clause = unaryFromClause(info) | LOOKAHEAD(subqueryFromClause(info)) clause = subqueryFromClause(info) | @@ -1889,7 +1891,7 @@ setFromClauseOptions(lparen, clause); } <RPAREN> - ) + ) ) [ (<MAKEDEP> { clause.setMakeDep(true); }) | (<MAKENOTDEP> { clause.setMakeNotDep(true); } ) @@ -1899,6 +1901,103 @@ } } +TextTable textTable(ParseInfo info) : +{ + Expression file = null; + String name = null; + Constant datatype = null; + List<TextTable.TextColumn> columns = new ArrayList<TextTable.TextColumn>(); + Integer width = null; + Character delimiter = null; + boolean escape = false; + Character quote = null; + Integer header = null; + Integer skip = null; + Token columnsToken = null; + String aliasID = null; +} +{ + <ID> <LPAREN> file = expression(info) + columnsToken = <ID> { if (!"columns".equalsIgnoreCase(columnsToken.image)) throw new ParseException(QueryPlugin.Util.getString("SQLParser.expected_columns")); } + name = id() + datatype = dataType() + [ + LOOKAHEAD(<ID>, { "width".equalsIgnoreCase(getToken(1).image) }) <ID> + width = intVal() + ] + { + columns.add(new TextTable.TextColumn(validateElementName(name), (String)datatype.getValue(), width)); + width = null; + } + (<COMMA> + name = id() + datatype = dataType() + [ + LOOKAHEAD(<ID>, { "width".equalsIgnoreCase(getToken(1).image) }) <ID> + width = intVal() + ] + { + columns.add(new TextTable.TextColumn(validateElementName(name), (String)datatype.getValue(), width)); + width = null; + } + )* + [ + LOOKAHEAD(<ID>, { "delimiter".equalsIgnoreCase(getToken(1).image) }) <ID> + delimiter = charVal(info, "DELMITER") + ] + [ + (<ESCAPE> quote = charVal(info, "ESCAPE") ) { escape = true; } + | + LOOKAHEAD(<ID>, { "quote".equalsIgnoreCase(getToken(1).image) }) + ( <ID> quote = charVal(info, "QUOTE") ) + ] + [ + LOOKAHEAD(<ID>, { "header".equalsIgnoreCase(getToken(1).image) }) <ID> + [ + header = intVal() + ] + { + if (header == null) { + header = 1; + } + } + ] + [ + LOOKAHEAD(<ID>, { "skip".equalsIgnoreCase(getToken(1).image) }) <ID> + skip = intVal() + ] + <RPAREN> + [<AS>] aliasID=id() + { + TextTable result = new TextTable(); + result.setFile(file); + result.setColumns(columns); + result.setDelimiter(delimiter); + result.setEscape(escape); + result.setHeader(header); + result.setSkip(skip); + result.setName(validateAlias(aliasID)); + result.setQuote(quote); + return result; + } +} + +int intVal() : +{ + Token t; +} +{ + t = <INTEGERVAL> + { + try { + return Integer.valueOf(t.image); + } catch(NumberFormatException e1) { + throw new ParseException(QueryPlugin.Util.getString("SQLParser.Integer_parse", t.image)); //$NON-NLS-1$ + } + } +} + + /** * Parse a unary from clause, which is just a single group name and an optional alias. * @return Unary from clause containing the group @@ -2228,8 +2327,8 @@ [<NOT> {negated = true;}] <LIKE> value=expression(info) - [ esc = escapeChar(info) | - (<LBRACE> esc = escapeChar(info) <RBRACE>) + [ <ESCAPE> esc = charVal(info, "LIKE ESCAPE") | + (<LBRACE> <ESCAPE> esc = charVal(info, "LIKE ESCAPE") <RBRACE>) ] ) { @@ -2242,15 +2341,15 @@ } } -Character escapeChar(ParseInfo info) : +Character charVal(ParseInfo info, String type) : { String escStr = null; } { - <ESCAPE> escStr=stringVal() + escStr=stringVal() { if (escStr.length() != 1) { - throw new ParseException(QueryPlugin.Util.getString("SQLParser.Invalid_escape_char", escStr)); //$NON-NLS-1$ + throw new ParseException(QueryPlugin.Util.getString("SQLParser.Invalid_char", type, escStr)); //$NON-NLS-1$ } return Character.valueOf(escStr.charAt(0)); } Modified: trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties =================================================================== --- trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/resources/org/teiid/query/execution/i18n.properties 2010-05-27 13:45:40 UTC (rev 2156) @@ -208,3 +208,11 @@ FileStorageManager.error_getting_free_disk_space=Error while seeking free disk space for directory: {0} FileStorageManager.can_not_save_lobs=Can not save *Reference Lobs* into persistent disk storage. FileStorageManager.batch_error=Error while converting batched data to file storage. + +TextTableNode.no_value=No value found for column \"{0}\" in the row ending on text line {1}. +TextTableNode.conversion_error=Could not convert value for column \"{0}\" in the row ending on text line {1}. +TextTableNode.header_missing=HEADER entry missing for column name \"{0}\". +TextTableNode.unclosed=Text parse error: Unclosed qualifier at end of text. +TextTableNode.character_not_allowed=Text parse error: Non-whitespace character found between the qualifier and the delimiter in text line {0}. +TextTableNode.unknown_escape=Text parse error: Unknown escape sequence \\{0} in text line {1}. +TextTableNode.invalid_width=Text parse error: Fixed width line width {0} is smaller than the expected {1} on text line {2} Modified: trunk/engine/src/main/resources/org/teiid/query/i18n.properties =================================================================== --- trunk/engine/src/main/resources/org/teiid/query/i18n.properties 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/main/resources/org/teiid/query/i18n.properties 2010-05-27 13:45:40 UTC (rev 2156) @@ -422,8 +422,8 @@ SQLParser.Invalid_id=Invalid metadata ID format: [{0}] SQLParser.Invalid_alias=Invalid alias format: [{0}] SQLParser.Invalid_short_name=Invalid simple identifier format: [{0}] -SQLParser.call_expected=Call keyword expected in callable statement -SQLParser.Invalid_escape_char=Like escape value must be a single character. +SQLParser.Invalid_char={0} value must be a single character: [{1}]. +SQLParser.expected_columns=Columns non-reserved word expected in TEXTTABLE. SystemSource.Add_desc=Add two numbers SystemSource.Add_result_desc=Left operand + right operand SystemSource.Subtract_desc=Subtract two numbers @@ -884,6 +884,11 @@ ValidationVisitor.drop_of_nontemptable=Cannot drop a non temporary table "{0}". ValidationVisitor.unrelated_orderby_xml=XML queries cannot order by an unrelated order by item. ValidationVisitor.orderby_expression_xml=XML queries cannot order by an expression. +ValidationVisitor.text_table_invalid_width=For a fixed width text table, all columns must have width set. +ValidationVisitor.text_table_width=Fixed width text tables should not have DELIMITER, QUOTE, ESCAPE, or HEADER specified. +ValidationVisitor.text_table_negative=Text table WIDTH, HEADER, or SKIP values must not be negative. +ValidationVisitor.text_table_delimiter=Text table DELIMITER cannot be the same as the QUOTE or ESCAPE characters. +ValidationVisitor.text_table_newline=Text table DELIMITER, QUOTE, and ESCAPE characters cannot be the new line character. UpdateProcedureResolver.only_variables=Element symbol "{0}" cannot be assigned a value. Only declared VARIABLES can be assigned values. wrong_result_type=No results found; or non-XML result object has been produced as a result of the execution of XQuery expression. Please note that only XML type results are supported. MappingLoader.unknown_node_type=Unknown Node Type "{0}" being loaded by the XML mapping document. Modified: trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java =================================================================== --- trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/test/java/org/teiid/query/parser/TestParser.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -71,9 +71,11 @@ import org.teiid.query.sql.lang.SubqueryCompareCriteria; import org.teiid.query.sql.lang.SubqueryFromClause; import org.teiid.query.sql.lang.SubquerySetCriteria; +import org.teiid.query.sql.lang.TextTable; import org.teiid.query.sql.lang.UnaryFromClause; import org.teiid.query.sql.lang.Update; import org.teiid.query.sql.lang.SetQuery.Operation; +import org.teiid.query.sql.lang.TextTable.TextColumn; import org.teiid.query.sql.proc.AssignmentStatement; import org.teiid.query.sql.proc.Block; import org.teiid.query.sql.proc.BreakStatement; @@ -2800,7 +2802,7 @@ } @Test public void testLikeWithEscapeException(){ - helpException("SELECT a from db.g where b like '#String' escape '#1'", "Parsing error: Like escape value must be a single character."); //$NON-NLS-1$ //$NON-NLS-2$ + helpException("SELECT a from db.g where b like '#String' escape '#1'", "Parsing error: LIKE ESCAPE value must be a single character: [#1]."); //$NON-NLS-1$ //$NON-NLS-2$ } /** SELECT "date"."time" from db.g */ @@ -6754,5 +6756,35 @@ query.setFrom(new From(Arrays.asList(sfc))); helpTest(sql, "SELECT * FROM TABLE(EXEC foo()) AS x", query); } + + @Test public void testTextTable() throws Exception { + String sql = "SELECT * from texttable(file columns x string WIDTH 1, y date width 10 skip 10) as x"; //$NON-NLS-1$ + Query query = new Query(); + query.setSelect(new Select(Arrays.asList(new AllSymbol()))); + TextTable tt = new TextTable(); + tt.setFile(new ElementSymbol("file")); + List<TextTable.TextColumn> columns = new ArrayList<TextTable.TextColumn>(); + columns.add(new TextTable.TextColumn("x", "string", 1)); + columns.add(new TextTable.TextColumn("y", "date", 10)); + tt.setColumns(columns); + tt.setSkip(10); + tt.setName("x"); + query.setFrom(new From(Arrays.asList(tt))); + helpTest(sql, "SELECT * FROM TEXTTABLE(file COLUMNS x string WIDTH 1, y date WIDTH 10 SKIP 10) AS x", query); + + sql = "SELECT * from texttable(file columns x string, y date delimiter ',' escape '\"' header skip 10) as x"; //$NON-NLS-1$ + tt.setDelimiter(','); + tt.setQuote('"'); + tt.setEscape(true); + tt.setHeader(1); + for (TextColumn textColumn : columns) { + textColumn.setWidth(null); + } + helpTest(sql, "SELECT * FROM TEXTTABLE(file COLUMNS x string, y date DELIMITER ',' ESCAPE '\"' HEADER SKIP 10) AS x", query); + } + + @Test public void testTextTableColumns() throws Exception { + helpException("SELECT * from texttable(foo x string)", "Parsing error: Columns non-reserved word expected in TEXTTABLE."); + } } Modified: trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java =================================================================== --- trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/test/java/org/teiid/query/processor/TestProcessor.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -93,7 +93,7 @@ import org.teiid.query.validator.Validator; import org.teiid.query.validator.ValidatorReport; - +@SuppressWarnings("nls") public class TestProcessor { // ################################## TEST HELPERS ################################ @@ -5254,21 +5254,28 @@ FunctionLibrary funcLibrary = new FunctionLibrary(SystemFunctionManager.getSystemFunctions(), new FunctionTree(new UDFSource(new FakeFunctionMetadataSource().getFunctionMethods()))); FakeMetadataFacade metadata = new FakeMetadataFacade(FakeMetadataFactory.example1Cached().getStore(), funcLibrary); - - Command command = helpParse(sql); + processPreparedStatement(sql, expected, dataManager, capFinder, + metadata, Arrays.asList("a ")); + } + + static void processPreparedStatement(String sql, List[] expected, + ProcessorDataManager dataManager, CapabilitiesFinder capFinder, + QueryMetadataInterface metadata, List<?> values) throws Exception { + Command command = helpParse(sql); CommandContext context = createCommandContext(); context.setMetadata(metadata); ProcessorPlan plan = helpGetPlan(command, metadata, capFinder, context); // Collect reference, set value - Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next(); VariableContext vc = new VariableContext(); - vc.setGlobalValue(ref.getContextSymbol(), "a "); //$NON-NLS-1$ + Iterator<?> valIter = values.iterator(); + for (Reference ref : ReferenceCollectorVisitor.getReferences(command)) { + vc.setGlobalValue(ref.getContextSymbol(), valIter.next()); //$NON-NLS-1$ + } context.setVariableContext(vc); // Run query - helpProcess(plan, context, dataManager, expected); - - } + helpProcess(plan, context, dataManager, expected); + } /** defect 15348 * @throws Exception */ @@ -5295,21 +5302,8 @@ FunctionLibrary funcLibrary = new FunctionLibrary(SystemFunctionManager.getSystemFunctions(), new FunctionTree(new UDFSource(new FakeFunctionMetadataSource().getFunctionMethods()))); FakeMetadataFacade metadata = new FakeMetadataFacade(FakeMetadataFactory.example1Cached().getStore(), funcLibrary); - - Command command = helpParse(sql); - CommandContext context = createCommandContext(); - context.setMetadata(metadata); - ProcessorPlan plan = helpGetPlan(command, metadata, capFinder, context); - - // Collect reference, set value - Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next(); - VariableContext vc = new VariableContext(); - vc.setGlobalValue(ref.getContextSymbol(), "a"); //$NON-NLS-1$ - context.setVariableContext(vc); - - // Run query - helpProcess(plan, context, dataManager, expected); - + processPreparedStatement(sql, expected, dataManager, capFinder, + metadata, Arrays.asList("a")); } @Test public void testSourceDoesntSupportGroupAlias() { @@ -7297,15 +7291,6 @@ FakeMetadataFacade metadata = FakeMetadataFactory.example1Cached(); - Command command = helpParse("select ?, e1 from pm1.g1"); //$NON-NLS-1$ - ProcessorPlan plan = helpGetPlan(command, metadata, capFinder); - - Reference ref = ReferenceCollectorVisitor.getReferences(command).iterator().next(); - VariableContext vc = new VariableContext(); - vc.setGlobalValue(ref.getContextSymbol(), "a"); //$NON-NLS-1$ - CommandContext context = createCommandContext(); - context.setVariableContext(vc); - List[] expected = new List[] { Arrays.asList("a", "b"), //$NON-NLS-1$ //$NON-NLS-2$ Arrays.asList("a", "c") //$NON-NLS-1$ //$NON-NLS-2$ @@ -7314,7 +7299,8 @@ HardcodedDataManager manager = new HardcodedDataManager(); manager.addData("SELECT 'a', pm1.g1.e1 FROM pm1.g1", expected); //$NON-NLS-1$ - helpProcess(plan, context, manager, expected); + processPreparedStatement("select ?, e1 from pm1.g1", expected, manager, capFinder, + metadata, Arrays.asList("a")); } @Test public void testCase6486() { Added: trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java =================================================================== --- trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java (rev 0) +++ trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,189 @@ +/* + * JBoss, Home of Professional Open Source. + * See the COPYRIGHT.txt file distributed with this work for information + * regarding copyright ownership. Some portions may be licensed + * to Red Hat, Inc. under one or more contributor license agreements. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA + * 02110-1301 USA. + */ + +package org.teiid.query.processor; + +import static org.teiid.query.processor.TestProcessor.*; + +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Arrays; +import java.util.List; + +import org.junit.Test; +import org.teiid.core.TeiidProcessingException; +import org.teiid.core.types.ClobImpl; +import org.teiid.core.types.ClobType; +import org.teiid.core.types.InputStreamFactory; +import org.teiid.core.util.UnitTestUtil; +import org.teiid.query.optimizer.capabilities.DefaultCapabilitiesFinder; +import org.teiid.query.unittest.FakeMetadataFactory; + +@SuppressWarnings({"unchecked", "nls"}) +public class TestTextTable { + + @Test public void testCorrelatedTextTable() throws Exception { + String sql = "select x.* from pm1.g1, texttable(e1 || ',' || e2 COLUMNS x string, y integer) x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList("a", 0), + Arrays.asList("a", 3), + Arrays.asList("c", 1), + Arrays.asList("b", 2), + Arrays.asList("a", 0), + }; + + process(sql, expected); + } + + @Test public void testTextTableFixed() throws Exception { + String sql = "select count(*) from texttable(? COLUMNS compkey string width 77, CDM_ID string width 14, CURRENCY string width 9, \"START\" string width 31, MATURITY string width 38, AMOUNT double width 13, RECORDSOURCE string width 13, SUMMIT_ID string width 25, RATE double width 21, SPREAD double width 9, DESK string width 13) x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList(52), + }; + + FakeDataManager dataManager = new FakeDataManager(); + sampleData1(dataManager); + + processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/cdm_dos.txt"))); + } + + @Test public void testNamedMultilineHeader() throws Exception { + String sql = "SELECT * from texttable(? COLUMNS Col3Head string HEADER) x"; + + List[] expected = new List[] { + Arrays.asList("VAL2"), + Arrays.asList("VAL4"), + Arrays.asList("abc"), + Arrays.asList("VAL9"), + }; + + FakeDataManager dataManager = new FakeDataManager(); + sampleData1(dataManager); + + processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/test-file.txt.csv"))); + } + + @Test public void testHeaderWithSkip() throws Exception { + String sql = "select count(*) from texttable(? COLUMNS PARTNAME string HEADER 3 SKIP 5) x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList(21), + }; + + FakeDataManager dataManager = new FakeDataManager(); + sampleData1(dataManager); + + processPreparedStatement(sql, expected, dataManager, new DefaultCapabilitiesFinder(), FakeMetadataFactory.example1Cached(), Arrays.asList(clobFromFile("text/TextParts_HeaderRow2.csv"))); + } + + @Test public void testEscape() throws Exception { + String sql = "select * from texttable('a\\,b,c\\\na\na,b\\\\' COLUMNS c1 string, c2 string ESCAPE '\\') x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList("a,b", "c\na"), + Arrays.asList("a", "b\\"), + }; + + process(sql, expected); + } + + @Test(expected=TeiidProcessingException.class) public void testEscapeError() throws Exception { + String sql = "select * from texttable('axq' COLUMNS c1 string ESCAPE 'x') x"; //$NON-NLS-1$ + + List[] expected = new List[] {}; + + process(sql, expected); + } + + @Test public void testDelimiter() throws Exception { + String sql = "select * from texttable('\na\\,b,c' COLUMNS c1 string, c2 string DELIMITER 'b') x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList("a\\,", ",c"), + }; + + process(sql, expected); + } + + @Test public void testNoRows() throws Exception { + String sql = "select * from texttable('' COLUMNS c1 string, c2 string SKIP 3) x"; //$NON-NLS-1$ + + List[] expected = new List[] {}; + + process(sql, expected); + } + + @Test public void testQuote() throws Exception { + String sql = "select * from texttable(' \" a\", \" \"\" \"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$ + + List[] expected = new List[] { + Arrays.asList(" a", " \" ") + }; + + process(sql, expected); + } + + @Test(expected=TeiidProcessingException.class) public void testUnclosedQuoteError() throws Exception { + String sql = "select * from texttable(' \" a\", \" \"\"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$ + + List[] expected = new List[] {}; + + process(sql, expected); + } + + @Test(expected=TeiidProcessingException.class) public void testQuoteError() throws Exception { + String sql = "select * from texttable(' \" a\", x\" \"\" \"' COLUMNS c1 string, c2 string) x"; //$NON-NLS-1$ + + List[] expected = new List[] {}; + + process(sql, expected); + } + + @Test(expected=TeiidProcessingException.class) public void testHeaderError() throws Exception { + String sql = "select * from texttable('notc1,notc2' COLUMNS c1 string, c2 string HEADER) x"; //$NON-NLS-1$ + + List[] expected = new List[] {}; + + process(sql, expected); + } + + public static void process(String sql, List[] expectedResults) throws Exception { + FakeDataManager dataManager = new FakeDataManager(); + sampleData1(dataManager); + ProcessorPlan plan = helpGetPlan(helpParse(sql), FakeMetadataFactory.example1Cached()); + helpProcess(plan, createCommandContext(), dataManager, expectedResults); + } + + public ClobType clobFromFile(final String file) { + return new ClobType(new ClobImpl(new InputStreamFactory() { + + @Override + public InputStream getInputStream() throws IOException { + return new FileInputStream(UnitTestUtil.getTestDataFile(file)); + } + }, -1)); + } + +} Property changes on: trunk/engine/src/test/java/org/teiid/query/processor/TestTextTable.java ___________________________________________________________________ Name: svn:mime-type + text/plain Modified: trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java =================================================================== --- trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/test/java/org/teiid/query/resolver/TestResolver.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -2941,4 +2941,9 @@ helpResolveException("select pm1.g1.e1 from pm1.g1, (exec pm1.sq2(pm1.g1.e2)) x"); //$NON-NLS-1$ } + @Test public void testCorrelatedTextTable() { + Command command = helpResolve("select x.* from pm1.g1, texttable(e1 COLUMNS x string) x"); //$NON-NLS-1$ + assertEquals(1, command.getProjectedSymbols().size()); + } + } \ No newline at end of file Modified: trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java =================================================================== --- trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java 2010-05-25 19:02:53 UTC (rev 2155) +++ trunk/engine/src/test/java/org/teiid/query/validator/TestValidator.java 2010-05-27 13:45:40 UTC (rev 2156) @@ -63,6 +63,7 @@ import org.teiid.query.unittest.FakeMetadataFactory; import org.teiid.query.unittest.FakeMetadataObject; import org.teiid.query.unittest.FakeMetadataStore; +import org.teiid.query.unittest.RealMetadataFactory; import org.teiid.query.validator.Validator; import org.teiid.query.validator.ValidatorReport; @@ -1983,6 +1984,17 @@ String userSql = "SELECT xpathValue('<?xml version=\"1.0\" encoding=\"utf-8\" ?><a><b><c>test</c></b></a>', '//*[local-name()=\"bookName\"]', 'xmlns==')"; //$NON-NLS-1$ helpValidate(userSql, new String[] {"xpathValue('<?xml version=\"1.0\" encoding=\"utf-8\" ?><a><b><c>test</c></b></a>', '//*[local-name()=\"bookName\"]', 'xmlns==')"}, FakeMetadataFactory.exampleBQTCached()); } + + @Test public void testTextTableNegativeWidth() { + helpValidate("SELECT * from texttable(null columns x string width -1) as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH -1) AS x"}, RealMetadataFactory.exampleBQTCached()); + } + + @Test public void testTextTableNoWidth() { + helpValidate("SELECT * from texttable(null columns x string width 1, y integer) as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH 1, y integer) AS x"}, RealMetadataFactory.exampleBQTCached()); + } - + @Test public void testTextTableInvalidDelimiter() { + helpValidate("SELECT * from texttable(null columns x string width 1 DELIMITER 'z') as x", new String[] {"TEXTTABLE(null COLUMNS x string WIDTH 1 DELIMITER 'z') AS x"}, RealMetadataFactory.exampleBQTCached()); + } + } Added: trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv =================================================================== --- trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv (rev 0) +++ trunk/engine/src/test/resources/text/TextParts_HeaderRow2.csv 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,26 @@ +. +. +PARTID,PARTNAME,PARTCOLOR,PARTWEIGHT +. +. +P301,Frame Base,Bisque,100.9 +P302,Tower,Honeydew,99.9 +P303,Axle Housing,Peach,98.9 +P304,Trough,Thistle,97.9 +P305,Outriggers,Lavender,96.9 +P306,Axle,Coral,95.9 +P307,Throwing Arm,Crimson,94.9 +P308,Sling and Pouch,Grey,93.9 +P309,Release Pin,Brick,92.9 +P310,Trigger,Rose,91.9 +P311,Counter-weight (fixed),Mauve,90.9 +P312,Counter-weight (hinged),Chartreuse,89.9 +P313,Dog ring,Salmon,88.9 +P314,Rack,Beige,87.9 +P315,Pinion,Red,86.9 +P316,Dzus fastener,Sepia,85.9 +P317,Half-shaft,Sienna,84.9 +P318,Input shaft,Carrot,83.9 +P319,Fuel port,Khaki,82.9 +P320,Caliper,Goldenrod,81.9 +P321,Splitter,Gold,80.9 Added: trunk/engine/src/test/resources/text/cdm_dos.txt =================================================================== --- trunk/engine/src/test/resources/text/cdm_dos.txt (rev 0) +++ trunk/engine/src/test/resources/text/cdm_dos.txt 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,52 @@ + 000183USD07/26/200107/26/2004 000183 USD 07/26/2001 07/26/2004 10000000.00 Summit 195205NY 0.005450 0.000000 MCREDCLN + 000184USD07/26/200107/26/2004 000184 USD 07/26/2001 07/26/2004 10000000.00 Summit 195219NY 0.005450 0.000000 NYBDUMMY + 000185USD07/26/200107/26/2004 000185 USD 07/26/2001 07/26/2004 10000000.00 Summit 195233NY 0.009700 0.000000 MCREDCLN + 000186USD07/26/200107/26/2004 000186 USD 07/26/2001 07/26/2004 10000000.00 Summit 195234NY 0.009700 0.000000 NYBDUMMY + 000258AUSD12/12/200209/15/2005 000258 USD 12/12/2002 09/15/2005 10000000.00 Summit 247299N 0.011500 0.000000 DCC2 + 000259AUSD01/17/200303/20/2005 000259 USD 01/17/2003 03/20/2005 10000000.00 Summit 247300N 0.011000 0.000000 DCC2 + 000273USD04/25/200307/07/2006 000273 USD 04/25/2003 07/07/2006 5000000.00 Summit 258123NY 0.008000 0.000000 LTINV + 000276AUSD05/10/200306/20/2008 000276 USD 05/10/2003 06/20/2008 10000000.00 Summit 260100N 0.033000 0.000000 DCC2 + 000277AUSD05/10/200306/20/2008 000277 USD 05/10/2003 06/20/2008 5000000.00 Summit 260101N 0.006500 0.000000 DCC2 + 000278AUSD05/13/200306/20/2008 000278 USD 05/13/2003 06/20/2008 5000000.00 Summit 260102N 0.033000 0.000000 DCC2 + 000279AUSD05/14/200306/20/2008 000279 USD 05/14/2003 06/20/2008 5000000.00 Summit 260131N 0.033000 0.000000 DCC2 + 000280USD06/11/200307/07/2006 000280 USD 06/11/2003 07/07/2006 5000000.00 Summit 264602NY 0.008900 0.000000 LTINV + 000289AUSD07/03/200309/20/2008 000289 USD 07/03/2003 09/20/2008 5000000.00 Summit 265496NY 0.004400 0.000000 LTINV + 000292USD07/19/200309/20/2008 000292 USD 07/19/2003 09/20/2008 5000000.00 Summit 271005NY 0.005000 0.000000 LTINV + 000350USD01/28/200403/20/2009 000350 USD 01/28/2004 03/20/2009 10000000.00 Summit 314996N 0.005200 0.000000 DCC2 + 000354USD03/10/200403/10/2007 000354 USD 03/10/2004 03/10/2007 9324082.77 Summit 318622NY 0.037500 0.000000 LTTRSCDS + 000357USD03/16/200403/16/2007 000357 USD 03/16/2004 03/16/2007 9924050.62 Summit 318638NY 0.030000 0.000000 LTTRSCDS + 000359USD03/09/200403/09/2007 000359 USD 03/09/2004 03/09/2007 9329349.59 Summit 318641NY 0.027500 0.000000 LTTRSCDS + 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 232935.47 Summit 318642NY 0.032500 0.000000 LTTRSCDS + 000352USD04/08/200404/08/2007 000352 USD 04/08/2004 04/08/2007 9788013.59 Summit 318643NY 0.032500 0.000000 LTTRSCDS + 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 809831.58 Summit 318644NY 0.032500 0.000000 LTTRSCDS + 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 705882.35 Summit 325357NY 0.032500 0.000000 LTTRSCDS + 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 156623.38 Summit 325358NY 0.032500 0.000000 LTTRSCDS + 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 278643.02 Summit 325363NY 0.032500 0.000000 LTTRSCDS + 000353USD03/04/200403/04/2007 000353 USD 03/04/2004 03/04/2007 711809.48 Summit 325364NY 0.032500 0.000000 LTTRSCDS + 000359USD03/09/200403/09/2007 000359 USD 03/09/2004 03/09/2007 76890.25 Summit 325366NY 0.027500 0.000000 LTTRSCDS + 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1049811.00 Summit 329001NY 0.032500 0.000000 LTTRSCDS + 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 414407.12 Summit 329002NY 0.032500 0.000000 LTTRSCDS + 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1204408.30 Summit 329003NY 0.032500 0.000000 LTTRSCDS + 000360USD03/18/200403/18/2007 000360 USD 03/18/2004 03/18/2007 1058626.60 Summit 329004NY 0.032500 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 194934.82 Summit 331567NY 0.040000 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 131062.23 Summit 335678NY 0.040000 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 1094136.66 Summit 335679NY 0.040000 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2785075.14 Summit 335680NY 0.040000 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2785075.14 Summit 335681NY 0.040000 0.000000 LTTRSCDS + 000362USD04/12/200404/12/2007 000362 USD 04/12/2004 04/12/2007 2984009.08 Summit 335682NY 0.040000 0.000000 LTTRSCDS + 000363USD05/03/200405/03/2007 000363 USD 05/03/2004 05/03/2007 10000000.00 Summit 338242NY 0.025000 0.000000 LTTRSCDS + 000364USD05/17/200405/17/2007 000364 USD 05/17/2004 05/17/2007 5000000.00 Summit 338820NY 0.032500 0.000000 LTTRSCDS + 000365USD05/07/200406/20/2009 000365 USD 05/07/2004 06/20/2009 5000000.00 Summit 340411N 0.004800 0.000000 DCC2 + 000099USD12/06/199912/06/2004 000099 USD 12/06/1999 12/06/2004 200000000.00 Summit 340805NY 0.000000 0.000000 PCTRSCDS + 000129USD04/03/200010/01/2008 000129 USD 04/03/2000 10/01/2008 200000000.00 Summit 340834NY 0.000000 0.000000 PCTRSCDS + 000148USD09/12/200012/19/2005 000148 USD 09/12/2000 12/19/2005 573667.12 Summit 340844NY 0.000000 0.000000 PCTRSCDS + 000149USD09/12/200012/19/2005 000149 USD 09/12/2000 12/19/2005 2118155.52 Summit 340847NY 0.000000 0.000000 PCTRSCDS + 000150USD09/12/200012/19/2005 000150 USD 09/12/2000 12/19/2005 4986491.88 Summit 340848NY 0.000000 0.000000 PCTRSCDS + 000366USD05/06/200405/06/2007 000366 USD 05/06/2004 05/06/2007 3058823.53 Summit 341369NY 0.030000 0.000000 LTTRSCDS + 000367USD05/06/200405/06/2007 000367 USD 05/06/2004 05/06/2007 6941176.47 Summit 341377NY 0.030000 0.000000 LTTRSCDS + 000213USD02/11/200202/11/2007 000213 USD 02/11/2002 02/11/2007 8000000.00 Summit 343117NY 0.056500 0.000000 CNA2 + 000272USD03/05/200301/31/2005 000272 USD 03/05/2003 01/31/2005 28000000.00 Summit 343123NY 0.006000 0.000000 CNA2 + 000333USD11/13/200311/20/2004 000333 USD 11/13/2003 11/20/2004 25000000.00 Summit 343151NY 0.003800 0.000000 CNA2 + 000368USD05/24/200405/24/2007 000368 USD 05/24/2004 05/24/2007 10000000.00 Summit 343169NY 0.025000 0.000000 LTTRSCDS + 000168USD12/18/200001/31/2006 000168 USD 12/18/2000 01/31/2006 672096851.46 Summit 344846NY 0.000000 0.000000 ASGCGD + 000369USD05/20/200405/20/2007 000369 USD 05/20/2004 05/20/2007 10000000.00 Summit 344851NY 0.032500 0.000000 LTTRSCDS Property changes on: trunk/engine/src/test/resources/text/cdm_dos.txt ___________________________________________________________________ Name: svn:mime-type + text/plain Added: trunk/engine/src/test/resources/text/test-file.txt.csv =================================================================== --- trunk/engine/src/test/resources/text/test-file.txt.csv (rev 0) +++ trunk/engine/src/test/resources/text/test-file.txt.csv 2010-05-27 13:45:40 UTC (rev 2156) @@ -0,0 +1,7 @@ +Col1Header,"Col2 +Header on newline",Col3Head +VAL1,,VAL2 +VAL3,,VAL4 +VAL5,Test,abc +VAL6,"VAL7 +VAL8 on a newline",VAL9