[teiid-commits] teiid SVN: r3242 - in branches/7.4.x: documentation/reference/src/main/docbook/en-US/content and 7 other directories.

Author: shawkins Date: 2011-06-10 13:50:13 -0400 (Fri, 10 Jun 2011) New Revision: 3242 Added: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/DocumentWrapper.java branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/NodeWrapper.java branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/StreamingUtils.java Modified: branches/7.4.x/build/kits/jboss-container/teiid-releasenotes.html branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/scalar_functions.xml branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/sql_support.xml branches/7.4.x/engine/pom.xml branches/7.4.x/engine/src/main/java/org/teiid/query/eval/Evaluator.java branches/7.4.x/engine/src/main/java/org/teiid/query/function/source/XMLSystemFunctions.java branches/7.4.x/engine/src/main/java/org/teiid/query/processor/relational/XMLTableNode.java branches/7.4.x/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/PathMapFilter.java branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/SaxonXQueryExpression.java branches/7.4.x/engine/src/test/java/org/teiid/query/processor/TestSQLXMLProcessing.java Log: TEIID-1448 adding the ability to stream docs for xquery processing Modified: branches/7.4.x/build/kits/jboss-container/teiid-releasenotes.html =================================================================== --- branches/7.4.x/build/kits/jboss-container/teiid-releasenotes.html 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/build/kits/jboss-container/teiid-releasenotes.html 2011-06-10 17:50:13 UTC (rev 3242) @@ -60,6 +60,7 @@ <LI><B>ODBC SSL</B> - added support for SSL encrypted ODBC connections. <LI><B>Reauthentication Statement</B> - SET SESSION AUTHORIZATION can now be used to perform a reauthentication via JDBC or ODBC. <LI><B>Pluggable Authorization</B> - an alternative PolicyDecider can be defined in the teiid-jboss-beans.xml file to customize authorization decisions. + <LI><B>Streaming XQuery</B> - in situations where document projection applies if the XMLQUERY/XMLTABLE path expressions meet certian conditions, then the incoming document will not only be projected, but the independent subtrees will be processed without loading the entire document. This allows for nearly arbitrarily large XML documents to be processed. See the Reference for more. </UL> <h2><a name="Compatibility">Compatibility Issues</a></h2> @@ -206,6 +207,11 @@ <h2><a name="LibraryUpdates">Thirdparty Library Updates</a></h2> The following components have been updated: + +<h4>From 7.4</h4> +<ul> + <li>nux 1.6, and xom 1.2 were added. +</ul> <h4>From 7.1</h4> <ul> <li>json-simple 1.1 was added. Modified: branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml =================================================================== --- branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/federated_planning.xml 2011-06-10 17:50:13 UTC (rev 3242) @@ -377,6 +377,41 @@ </listitem> </itemizedlist> </section> + <section id="xquery_optimization"> + <title>XQuery Optimization</title> + <para>A technique known as document projection is used to reduce the memory footprint of the context item document. + Document projection loads only the parts of the document needed by the relevant XQuery and path expressions. + Since document projection analysis uses all relevant path expressions, even 1 expression that could potentially use many nodes, e.g. //x rather than /a/b/x will cause a larger memory footprint. + With the relevant content removed the entire document will still be loaded into memory for processing. + Document projection will only be used when there is a context item (unnamed PASSING clause item) passed to XMLTABLE/XMLQUERY. A named variable will not have document projection performed. + In some cases the expressions used may be too complex for the optimizer to use document projection. You should check the SHOWPLAN DEBUG full plan output to see if the appropriate optimization has been performed.</para> + <para>With additional restrictions, simple context path expressions allow the processor to evaluate document subtrees independently - without loading the full document in memory. + A simple context path expression can be of the form "[/][ns:]root/[ns1:]elem/...", where a namespace prefix or element name can also be the * wild card. As with normal XQuery processing if namespace prefixes are used in the XQuery expression, they should be declared using the XMLNAMESPACES clause.</para> + <example> + <title>Streaming Eligible XMLQUERY</title> + <programlisting>XMLQUERY('/*:root/*:child' PASSING doc)</programlisting> + <para>Rather than loading the entire doc in-memory as a DOM tree, each child element will be independently added to the result.</para> + </example> + <example> + <title>Streaming Ineligible XMLQUERY</title> + <programlisting>XMLQUERY('//child' PASSING doc)</programlisting> + <para>The use of the descendent axis prevents the streaming optimization, but document projection can still be performed.</para> + </example> + <para>When using XMLTABLE, the COLUMN PATH expressions have additional restrictions. + They are allowed to reference any part of the element subtree formed by the context expression and they may use any attribute value from their direct parentage. + Any path expression where it is possible to reference a non-direct ancestor or sibling of the current context item prevent streaming from being used. + </para> + <example> + <title>Streaming Eligible XMLTABLE</title> + <programlisting>XMLTABLE('/*:root/*:child' PASSING doc COLUMNS fullchild XML PATH '.', parent_attr string PATH '../@attr', child_val integer)</programlisting> + <para>The context XQuery and the column path expression allow the streaming optimization, rather than loading the entire doc in-memory as a DOM tree, each child element will be independently added to the result.</para> + </example> + <example> + <title>Streaming Ineligible XMLTABLE</title> + <programlisting>XMLTABLE('/*:root/*:child' PASSING doc COLUMNS sibling_attr string PATH '../other_child/@attr')</programlisting> + <para>The reference of an element outside of the child subtree in the sibling_attr path prevents the streaming optimization from being used, but document projection can still be performed.</para> + </example> + </section> <section> <title>Federated Failure Modes</title> <section> Modified: branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/scalar_functions.xml =================================================================== --- branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/scalar_functions.xml 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/scalar_functions.xml 2011-06-10 17:50:13 UTC (rev 3242) @@ -2036,9 +2036,7 @@ <para>xquery in string. Return value is xml.</para> <para>XMLQUERY is part of the SQL/XML 2006 specification.</para> <para>See also <link linkend="xmltable">XMLTABLE</link></para> - <note><para>A technique known as document projection is used to reduce the memory footprint of the context item document. - Only the parts of the document needed by the XQuery path expressions will be loaded into memory. Since document projection analysis uses all relevant path expressions, even 1 expression that could potentially use many nodes, e.g. //x rather than /a/b/x will cause a larger memory footprint.</para></note> - + <note><para>See also <xref linkend="xquery_optimization"/></para></note> </section> <section id="xmlserialize"> <title>XMLSERIALIZE</title> Modified: branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/sql_support.xml =================================================================== --- branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/documentation/reference/src/main/docbook/en-US/content/sql_support.xml 2011-06-10 17:50:13 UTC (rev 3242) @@ -838,6 +838,7 @@ <para>See XMLELEMENT for the definition of NSP - <link linkend="xmlnamespaces">XMLNAMESPACES</link>.</para> <para>See XMLQUERY for the definition of <link linkend="passing">PASSING</link>.</para> <para>See also <link linkend="xmlquery">XMLQUERY</link></para> + <note><para>See also <xref linkend="xquery_optimization"/></para></note> <itemizedlist> <para>Parameters</para> <listitem> @@ -849,10 +850,21 @@ </para> </listitem> <listitem> - <para>If COLUMNS is not specified, then that is the same as having the COLUMNS clause: "COLUMNS OBJECT_VALUE XML PATH '.'", which returns the entire item as an XML value. Each non-ordinality column specifies a type and optionally a PATH and a DEFAULT expression. - If PATH is not specified, then the path will be the same as the column name. A FOR ORDINALITY column is typed as integer and will return the 1-based item number as its value. + <para>If COLUMNS is not specified, then that is the same as having the COLUMNS clause: "COLUMNS OBJECT_VALUE XML PATH '.'", which returns the entire item as an XML value. </para> </listitem> + <listitem> + <para>A FOR ORDINALITY column is typed as integer and will return the 1-based item number as its value. + </para> + </listitem> + <listitem> + <para>Each non-ordinality column specifies a type and optionally a PATH and a DEFAULT expression. + </para> + </listitem> + <listitem> + <para>If PATH is not specified, then the path will be the same as the column name. + </para> + </listitem> </itemizedlist> <itemizedlist> <para>Syntax Rules: Modified: branches/7.4.x/engine/pom.xml =================================================================== --- branches/7.4.x/engine/pom.xml 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/pom.xml 2011-06-10 17:50:13 UTC (rev 3242) @@ -91,7 +91,19 @@ <groupId>com.googlecode.json-simple</groupId> <artifactId>json-simple</artifactId> </dependency> + + <dependency> + <groupId>nux</groupId> + <artifactId>nux</artifactId> + <version>1.6</version> + </dependency> + <dependency> + <groupId>xom</groupId> + <artifactId>xom</artifactId> + <version>1.2</version> + </dependency> + </dependencies> </project> \ No newline at end of file Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/eval/Evaluator.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/eval/Evaluator.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/eval/Evaluator.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -42,6 +42,10 @@ import java.util.regex.Pattern; import java.util.regex.PatternSyntaxException; +import javax.xml.transform.stream.StreamResult; + +import net.sf.saxon.om.NodeInfo; +import net.sf.saxon.query.QueryResult; import net.sf.saxon.trans.XPathException; import org.teiid.api.exception.query.ExpressionEvaluationException; @@ -50,6 +54,7 @@ import org.teiid.core.ComponentNotFoundException; import org.teiid.core.TeiidComponentException; import org.teiid.core.TeiidProcessingException; +import org.teiid.core.TeiidRuntimeException; import org.teiid.core.types.BlobType; import org.teiid.core.types.ClobImpl; import org.teiid.core.types.ClobType; @@ -68,6 +73,7 @@ import org.teiid.query.function.FunctionDescriptor; import org.teiid.query.function.FunctionLibrary; import org.teiid.query.function.source.XMLSystemFunctions; +import org.teiid.query.function.source.XMLSystemFunctions.XmlConcat; import org.teiid.query.processor.ProcessorDataManager; import org.teiid.query.sql.LanguageObject; import org.teiid.query.sql.lang.AbstractSetCriteria; @@ -113,11 +119,37 @@ import org.teiid.query.util.CommandContext; import org.teiid.query.xquery.saxon.SaxonXQueryExpression; import org.teiid.query.xquery.saxon.SaxonXQueryExpression.Result; +import org.teiid.query.xquery.saxon.SaxonXQueryExpression.RowProcessor; import org.teiid.translator.WSConnection.Util; public class Evaluator { - private final class SequenceReader extends Reader { + private final class XMLQueryRowProcessor implements RowProcessor { + XmlConcat concat; //just used to get a writer + Type type; + private javax.xml.transform.Result result; + + private XMLQueryRowProcessor() throws TeiidProcessingException { + concat = new XmlConcat(context.getBufferManager()); + result = new StreamResult(concat.getWriter()); + } + + @Override + public void processRow(NodeInfo row) { + if (type == null) { + type = SaxonXQueryExpression.getType(row); + } else { + type = Type.CONTENT; + } + try { + QueryResult.serialize(row, result, SaxonXQueryExpression.DEFAULT_OUTPUT_PROPERTIES); + } catch (XPathException e) { + throw new TeiidRuntimeException(e); + } + } + } + + private final class SequenceReader extends Reader { private LinkedList<Reader> readers; private Reader current = null; @@ -759,13 +791,33 @@ private Object evaluateXMLQuery(List<?> tuple, XMLQuery xmlQuery) throws BlockedException, TeiidComponentException, FunctionExecutionException { - boolean emptyOnEmpty = true; - if (xmlQuery.getEmptyOnEmpty() != null) { - emptyOnEmpty = xmlQuery.getEmptyOnEmpty(); - } + boolean emptyOnEmpty = xmlQuery.getEmptyOnEmpty() == null || xmlQuery.getEmptyOnEmpty(); Result result = null; try { - result = evaluateXQuery(xmlQuery.getXQueryExpression(), xmlQuery.getPassing(), tuple); + XMLQueryRowProcessor rp = null; + if (xmlQuery.getXQueryExpression().isStreaming()) { + rp = new XMLQueryRowProcessor(); + } + try { + result = evaluateXQuery(xmlQuery.getXQueryExpression(), xmlQuery.getPassing(), tuple, rp); + } catch (TeiidRuntimeException e) { + if (e.getCause() instanceof XPathException) { + throw (XPathException)e.getCause(); + } + throw e; + } + if (rp != null) { + XMLType.Type type = rp.type; + if (type == null) { + if (!emptyOnEmpty) { + return null; + } + type = Type.CONTENT; + } + XMLType val = rp.concat.close(); + val.setType(rp.type); + return val; + } return xmlQuery.getXQueryExpression().createXMLType(result.iter, this.context.getBufferManager(), emptyOnEmpty); } catch (TeiidProcessingException e) { throw new FunctionExecutionException(e, QueryPlugin.Util.getString("Evaluator.xmlquery", e.getMessage())); //$NON-NLS-1$ @@ -859,7 +911,7 @@ } } - public Result evaluateXQuery(SaxonXQueryExpression xquery, List<DerivedColumn> cols, List<?> tuple) + public Result evaluateXQuery(SaxonXQueryExpression xquery, List<DerivedColumn> cols, List<?> tuple, RowProcessor processor) throws BlockedException, TeiidComponentException, TeiidProcessingException { HashMap<String, Object> parameters = new HashMap<String, Object>(); Object contextItem = null; @@ -871,7 +923,7 @@ parameters.put(passing.getAlias(), value); } } - return xquery.evaluateXQuery(contextItem, parameters); + return xquery.evaluateXQuery(contextItem, parameters, processor, context); } private Evaluator.NameValuePair<Object>[] getNameValuePairs(List<?> tuple, List<DerivedColumn> args, boolean xmlNames) Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/function/source/XMLSystemFunctions.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/function/source/XMLSystemFunctions.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/function/source/XMLSystemFunctions.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -401,6 +401,7 @@ private Writer writer; private FileStoreInputStreamFactory fsisf; private FileStore fs; + private Type type; public XmlConcat(BufferManager bm) throws TeiidProcessingException { fs = bm.createFileStore("xml"); //$NON-NLS-1$ @@ -417,6 +418,13 @@ } public void addValue(Object object) throws TeiidProcessingException { + if (type == null) { + if (object instanceof XMLType) { + type = ((XMLType)object).getType(); + } + } else { + type = Type.CONTENT; + } try { convertValue(writer, eventWriter, eventFactory, object); } catch (IOException e) { @@ -431,6 +439,10 @@ } } + public Writer getWriter() { + return writer; + } + public XMLType close() throws TeiidProcessingException { try { eventWriter.flush(); @@ -443,7 +455,11 @@ throw new TeiidProcessingException(e); } XMLType result = new XMLType(new SQLXMLImpl(fsisf)); - result.setType(Type.CONTENT); + if (type == null) { + result.setType(Type.CONTENT); + } else { + result.setType(type); + } return result; } Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/processor/relational/XMLTableNode.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/processor/relational/XMLTableNode.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/processor/relational/XMLTableNode.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -29,6 +29,7 @@ import java.util.Map; import net.sf.saxon.om.Item; +import net.sf.saxon.om.NodeInfo; import net.sf.saxon.om.SequenceIterator; import net.sf.saxon.sxpath.XPathDynamicContext; import net.sf.saxon.sxpath.XPathExpression; @@ -42,8 +43,12 @@ import org.teiid.api.exception.query.ExpressionEvaluationException; import org.teiid.common.buffer.BlockedException; import org.teiid.common.buffer.TupleBatch; +import org.teiid.common.buffer.TupleBuffer; +import org.teiid.common.buffer.BufferManager.TupleSourceType; import org.teiid.core.TeiidComponentException; +import org.teiid.core.TeiidException; import org.teiid.core.TeiidProcessingException; +import org.teiid.core.TeiidRuntimeException; import org.teiid.core.types.DataTypeManager; import org.teiid.core.types.XMLType; import org.teiid.query.QueryPlugin; @@ -51,11 +56,15 @@ import org.teiid.query.sql.lang.XMLTable; import org.teiid.query.sql.lang.XMLTable.XMLColumn; import org.teiid.query.xquery.saxon.SaxonXQueryExpression.Result; +import org.teiid.query.xquery.saxon.SaxonXQueryExpression.RowProcessor; /** * Handles xml table processing. + * + * When streaming the results will be fully built and stored in a buffer + * before being returned */ -public class XMLTableNode extends SubqueryAwareRelationalNode { +public class XMLTableNode extends SubqueryAwareRelationalNode implements RowProcessor { private static Map<Class<?>, BuiltInAtomicType> typeMapping = new HashMap<Class<?>, BuiltInAtomicType>(); @@ -74,6 +83,10 @@ private int rowCount = 0; private Item item; + private TupleBuffer buffer; + private int outputRow = 1; + private boolean usingOutput; + public XMLTableNode(int nodeID) { super(nodeID); } @@ -81,6 +94,10 @@ @Override public void closeDirect() { super.closeDirect(); + if(this.buffer != null) { + this.buffer.remove(); + this.buffer = null; + } reset(); } @@ -93,6 +110,9 @@ } item = null; rowCount = 0; + outputRow = 1; + usingOutput = false; + this.buffer = null; } public void setTable(XMLTable table) { @@ -116,31 +136,62 @@ protected TupleBatch nextBatchDirect() throws BlockedException, TeiidComponentException, TeiidProcessingException { - if (result == null) { - setReferenceValues(this.table); - result = getEvaluator(Collections.emptyMap()).evaluateXQuery(this.table.getXQueryExpression(), this.table.getPassing(), null); + evaluate(); + + if (this.table.getXQueryExpression().isStreaming()) { + TupleBatch batch = this.buffer.getBatch(outputRow); + outputRow = batch.getEndRow() + 1; + return batch; } while (!isBatchFull() && !isLastBatch()) { - processRow(); + if (item == null) { + try { + item = result.iter.next(); + } catch (XPathException e) { + throw new TeiidProcessingException(e, QueryPlugin.Util.getString("XMLTableNode.error", e.getMessage())); //$NON-NLS-1$ + } + rowCount++; + if (item == null) { + terminateBatches(); + break; + } + } + addBatchRow(processRow()); } return pullBatch(); } - private void processRow() throws ExpressionEvaluationException, BlockedException, - TeiidComponentException, TeiidProcessingException { - if (item == null) { + private void evaluate() throws TeiidComponentException, + ExpressionEvaluationException, BlockedException, + TeiidProcessingException { + if (result == null) { + if (this.buffer == null && this.table.getXQueryExpression().isStreaming()) { + this.buffer = this.getBufferManager().createTupleBuffer(getOutputElements(), getConnectionID(), TupleSourceType.PROCESSOR); + } + setReferenceValues(this.table); try { - item = result.iter.next(); - } catch (XPathException e) { - throw new TeiidProcessingException(e, QueryPlugin.Util.getString("XMLTableNode.error", e.getMessage())); //$NON-NLS-1$ + result = getEvaluator(Collections.emptyMap()).evaluateXQuery(this.table.getXQueryExpression(), this.table.getPassing(), null, this); + if (this.buffer != null) { + this.buffer.close(); + if (!usingOutput) { + this.buffer.setForwardOnly(true); + } + } + } catch (TeiidRuntimeException e) { + if (e.getCause() instanceof TeiidComponentException) { + throw (TeiidComponentException)e.getCause(); + } + if (e.getCause() instanceof TeiidProcessingException) { + throw (TeiidProcessingException)e.getCause(); + } + throw e; } - rowCount++; - if (item == null) { - terminateBatches(); - return; - } } + } + + private List<?> processRow() throws ExpressionEvaluationException, BlockedException, + TeiidComponentException, TeiidProcessingException { List<Object> tuple = new ArrayList<Object>(projectedColumns.size()); for (XMLColumn proColumn : projectedColumns) { if (proColumn.isOrdinal()) { @@ -167,8 +218,10 @@ if (pathIter.next() != null) { throw new TeiidProcessingException(QueryPlugin.Util.getString("XMLTableName.multi_value", proColumn.getName())); //$NON-NLS-1$ } - Object value = Value.convertToJava(colItem); - if (value instanceof Item) { + Object value = colItem; + if (value instanceof AtomicValue) { + value = Value.convertToJava(colItem); + } else if (value instanceof Item) { Item i = (Item)value; BuiltInAtomicType bat = typeMapping.get(proColumn.getSymbol().getType()); if (bat != null) { @@ -190,7 +243,37 @@ } } item = null; - addBatchRow(tuple); + return tuple; } + + @Override + public boolean hasFinalBuffer() { + return this.table.getXQueryExpression().isStreaming(); + } + + @Override + public TupleBuffer getFinalBuffer() throws BlockedException, + TeiidComponentException, TeiidProcessingException { + evaluate(); + usingOutput = true; + TupleBuffer finalBuffer = this.buffer; + this.buffer = null; + close(); + return finalBuffer; + } + + @Override + public void processRow(NodeInfo row) { + this.item = row; + rowCount++; + if (rowCount % 100 == 0) { + System.out.println(System.currentTimeMillis() + " " + rowCount); + } + try { + this.buffer.addTuple(processRow()); + } catch (TeiidException e) { + throw new TeiidRuntimeException(e); + } + } } \ No newline at end of file Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/validator/ValidationVisitor.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -789,8 +789,8 @@ } protected void validateInsert(Insert obj) { - Collection vars = obj.getVariables(); - Iterator varIter = vars.iterator(); + Collection<ElementSymbol> vars = obj.getVariables(); + Iterator<ElementSymbol> varIter = vars.iterator(); Collection values = obj.getValues(); Iterator valIter = values.iterator(); GroupSymbol insertGroup = obj.getGroup(); @@ -798,9 +798,7 @@ try { // Validate that all elements in variable list are updatable - Iterator elementIter = vars.iterator(); - while(elementIter.hasNext()) { - ElementSymbol insertElem = (ElementSymbol) elementIter.next(); + for (ElementSymbol insertElem : vars) { if(! getMetadata().elementSupports(insertElem.getMetadataID(), SupportConstants.Element.UPDATE)) { handleValidationError(QueryPlugin.Util.getString("ERR.015.012.0052", insertElem), insertElem); //$NON-NLS-1$ } @@ -825,7 +823,7 @@ // if any of the value present in the insert are null while(valIter.hasNext() && varIter.hasNext()) { Expression nextValue = (Expression) valIter.next(); - ElementSymbol nextVar = (ElementSymbol) varIter.next(); + ElementSymbol nextVar = varIter.next(); if (!(nextValue instanceof Constant) && getMetadata().isMultiSourceElement(nextVar.getMetadataID())) { handleValidationError(QueryPlugin.Util.getString("ValidationVisitor.multisource_constant", nextVar), nextVar); //$NON-NLS-1$ Added: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/DocumentWrapper.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/DocumentWrapper.java (rev 0) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/DocumentWrapper.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -0,0 +1,268 @@ +package org.teiid.query.xquery.saxon; + +import java.lang.reflect.Method; +import java.util.Collections; +import java.util.HashMap; +import java.util.Iterator; + +import net.sf.saxon.Configuration; +import net.sf.saxon.om.DocumentInfo; +import net.sf.saxon.om.NamePool; +import net.sf.saxon.om.NodeInfo; +import net.sf.saxon.type.Type; +import nu.xom.Attribute; +import nu.xom.Document; +import nu.xom.Element; +import nu.xom.Node; + +/** + * The root node of an XPath tree. (Or equivalently, the tree itself). + * <P> + * This class is used not only for a document, but also for the root + * of a document-less tree fragment. + * + * @author Michael H. Kay + * @author Wolfgang Hoschek (ported net.sf.saxon.jdom to XOM) + * @author Steve Hawkins (Ported to Saxon 9.1 for Teiid) + */ + +public class DocumentWrapper extends NodeWrapper implements DocumentInfo { + + protected Configuration config; + + protected String baseURI; + + protected int documentNumber; + + private HashMap idIndex; + + /** + * Create a Saxon wrapper for a XOM root node + * + * @param root + * The XOM root node + * @param baseURI + * The base URI for all the nodes in the tree + * @param config + * The configuration which defines the name pool used for all + * names in this tree + */ + public DocumentWrapper(Node root, String baseURI, Configuration config) { + super(root, null, 0); + if (root.getParent() != null) + throw new IllegalArgumentException("root node must not have a parent node"); + this.baseURI = baseURI; + this.docWrapper = this; + setConfiguration(config); + } + + /** + * Wrap a node in the XOM document. + * + * @param node + * The node to be wrapped. This must be a node in the same + * document (the system does not check for this). + * @return the wrapping NodeInfo object + */ + + public NodeInfo wrap(Node node) { + if (node == this.node) { + return this; + } + return makeWrapper(node, this); + } + + /** + * Set the configuration, which defines the name pool used for all names in + * this document. This is always called after a new document has been + * created. The implementation must register the name pool with the + * document, so that it can be retrieved using getNamePool(). It must also + * call NamePool.allocateDocumentNumber(), and return the relevant document + * number when getDocumentNumber() is subsequently called. + * + * @param config + * The configuration to be used + */ + + public void setConfiguration(Configuration config) { + this.config = config; + this.documentNumber = allocateDocumentNumber(config); + } + + /** + * Get the configuration previously set using setConfiguration + */ + + public Configuration getConfiguration() { + return config; + } + + /** + * Get the name pool used for the names in this document + * + * @return the name pool in which all the names used in this document are + * registered + */ + + public NamePool getNamePool() { + return config.getNamePool(); + } + + /** + * Get the unique document number for this document (the number is unique + * for all documents within a NamePool) + * + * @return the unique number identifying this document within the name pool + */ + + public int getDocumentNumber() { + return documentNumber; + } + + /** + * Get the element with a given ID, if any + * + * @param id + * the required ID value + * @return the element with the given ID, or null if there is no such ID + * present (or if the parser has not notified attributes as being of + * type ID). + */ + + public NodeInfo selectID(String id) { + if (idIndex == null) { + Element elem; + switch (nodeKind) { + case Type.DOCUMENT : + elem = ((Document) node).getRootElement(); + break; + case Type.ELEMENT : + elem = (Element) node; + break; + default: + return null; + } + idIndex = new HashMap(50); + buildIDIndex(elem); + } + return (NodeInfo) idIndex.get(id); + } + + + private void buildIDIndex(Element elem) { + // walk the tree in reverse document order, to satisfy the XPath 1.0 rule + // that says if an ID appears twice, the first one wins + for (int i=elem.getChildCount(); --i >= 0 ; ) { + Node child = elem.getChild(i); + if (child instanceof Element) { + buildIDIndex((Element)child); + } + } + for (int i=elem.getAttributeCount(); --i >= 0 ; ) { + Attribute att = elem.getAttribute(i); + if (att.getType() == Attribute.Type.ID) { + idIndex.put(att.getValue(), wrap(elem)); + } + } + } + + /** + * Get the unparsed entity with a given name + * + * @param name + * the name of the entity + * @return null: XOM does not provide access to unparsed entities + * @return if the entity exists, return an array of two Strings, the first + * holding the system ID of the entity, the second holding the + * public ID if there is one, or null if not. If the entity does not + * exist, return null. + */ + + public String[] getUnparsedEntity(String name) { + return null; + } + + private static final Method saxon85Method = findAllocateDocumentNumberMethod85(); + + // work-around for incompatibility introduced in saxon-8.5.1 + private int allocateDocumentNumber(Configuration config) { + if (saxon85Method == null) { + try { // saxon >= 8.5.1 + return allocateDocumentNumber851(config); + } catch (Throwable t) { + throw new RuntimeException(t); + } + } + + // saxon < 8.5.1 + try { + // return config.getNamePool().allocateDocumentNumber(this); + Object result = saxon85Method.invoke(config.getNamePool(), new Object[] {this}); + return ((Integer) result).intValue(); + } catch (Throwable t) { + throw new RuntimeException(t); + } + + } + + // saxon >= 8.5.1 + private int allocateDocumentNumber851(Configuration config) { + return config.getDocumentNumberAllocator().allocateDocumentNumber(); + } + + private static Method findAllocateDocumentNumberMethod85() { + try { + return NamePool.class.getMethod("allocateDocumentNumber", new Class[] {NodeInfo.class}); + } catch (Throwable t) { + return null; + } + } + + @Override + public Iterator getUnparsedEntityNames() { + return Collections.EMPTY_LIST.iterator(); + } + + @Override + public int getColumnNumber() { + return -1; + } + + @Override + public boolean isId() { + return false; + } + + @Override + public boolean isIdref() { + return false; + } + + @Override + public boolean isNilled() { + return false; + } + +} + +// +// The contents of this file are subject to the Mozilla Public License Version +// 1.0 (the "License"); +// you may not use this file except in compliance with the License. You may +// obtain a copy of the +// License at http://www.mozilla.org/MPL/ +// +// Software distributed under the License is distributed on an "AS IS" basis, +// WITHOUT WARRANTY OF ANY KIND, either express or implied. +// See the License for the specific language governing rights and limitations +// under the License. +// +// The Original Code is: all this file. +// +// The Initial Developer of the Original Code is Michael Kay +// +// Portions created by (your name) are Copyright (C) (your legal entity). All +// Rights Reserved. +// +// Contributor(s): none. +// Property changes on: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/DocumentWrapper.java ___________________________________________________________________ Added: svn:mime-type + text/plain Added: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/NodeWrapper.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/NodeWrapper.java (rev 0) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/NodeWrapper.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -0,0 +1,1387 @@ +package org.teiid.query.xquery.saxon; + +import net.sf.saxon.Configuration; +import net.sf.saxon.event.Receiver; +import net.sf.saxon.om.Axis; +import net.sf.saxon.om.AxisIterator; +import net.sf.saxon.om.AxisIteratorImpl; +import net.sf.saxon.om.DocumentInfo; +import net.sf.saxon.om.EmptyIterator; +import net.sf.saxon.om.FastStringBuffer; +import net.sf.saxon.om.Item; +import net.sf.saxon.om.NamePool; +import net.sf.saxon.om.NamespaceIterator; +import net.sf.saxon.om.Navigator; +import net.sf.saxon.om.NodeInfo; +import net.sf.saxon.om.SequenceIterator; +import net.sf.saxon.om.SiblingCountingNode; +import net.sf.saxon.om.SingleNodeIterator; +import net.sf.saxon.om.SingletonIterator; +import net.sf.saxon.om.StandardNames; +import net.sf.saxon.om.VirtualNode; +import net.sf.saxon.pattern.AnyNodeTest; +import net.sf.saxon.pattern.NameTest; +import net.sf.saxon.pattern.NodeKindTest; +import net.sf.saxon.pattern.NodeTest; +import net.sf.saxon.trans.XPathException; +import net.sf.saxon.type.Type; +import net.sf.saxon.value.AtomicValue; +import net.sf.saxon.value.StringValue; +import net.sf.saxon.value.UntypedAtomicValue; +import net.sf.saxon.value.Value; +import nu.xom.Attribute; +import nu.xom.Comment; +import nu.xom.DocType; +import nu.xom.Document; +import nu.xom.Element; +import nu.xom.Node; +import nu.xom.ParentNode; +import nu.xom.ProcessingInstruction; +import nu.xom.Text; + +/** + * A node in the XML parse tree representing an XML element, character content, + * or attribute. + * <P> + * This is the implementation of the NodeInfo interface used as a wrapper for + * XOM nodes. + * + * @author Michael H. Kay + * @author Wolfgang Hoschek (ported net.sf.saxon.jdom to XOM) + * @author Steve Hawkins (Ported to Saxon 9.1 for Teiid and fixed a bug with the buffer usage in getDeclaredNamespaces) + */ + +public class NodeWrapper implements NodeInfo, VirtualNode, SiblingCountingNode { + + protected Node node; + + protected short nodeKind; + + private NodeWrapper parent; // null means unknown + + protected DocumentWrapper docWrapper; + + protected int index; // -1 means unknown + + /** + * This constructor is protected: nodes should be created using the wrap + * factory method on the DocumentWrapper class + * + * @param node + * The XOM node to be wrapped + * @param parent + * The NodeWrapper that wraps the parent of this node + * @param index + * Position of this node among its siblings + */ + protected NodeWrapper(Node node, NodeWrapper parent, int index) { + short kind; + if (node instanceof Element) { + kind = Type.ELEMENT; + } else if (node instanceof Text) { + kind = Type.TEXT; + } else if (node instanceof Attribute) { + kind = Type.ATTRIBUTE; + } else if (node instanceof Comment) { + kind = Type.COMMENT; + } else if (node instanceof ProcessingInstruction) { + kind = Type.PROCESSING_INSTRUCTION; + } else if (node instanceof Document) { + kind = Type.DOCUMENT; + } else { + throwIllegalNode(node); // moved out of fast path to enable better inlining + return; // keep compiler happy + } + this.nodeKind = kind; + this.node = node; + this.parent = parent; + this.index = index; + } + + /** + * Factory method to wrap a XOM node with a wrapper that implements the + * Saxon NodeInfo interface. + * + * @param node + * The XOM node + * @param docWrapper + * The wrapper for the Document containing this node + * @return The new wrapper for the supplied node + */ + protected final NodeWrapper makeWrapper(Node node, DocumentWrapper docWrapper) { + return makeWrapper(node, docWrapper, null, -1); + } + + /** + * Factory method to wrap a XOM node with a wrapper that implements the + * Saxon NodeInfo interface. + * + * @param node + * The XOM node + * @param docWrapper + * The wrapper for the Document containing this node + * @param parent + * The wrapper for the parent of the XOM node + * @param index + * The position of this node relative to its siblings + * @return The new wrapper for the supplied node + */ + + protected final NodeWrapper makeWrapper(Node node, DocumentWrapper docWrapper, + NodeWrapper parent, int index) { + + if (node == docWrapper.node) return docWrapper; + NodeWrapper wrapper = new NodeWrapper(node, parent, index); + wrapper.docWrapper = docWrapper; + return wrapper; + } + + private static void throwIllegalNode(Node node) { + String str = node == null ? + "NULL" : + node.getClass() + " instance " + node.toString(); + throw new IllegalArgumentException("Bad node type in XOM! " + str); + } + + /** + * Get the configuration + */ + + public Configuration getConfiguration() { + return docWrapper.getConfiguration(); + } + + /** + * Get the underlying XOM node, to implement the VirtualNode interface + */ + + public Object getUnderlyingNode() { + return node; + } + + /** + * Get the name pool for this node + * + * @return the NamePool + */ + + public NamePool getNamePool() { + return docWrapper.getNamePool(); + } + + /** + * Return the type of node. + * + * @return one of the values Node.ELEMENT, Node.TEXT, Node.ATTRIBUTE, etc. + */ + + public int getNodeKind() { + return nodeKind; + } + + /** + * Get the typed value of the item + */ + + public SequenceIterator getTypedValue() { + return SingletonIterator.makeIterator((AtomicValue)atomize()); + } + + /** + * Get the typed value. The result of this method will always be consistent + * with the method {@link net.sf.saxon.om.Item#getTypedValue()}. However, + * this method is often more convenient and may be more efficient, + * especially in the common case where the value is expected to be a + * singleton. + * + * @return the typed value. If requireSingleton is set to true, the result + * will always be an AtomicValue. In other cases it may be a Value + * representing a sequence whose items are atomic values. + * @since 8.5 + */ + + public Value atomize() { + switch (getNodeKind()) { + case Type.COMMENT: + case Type.PROCESSING_INSTRUCTION: + return new StringValue(getStringValueCS()); + default: + return new UntypedAtomicValue(getStringValueCS()); + } + } + + /** + * Get the type annotation of this node, if any. Returns -1 for kinds of + * nodes that have no annotation, and for elements annotated as untyped, and + * attributes annotated as untypedAtomic. + * + * @return the type annotation of the node. + * @see net.sf.saxon.type.Type + */ + + public int getTypeAnnotation() { + if (getNodeKind() == Type.ATTRIBUTE) { + return StandardNames.XS_UNTYPED_ATOMIC; + } + return StandardNames.XS_UNTYPED; + } + + /** + * Determine whether this is the same node as another node. <br /> + * Note: a.isSameNode(b) if and only if generateId(a)==generateId(b) + * + * @return true if this Node object and the supplied Node object represent + * the same node in the tree. + */ + + public boolean isSameNodeInfo(NodeInfo other) { + if (other instanceof NodeWrapper) { + return node == ((NodeWrapper) other).node; // In XOM equality means identity + } + return false; + } + + /** + * The equals() method compares nodes for identity. It is defined to give the same result + * as isSameNodeInfo(). + * + * @param other the node to be compared with this node + * @return true if this NodeInfo object and the supplied NodeInfo object represent + * the same node in the tree. + * @since 8.7 Previously, the effect of the equals() method was not defined. Callers + * should therefore be aware that third party implementations of the NodeInfo interface may + * not implement the correct semantics. It is safer to use isSameNodeInfo() for this reason. + * The equals() method has been defined because it is useful in contexts such as a Java Set or HashMap. + */ + + public boolean equals(Object other) { + if (other instanceof NodeInfo) { + return isSameNodeInfo((NodeInfo)other); + } else { + return false; + } + } + + /** + * The hashCode() method obeys the contract for hashCode(): that is, if two objects are equal + * (represent the same node) then they must have the same hashCode() + * @since 8.7 Previously, the effect of the equals() and hashCode() methods was not defined. Callers + * should therefore be aware that third party implementations of the NodeInfo interface may + * not implement the correct semantics. + */ + + public int hashCode() { + return node.hashCode(); + } + + /** + * Get the System ID for the node. + * + * @return the System Identifier of the entity in the source document + * containing the node, or null if not known. Note this is not the + * same as the base URI: the base URI can be modified by xml:base, + * but the system ID cannot. + */ + + public String getSystemId() { + return docWrapper.baseURI; + } + + public void setSystemId(String uri) { + docWrapper.baseURI = uri; + } + + /** + * Get the Base URI for the node, that is, the URI used for resolving a + * relative URI contained in the node. + */ + + public String getBaseURI() { + return node.getBaseURI(); + } + + /** + * Get line number + * + * @return the line number of the node in its original source document; or + * -1 if not available + */ + + public int getLineNumber() { + return -1; + } + + /** + * Determine the relative position of this node and another node, in + * document order. The other node will always be in the same document. + * + * @param other + * The other node, whose position is to be compared with this + * node + * @return -1 if this node precedes the other node, +1 if it follows the + * other node, or 0 if they are the same node. (In this case, + * isSameNode() will always return true, and the two nodes will + * produce the same result for generateId()) + */ + + public int compareOrder(NodeInfo other) { + if (other instanceof NodeWrapper) { + return compareOrderFast(node,((NodeWrapper) other).node); +// } +// if (other instanceof SiblingCountingNode) { +// return Navigator.compareOrder(this, (SiblingCountingNode) other); + } else { + // it must be a namespace node + return -other.compareOrder(this); + } + } + + private static int compareOrderFast(Node first, Node second) { + /* + * Unfortunately we do not have a sequence number for each node at hand; + * this would allow to turn the comparison into a simple sequence number + * subtraction. Walking the entire tree and batch-generating sequence + * numbers on the fly is no good option either. However, this rewritten + * implementation turns out to be more than fast enough. + */ + + // assert first != null && second != null + // assert first and second MUST NOT be namespace nodes + if (first == second) return 0; + + ParentNode firstParent = first.getParent(); + ParentNode secondParent = second.getParent(); + if (firstParent == null) { + if (secondParent != null) return -1; // first node is the root + // both nodes are parentless, use arbitrary but fixed order: + return first.hashCode() - second.hashCode(); + } + + if (secondParent == null) return +1; // second node is the root + + // do they have the same parent (common case)? + if (firstParent == secondParent) { + int i1 = firstParent.indexOf(first); + int i2 = firstParent.indexOf(second); + + // note that attributes and namespaces are not children + // of their own parent (i = -1). + // attribute (if any) comes before child + if (i1 != -1) return (i2 != -1) ? i1 - i2 : +1; + if (i2 != -1) return -1; + + // assert: i1 == -1 && i2 == -1 + // i.e. both nodes are attributes + Element elem = (Element) firstParent; + for (int i = elem.getAttributeCount(); --i >= 0;) { + Attribute attr = elem.getAttribute(i); + if (attr == second) return -1; + if (attr == first) return +1; + } + throw new IllegalStateException("should be unreachable"); + } + + // find the depths of both nodes in the tree + int depth1 = 0; + int depth2 = 0; + Node p1 = first; + Node p2 = second; + while (p1 != null) { + depth1++; + p1 = p1.getParent(); + if (p1 == second) return +1; + } + while (p2 != null) { + depth2++; + p2 = p2.getParent(); + if (p2 == first) return -1; + } + + // move up one branch of the tree so we have two nodes on the same level + p1 = first; + while (depth1 > depth2) { + p1 = p1.getParent(); + depth1--; + } + p2 = second; + while (depth2 > depth1) { + p2 = p2.getParent(); + depth2--; + } + + // now move up both branches in sync until we find a common parent + while (true) { + firstParent = p1.getParent(); + secondParent = p2.getParent(); + if (firstParent == null || secondParent == null) { + // both nodes are documentless, use arbitrary but fixed order + // based on their root elements + return p1.hashCode() - p2.hashCode(); + // throw new NullPointerException("XOM tree compare - internal error"); + } + if (firstParent == secondParent) { + return firstParent.indexOf(p1) - firstParent.indexOf(p2); + } + p1 = firstParent; + p2 = secondParent; + } + } + + /** + * Return the string value of the node. The interpretation of this depends + * on the type of node. For an element it is the accumulated character + * content of the element, including descendant elements. + * + * @return the string value of the node + */ + + public String getStringValue() { + return node.getValue(); + } + + /** + * Get the value of the item as a CharSequence. This is in some cases more efficient than + * the version of the method that returns a String. + */ + + public CharSequence getStringValueCS() { + return node.getValue(); + } + + /** + * Get name code. The name code is a coded form of the node name: two nodes + * with the same name code have the same namespace URI, the same local name, + * and the same prefix. By masking the name code with &0xfffff, you get a + * fingerprint: two nodes with the same fingerprint have the same local name + * and namespace URI. + * + * @see net.sf.saxon.om.NamePool#allocate allocate + */ + + public int getNameCode() { + switch (nodeKind) { + case Type.ELEMENT: + case Type.ATTRIBUTE: + case Type.PROCESSING_INSTRUCTION: + return docWrapper.getNamePool().allocate(getPrefix(), getURI(), + getLocalPart()); + default: + return -1; + } + } + + /** + * Get fingerprint. The fingerprint is a coded form of the expanded name of + * the node: two nodes with the same name code have the same namespace URI + * and the same local name. A fingerprint of -1 should be returned for a + * node with no name. + */ + + public int getFingerprint() { + int nc = getNameCode(); + if (nc == -1) return -1; + return nc & 0xfffff; + } + + /** + * Get the local part of the name of this node. This is the name after the + * ":" if any. + * + * @return the local part of the name. For an unnamed node, returns "". + */ + + public String getLocalPart() { + switch (nodeKind) { + case Type.ELEMENT: + return ((Element) node).getLocalName(); + case Type.ATTRIBUTE: + return ((Attribute) node).getLocalName(); + case Type.PROCESSING_INSTRUCTION: + return ((ProcessingInstruction) node).getTarget(); + default: + return ""; + } + } + + /** + * Get the prefix of the name of the node. This is defined only for elements and attributes. + * If the node has no prefix, or for other kinds of node, return a zero-length string. + * @return The prefix of the name of the node. + */ + + public String getPrefix() { + switch (nodeKind) { + case Type.ELEMENT: + return ((Element) node).getNamespacePrefix(); + case Type.ATTRIBUTE: + return ((Attribute) node).getNamespacePrefix(); + default: + return ""; + } + } + + /** + * Get the URI part of the name of this node. This is the URI corresponding + * to the prefix, or the URI of the default namespace if appropriate. + * + * @return The URI of the namespace of this node. For an unnamed node, or + * for a node with an empty prefix, return an empty string. + */ + + public String getURI() { + switch (nodeKind) { + case Type.ELEMENT: + return ((Element) node).getNamespaceURI(); + case Type.ATTRIBUTE: + return ((Attribute) node).getNamespaceURI(); + default: + return ""; + } + } + + /** + * Get the display name of this node. For elements and attributes this is + * [prefix:]localname. For unnamed nodes, it is an empty string. + * + * @return The display name of this node. For a node with no name, return an + * empty string. + */ + + public String getDisplayName() { + switch (nodeKind) { + case Type.ELEMENT: + return ((Element) node).getQualifiedName(); + case Type.ATTRIBUTE: + return ((Attribute) node).getQualifiedName(); + case Type.PROCESSING_INSTRUCTION: + return ((ProcessingInstruction) node).getTarget(); + default: + return ""; + } + } + + /** + * Get the NodeInfo object representing the parent of this node + */ + + public NodeInfo getParent() { + if (parent == null) { + ParentNode p = node.getParent(); + if (p != null) parent = makeWrapper(p, docWrapper); + } + return parent; + } + + /** + * Get the index position of this node among its siblings (starting from 0) + */ + + public int getSiblingPosition() { + if (index != -1) return index; + switch (nodeKind) { + case Type.ATTRIBUTE: { + Attribute att = (Attribute) node; + Element p = (Element) att.getParent(); + if (p == null) return 0; + for (int i=p.getAttributeCount(); --i >= 0;) { + if (p.getAttribute(i) == att) { + index = i; + return i; + } + } + throw new IllegalStateException("XOM node not linked to parent node"); + } + + default: { + ParentNode p = node.getParent(); + int i = (p == null ? 0 : p.indexOf(node)); + if (i == -1) throw new IllegalStateException("XOM node not linked to parent node"); + index = i; + return index; + } + } + } + + /** + * Return an iteration over the nodes reached by the given axis from this + * node + * + * @param axisNumber + * the axis to be used + * @return a SequenceIterator that scans the nodes reached by the axis in + * turn. + */ + + public AxisIterator iterateAxis(byte axisNumber) { + return iterateAxis(axisNumber, AnyNodeTest.getInstance()); + } + + /** + * Return an iteration over the nodes reached by the given axis from this + * node + * + * @param axisNumber + * the axis to be used + * @param nodeTest + * A pattern to be matched by the returned nodes + * @return a SequenceIterator that scans the nodes reached by the axis in + * turn. + */ + + public AxisIterator iterateAxis(byte axisNumber, NodeTest nodeTest) { + // for clarifications, see the W3C specs or: + // http://msdn.microsoft.com/library/default.asp?url=/library/en-us/xmlsdk/h... + switch (axisNumber) { + case Axis.ANCESTOR: + return new AncestorAxisIterator(this, false, nodeTest); + + case Axis.ANCESTOR_OR_SELF: + return new AncestorAxisIterator(this, true, nodeTest); + + case Axis.ATTRIBUTE: + if (nodeKind != Type.ELEMENT || ((Element) node).getAttributeCount() == 0) { + return EmptyIterator.getInstance(); + } else { + return new AttributeAxisIterator(this, nodeTest); + } + + case Axis.CHILD: + if (hasChildNodes()) { + return new ChildAxisIterator(this, true, true, nodeTest); + } else { + return EmptyIterator.getInstance(); + } + + case Axis.DESCENDANT: + if (hasChildNodes()) { + return new DescendantAxisIterator(this, false, false, nodeTest); + } else { + return EmptyIterator.getInstance(); + } + + case Axis.DESCENDANT_OR_SELF: + if (hasChildNodes()) { + return new DescendantAxisIterator(this, true, false, nodeTest); + } else { + return filteredSingleton(this, nodeTest); + } + + case Axis.FOLLOWING: + if (getParent() == null) { + return EmptyIterator.getInstance(); + } else { + return new DescendantAxisIterator(this, false, true, nodeTest); + } + + case Axis.FOLLOWING_SIBLING: + if (nodeKind == Type.ATTRIBUTE || getParent() == null) { + return EmptyIterator.getInstance(); + } else { + return new ChildAxisIterator(this, false, true, nodeTest); + } + + case Axis.NAMESPACE: + if (nodeKind == Type.ELEMENT) { + return NamespaceIterator.makeIterator(this, nodeTest); + } else { + return EmptyIterator.getInstance(); + } + + case Axis.PARENT: + if (getParent() == null) { + return EmptyIterator.getInstance(); + } else { + return filteredSingleton(getParent(), nodeTest); + } + + case Axis.PRECEDING: + return new PrecedingAxisIterator(this, false, nodeTest); +// return new Navigator.AxisFilter( +// new Navigator.PrecedingEnumeration(this, false), nodeTest); + + case Axis.PRECEDING_SIBLING: + if (nodeKind == Type.ATTRIBUTE || getParent() == null) { + return EmptyIterator.getInstance(); + } else { + return new ChildAxisIterator(this, false, false, nodeTest); + } + + case Axis.SELF: + return filteredSingleton(this, nodeTest); + + case Axis.PRECEDING_OR_ANCESTOR: + // This axis is used internally by saxon for the xsl:number implementation, + // it returns the union of the preceding axis and the ancestor axis. + return new PrecedingAxisIterator(this, true, nodeTest); +// return new Navigator.AxisFilter(new Navigator.PrecedingEnumeration( +// this, true), nodeTest); + + default: + throw new IllegalArgumentException("Unknown axis number " + axisNumber); + } + } + +// private static AxisIterator makeSingleIterator(NodeWrapper wrapper, NodeTest nodeTest) { +// if (nodeTest == AnyNodeTest.getInstance() || nodeTest.matches(wrapper)) +// return SingletonIterator.makeIterator(wrapper); +// else +// return EmptyIterator.getInstance(); +// } + + /** + * Get the value of a given attribute of this node + * + * @param fingerprint + * The fingerprint of the attribute name + * @return the attribute value if it exists or null if not + */ + + public String getAttributeValue(int fingerprint) { + if (nodeKind == Type.ELEMENT) { + NamePool pool = docWrapper.getNamePool(); + String localName = pool.getLocalName(fingerprint); + String uri = pool.getURI(fingerprint); + Attribute att = ((Element) node).getAttribute(localName, uri); + if (att != null) return att.getValue(); + } + return null; + } + + /** + * Get the root node of the tree containing this node + * + * @return the NodeInfo representing the top-level ancestor of this node. + * This will not necessarily be a document node + */ + + public NodeInfo getRoot() { + return docWrapper; + } + + /** + * Get the root node, if it is a document node. + * + * @return the DocumentInfo representing the containing document. + */ + + public DocumentInfo getDocumentRoot() { + if (docWrapper.node instanceof Document) { + return docWrapper; + } else { + return null; + } + } + + /** + * Determine whether the node has any children. <br /> + * Note: the result is equivalent to <br /> + * getEnumeration(Axis.CHILD, AnyNodeTest.getInstance()).hasNext() + */ + + public boolean hasChildNodes() { + return node.getChildCount() > 0; + } + + /** + * Get a character string that uniquely identifies this node. Note: + * a.isSameNode(b) if and only if generateId(a)==generateId(b) + * + * @param buffer a buffer to contain a string that uniquely identifies this node, across all documents + */ + + public void generateId(FastStringBuffer buffer) { + Navigator.appendSequentialKey(this, buffer, true); + //buffer.append(Navigator.getSequentialKey(this)); + } + + /** + * Get the document number of the document containing this node. For a + * free-standing orphan node, just return the hashcode. + */ + + public int getDocumentNumber() { + return docWrapper.getDocumentNumber(); + } + + /** + * Copy this node to a given outputter (deep copy) + */ + + public void copy(Receiver out, int whichNamespaces, + boolean copyAnnotations, int locationId) throws XPathException { + Navigator.copy(this, out, docWrapper.getNamePool(), whichNamespaces, + copyAnnotations, locationId); + } + + /** + * Get all namespace undeclarations and undeclarations defined on this element. + * + * @param buffer If this is non-null, and the result array fits in this buffer, then the result + * may overwrite the contents of this array, to avoid the cost of allocating a new array on the heap. + * @return An array of integers representing the namespace declarations and undeclarations present on + * this element. For a node other than an element, return null. Otherwise, the returned array is a + * sequence of namespace codes, whose meaning may be interpreted by reference to the name pool. The + * top half word of each namespace code represents the prefix, the bottom half represents the URI. + * If the bottom half is zero, then this is a namespace undeclaration rather than a declaration. + * The XML namespace is never included in the list. If the supplied array is larger than required, + * then the first unused entry will be set to -1. + * <p/> + * <p>For a node other than an element, the method returns null.</p> + */ + + public int[] getDeclaredNamespaces(int[] buffer) { + if (node instanceof Element) { + Element elem = (Element)node; + int size = elem.getNamespaceDeclarationCount(); + if (size == 0) { + return EMPTY_NAMESPACE_LIST; + } + int[] result = (buffer != null && size <= buffer.length ? buffer : new int[size]); + NamePool pool = getNamePool(); + for (int i=0; i < size; i++) { + String prefix = elem.getNamespacePrefix(i); + String uri = elem.getNamespaceURI(prefix); + result[i] = pool.allocateNamespaceCode(prefix, uri); + } + if (size < result.length) { + result[size] = -1; + } + return result; + } else { + return null; + } + } + + /////////////////////////////////////////////////////////////////////////////// + // Axis enumeration classes + /////////////////////////////////////////////////////////////////////////////// + + /** + * Handles the ancestor axis in a rather direct manner. + */ + private final class AncestorAxisIterator extends AxisIteratorImpl { + + private NodeWrapper start; + private boolean includeSelf; + + private NodeTest nodeTest; + + public AncestorAxisIterator(NodeWrapper start, boolean includeSelf, NodeTest test) { + // use lazy instead of eager materialization (performance) + this.start = start; + if (test == AnyNodeTest.getInstance()) test = null; + this.nodeTest = test; + if (!includeSelf) this.current = start; + this.includeSelf = includeSelf; + this.position = 0; + } + + public Item next() { + NodeInfo curr; + do { // until we find a match + curr = advance(); + } + while (curr != null && nodeTest != null && (! nodeTest.matches(curr))); + + if (curr != null) position++; + current = curr; + return curr; + } + + private NodeInfo advance() { + if (current == null) + current = start; + else + current = current.getParent(); + + return current; + } + + public Item current() { + return current; + } + + public SequenceIterator getAnother() { + return new AncestorAxisIterator(start, includeSelf, nodeTest); + } + + public int getProperties() { + return 0; + } + + } // end of class AncestorAxisIterator + + /** + * Handles the attribute axis in a rather direct manner. + */ + private final class AttributeAxisIterator extends AxisIteratorImpl { + + private NodeWrapper start; + + private int cursor; + + private NodeTest nodeTest; + + public AttributeAxisIterator(NodeWrapper start, NodeTest test) { + // use lazy instead of eager materialization (performance) + this.start = start; + if (test == AnyNodeTest.getInstance()) test = null; + this.nodeTest = test; + this.position = 0; + this.cursor = 0; + } + + public Item next() { + NodeInfo curr; + do { // until we find a match + curr = advance(); + } + while (curr != null && nodeTest != null && (! nodeTest.matches(curr))); + + if (curr != null) position++; + current = curr; + return curr; + } + + private NodeInfo advance() { + Element elem = (Element) start.node; + if (cursor == elem.getAttributeCount()) return null; + NodeInfo curr = makeWrapper(elem.getAttribute(cursor), docWrapper, start, cursor); + cursor++; + return curr; + } + + public Item current() { + return current; + } + + public SequenceIterator getAnother() { + return new AttributeAxisIterator(start, nodeTest); + } + + public int getProperties() { + return 0; + } + + } // end of class AttributeAxisIterator + + /** + * The class ChildAxisIterator handles not only the child axis, but also the + * following-sibling and preceding-sibling axes. It can also iterate the + * children of the start node in reverse order, something that is needed to + * support the preceding and preceding-or-ancestor axes (the latter being + * used by xsl:number) + */ + private final class ChildAxisIterator extends AxisIteratorImpl { + + private NodeWrapper start; + private NodeWrapper commonParent; + private int ix; + private boolean downwards; // iterate children of start node (not siblings) + private boolean forwards; // iterate in document order (not reverse order) + + private ParentNode par; + private int cursor; + + private NodeTest nodeTest; + + private ChildAxisIterator(NodeWrapper start, boolean downwards, boolean forwards, NodeTest test) { + this.start = start; + this.downwards = downwards; + this.forwards = forwards; + + if (test == AnyNodeTest.getInstance()) test = null; + this.nodeTest = test; + this.position = 0; + + if (downwards) + commonParent = start; + else + commonParent = (NodeWrapper) start.getParent(); + + par = (ParentNode) commonParent.node; + if (downwards) { + ix = (forwards ? 0 : par.getChildCount()); + } else { + // find the start node among the list of siblings +// ix = start.getSiblingPosition(); + ix = par.indexOf(start.node); + if (forwards) ix++; + } + cursor = ix; + if (!downwards && !forwards) ix--; + } + + public Item next() { + NodeInfo curr; + do { // until we find a match + curr = advance(); + } + while (curr != null && nodeTest != null && (! nodeTest.matches(curr))); + + if (curr != null) position++; + current = curr; + return curr; + } + + private NodeInfo advance() { + Node nextChild; + do { + if (forwards) { + if (cursor == par.getChildCount()) return null; + nextChild = par.getChild(cursor++); + } else { // backwards + if (cursor == 0) return null; + nextChild = par.getChild(--cursor); + } + } while (nextChild instanceof DocType); + // DocType is not an XPath node; can occur for /child::node() + + NodeInfo curr = makeWrapper(nextChild, docWrapper, commonParent, ix); + ix += (forwards ? 1 : -1); + return curr; + } + + public Item current() { + return current; + } + + public SequenceIterator getAnother() { + return new ChildAxisIterator(start, downwards, forwards, nodeTest); + } + + public int getProperties() { + return 0; + } + } + + /** + * A bit of a misnomer; efficiently takes care of descendants, + * descentants-or-self as well as "following" axis. + * "includeSelf" must be false for the following axis. + * Uses simple and effective O(1) backtracking via indexOf(). + */ + private final class DescendantAxisIterator extends AxisIteratorImpl { + + private NodeWrapper start; + private boolean includeSelf; + private boolean following; + + private Node anchor; // so we know where to stop the scan + private Node currNode; + private boolean moveToNextSibling; + + private NodeTest nodeTest; + + private String testLocalName; + private String testURI; + + public DescendantAxisIterator(NodeWrapper start, boolean includeSelf, boolean following, NodeTest test) { + this.start = start; + this.includeSelf = includeSelf; + this.following = following; + this.moveToNextSibling = following; + + if (!following) anchor = start.node; + if (!includeSelf) currNode = start.node; + + if (test == AnyNodeTest.getInstance()) { // performance hack + test = null; // mark as AnyNodeTest + } + else if (test instanceof NameTest) { + NameTest nt = (NameTest) test; + if (nt.getPrimitiveType() == Type.ELEMENT) { // performance hack + // mark as element name test + NamePool pool = getNamePool(); + this.testLocalName = pool.getLocalName(nt.getFingerprint()); + this.testURI = pool.getURI(nt.getFingerprint()); + } + } + else if (test instanceof NodeKindTest) { + if (test.getPrimitiveType() == Type.ELEMENT) { // performance hack + // mark as element type test + this.testLocalName = ""; + this.testURI = null; + } + } + this.nodeTest = test; + this.position = 0; + } + + public Item next() { + NodeInfo curr; + do { // until we find a match + curr = advance(); + } + while (curr != null && nodeTest != null && (! nodeTest.matches(curr))); + + if (curr != null) position++; + current = curr; + return curr; + } + + // might look expensive at first glance - but it's not + private NodeInfo advance() { + if (currNode == null) { // if includeSelf + currNode = start.node; + return start; + } + + int i; + do { + i = 0; + Node p = currNode; + + if (p.getChildCount() == 0 || moveToNextSibling) { // move to next sibling + + moveToNextSibling = false; // do it just once + while (true) { + // if we've reached the root we're done scanning + p = currNode.getParent(); + if (p == null) return null; + + // Note: correct even if currNode is an attribute. + // Performance is particularly good with the O(1) patch + // for XOM's ParentNode.indexOf() + i = currNode.getParent().indexOf(currNode) + 1; + + if (i < p.getChildCount()) { + break; // break out of while(true) loop; move to next sibling + } + else { // reached last sibling; move up + currNode = p; + // if we've come all the way back to the start anchor we're done + if (p == anchor) return null; + } + } + } + currNode = p.getChild(i); + } while (!conforms(currNode)); + + // note the null here: makeNodeWrapper(parent, ...) is fast, so it + // doesn't really matter that we don't keep a link to it. + // In fact, it makes objects more short lived, easing pressure on + // the VM allocator and collector for tenured heaps. + return makeWrapper(currNode, docWrapper, null, i); + } + + // avoids NodeWrapper allocation when there's clearly a mismatch (common case) + private boolean conforms(Node node) { + if (this.testLocalName != null) { // element test? + if (!(node instanceof Element)) return false; + if (this.testURI == null) return true; // pure element type test + + // element name test + Element elem = (Element) node; + return this.testLocalName.equals(elem.getLocalName()) && + this.testURI.equals(elem.getNamespaceURI()); + } + else { // DocType is not an XPath node; can occur for /descendants::node() + return !(node instanceof DocType); + } + } + + public Item current() { + return current; + } + + public SequenceIterator getAnother() { + return new DescendantAxisIterator(start, includeSelf, following, nodeTest); + } + + public int getProperties() { + return 0; + } + } + + /** + * Efficiently takes care of preceding axis and Saxon internal preceding-or-ancestor axis. + * Uses simple and effective O(1) backtracking via indexOf(). + * Implemented along similar lines as DescendantAxisIterator. + */ + private final class PrecedingAxisIterator extends AxisIteratorImpl { + + private NodeWrapper start; + private boolean includeAncestors; + + private Node currNode; + private ParentNode nextAncestor; // next ancestors to skip if !includeAncestors + + private NodeTest nodeTest; + + private String testLocalName; + private String testURI; + + public PrecedingAxisIterator(NodeWrapper start, boolean includeAncestors, NodeTest test) { + this.start = start; + this.includeAncestors = includeAncestors; + this.currNode = start.node; + if (includeAncestors) + nextAncestor = null; + else + nextAncestor = start.node.getParent(); + + if (test == AnyNodeTest.getInstance()) { // performance hack + test = null; // mark as AnyNodeTest + } + else if (test instanceof NameTest) { + NameTest nt = (NameTest) test; + if (nt.getPrimitiveType() == Type.ELEMENT) { // performance hack + // mark as element name test + NamePool pool = getNamePool(); + this.testLocalName = pool.getLocalName(nt.getFingerprint()); + this.testURI = pool.getURI(nt.getFingerprint()); + } + } + else if (test instanceof NodeKindTest) { + if (test.getPrimitiveType() == Type.ELEMENT) { // performance hack + // mark as element type test + this.testLocalName = ""; + this.testURI = null; + } + } + this.nodeTest = test; + this.position = 0; + } + + public Item next() { + NodeInfo curr; + do { // until we find a match + curr = advance(); + } + while (curr != null && nodeTest != null && (! nodeTest.matches(curr))); + + if (curr != null) position++; + current = curr; + return curr; + } + + // might look expensive at first glance - but it's not + private NodeInfo advance() { + int i; + do { + Node p; + + while (true) { + // if we've reached the root we're done scanning +// System.out.println("p="+p); + p = currNode.getParent(); + if (p == null) return null; + + // Note: correct even if currNode is an attribute. + // Performance is particularly good with the O(1) patch + // for XOM's ParentNode.indexOf() + i = currNode.getParent().indexOf(currNode) - 1; + + if (i >= 0) { // move to next sibling's last descendant node + p = p.getChild(i); // move to next sibling + int j; + while ((j = p.getChildCount()-1) >= 0) { // move to last descendant node + p = p.getChild(j); + i = j; + } + break; // break out of while(true) loop + } + else { // there are no more siblings; move up + // if !includeAncestors skip the ancestors of the start node + // assert p != null + if (p != nextAncestor) break; // break out of while(true) loop + + nextAncestor = nextAncestor.getParent(); + currNode = p; + } + } + currNode = p; + + } while (!conforms(currNode)); + + // note the null here: makeNodeWrapper(parent, ...) is fast, so it + // doesn't really matter that we don't keep a link to it. + // In fact, it makes objects more short lived, easing pressure on + // the VM allocator and collector for tenured heaps. + return makeWrapper(currNode, docWrapper, null, i); + } + + // avoids NodeWrapper allocation when there's clearly a mismatch (common case) + // same as for DescendantAxisIterator + private boolean conforms(Node node) { + if (this.testLocalName != null) { // element test? + if (!(node instanceof Element)) return false; + if (this.testURI == null) return true; // pure element type test + + // element name test + Element elem = (Element) node; + return this.testLocalName.equals(elem.getLocalName()) && + this.testURI.equals(elem.getNamespaceURI()); + } + else { // DocType is not an XPath node + return !(node instanceof DocType); + } + } + + public Item current() { + return current; + } + + public SequenceIterator getAnother() { + return new PrecedingAxisIterator(start, includeAncestors, nodeTest); + } + + public int getProperties() { + return 0; + } + } + + private static AxisIterator filteredSingleton(NodeInfo node, NodeTest nodeTest) { +// return Navigator.filteredSingleton(node, nodeTest); // saxon >= 8.7 + if (node != null && (nodeTest == AnyNodeTest.getInstance() || nodeTest.matches(node))) { + return SingleNodeIterator.makeIterator(node); + } else { + return EmptyIterator.getInstance(); + } + } + + @Override + public int getColumnNumber() { + return -1; + } + + @Override + public boolean isId() { + return false; + } + + @Override + public boolean isIdref() { + return false; + } + + @Override + public boolean isNilled() { + return false; + } + +} + +// +// The contents of this file are subject to the Mozilla Public License Version +// 1.0 (the "License"); +// you may not use this file except in compliance with the License. You may +// obtain a copy of the +// License at http://www.mozilla.org/MPL/ +// +// Software distributed under the License is distributed on an "AS IS" basis, +// WITHOUT WARRANTY OF ANY KIND, either express or implied. +// See the License for the specific language governing rights and limitations +// under the License. +// +// The Original Code is: all this file. +// +// The Initial Developer of the Original Code is Michael Kay, with extensive +// rewriting by Wolfgang Hoschek +// +// Portions created by (your name) are Copyright (C) (your legal entity). All +// Rights Reserved. +// +// Contributor(s): none. +// Property changes on: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/NodeWrapper.java ___________________________________________________________________ Added: svn:mime-type + text/plain Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/PathMapFilter.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/PathMapFilter.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/PathMapFilter.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -184,5 +184,13 @@ super.endElement(); } } + + @Override + public void startContent() throws XPathException { + MatchContext context = matchContext.getLast(); + if (context.matchedElement) { + super.startContent(); + } + } } \ No newline at end of file Modified: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/SaxonXQueryExpression.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/SaxonXQueryExpression.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/SaxonXQueryExpression.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -23,9 +23,11 @@ package org.teiid.query.xquery.saxon; import java.io.IOException; +import java.io.InputStream; import java.io.Writer; import java.sql.SQLXML; import java.util.Arrays; +import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import java.util.LinkedList; @@ -61,6 +63,7 @@ import net.sf.saxon.query.DynamicQueryContext; import net.sf.saxon.query.QueryResult; import net.sf.saxon.query.StaticQueryContext; +import net.sf.saxon.query.XQueryExpression; import net.sf.saxon.sxpath.IndependentContext; import net.sf.saxon.sxpath.XPathEvaluator; import net.sf.saxon.sxpath.XPathExpression; @@ -69,6 +72,12 @@ import net.sf.saxon.type.ItemType; import net.sf.saxon.type.TypeHierarchy; import net.sf.saxon.value.SequenceType; +import nu.xom.Builder; +import nu.xom.Element; +import nu.xom.Nodes; +import nu.xom.ParsingException; +import nux.xom.xquery.StreamingPathFilter; +import nux.xom.xquery.StreamingTransform; import org.teiid.api.exception.query.QueryResolverException; import org.teiid.common.buffer.BufferManager; @@ -80,6 +89,9 @@ import org.teiid.core.types.XMLTranslator; import org.teiid.core.types.XMLType; import org.teiid.core.types.XMLType.Type; +import org.teiid.logging.LogConstants; +import org.teiid.logging.LogManager; +import org.teiid.logging.MessageLevel; import org.teiid.query.QueryPlugin; import org.teiid.query.analysis.AnalysisRecord; import org.teiid.query.function.source.XMLSystemFunctions; @@ -88,11 +100,34 @@ import org.teiid.query.sql.symbol.DerivedColumn; import org.teiid.query.sql.symbol.XMLNamespaces; import org.teiid.query.sql.symbol.XMLNamespaces.NamespaceItem; +import org.teiid.query.util.CommandContext; import org.teiid.translator.WSConnection.Util; @SuppressWarnings("serial") public class SaxonXQueryExpression { + public static final Properties DEFAULT_OUTPUT_PROPERTIES = new Properties(); + { + DEFAULT_OUTPUT_PROPERTIES.setProperty(OutputKeys.METHOD, "xml"); //$NON-NLS-1$ + //props.setProperty(OutputKeys.INDENT, "yes"); //$NON-NLS-1$ + DEFAULT_OUTPUT_PROPERTIES.setProperty(OutputKeys.OMIT_XML_DECLARATION, "yes"); //$NON-NLS-1$ + } + + private static Nodes NONE = new Nodes(); + private static InputStream FAKE_IS = new InputStream() { + + @Override + public int read() throws IOException { + return 0; + } + }; + + public interface RowProcessor { + + void processRow(NodeInfo row); + + } + public static class Result { public SequenceIterator iter; public List<Source> sources = new LinkedList<Source>(); @@ -147,16 +182,20 @@ } }; - private net.sf.saxon.query.XQueryExpression xQuery; + private XQueryExpression xQuery; + private String xQueryString; + private Map<String, String> namespaceMap = new HashMap<String, String>(); private Configuration config = new Configuration(); private PathMapRoot contextRoot; + private StreamingPathFilter streamingPathFilter; public SaxonXQueryExpression(String xQueryString, XMLNamespaces namespaces, List<DerivedColumn> passing, List<XMLTable.XMLColumn> columns) throws QueryResolverException { config.setErrorListener(ERROR_LISTENER); + this.xQueryString = xQueryString; StaticQueryContext context = new StaticQueryContext(config); IndependentContext ic = new IndependentContext(config); - + namespaceMap.put("", ""); //$NON-NLS-1$ //$NON-NLS-2$ if (namespaces != null) { for (NamespaceItem item : namespaces.getNamespaceItems()) { if (item.getPrefix() == null) { @@ -166,10 +205,12 @@ } else { context.setDefaultElementNamespace(item.getUri()); ic.setDefaultElementNamespace(item.getUri()); + namespaceMap.put("", item.getUri()); //$NON-NLS-1$ } } else { context.declareNamespace(item.getPrefix(), item.getUri()); ic.declareNamespace(item.getPrefix(), item.getUri()); + namespaceMap.put(item.getPrefix(), item.getUri()); } } } @@ -211,6 +252,13 @@ } public void useDocumentProjection(List<XMLTable.XMLColumn> columns, AnalysisRecord record) { + try { + streamingPathFilter = StreamingUtils.getStreamingPathFilter(xQueryString, namespaceMap); + } catch (IllegalArgumentException e) { + if (record.recordDebug()) { + record.println("Document streaming will not be used: " + e.getMessage()); //$NON-NLS-1$ + } + } this.contextRoot = null; PathMap map = this.xQuery.getPathMap(); PathMapRoot parentRoot; @@ -246,8 +294,8 @@ return; } } else { - for (Iterator iter = finalNodes.iterator(); iter.hasNext(); ) { - PathMapNode subNode = (PathMapNode)iter.next(); + for (Iterator<PathMapNode> iter = finalNodes.iterator(); iter.hasNext(); ) { + PathMapNode subNode = iter.next(); subNode.createArc(new AxisExpression(Axis.DESCENDANT_OR_SELF, AnyNodeTest.getInstance())); } } @@ -265,6 +313,24 @@ } this.contextRoot = parentRoot; } + + public static final boolean[] isValidAncestorAxis = + { + false, // ANCESTOR + false, // ANCESTOR_OR_SELF; + true, // ATTRIBUTE; + false, // CHILD; + false, // DESCENDANT; + false, // DESCENDANT_OR_SELF; + false, // FOLLOWING; + false, // FOLLOWING_SIBLING; + true, // NAMESPACE; + true, // PARENT; + false, // PRECEDING; + false, // PRECEDING_SIBLING; + true, // SELF; + false, // PRECEDING_OR_ANCESTOR; + }; private PathMapRoot projectColumns(PathMapRoot parentRoot, List<XMLTable.XMLColumn> columns, PathMapNode finalNode, AnalysisRecord record) { for (XMLColumn xmlColumn : columns) { @@ -293,6 +359,9 @@ continue; } for (PathMapArc arc : subContextRoot.getArcs()) { + if (streamingPathFilter != null && !validateColumnForStreaming(record, xmlColumn, arc)) { + streamingPathFilter = null; + } finalNode.createArc(arc.getStep(), arc.getTarget()); } HashSet<PathMapNode> subFinalNodes = new HashSet<PathMapNode>(); @@ -319,6 +388,54 @@ return newMap.reduceToDownwardsAxes(newRoot); } + private boolean validateColumnForStreaming(AnalysisRecord record, + XMLColumn xmlColumn, PathMapArc arc) { + boolean ancestor = false; + LinkedList<PathMapArc> arcStack = new LinkedList<PathMapArc>(); + arcStack.add(arc); + while (!arcStack.isEmpty()) { + PathMapArc current = arcStack.removeFirst(); + byte axis = current.getStep().getAxis(); + if (ancestor) { + if (current.getTarget().isReturnable()) { + if (axis != Axis.NAMESPACE && axis != Axis.ATTRIBUTE) { + if (record.recordDebug()) { + record.println("Document streaming will not be used, since the column path contains an invalid reverse axis " + xmlColumn.getPath()); //$NON-NLS-1$ + } + return false; + } + } + if (!isValidAncestorAxis[axis]) { + if (record.recordDebug()) { + record.println("Document streaming will not be used, since the column path contains an invalid reverse axis " + xmlColumn.getPath()); //$NON-NLS-1$ + } + return false; + } + } else if (!Axis.isSubtreeAxis[axis]) { + if (axis == Axis.PARENT + || axis == Axis.ANCESTOR + || axis == Axis.ANCESTOR_OR_SELF) { + if (current.getTarget().isReturnable()) { + if (record.recordDebug()) { + record.println("Document streaming will not be used, since the column path contains an invalid reverse axis " + xmlColumn.getPath()); //$NON-NLS-1$ + } + return false; + } + ancestor = true; + } else { + if (record.recordDebug()) { + record.println("Document streaming will not be used, since the column path may not reference an ancestor or subtree " + xmlColumn.getPath()); //$NON-NLS-1$ + } + return false; + } + } + for (PathMapArc pathMapArc : current.getTarget().getArcs()) { + arcStack.add(pathMapArc); + } + } + return true; + } + private void addReturnedArcs(XMLColumn xmlColumn, PathMapNode subNode) { if (xmlColumn.getSymbol().getType() == DataTypeManager.DefaultDataClasses.XML) { subNode.createArc(new AxisExpression(Axis.DESCENDANT_OR_SELF, AnyNodeTest.getInstance())); @@ -371,7 +488,7 @@ } } - public Result evaluateXQuery(Object context, Map<String, Object> parameterValues) throws TeiidProcessingException { + public Result evaluateXQuery(Object context, Map<String, Object> parameterValues, final RowProcessor processor, CommandContext commandContext) throws TeiidProcessingException { DynamicQueryContext dynamicContext = new DynamicQueryContext(config); Result result = new Result(); @@ -399,6 +516,39 @@ AugmentedSource sourceInput = AugmentedSource.makeAugmentedSource(source); sourceInput.addFilter(filter); source = sourceInput; + + //use streamable processing instead + if (streamingPathFilter != null && processor != null) { + if (LogManager.isMessageToBeRecorded(LogConstants.CTX_DQP, MessageLevel.DETAIL)) { + LogManager.logDetail(LogConstants.CTX_DQP, "Using stream processing for evaluation of", this.xQueryString); //$NON-NLS-1$ + } + //set to non-blocking in case default expression evaluation blocks + boolean isNonBlocking = commandContext.isNonBlocking(); + commandContext.setNonBlocking(true); + + final StreamingTransform myTransform = new StreamingTransform() { + public Nodes transform(Element elem) { + processor.processRow(StreamingUtils.wrap(elem, config)); + return NONE; + } + }; + + Builder builder = new Builder(new SaxonReader(config, sourceInput), false, + streamingPathFilter.createNodeFactory(null, myTransform)); + try { + //the builder is hard wired to parse the source, but the api will throw an exception if the stream is null + builder.build(FAKE_IS); + return result; + } catch (ParsingException e) { + throw new TeiidProcessingException(e, QueryPlugin.Util.getString("SaxonXQueryExpression.bad_context")); //$NON-NLS-1$ + } catch (IOException e) { + throw new TeiidProcessingException(e, QueryPlugin.Util.getString("SaxonXQueryExpression.bad_context")); //$NON-NLS-1$ + } finally { + if (!isNonBlocking) { + commandContext.setNonBlocking(false); + } + } + } } DocumentInfo doc; try { @@ -429,17 +579,7 @@ XMLType.Type type = Type.CONTENT; if (item instanceof NodeInfo) { NodeInfo info = (NodeInfo)item; - switch (info.getNodeKind()) { - case net.sf.saxon.type.Type.DOCUMENT: - type = Type.DOCUMENT; - break; - case net.sf.saxon.type.Type.ELEMENT: - type = Type.ELEMENT; - break; - case net.sf.saxon.type.Type.TEXT: - type = Type.TEXT; - break; - } + type = getType(info); } Item next = iter.next(); if (next != null) { @@ -450,17 +590,29 @@ @Override public void translate(Writer writer) throws TransformerException, IOException { - Properties props = new Properties(); - props.setProperty(OutputKeys.METHOD, "xml"); //$NON-NLS-1$ - //props.setProperty(OutputKeys.INDENT, "yes"); //$NON-NLS-1$ - props.setProperty(OutputKeys.OMIT_XML_DECLARATION, "yes"); //$NON-NLS-1$ - QueryResult.serializeSequence(iter.getAnother(), config, writer, props); + QueryResult.serializeSequence(iter.getAnother(), config, writer, DEFAULT_OUTPUT_PROPERTIES); } }); XMLType value = new XMLType(xml); value.setType(type); return value; } + + public static XMLType.Type getType(NodeInfo info) { + switch (info.getNodeKind()) { + case net.sf.saxon.type.Type.DOCUMENT: + return Type.DOCUMENT; + case net.sf.saxon.type.Type.ELEMENT: + return Type.ELEMENT; + case net.sf.saxon.type.Type.TEXT: + return Type.TEXT; + case net.sf.saxon.type.Type.COMMENT: + return Type.COMMENT; + case net.sf.saxon.type.Type.PROCESSING_INSTRUCTION: + return Type.PI; + } + return Type.CONTENT; + } public Configuration getConfig() { return config; @@ -477,5 +629,9 @@ showArcs(sb, node, level + 1); } } + + public boolean isStreaming() { + return streamingPathFilter != null; + } } Added: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/StreamingUtils.java =================================================================== --- branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/StreamingUtils.java (rev 0) +++ branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/StreamingUtils.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -0,0 +1,322 @@ +/* + * JBoss, Home of Professional Open Source. + * See the COPYRIGHT.txt file distributed with this work for information + * regarding copyright ownership. Some portions may be licensed + * to Red Hat, Inc. under one or more contributor license agreements. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA + * 02110-1301 USA. + */ + +package org.teiid.query.xquery.saxon; + +import java.io.IOException; +import java.util.Map; + +import net.sf.saxon.AugmentedSource; +import net.sf.saxon.Configuration; +import net.sf.saxon.event.ContentHandlerProxy; +import net.sf.saxon.event.PipelineConfiguration; +import net.sf.saxon.event.ProxyReceiver; +import net.sf.saxon.event.Receiver; +import net.sf.saxon.om.Name11Checker; +import net.sf.saxon.om.NodeInfo; +import net.sf.saxon.trans.XPathException; +import nu.xom.DocType; +import nu.xom.Node; +import nux.xom.xquery.StreamingPathFilter; + +import org.xml.sax.ContentHandler; +import org.xml.sax.DTDHandler; +import org.xml.sax.EntityResolver; +import org.xml.sax.ErrorHandler; +import org.xml.sax.InputSource; +import org.xml.sax.SAXException; +import org.xml.sax.SAXNotRecognizedException; +import org.xml.sax.SAXNotSupportedException; +import org.xml.sax.XMLReader; +import org.xml.sax.ext.LexicalHandler; + +final class StreamingUtils { + /** + * Converts a xom node into something readable by Saxon + * @param node + * @param config + * @return + */ + static NodeInfo wrap(Node node, Configuration config) { + if (node == null) + throw new IllegalArgumentException("node must not be null"); //$NON-NLS-1$ + if (node instanceof DocType) + throw new IllegalArgumentException("DocType can't be queried by XQuery/XPath"); //$NON-NLS-1$ + + Node root = node; + while (root.getParent() != null) { + root = root.getParent(); + } + + DocumentWrapper docWrapper = new DocumentWrapper(root, root.getBaseURI(), config); + + return docWrapper.wrap(node); + } + + /** + * Pre-parser that adds validation and handles a default name space + * + * TODO: add support for more general paths including node tests + * this could be done as a secondary expression applied to the + * context item + * + * @param locationPath + * @param prefixMap + * @return + */ + public static StreamingPathFilter getStreamingPathFilter(String locationPath, Map<String, String> prefixMap) { + if (locationPath.indexOf("//") >= 0) //$NON-NLS-1$ + throw new IllegalArgumentException("DESCENDANT axis is not supported"); //$NON-NLS-1$ + + String path = locationPath.trim(); + if (path.startsWith("/")) path = path.substring(1); //$NON-NLS-1$ + if (path.endsWith("/")) path = path.substring(0, path.length() - 1); //$NON-NLS-1$ + path = path.trim(); + String[] localNames = path.split("/"); //$NON-NLS-1$ + + if (localNames.length == 1) { + throw new IllegalArgumentException(locationPath + " refers to only the root element"); //$NON-NLS-1$ + } + + String fixedPath = ""; //$NON-NLS-1$ + + // parse prefix:localName pairs and resolve prefixes to namespaceURIs + for (int i = 0; i < localNames.length; i++) { + fixedPath += "/"; //$NON-NLS-1$ + int k = localNames[i].indexOf(':'); + if (k >= 0 && localNames[i].indexOf(':', k+1) >= 0) + throw new IllegalArgumentException( + "QName must not contain more than one colon: " //$NON-NLS-1$ + + "qname='" + localNames[i] + "', path='" + path + "'"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ + if (k <= 0) { + fixedPath += " :"; //$NON-NLS-1$ + } else { + String prefix = localNames[i].substring(0, k).trim(); + if (k >= localNames[i].length() - 1) + throw new IllegalArgumentException( + "Missing localName for prefix: " + "prefix='" //$NON-NLS-1$ //$NON-NLS-2$ + + prefix + "', path='" + path + "', prefixes=" + prefixMap); //$NON-NLS-1$ //$NON-NLS-2$ + fixedPath += prefix + ":"; //$NON-NLS-1$ + } // end if + + localNames[i] = localNames[i].substring(k + 1).trim(); + if (!localNames[i].equals("*") && !Name11Checker.getInstance().isValidNCName(localNames[i])) { //$NON-NLS-1$ + throw new IllegalArgumentException(localNames[i] + " is not a valid local name."); //$NON-NLS-1$ + } + fixedPath += localNames[i]; + } + return new StreamingPathFilter(fixedPath, prefixMap); + } + +} + +/** + * An {@link XMLReader} designed to bridge between the Saxon document projection logic and the XOM/NUX streaming logic. + */ +final class SaxonReader implements XMLReader { + + private ContentHandler handler; + private LexicalHandler lexicalHandler; + + private Configuration config; + private AugmentedSource source; + + public SaxonReader(Configuration config, AugmentedSource source) { + this.config = config; + this.source = source; + } + + @Override + public void setProperty(String name, Object value) + throws SAXNotRecognizedException, SAXNotSupportedException { + if ("http://xml.org/sax/properties/lexical-handler".equals(name)) { //$NON-NLS-1$ + this.lexicalHandler = (LexicalHandler) value; + } + } + + @Override + public void setFeature(String name, boolean value) + throws SAXNotRecognizedException, SAXNotSupportedException { + } + + @Override + public void setErrorHandler(ErrorHandler handler) { + throw new UnsupportedOperationException(); + + } + + @Override + public void setEntityResolver(EntityResolver resolver) { + throw new UnsupportedOperationException(); + + } + + @Override + public void setDTDHandler(DTDHandler handler) { + + } + + @Override + public void setContentHandler(ContentHandler handler) { + this.handler = handler; + } + + @Override + public void parse(String systemId) throws IOException, SAXException { + throw new UnsupportedOperationException(); + } + + @Override + public void parse(InputSource input) throws IOException, SAXException { + ContentHandlerProxy chp = new ContentHandlerProxy(); + chp.setLexicalHandler(lexicalHandler); + chp.setUnderlyingContentHandler(handler); + this.source.addFilter(new ContentHandlerProxyReceiver(chp)); + try { + config.buildDocument(source); + } catch (XPathException e) { + throw new SAXException(e); + } + } + + @Override + public Object getProperty(String name) throws SAXNotRecognizedException, + SAXNotSupportedException { + throw new UnsupportedOperationException(); + } + + @Override + public boolean getFeature(String name) throws SAXNotRecognizedException, + SAXNotSupportedException { + throw new UnsupportedOperationException(); + } + + @Override + public ErrorHandler getErrorHandler() { + return null; + } + + @Override + public EntityResolver getEntityResolver() { + throw new UnsupportedOperationException(); + } + + @Override + public DTDHandler getDTDHandler() { + throw new UnsupportedOperationException(); + } + + @Override + public ContentHandler getContentHandler() { + return this.handler; + } + +} + +/** + * Adapts the {@link ContentHandlerProxy} to be a {@link ProxyReceiver} + */ +final class ContentHandlerProxyReceiver extends ProxyReceiver { + + private Receiver reciever; + + public ContentHandlerProxyReceiver(Receiver reciever) { + this.reciever = reciever; + } + + public void attribute(int nameCode, int typeCode, CharSequence value, + int locationId, int properties) throws XPathException { + reciever.attribute(nameCode, typeCode, value, locationId, + properties); + } + + public void characters(CharSequence chars, int locationId, + int properties) throws XPathException { + reciever.characters(chars, locationId, properties); + } + + public void close() throws XPathException { + reciever.close(); + } + + public void comment(CharSequence content, int locationId, int properties) + throws XPathException { + reciever.comment(content, locationId, properties); + } + + public void endDocument() throws XPathException { + reciever.endDocument(); + } + + public void endElement() throws XPathException { + reciever.endElement(); + } + + public PipelineConfiguration getPipelineConfiguration() { + return reciever.getPipelineConfiguration(); + } + + public String getSystemId() { + return reciever.getSystemId(); + } + + public void namespace(int namespaceCode, int properties) + throws XPathException { + reciever.namespace(namespaceCode, properties); + } + + public void open() throws XPathException { + reciever.open(); + } + + public void processingInstruction(String name, CharSequence data, + int locationId, int properties) throws XPathException { + reciever.processingInstruction(name, data, locationId, properties); + } + + public void setPipelineConfiguration(PipelineConfiguration config) { + reciever.setPipelineConfiguration(config); + } + + public void setSystemId(String systemId) { + reciever.setSystemId(systemId); + } + + public void setUnparsedEntity(String name, String systemID, + String publicID) throws XPathException { + reciever.setUnparsedEntity(name, systemID, publicID); + } + + public void startContent() throws XPathException { + reciever.startContent(); + } + + public void startDocument(int properties) throws XPathException { + reciever.startDocument(properties); + } + + public void startElement(int nameCode, int typeCode, int locationId, + int properties) throws XPathException { + reciever.startElement(nameCode, typeCode, locationId, properties); + } + +} \ No newline at end of file Property changes on: branches/7.4.x/engine/src/main/java/org/teiid/query/xquery/saxon/StreamingUtils.java ___________________________________________________________________ Added: svn:mime-type + text/plain Modified: branches/7.4.x/engine/src/test/java/org/teiid/query/processor/TestSQLXMLProcessing.java =================================================================== --- branches/7.4.x/engine/src/test/java/org/teiid/query/processor/TestSQLXMLProcessing.java 2011-06-10 16:28:08 UTC (rev 3241) +++ branches/7.4.x/engine/src/test/java/org/teiid/query/processor/TestSQLXMLProcessing.java 2011-06-10 17:50:13 UTC (rev 3242) @@ -271,7 +271,27 @@ process(sql, expected); } + + @Test public void testXmlQueryEmptyNullString() throws Exception { + String sql = "select xmlquery('/a/b' passing xmlparse(document '<x/>') null on empty)"; //$NON-NLS-1$ + + List<?>[] expected = new List<?>[] { + Arrays.asList((String)null) + }; + process(sql, expected); + } + + @Test public void testXmlQueryStreaming() throws Exception { + String sql = "select xmlquery('/a/b' passing xmlparse(document '<a><b x=''1''/><b x=''2''/></a>') null on empty)"; //$NON-NLS-1$ + + List<?>[] expected = new List<?>[] { + Arrays.asList("<b x=\"1\"/><b x=\"2\"/>") + }; + + process(sql, expected); + } + @Test public void testXmlNameEscaping() throws Exception { String sql = "select xmlforest(\"xml\") from (select 1 as \"xml\") x"; //$NON-NLS-1$ @@ -371,6 +391,27 @@ process(sql, expected); } + @Test public void testXmlTableStreamingParentAttributes() throws Exception { + String sql = "select * from xmltable('/a/b' passing xmlparse(document '<a x=''1''><b>foo</b></a>') columns y string path '.', x integer path '../@x') as x"; //$NON-NLS-1$ + List<?>[] expected = new List<?>[] { + Arrays.asList("foo", 1), + }; + process(sql, expected); + } + + /** + * Highlights that the PathMapFilter needs to be selective in calling startContent + * @throws Exception + */ + @Test public void testXmlStreamingError() throws Exception { + String sql = "select * from xmltable('/a/a' passing xmlparse(document '<a><a>2000-01-01T01:01:00.2-06:00<a></a></a></a>') columns x timestamp path 'xs:dateTime(./text())') as x"; //$NON-NLS-1$ + Timestamp ts = TimestampUtil.createTimestamp(100, 0, 1, 1, 1, 0, 200000000); + List<?>[] expected = new List<?>[] { + Arrays.asList(ts), + }; + process(sql, expected); + } + @Test public void testXmlTableSubquery() throws Exception { String sql = "select * from xmltable('/a/b' passing convert('<a><b>first</b><b x=\"attr\">c</b></a>', xml) columns x string path '@x', val string path '/.') as x where val = (select max(e1) from pm1.g1 as x)";