[JBoss JIRA] (TEIID-2159) Provide language objects to produce a JSON document
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2159?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2159:
---------------------------------------
This issue for the production side from Teiid. There would be another issue to tie this into the rest war generation (either automatic or from Designer). Teiid already has jsonToXml for processing incoming json and he google spreadsheet translator that consumes google json, and the odata in/out logic.
We'll probably revisit the jsonToXml logic at some point do something more idomatic to json.
> Provide language objects to produce a JSON document
> ---------------------------------------------------
>
> Key: TEIID-2159
> URL: https://issues.jboss.org/browse/TEIID-2159
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Fix For: 8.3
>
>
> SQLXML is standard that lets users define XML constructs to define a XML documents using SQL. However there is such standard exists for producing the JSON based documents.
> It would be nice if Teiid provides such SQL language extensions produce and deal with JSON documents. The constructs equivalent from SQLXML are
> JsonParse
> JsonObject
> JsonArray
> JsonForrest
> JsonAgg
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2384) Managing Spatial Data Types
by luca gioppo (JIRA)
luca gioppo created TEIID-2384:
----------------------------------
Summary: Managing Spatial Data Types
Key: TEIID-2384
URL: https://issues.jboss.org/browse/TEIID-2384
Project: Teiid
Issue Type: Feature Request
Components: Query Engine
Affects Versions: 8.2
Reporter: luca gioppo
Assignee: Steven Hawkins
It would be useful to be able to consume data from spatial database exposing the VDB as a spatial database to other application (imagine geoserver).
TEIID could be strategic for merging georeferenced data and make it available to those systems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2159) Provide language objects to produce a JSON document
by luca gioppo (JIRA)
[ https://issues.jboss.org/browse/TEIID-2159?page=com.atlassian.jira.plugin... ]
luca gioppo commented on TEIID-2159:
------------------------------------
Will this enable us to produce Rest API that return json objects or just consume json API?
Thanks
Luca
> Provide language objects to produce a JSON document
> ---------------------------------------------------
>
> Key: TEIID-2159
> URL: https://issues.jboss.org/browse/TEIID-2159
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Fix For: 8.3
>
>
> SQLXML is standard that lets users define XML constructs to define a XML documents using SQL. However there is such standard exists for producing the JSON based documents.
> It would be nice if Teiid provides such SQL language extensions produce and deal with JSON documents. The constructs equivalent from SQLXML are
> JsonParse
> JsonObject
> JsonArray
> JsonForrest
> JsonAgg
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2159) Provide language objects to produce a JSON document
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2159?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2159:
---------------------------------------
After starting the implementation I'm leaning toward just using a subtype of Clob (similar to the XMLType.Type subtypes) to represent JSON rather than introducing a full new type. This allows us to sidestep whether json needs to be a keyword and a host of logic that needs added related to types.
An example usage:
{code}select jsonArray(1, null, true, {d '2007-01-01'}, jsonParse('{"name":123}', true), unescape('\t\n?')){code}
returns
{code}[1,null,true,"2007-01-01",{"name":123},"\t\n?"]{code}
note the conversion of date to string
JsonParse/JsonArray will be added as normal functions. While JsonObject and JsonArrayAgg (which can be deprecated when we generally allow array type usage) will need parsing/language object support.
The initial work should be in for beta2.
> Provide language objects to produce a JSON document
> ---------------------------------------------------
>
> Key: TEIID-2159
> URL: https://issues.jboss.org/browse/TEIID-2159
> Project: Teiid
> Issue Type: Feature Request
> Components: Query Engine
> Reporter: Ramesh Reddy
> Assignee: Steven Hawkins
> Fix For: 8.3
>
>
> SQLXML is standard that lets users define XML constructs to define a XML documents using SQL. However there is such standard exists for producing the JSON based documents.
> It would be nice if Teiid provides such SQL language extensions produce and deal with JSON documents. The constructs equivalent from SQLXML are
> JsonParse
> JsonObject
> JsonArray
> JsonForrest
> JsonAgg
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2383) InputStreamReader throws MalformedInputException when handling multi-byte characters
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2383?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-2383.
-----------------------------------
Fix Version/s: 8.3
Resolution: Done
This occurs when the buffer is not completely used with a dangling multi-byte character, but a later usage appears to completely use the buffer and thus clears the partial multi-byte values. So this can affect multi-byte files that are 24KB and over.
The fix is different than the suggested so that the buffer is still cleared as much as possible for the next read.
> InputStreamReader throws MalformedInputException when handling multi-byte characters
> ------------------------------------------------------------------------------------
>
> Key: TEIID-2383
> URL: https://issues.jboss.org/browse/TEIID-2383
> Project: Teiid
> Issue Type: Bug
> Components: Common
> Affects Versions: 7.7.1
> Reporter: Hisanobu Okuda
> Assignee: Steven Hawkins
> Fix For: 8.3
>
> Attachments: sample.zip, simple_client.tar
>
>
> My VDB uses File translator to handle a csv file which contains multi-byte characters. Reading some csv file via the VDB, MalformedInputException always occurs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2383) InputStreamReader throws MalformedInputException when handling multi-byte characters
by Hisanobu Okuda (JIRA)
[ https://issues.jboss.org/browse/TEIID-2383?page=com.atlassian.jira.plugin... ]
Hisanobu Okuda commented on TEIID-2383:
---------------------------------------
This works fine on my setup:-
{code}
[hokuda@localhost xxx]$ git diff
diff --git a/teiid-7.7.1.GA/common-core/src/main/java/org/teiid/core/util/InputStreamReader.java b/teiid-7.7.1.GA/commo
index eca54fe..3c6ce70 100644
--- a/teiid-7.7.1.GA/common-core/src/main/java/org/teiid/core/util/InputStreamReader.java
+++ b/teiid-7.7.1.GA/common-core/src/main/java/org/teiid/core/util/InputStreamReader.java
@@ -90,7 +90,7 @@ public class InputStreamReader extends Reader {
}
done = true;
}
- if (bb.position() != read) {
+ if (bb.hasRemaining()) {
bb.compact();
} else {
bb.clear();
@@ -105,4 +105,4 @@ public class InputStreamReader extends Reader {
return len;
}
-}
\ No newline at end of file
+}
{code}
> InputStreamReader throws MalformedInputException when handling multi-byte characters
> ------------------------------------------------------------------------------------
>
> Key: TEIID-2383
> URL: https://issues.jboss.org/browse/TEIID-2383
> Project: Teiid
> Issue Type: Bug
> Components: Common
> Affects Versions: 7.7.1
> Reporter: Hisanobu Okuda
> Assignee: Steven Hawkins
> Attachments: sample.zip, simple_client.tar
>
>
> My VDB uses File translator to handle a csv file which contains multi-byte characters. Reading some csv file via the VDB, MalformedInputException always occurs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2383) InputStreamReader throws MalformedInputException when handling multi-byte characters
by Hisanobu Okuda (JIRA)
[ https://issues.jboss.org/browse/TEIID-2383?page=com.atlassian.jira.plugin... ]
Hisanobu Okuda commented on TEIID-2383:
---------------------------------------
File translator uses buffered reader to read file. In the sample, you can see the Japanese character '道'={0xe9 0x81 0x93} is buffered separately, i.e., {0xe9} and {0x81 0x93}.
{code}
[hokuda@localhost 00769999]$ rlwrap -l jdb.log -pRED -C jdb_teiid /opt/jdk1.6.0_37/bin/jdb -attach localhost:8787
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
> stop at org.teiid.core.util.InputStreamReader:93
Set breakpoint org.teiid.core.util.InputStreamReader:93
>
Breakpoint hit: "thread=Worker2_QueryProcessorQueue16", org.teiid.core.util.InputStreamReader.read(), line=93 bci=178
Worker2_QueryProcessorQueue16[1] cont
>
Breakpoint hit: "thread=Worker2_QueryProcessorQueue16", org.teiid.core.util.InputStreamReader.read(), line=93 bci=178
Worker2_QueryProcessorQueue16[1] cont
>
Breakpoint hit: "thread=Worker2_QueryProcessorQueue16", org.teiid.core.util.InputStreamReader.read(), line=93 bci=178
Worker2_QueryProcessorQueue16[1] cont
>
Breakpoint hit: "thread=Worker2_QueryProcessorQueue16", org.teiid.core.util.InputStreamReader.read(), line=93 bci=178
Worker2_QueryProcessorQueue16[1] cont
>
Breakpoint hit: "thread=Worker3_QueryProcessorQueue18", org.teiid.core.util.InputStreamReader.read(), line=93 bci=178
Worker3_QueryProcessorQueue18[1] print this.bb
this.bb = "java.nio.HeapByteBuffer[pos=8191 lim=8192 cap=8192]"
Worker3_QueryProcessorQueue18[1] print this.bb.hb[8191]
this.bb.hb[8191] = -23 (=0xe9)
{code}
Before executing line 93, position=8191 and limit=8192 in this.bb. This means the last byte remained to be decoded at next turn. The value of the last byte is -23 = 0xe9. It is the first byte of Japanese character '道'={0xe9 0x81 0x93}.
{code}
Worker3_QueryProcessorQueue18[1] locals
Method arguments:
cbuf = instance of char[8192] (id=10433)
off = 0
len = 8192
Local variables:
read = 8191
cr = instance of java.nio.charset.CoderResult(id=10434)
Since bb.position()=8191 and read=8191, bb.clear() is invoked. Actually, bb.compact() should be invoked though.
Worker3_QueryProcessorQueue18[1] next
>
Step completed: "thread=Worker3_QueryProcessorQueue18", org.teiid.core.util.InputStreamReader.read(), line=96 bci=201
Worker3_QueryProcessorQueue18[1] next
>
Step completed: "thread=Worker3_QueryProcessorQueue18", org.teiid.core.util.InputStreamReader.read(), line=98 bci=209
Worker3_QueryProcessorQueue18[1] print this.bb
this.bb = "java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192]"
{code}
Therefore read=8191 and bb.position()=8191, bb.clear() is invoked, the position of bb is reset to 0, and finally the last byte this.bb.hb[8191] = -23 (=0xe9) is discarded. In next blocking read, 0x81 and 0x93 are stored in bb.
{code}
Worker3_QueryProcessorQueue18[1] stop at org.teiid.core.util.InputStreamReader:82
Set breakpoint org.teiid.core.util.InputStreamReader:82
Worker3_QueryProcessorQueue18[1] cont
>
Breakpoint hit: "thread=Worker3_QueryProcessorQueue18", org.teiid.core.util.InputStreamReader.read(), line=82 bci=100
Worker3_QueryProcessorQueue18[1] print this.bb
this.bb = "java.nio.HeapByteBuffer[pos=0 lim=4 cap=8192]"
Worker3_QueryProcessorQueue18[1] print this.bb.hb[0]
this.bb.hb[0] = -127 (=0x81)
Worker3_QueryProcessorQueue18[1] print this.bb.hb[1]
this.bb.hb[1] = -109 (=0x93)
Worker3_QueryProcessorQueue18[1] print this.bb.hb[2]
this.bb.hb[2] = 39
Worker3_QueryProcessorQueue18[1] print this.bb.hb[3]
this.bb.hb[3] = 10
Worker3_QueryProcessorQueue18[1]
{code}
The first byte 0xe9 is missing, decoding fails.
> InputStreamReader throws MalformedInputException when handling multi-byte characters
> ------------------------------------------------------------------------------------
>
> Key: TEIID-2383
> URL: https://issues.jboss.org/browse/TEIID-2383
> Project: Teiid
> Issue Type: Bug
> Components: Common
> Affects Versions: 7.7.1
> Reporter: Hisanobu Okuda
> Assignee: Steven Hawkins
> Attachments: sample.zip, simple_client.tar
>
>
> My VDB uses File translator to handle a csv file which contains multi-byte characters. Reading some csv file via the VDB, MalformedInputException always occurs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (TEIID-2383) InputStreamReader throws MalformedInputException when handling multi-byte characters
by Hisanobu Okuda (JIRA)
[ https://issues.jboss.org/browse/TEIID-2383?page=com.atlassian.jira.plugin... ]
Hisanobu Okuda updated TEIID-2383:
----------------------------------
Steps to Reproduce:
1. unzip sample.zip
2. copy files in sample.zip
{code}
sample
|-- datafile -> $JBOSS_HOME/servers/$profile
| `-- postal_utf8.csv
|-- file-test-vdb.xml -> $JBOSS_HOME/servers/$profile/deploy
`-- test-ds.xml -> $JBOSS_HOME/servers/$profile/deploy
{code}
3. run simple client
ex)
{code}
./run.sh
{code}
4. After 256 raws fetched, MalformedInputException occurs as follow:-
{code}
11:42:16,219 DEBUG [PROCESSOR] Request Thread x8XReh9a7kIY.0 - error occurred
org.teiid.core.TeiidProcessingException: Input length = 1
at org.teiid.query.processor.relational.TextTableNode.readChar(TextTableNode.java:267)
at org.teiid.query.processor.relational.TextTableNode.readLine(TextTableNode.java:207)
at org.teiid.query.processor.relational.TextTableNode.nextBatchDirect(TextTableNode.java:166)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.hasNext(AbstractTupleSource.java:91)
at org.teiid.query.processor.relational.NestedTableJoinStrategy.process(NestedTableJoinStrategy.java:120)
at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:199)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:155)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:176)
at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:139)
at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)
at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:147)
at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:375)
at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:288)
at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)
at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:216)
at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:240)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:122)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at org.teiid.core.util.InputStreamReader.read(InputStreamReader.java:84)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at org.teiid.query.processor.relational.TextTableNode.readChar(TextTableNode.java:245)
... 26 more
{code}
was:
1. unzip sample.zip
2. copy files in sample.zip
sample
|-- datafile -> $JBOSS_HOME/servers/$profile
| `-- postal_utf8.csv
|-- file-test-vdb.xml -> $JBOSS_HOME/servers/$profile/deploy
`-- test-ds.xml -> $JBOSS_HOME/servers/$profile/deploy
3. run simple client
ex)
./run.sh localhost 31000 DynamicPortfolio "select test.* from (call TestData.getTextFiles('postal_utf8.csv')) f, TEXTTABLE(f.file COLUMNS id string, Town1 string, City1 string, State1 string HEADER) test"
4. After 256 raws fetched, MalformedInputException occurs as follow:-
11:42:16,219 DEBUG [PROCESSOR] Request Thread x8XReh9a7kIY.0 - error occurred
org.teiid.core.TeiidProcessingException: Input length = 1
at org.teiid.query.processor.relational.TextTableNode.readChar(TextTableNode.java:267)
at org.teiid.query.processor.relational.TextTableNode.readLine(TextTableNode.java:207)
at org.teiid.query.processor.relational.TextTableNode.nextBatchDirect(TextTableNode.java:166)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.hasNext(AbstractTupleSource.java:91)
at org.teiid.query.processor.relational.NestedTableJoinStrategy.process(NestedTableJoinStrategy.java:120)
at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:199)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:155)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:280)
at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:176)
at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:139)
at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)
at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:147)
at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:375)
at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:288)
at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)
at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:216)
at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:240)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:122)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at org.teiid.core.util.InputStreamReader.read(InputStreamReader.java:84)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
at org.teiid.query.processor.relational.TextTableNode.readChar(TextTableNode.java:245)
... 26 more
> InputStreamReader throws MalformedInputException when handling multi-byte characters
> ------------------------------------------------------------------------------------
>
> Key: TEIID-2383
> URL: https://issues.jboss.org/browse/TEIID-2383
> Project: Teiid
> Issue Type: Bug
> Components: Common
> Affects Versions: 7.7.1
> Reporter: Hisanobu Okuda
> Assignee: Steven Hawkins
>
> My VDB uses File translator to handle a csv file which contains multi-byte characters. Reading some csv file via the VDB, MalformedInputException always occurs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month