10:51 a.m.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:55 a.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Mark Addleman commented on TEIID-1801:
--------------------------------------
When processing a streaming plan, the engine should only allow aggregate operations within
a time period. Global aggregate operations (including as ORDER BY) can be treated as an
error by the planner.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Mark Addleman edited comment on TEIID-1801 at 11/11/11 11:32 PM:
-----------------------------------------------------------------
Suppose I have a query that does a left join between a constant data source and a
streaming data source such as
SELECT constant.pkey, stream.b FROM constant LEFT JOIN stream ON
constant.pkey=stream.fkey
In time period zero, constant.pkey is "abc" but stream.b produces nothing so the
result set is naturally ("abc", null). In time period one, stream.a produces
"123" but, since the constant data source is not streaming, it produces nothing.
Arguably, there are two possible correct behaviors: Either produce ("abc",
"123") or produce an empty result set. Based on our discussions, I believe
Teiid's behavior would be to produce the empty result set in time period one since the
engine doesn't have a concept of constant.
Do I understand correctly? Is there a large cost in supporting a "constantly"
hint so I could express something like:
SELECT constant.pkey, stream.b FROM /* constantly */ constant LEFT JOIN stream on
constant.pkey=stream.fkey
The semantics would be to repeat the initial values from the constant data source every
time the join is evaluated.
was (Author: markaddleman):
Suppose I have a query that does a left join between a constant data source and a
streaming data source such as
SELECT constant.pkey, stream.b FROM constant LEFT JOIN stream ON
constant.pkey=stream.fkey
In time period zero, constant.pkey is "abc" but stream.b is null so the result
set is naturally ("abc", null). In time period one, stream.a produces
"123" but, since the constant data source is not streaming, it produces nothing.
Arguably, there are two possible correct behaviors: Either produce ("abc",
"123") or produce an empty result set. Based on our discussions, I believe
Teiid's behavior would be to produce the empty result set in time period one since the
engine doesn't have a concept of constant.
Do I understand correctly? Is there a large cost in supporting a "constantly"
hint so I could express something like:
SELECT constant.pkey, stream.b FROM /* constantly */ constant LEFT JOIN stream on
constant.pkey=stream.fkey
The semantics would be to repeat the initial values from the constant data source every
time the join is evaluated.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:33 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Mark Addleman commented on TEIID-1801:
--------------------------------------
Suppose I have a query that does a left join between a constant data source and a
streaming data source such as
SELECT constant.pkey, stream.b FROM constant LEFT JOIN stream ON
constant.pkey=stream.fkey
In time period zero, constant.pkey is "abc" but stream.b is null so the result
set is naturally ("abc", null). In time period one, stream.a produces
"123" but, since the constant data source is not streaming, it produces nothing.
Arguably, there are two possible correct behaviors: Either produce ("abc",
"123") or produce an empty result set. Based on our discussions, I believe
Teiid's behavior would be to produce the empty result set in time period one since the
engine doesn't have a concept of constant.
Do I understand correctly? Is there a large cost in supporting a "constantly"
hint so I could express something like:
SELECT constant.pkey, stream.b FROM /* constantly */ constant LEFT JOIN stream on
constant.pkey=stream.fkey
The semantics would be to repeat the initial values from the constant data source every
time the join is evaluated.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:35 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Mark Addleman edited comment on TEIID-1801 at 11/11/11 11:33 PM:
-----------------------------------------------------------------
Suppose I have a query that does a left join between a constant data source and a
streaming data source such as
SELECT constant.pkey, stream.b FROM constant LEFT JOIN stream ON
constant.pkey=stream.fkey
In time period zero, constant.pkey is "abc" but stream.fkey produces nothing so
the result set is naturally ("abc", null). In time period one, stream.fkey
produces "abc" and stream.b produces "123" but, since the constant
data source is not streaming, it produces nothing. Arguably, there are two possible
correct behaviors: Either produce ("abc", "123") or produce an empty
result set. Based on our discussions, I believe Teiid's behavior would be to produce
the empty result set in time period one since the engine doesn't have a concept of
constant.
Do I understand correctly? Is there a large cost in supporting a "constantly"
hint so I could express something like:
SELECT constant.pkey, stream.b FROM /* constantly */ constant LEFT JOIN stream on
constant.pkey=stream.fkey
The semantics would be to repeat the initial values from the constant data source every
time the join is evaluated.
was (Author: markaddleman):
Suppose I have a query that does a left join between a constant data source and a
streaming data source such as
SELECT constant.pkey, stream.b FROM constant LEFT JOIN stream ON
constant.pkey=stream.fkey
In time period zero, constant.pkey is "abc" but stream.b produces nothing so the
result set is naturally ("abc", null). In time period one, stream.a produces
"123" but, since the constant data source is not streaming, it produces nothing.
Arguably, there are two possible correct behaviors: Either produce ("abc",
"123") or produce an empty result set. Based on our discussions, I believe
Teiid's behavior would be to produce the empty result set in time period one since the
engine doesn't have a concept of constant.
Do I understand correctly? Is there a large cost in supporting a "constantly"
hint so I could express something like:
SELECT constant.pkey, stream.b FROM /* constantly */ constant LEFT JOIN stream on
constant.pkey=stream.fkey
The semantics would be to repeat the initial values from the constant data source every
time the join is evaluated.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:13 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Steven Hawkins commented on TEIID-1801:
---------------------------------------
With a reset/re-run approach, something like an order by would be particular to the window
and so does not need to be handled as an error.
Now that we have exposed an asynch method in JDBC, that would probably be a good spot to
initiate a re-runable plan,
e.g. void submitExecute(String sql, StatementCallback callback, boolean rerun) throws
SQLException
This could even be the place to indicate the time window. We would just have to add
another piece of information to the ExectionContext. Another possible approach to a
configurable time window would be to use the source hint
https://issues.jboss.org/browse/TEIID-1726 feature "SELECT /*+ sh:window(10) */ ...
".
I think the caching concerns can be handled more generally, such as translator result set
caching, with a session temp table, or possibly by having the with clause not be reset in
streaming mode - but that may need to be hint driven.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Reporter: Mark Addleman
Assignee: Steven Hawkins
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:10 a.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Steven Hawkins updated TEIID-1801:
----------------------------------
Fix Version/s: 8.0
Component/s: JDBC Driver
Query Engine
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Components: JDBC Driver, Query Engine
Reporter: Mark Addleman
Assignee: Steven Hawkins
Fix For: 8.0
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:05 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Mark Addleman commented on TEIID-1801:
--------------------------------------
Regarding the time window & re-runing: I don't particularly care if these options
are exposed to the client via a method or a query hint as long as the data is available to
the translator via the execution context. My gut feel is that a query hint is
conceptually better.
Regarding the result set caching: This issue seems dominated by internal Teiid
considerations. My only input is that the result sets are potentially large so keeping
the clause open may not a good solution. I'm perfectly comfortable with requiring a
hint to indicate that a cache should be used.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Components: JDBC Driver, Query Engine
Reporter: Mark Addleman
Assignee: Steven Hawkins
Fix For: 8.0
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:22 a.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Work on TEIID-1801 started by Steven Hawkins.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Components: JDBC Driver, Query Engine
Reporter: Mark Addleman
Assignee: Steven Hawkins
Fix For: 8.0
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:52 p.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Steven Hawkins resolved TEIID-1801.
-----------------------------------
Resolution: Done
Added a RequestOptions object to the asynch methods on
TeiidStatement/TeiidPreparedStatement. It can be used to specify that a query should be
executed continuously. Thus continuous queries are currently limited to only embedded
jdbc using the Teiid extensions. If other modes are needed, non-embedded, odbc, etc.,
then we'll have to consider adding hints or other SQL based mechanisms.
Continuous queries will not be counted against the max active plans. Also, no built-in
mechanism is provided for specifying time windows. The general source hint mechanism can
be used for that purpose. If needed additional options can be added to the RequestOptions
object.
Note - I'm not sure if continuous is the best term or not.
At the translator level a new ReusableExecution interface was added so that custom
translators my return Executions that can be reused for the lifetime of the user query.
Close is still called as normal, but there is also a new ReusableExecution.dispose method
that is called when the user command finishes. As part of TEIID-1604 I'll also be
adding general callback mechanisms for notification of the end of user query and end of
session.
There is still more work to do on how to cache non-streaming results. The current
workaround is to use a session temp table or a matview to prevent unnecessary source
re-executions. Other possibilities include giving special meaning to the WITH clause
and/or some form of translator result set caching TEIID-1598. In general processing will
not be as efficient as it could be since we are not saving intermediate results, for
example if one side of a join is non-streaming and requires a Teiid sort, we will need to
perform the sort with each processing run. On the plus side though, the logic minimally
impacts the client and processing logic.
The rows returned through a continuous query cannot currently exceed 2^31-1. If this
limitation ever needs addressed, it would be far easier to modifying our result logic to
support longs rather than forcing the client to have more knowledge of continuous queries.
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Components: JDBC Driver, Query Engine
Reporter: Mark Addleman
Assignee: Steven Hawkins
Fix For: 8.0
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:19 a.m.
[
https://issues.jboss.org/browse/TEIID-1801?page=com.atlassian.jira.plugin...
]
Steven Hawkins closed TEIID-1801.
---------------------------------
Async Event Processing: Reset and Re-run plan
----------------------------------------------
Key: TEIID-1801
URL: https://issues.jboss.org/browse/TEIID-1801
Project: Teiid
Issue Type: Feature Request
Components: JDBC Driver, Query Engine
Reporter: Mark Addleman
Assignee: Steven Hawkins
Fix For: 8.0
In order to processing a stream of events aggregated within a specified time window, the
engine should provide some way of specifying the time period (analogous to a window),
reset and re-rerun the plan for every time period. There should also be an engine to
translator callback indicating that the plan has been reset as opposed to closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4430
days inactive
4629
days old
10 comments
8 participants
participants (8)
-
Mark Addleman (Commented) (JIRA) -
Mark Addleman (Created) (JIRA)
-
Mark Addleman (Issue Comment Edited) (JIRA)
-
Steven Hawkins (Commented) (JIRA)
-
Steven Hawkins (JIRA)
-
Steven Hawkins (Resolved) (JIRA)
-
Steven Hawkins (Updated) (JIRA)
-
Steven Hawkins (Work started) (JIRA)