[JBoss JIRA] (TEIID-2461) Better determine when to block on the output buffer
by Steven Hawkins (JIRA)
Steven Hawkins created TEIID-2461:
-------------------------------------
Summary: Better determine when to block on the output buffer
Key: TEIID-2461
URL: https://issues.jboss.org/browse/TEIID-2461
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 8.4
The output buffer size limitation helps prevent forward only results from unnecessarily consuming resources, but it also may slow down the processing of otherwise already buffered results - which will hold those and other downstream resources unnecessarily. We should examine the plan to see if this is possible before allowing the output buffer to be size limited.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2326) Allow importing vdbs to share the materialized view of imported vdbs
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2326?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-2326.
-----------------------------------
Resolution: Done
updated the logic to ensure that an imported vdb mat view is loaded as if it were in the imported vdb. also added a combined classloader for vdb with imports.
It seems like a reasonable default to always share imported materialized views. There may at some point still need to be a mechanism to tell teiid to not import an internal mat view, but ideally it would be more granular than an entire vdb (perhaps a VDBImport.excludeMatViews that is a list). Otherwise the workaround is to use an external mat view and use alternative behavior based upon the vdb name/version.
> Allow importing vdbs to share the materialized view of imported vdbs
> --------------------------------------------------------------------
>
> Key: TEIID-2326
> URL: https://issues.jboss.org/browse/TEIID-2326
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Fix For: 8.4
>
>
> Currently the end importing vdb will be the owner of a distinct set of internal materialized views regardless of whether those materialized views come from an imported vdb.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2381) Expanded source hint support
by Mark Addleman (JIRA)
[ https://issues.jboss.org/browse/TEIID-2381?page=com.atlassian.jira.plugin... ]
Mark Addleman commented on TEIID-2381:
--------------------------------------
For general hints, I think the same accumulate and deliver-to-translator approach would work.
> Expanded source hint support
> ----------------------------
>
> Key: TEIID-2381
> URL: https://issues.jboss.org/browse/TEIID-2381
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Attachments: HintsExecutionFactory.java, RandomExecutionFactory.java, RandomNumberStoredProcedureExecution.java, SysviewHintsExecutionTest.java
>
>
> We currently look at the source hint in only the root user query (not in subqueries nor the with clause) and only consider it in a very narrow set of circumstances when it's used in a view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2381) Expanded source hint support
by Mark Addleman (JIRA)
[ https://issues.jboss.org/browse/TEIID-2381?page=com.atlassian.jira.plugin... ]
Mark Addleman commented on TEIID-2381:
--------------------------------------
Thanks for the examples. Much clearer to me now.
> Do we accumulate source hints, or does the top level one take precedence?
We have cases where the upper-most hint takes precedence and another case where the lowest hint takes precendence. Any non-arbitrary solution is going to require some policy for picking the right one(s). Ultimately, that policy needs to be encoded in the translator or some new API. I don't see any advantage of it being a separate API. Thus, I like the accumulate approach.
My first cut at the accumulation algorithm would be something like: (1) from the pre-optimized structure, accumulate all hints grouped by source. For each source in the optimized structure, send the translator the appropriate group from #1. If the translator needs additional information and/or mapping, it's up to the programmer to encode that information into the hint.
> Expanded source hint support
> ----------------------------
>
> Key: TEIID-2381
> URL: https://issues.jboss.org/browse/TEIID-2381
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Attachments: HintsExecutionFactory.java, RandomExecutionFactory.java, RandomNumberStoredProcedureExecution.java, SysviewHintsExecutionTest.java
>
>
> We currently look at the source hint in only the root user query (not in subqueries nor the with clause) and only consider it in a very narrow set of circumstances when it's used in a view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2381) Expanded source hint support
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2381?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2381:
---------------------------------------
> Can you provide a specific example?
It's easier to illustrate with inline views:
SELECT /\*+ sh ... \*/ col1, col2 FROM (SELECT /\*+ sh ... \*/ col1 FROM t1) as v1, (SELECT /\*+ sh ... \*/ col2 FROM t2) as v2 WHERE col1 = col2
could be pushed as with the views removed as:
SELECT col1, col2 FROM t1, t2 WHERE col1 = col2
Do we accumulate source hints, or does the top level one take precedence?
Or a scenario like:
SELECT /\*+ sh ... \*/ col1, col2 FROM (SELECT /\*+ sh ... \*/ col1 FROM t1, t2 ...) as v1, (SELECT /\*+ sh ... \*/ col2 FROM t3, t4 ...) as v2 WHERE col1 = col2
with pushdowns that cross the original view layers (t1 joined with t3 and t2 joined with t4). Similar restructuring can happen with unions. The main trust here is that the pre-optimization structure unless other hints to preserve views/joins are used does not give an indication of the final structure most of the time. That is why the source hint was originally limited to simple query propagation and mainly targeted at the top-level query.
> Expanded source hint support
> ----------------------------
>
> Key: TEIID-2381
> URL: https://issues.jboss.org/browse/TEIID-2381
> Project: Teiid
> Issue Type: Enhancement
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Attachments: HintsExecutionFactory.java, RandomExecutionFactory.java, RandomNumberStoredProcedureExecution.java, SysviewHintsExecutionTest.java
>
>
> We currently look at the source hint in only the root user query (not in subqueries nor the with clause) and only consider it in a very narrow set of circumstances when it's used in a view.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2430) Add faster resource cleanup for failed sortutility runs
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2430?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-2430.
-----------------------------------
Resolution: Done
Where simple, exiting sortutilities are held statefully and cleanuped when the relevant node is closed (this catches failure cases that happen outside of the sortutility processing). Also added try catch blocks for the main sorting methods to ensure that failures are immediately cleaned up.
There are still 2 less than ideal scenarios where we are not statefully holding a sortutility, which have been updated to not block on memory Due to TEIID-2429 this means that there is a minor chance that the processing could block on memory. Prior to this check-in that means that the current working results would need to be cleaned up by reference and everything reprocessed again.
> Add faster resource cleanup for failed sortutility runs
> -------------------------------------------------------
>
> Key: TEIID-2430
> URL: https://issues.jboss.org/browse/TEIID-2430
> Project: Teiid
> Issue Type: Quality Risk
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Fix For: 8.4
>
>
> Failure during the run of sort utility (which can occur in the sort utility or outside due to blocked exceptions / streaming dup remove) will just rely on garbage collection clean-up of intermediate buffers. There should better handling in calling code to clean up proactively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2430) Add faster resource cleanup for failed sortutility runs
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2430?page=com.atlassian.jira.plugin... ]
Steven Hawkins edited comment on TEIID-2430 at 4/8/13 1:48 PM:
---------------------------------------------------------------
Where simple, exiting sortutilities are held statefully and cleaned up when the relevant node is closed (this catches failure cases that happen outside of the sortutility processing). Also added try catch blocks for the main sorting methods to ensure that failures are immediately cleaned up.
There are still 2 less than ideal scenarios where we are not statefully holding a sortutility, which have been updated to not block on memory Due to TEIID-2429 this means that there is a minor chance that the processing could block on memory. Prior to this check-in that means that the current working results would need to be cleaned up by reference and everything reprocessed again.
was (Author: shawkins):
Where simple, exiting sortutilities are held statefully and cleanuped when the relevant node is closed (this catches failure cases that happen outside of the sortutility processing). Also added try catch blocks for the main sorting methods to ensure that failures are immediately cleaned up.
There are still 2 less than ideal scenarios where we are not statefully holding a sortutility, which have been updated to not block on memory Due to TEIID-2429 this means that there is a minor chance that the processing could block on memory. Prior to this check-in that means that the current working results would need to be cleaned up by reference and everything reprocessed again.
> Add faster resource cleanup for failed sortutility runs
> -------------------------------------------------------
>
> Key: TEIID-2430
> URL: https://issues.jboss.org/browse/TEIID-2430
> Project: Teiid
> Issue Type: Quality Risk
> Components: Query Engine
> Reporter: Steven Hawkins
> Assignee: Steven Hawkins
> Fix For: 8.4
>
>
> Failure during the run of sort utility (which can occur in the sort utility or outside due to blocked exceptions / streaming dup remove) will just rely on garbage collection clean-up of intermediate buffers. There should better handling in calling code to clean up proactively.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2460) Weird behavior when Max buffer space restriction is hit
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2460?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-2460:
---------------------------------------
I should add that it is still possible due to event timing to have a small query fail after a large one, but between these changes, the sorting changes, and TEIID-2430, the chances of this happening have been greatly reduced.
> Weird behavior when Max buffer space restriction is hit
> --------------------------------------------------------
>
> Key: TEIID-2460
> URL: https://issues.jboss.org/browse/TEIID-2460
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.7.6
> Reporter: Filip Nguyen
> Assignee: Steven Hawkins
> Fix For: 8.4
>
> Attachments: teiid-jboss-beans.xml
>
>
> I was trying to restrict the disk space used by buffer manager (see steps to reproduce for my methodology). When the disk limit is hit, it really tries to stop the query, but doesn't succeed immediately.
> There is a big amount of exceptions [1] for relatively long period of time (minutes for big files), until it fails eventually with [2]. The error [2] is also given back to the JDBC client.
> Problem is that after the query fails in this fashion, the whole buffer disk space is still occupied and any new query, that needs even small (acceptable) buffer disk space, fails.
> Only way that I have found to make the buffer space usable again is to restart the server.
> [1] Error transferring block to storage 149742
> java.io.IOException: Max buffer space of 31,457,280 bytes has been exceed. The current operation will be aborted.
> [2] org.teiid.jdbc.TeiidSQLException: Batch not found in storage 50937
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (TEIID-2460) Weird behavior when Max buffer space restriction is hit
by Steven Hawkins (JIRA)
[ https://issues.jboss.org/browse/TEIID-2460?page=com.atlassian.jira.plugin... ]
Steven Hawkins resolved TEIID-2460.
-----------------------------------
Fix Version/s: 8.4
Resolution: Done
#1) Did not address - between the 8.4 sorting improvements and the initial changes here it did not seem necessary to tweak the default logging.
#2) added a callback into the buffer manager to clear out all memory related to the failing tuple batch and ensured if additional batches are added that the process will fail (not just if a batch is retrieved).
#4) this would still be useful, but was not added. This would require tracking the processor/workitem by cache group and marking the work as failing and calling more work so that the process resumes/fails quickly. This can be addressed later if a more graceful mechanism is needed (which would probably require readdressing the whole issue as we would want to make priority based decisions about what gets invalidated).
> Weird behavior when Max buffer space restriction is hit
> --------------------------------------------------------
>
> Key: TEIID-2460
> URL: https://issues.jboss.org/browse/TEIID-2460
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 7.7.6
> Reporter: Filip Nguyen
> Assignee: Steven Hawkins
> Fix For: 8.4
>
> Attachments: teiid-jboss-beans.xml
>
>
> I was trying to restrict the disk space used by buffer manager (see steps to reproduce for my methodology). When the disk limit is hit, it really tries to stop the query, but doesn't succeed immediately.
> There is a big amount of exceptions [1] for relatively long period of time (minutes for big files), until it fails eventually with [2]. The error [2] is also given back to the JDBC client.
> Problem is that after the query fails in this fashion, the whole buffer disk space is still occupied and any new query, that needs even small (acceptable) buffer disk space, fails.
> Only way that I have found to make the buffer space usable again is to restart the server.
> [1] Error transferring block to storage 149742
> java.io.IOException: Max buffer space of 31,457,280 bytes has been exceed. The current operation will be aborted.
> [2] org.teiid.jdbc.TeiidSQLException: Batch not found in storage 50937
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months