[JBoss JIRA] (TEIID-5525) add a flag to revert to the prior behavior (TEIID-4557)
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5525?page=com.atlassian.jira.plugin... ]
Steven Hawkins edited comment on TEIID-5525 at 11/1/18 12:06 PM:
-----------------------------------------------------------------
> However that should only be off by a factor of 2-3 (mostly due to string estimation).
Actually object overhead alone adds a factor of 2. Using the engine tests show that the heap memory estimate averages out to be 5 times higher than disk size. With the performance tests it's an average of 8 times higher. So that is likely the root of the issue - in scenarios that spill to disk, using an entirely heap based estimate of memory usage overstates memory usage more than I had anticipated.
was (Author: shawkins):
> However that should only be off by a factor of 2-3 (mostly due to string estimation).
Actually object overhead in general adds a factor of 2. Using the engine tests show that the heap memory estimate averages out to be 5 times higher than disk size. With the performance tests it's an average of 8 times higher. So that is likely the root of the issue - in scenarios that spill to disk, using an entirely heap based estimate of memory usage overstates memory usage more than I had anticipated.
> add a flag to revert to the prior behavior (TEIID-4557)
> -------------------------------------------------------
>
> Key: TEIID-5525
> URL: https://issues.jboss.org/browse/TEIID-5525
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.14.6_4, 8.12.15.6_4
> Reporter: Debbie Steigner
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.0, 11.2.1
>
>
> Several customers are seeing errors[1] after https://issues.jboss.org/browse/TEIID-4557 was implemented and generally have to drop the max-active-plans to 1 for their queries to succeed. Can a flag be added to revert to the behavior prior to TEIID-4557?
> [1]
> ERROR [org.teiid.PROCESSOR] TEIID30019 Unexpected exception for request rcX003UtoyEg.0: org.teiid.core.TeiidComponentException: TEIID31261 Max estimated size 4,963,628,721 for a single operation/table id 85 has been exceeded. The server may need to increase the amount of disk or memory available, or decrease the number of max active plans.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (TEIID-5525) add a flag to revert to the prior behavior (TEIID-4557)
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5525?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-5525:
---------------------------------------
> However that should only be off by a factor of 2-3 (mostly due to string estimation).
Actually object overhead in general adds a factor of 2. Using the engine tests show that the heap memory estimate averages out to be 5 times higher than disk size. With the performance tests it's an average of 8 times higher. So that is likely the root of the issue - in scenarios that spill to disk, using an entirely heap based estimate of memory usage overstates memory usage more than I had anticipated.
> add a flag to revert to the prior behavior (TEIID-4557)
> -------------------------------------------------------
>
> Key: TEIID-5525
> URL: https://issues.jboss.org/browse/TEIID-5525
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.14.6_4, 8.12.15.6_4
> Reporter: Debbie Steigner
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.0, 11.2.1
>
>
> Several customers are seeing errors[1] after https://issues.jboss.org/browse/TEIID-4557 was implemented and generally have to drop the max-active-plans to 1 for their queries to succeed. Can a flag be added to revert to the behavior prior to TEIID-4557?
> [1]
> ERROR [org.teiid.PROCESSOR] TEIID30019 Unexpected exception for request rcX003UtoyEg.0: org.teiid.core.TeiidComponentException: TEIID31261 Max estimated size 4,963,628,721 for a single operation/table id 85 has been exceeded. The server may need to increase the amount of disk or memory available, or decrease the number of max active plans.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (TEIID-5525) add a flag to revert to the prior behavior (TEIID-4557)
by Steven Hawkins (Jira)
[ https://issues.jboss.org/browse/TEIID-5525?page=com.atlassian.jira.plugin... ]
Steven Hawkins commented on TEIID-5525:
---------------------------------------
> Is disk also a limitation with the above issue? I did not realize there is per query disk limitation.
There are two limits - a session max, and a per buffer max. The session max is not enforced by default and must be set via a system property. The per buffer max is enforced by default using a value that is derived from all memory - heap, memory buffer, and storage - and the max active plans (see the other issues for the forumla). The idea being that a single table or result set (final or intermediate) that exceeds that threshold will likely cause memory issues for the system. There are a couple of potential issues:
1. we over estimate to a degree that is not representative of the actual size. This is somewhat the case already for anything that spills into the memory buffer or disk as the serialized size will be smaller than the heap memory estimate. However that should only be off by a factor of 2-3 (mostly due to string estimation).
2. There's a book keeping issue that is causing the estimate to be off.
Either way it would be best to know what the situation is and correct accordingly.
On the other hand there's the question of what their scenarios are actually doing. If the query for example really is creating a buffer that takes up half or more of all of the systems resources including disk that is something that could be an issue with any additional load or buffering.
> The right solution we either revert above or not seems to be when the query gets to a stage with the above limitation, reducing the buffer/batch sizes to half and try again, not sure if that is possible though with query processor.
I'm not sure what you are saying here. Reducing the batch size will not help the overall amount of committed memory.
> add a flag to revert to the prior behavior (TEIID-4557)
> -------------------------------------------------------
>
> Key: TEIID-5525
> URL: https://issues.jboss.org/browse/TEIID-5525
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.14.6_4, 8.12.15.6_4
> Reporter: Debbie Steigner
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.0, 11.2.1
>
>
> Several customers are seeing errors[1] after https://issues.jboss.org/browse/TEIID-4557 was implemented and generally have to drop the max-active-plans to 1 for their queries to succeed. Can a flag be added to revert to the behavior prior to TEIID-4557?
> [1]
> ERROR [org.teiid.PROCESSOR] TEIID30019 Unexpected exception for request rcX003UtoyEg.0: org.teiid.core.TeiidComponentException: TEIID31261 Max estimated size 4,963,628,721 for a single operation/table id 85 has been exceeded. The server may need to increase the amount of disk or memory available, or decrease the number of max active plans.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months
[JBoss JIRA] (TEIID-5525) add a flag to revert to the prior behavior (TEIID-4557)
by Ramesh Reddy (Jira)
[ https://issues.jboss.org/browse/TEIID-5525?page=com.atlassian.jira.plugin... ]
Ramesh Reddy commented on TEIID-5525:
-------------------------------------
Is disk also a limitation with the above issue? I did not realize there is per query disk limitation. The right solution we either revert above or not seems to be when the query gets to a stage with the above limitation, reducing the buffer/batch sizes to half and try again, not sure if that is possible though with query processor.
> add a flag to revert to the prior behavior (TEIID-4557)
> -------------------------------------------------------
>
> Key: TEIID-5525
> URL: https://issues.jboss.org/browse/TEIID-5525
> Project: Teiid
> Issue Type: Bug
> Components: Query Engine
> Affects Versions: 8.12.14.6_4, 8.12.15.6_4
> Reporter: Debbie Steigner
> Assignee: Steven Hawkins
> Priority: Major
> Fix For: 12.0, 11.2.1
>
>
> Several customers are seeing errors[1] after https://issues.jboss.org/browse/TEIID-4557 was implemented and generally have to drop the max-active-plans to 1 for their queries to succeed. Can a flag be added to revert to the behavior prior to TEIID-4557?
> [1]
> ERROR [org.teiid.PROCESSOR] TEIID30019 Unexpected exception for request rcX003UtoyEg.0: org.teiid.core.TeiidComponentException: TEIID31261 Max estimated size 4,963,628,721 for a single operation/table id 85 has been exceeded. The server may need to increase the amount of disk or memory available, or decrease the number of max active plans.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 6 months