1:40 p.m.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results for
greater performance - and so that they can be paged out by the buffermanager rather than
being held by a lobmanager.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:26 p.m.
New subject: [JBoss JIRA] Commented: (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins commented on TEIID-1750:
---------------------------------------
Also added logic to select better batch sizes based upon the element types. This allows
for power of two batch sizes between 128 and 2048 rows depending upon the types/width of
the row (that range can very based upon the base setting for processor batch size).
Planning logic is largely unaware of this distinction however. Also object and lob typed
values can produce very inaccurate estimates.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:45 a.m.
New subject: [JBoss JIRA] Commented: (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins commented on TEIID-1750:
---------------------------------------
Changed the buffermanager to only be concerned with the memory state. Changed the
buffering algorithm to be a tiered lru, which significantly reduces reads/writes when
accessing table structures. the old lru/mru/working set model was specifically designed
for tuplebuffers which have a linear insertion and forward iteration model. there will
still need to be some tuning made to better support tuplebuffers.
tuplebuffers/tables no longer directly hold managedbatches, which reduces the overall
memory footprint. This also helps effectively limit the softreference population.
put in a rough cut of a block storage scheme. this is likely to be further refined. the
code should naturally allow for off-heap direct memory buffers to be used for large vm
scenarios.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:19 a.m.
New subject: [JBoss JIRA] Commented: (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins commented on TEIID-1750:
---------------------------------------
Changed the buffering algorithm to LRFU and reusing the crf value for page updates and
reusing the value even after an item is persisted (although only the initial persistence
value is used - it would be better to keep updating this value for subsequent removals
from memory as crf values are increasing with time).
Also added a custom arraylist that grows more slower than the JRE arraylist, but stays on
powers of 2 to match batch/page sizes.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:55 a.m.
New subject: [JBoss JIRA] (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins commented on TEIID-1750:
---------------------------------------
Reducing the memory overhead of ManagedBatches, using variable sized batches, switching to
a concurrent LRFU, etc. In initial testing so little have changed in the single threaded
all memory case (depending on how trivial the data set is there can be a small degredation
to a small improvement). Minor additional overhead is to be expected due to maintaining
the cache ordering.
In multi-threaded testing even with small amounts of data (2 threads creating/manipulating
2 256k row tables with 128byte rows) both the all memory and a constrained buffer memory
test show marked improvement of 30% or more.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:40 p.m.
New subject: [JBoss JIRA] (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins resolved TEIID-1750.
-----------------------------------
Resolution: Done
Also added tracking of batch overhead and refined the crf function.
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:06 a.m.
New subject: [JBoss JIRA] (TEIID-1750) Memory handling improvements
[
https://issues.jboss.org/browse/TEIID-1750?page=com.atlassian.jira.plugin...
]
Steven Hawkins closed TEIID-1750.
---------------------------------
Memory handling improvements
----------------------------
Key: TEIID-1750
URL: https://issues.jboss.org/browse/TEIID-1750
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine
Affects Versions: 7.0
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Fix For: 7.6
The buffermanager will create softreferences for batches that prefer memory, but does not
track the memory as held once the value has been passivated.
Also small lobs, such as xml values created via sqlxml, should be inlined into results
for greater performance - and so that they can be paged out by the buffermanager rather
than being held by a lobmanager.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4687
days inactive
4854
days old
6 comments
3 participants
participants (3)
-
Steven Hawkins (Commented) (JIRA) -
Steven Hawkins (JIRA)
-
Steven Hawkins (Resolved) (JIRA)