[
https://issues.jboss.org/browse/TEIID-5574?page=com.atlassian.jira.plugin...
]
Steven Hawkins edited comment on TEIID-5574 at 12/18/18 2:58 PM:
-----------------------------------------------------------------
Starting with the bufferservice I'll work in both directions to rationalize names.
What we want to convey is that there are three tiers of caching / managed buffers.
We'll stick with the terms buffer / buffer manager rather than cache. Things break
down along the lines of:
* Heap Buffer
* Fixed Memory Buffer
* Disk Buffer
Also the notion of memory usage by the heap buffer is complicated because that is shared
with the vm and by processing work. So we've used the term reserve to talk about what
plans and the vm can "reserve" of the heap space.
While heap holds direct object references, Fixed Memory and Disk tiers hold much denser
columnar serialized forms of the batches / pages. Side note: ideally that could be more
fluid - years ago I started on the changes that would allow serialized batches to be used
directly, but abandoned those changes. Spark has the ability to do that which can help
cut down quite a bit of serialization overhead in things like multi-pass sorts.
The initial changes would be:
Heap Buffer
maxReservedKb -> maxReservedHeapKb
heapCacheMemoryInUseKB -> heapBufferInUseKb (I'll add a note that this can be
greater than the max reserved - which adds back pressure on plans to wait for memory
reservations and forces more evictions by workers and the cleaner lower tiers)
heapMemoryInUseByActivePlansKB -> heapReservedByActivePlansKb
Fixed Memory
memoryBufferOffHeap -> fixedMemoryBufferOffHeap
memoryBufferSpace -> fixedMemeoryBufferSpaceMb
Disk
maxBufferSpace -> maxDiskBufferSpaceMb
Other
maxStorageObjectSize -> maxStorageObjectSizeBytes
cacheReadCount/WriteCount -> storageReadCount/WriteCount
It probably also makes sense to normalize to Mb, so maxReservedHeapMb instead.
The initial target would be to keep and deprecate the legacy properties.
[~rareddy] [~rhn-engineering-vhalbert] Any thoughts or suggestions?
was (Author: shawkins):
Starting with the bufferservice I'll work in both directions to rationalize names.
What we want to convey is that there are three tiers of caching / managed buffers.
We'll stick with the terms buffer / buffer manager rather than cache. Things break
down along the lines of:
* Heap Buffer
* Fixed Memory Buffer
* Disk Buffer
Also the notion of memory usage by the heap buffer is complicated because that is shared
with the vm and by processing work. So we've used the term reserve to talk about what
plans and the vm can "reserve" of the heap space.
While heap holds direct object references, Fixed Memory and Disk tiers hold much denser
columnar serialized forms of the batches / pages. Side note: ideally that could be more
fluid - years ago I started on the changes that would allow serialized batches to be used
directly, but abandoned those changes. Spark has the ability to do that which can help
cut down quite a bit of serialization overhead in things like multi-pass sorts.
The initial changes would be:
Heap Buffer
maxReservedKb -> maxReservedHeapKb
heapCacheMemoryInUseKB -> heapBufferInUseKb (I'll add a note that this can be
greater than the max reserved - which adds back pressure on plans to wait for memory
reservations and forces more evictions by workers and the cleaner lower tiers)
heapMemoryInUseByActivePlansKB -> heapReservedByActivePlansKb
Fixed Memory
memoryBufferOffHeap -> fixedMemoryBufferOffHeap
memoryBufferSpace -> fixedMemeoryBufferSpaceMb
Disk
maxBufferSpace -> maxDiskBufferSpaceMb
Other
maxStorageObjectSize -> maxStorageObjectSizeBytes
cacheReadCount/WriteCount -> serializedReadCount/WriteCount (see side note above, if
things are more fluid serialized may not be the right term here)
It probably also makes sense to normalize to Mb, so maxReservedHeapMb instead.
The initial target would be to keep and deprecate the legacy properties.
[~rareddy] [~rhn-engineering-vhalbert] Any thoughts or suggestions?
Clarify buffer manager property names
-------------------------------------
Key: TEIID-5574
URL:
https://issues.jboss.org/browse/TEIID-5574
Project: Teiid
Issue Type: Quality Risk
Components: Query Engine, Server
Reporter: Steven Hawkins
Assignee: Steven Hawkins
Priority: Major
Fix For: 12.0
Where possible the buffer manager properties should be better differentiated. Instead of
max-buffer-space for example, we could have max-buffer-disk-space-mb.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)