[teiid-issues] [JBoss JIRA] (TEIID-5574) Clarify buffer manager property names

[Steven Hawkins (Jira)]

    [ https://issues.jboss.org/browse/TEIID-5574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675983#comment-13675983 ] 

Steven Hawkins edited comment on TEIID-5574 at 12/18/18 3:14 PM:
-----------------------------------------------------------------

Starting with the bufferservice I'll work in both directions to rationalize names.  What we want to convey is that there are three tiers of caching / managed buffers.  We'll stick with the terms buffer / buffer manager rather than cache.  Things break down along the lines of:

* Heap Buffer
* Fixed Memory Buffer
* Disk Buffer

Also the notion of memory usage by the heap buffer is complicated because that is shared with the vm and by processing work.  So we've used the term reserve to talk about what plans and the vm can "reserve" of the heap space.

While heap holds direct object references, Fixed Memory and Disk tiers hold much denser columnar serialized forms of the batches / pages.  Side note:  ideally that could be more fluid - years ago I started on the changes that would allow serialized batches to be used directly, but abandoned those changes.  Spark has the ability to do that which can help cut down quite a bit of serialization overhead in things like multi-pass sorts.

The initial changes would be:

Heap Buffer

maxReservedKb -> maxReservedHeapKb
heapCacheMemoryInUseKB -> heapBufferInUseKb (I'll add a note that this can be greater than the max reserved - which adds back pressure on plans to wait for memory reservations and forces more evictions by workers and the cleaner lower tiers)
heapMemoryInUseByActivePlansKB -> heapReservedByActivePlansKb

Fixed Memory

memoryBufferOffHeap -> fixedMemoryBufferOffHeap
memoryBufferSpace -> fixedMemeoryBufferSpaceMb

Disk

maxBufferSpace -> maxDiskBufferSpaceMb

Other

maxStorageObjectSize -> maxStorageObjectSizeBytes
cacheReadCount/WriteCount -> fixedMemoryReadCount/WriteCount

It probably also makes sense to normalize to Mb, so maxReservedHeapMb instead.

The initial target would be to keep and deprecate the legacy properties.

[~rareddy] [~rhn-engineering-vhalbert] Any thoughts or suggestions?


was (Author: shawkins):
Starting with the bufferservice I'll work in both directions to rationalize names.  What we want to convey is that there are three tiers of caching / managed buffers.  We'll stick with the terms buffer / buffer manager rather than cache.  Things break down along the lines of:

* Heap Buffer
* Fixed Memory Buffer
* Disk Buffer

Also the notion of memory usage by the heap buffer is complicated because that is shared with the vm and by processing work.  So we've used the term reserve to talk about what plans and the vm can "reserve" of the heap space.

While heap holds direct object references, Fixed Memory and Disk tiers hold much denser columnar serialized forms of the batches / pages.  Side note:  ideally that could be more fluid - years ago I started on the changes that would allow serialized batches to be used directly, but abandoned those changes.  Spark has the ability to do that which can help cut down quite a bit of serialization overhead in things like multi-pass sorts.

The initial changes would be:

Heap Buffer

maxReservedKb -> maxReservedHeapKb
heapCacheMemoryInUseKB -> heapBufferInUseKb (I'll add a note that this can be greater than the max reserved - which adds back pressure on plans to wait for memory reservations and forces more evictions by workers and the cleaner lower tiers)
heapMemoryInUseByActivePlansKB -> heapReservedByActivePlansKb

Fixed Memory

memoryBufferOffHeap -> fixedMemoryBufferOffHeap
memoryBufferSpace -> fixedMemeoryBufferSpaceMb

Disk

maxBufferSpace -> maxDiskBufferSpaceMb

Other

maxStorageObjectSize -> maxStorageObjectSizeBytes
cacheReadCount/WriteCount -> storageReadCount/WriteCount

It probably also makes sense to normalize to Mb, so maxReservedHeapMb instead.

The initial target would be to keep and deprecate the legacy properties.

[~rareddy] [~rhn-engineering-vhalbert] Any thoughts or suggestions?

> Clarify buffer manager property names
> -------------------------------------
>
>                 Key: TEIID-5574
>                 URL: https://issues.jboss.org/browse/TEIID-5574
>             Project: Teiid
>          Issue Type: Quality Risk
>          Components: Query Engine, Server
>            Reporter: Steven Hawkins
>            Assignee: Steven Hawkins
>            Priority: Major
>             Fix For: 12.0
>
>
> Where possible the buffer manager properties should be better differentiated.  Instead of max-buffer-space for example, we could have max-buffer-disk-space-mb.



--
This message was sent by Atlassian Jira
(v7.12.1#712002)