[teiid-issues] [JBoss JIRA] Commented: (TEIID-1509) Change the default for value caching

[Steven Hawkins (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588737#comment-12588737 ] 

Steven Hawkins commented on TEIID-1509:
---------------------------------------

Under the umbrella of better memory management, I would propose a larger set of changes:

switch the default for value caching to disabled
change the default for maxReserveBatchColumns and maxProcessingBatchColumns to 0 to indicate that they should be calculated values.

internally we will switch back to computing batch sizes in bytes (or KB) and assume 64 bits, but we won't change the config at least until Teiid 8.

non-zero values for maxReserveBatchColumns and maxProcessingBatchColumns will get converted to bytes by assuming 64 bytes per value, i.e. 
bytes = (maxReserveBatchColumns) * (64 bytes per value) * (processor batch size)

For the old default of 16384, that would imply 512MB.

Assuming max available memory = vm max heap - 300 MB (AS/Teiid overhead). 

The calculated byte value for maxReserveBatchColumns=0 would be: 1/2 of the max available memory under 1 gig and 75% of the max available over 1 gig

The calculate byte value for maxProcessingBatchColumns=0 would be: 10% of the max available memory / max active plans

The schema size calculation would also be converted to bytes and would take into account whether value caching was enabled, so that the affect of value caching is automatically considered.

These assumptions align well with a typical dedicated server install.  There are two scenarios that would need to be doc'ed:

1. A memory constrained 32 bit VM, which implies that value caching should be enabled.  The admin may also want to adjust the maxReserveBatchColumns higher to compensate for the 64 bit assumption.

2. A large memory non-dedicated server.  In this case assuming 75% of each additional gig of memory may be too high.




> Change the default for value caching
> ------------------------------------
>
>                 Key: TEIID-1509
>                 URL: https://issues.jboss.org/browse/TEIID-1509
>             Project: Teiid
>          Issue Type: Quality Risk
>          Components: Query Engine
>    Affects Versions: 7.4
>            Reporter: Steven Hawkins
>            Assignee: Steven Hawkins
>             Fix For: 7.4
>
>
> There is a larger performance impact from having value caching enabled with large data sets (which is the more typical deployment scenario) than having value caching disabled and a larger memory footprint.
> So the value caching default should change to false.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira