[teiid-issues] [JBoss JIRA] (TEIID-2316) Give translators a better batch size

[Steven Hawkins (JIRA)]

     [ https://issues.jboss.org/browse/TEIID-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Hawkins resolved TEIID-2316.
-----------------------------------

    Resolution: Done


Removed the connector batch size setting and added a release note.  

We will now push twice the working batch size on the execution context.  The working batch size is determined from the processor batch size and the estimated data size.  We'll adjust the working batch size up or down from 1/8 to 8x the processor batch size.  With the default processor batch size is 256, which means that the connector will be asked for batch sizes between 64 and 4096 rows.

For most scenarios this should be sufficient to prevent issues with allocating too much space based upon a single connector batch size setting.

The google connector was modified so that batch size refers to the max batch size.

There may need to be some other provisions for sources, such as oracle that perfer extremely small fetch sizes.

Also updated the access node logic not to block if partial batches are delivered to it (which can also happen in continuous mode).  This allows translators to effectively ignore the batch size and deliver whatever number of rows they feel is appropriate.
                
> Give translators a better batch size
> ------------------------------------
>
>                 Key: TEIID-2316
>                 URL: https://issues.jboss.org/browse/TEIID-2316
>             Project: Teiid
>          Issue Type: Enhancement
>          Components: Query Engine
>    Affects Versions: 8.3
>            Reporter: Steven Hawkins
>            Assignee: Steven Hawkins
>             Fix For: 8.3
>
>
> 7.5 and later releases refined logic in the engine to shape batch sizes based upon the data, however we still only use the default connector batch size when requesting data.
> This value currently defaults to 512, however that can be much too large for many scenarios given how the underlying sources handling buffering.  At the least we should provide the engine adjusted batch size.  It would be good as well to let the translator specify a scaling factor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira