[teiid-issues] [JBoss JIRA] (TEIID-3422) Setting more than 513MB for buffer-service-max-storage-object-size, OOME occurs at org.teiid.common.buffer.impl.BufferFrontedFileStoreCache.initialize()

[Steven Hawkins (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057749#comment-13057749 ] 

Steven Hawkins commented on TEIID-3422:
---------------------------------------

> By taking look at the source code[2], if the value of maxStorageObjectSize is more than equals with 536870912, the loop becomes infinit, and will cause OOME.

Yes there is an issue there due to the integer overflow.  That will be corrected.

> One of our user needs to set more than 512MB for buffer-service-max-storage-object-size to avoid TEIID30001[1], this limitation becomes a critical issue for them.

How big is the vm, processor batch size, and what kind of data is being serialized?  Setting a value that high probably means that there is data inline in the batches that should not be.

> Setting more than 513MB for buffer-service-max-storage-object-size, OOME occurs at org.teiid.common.buffer.impl.BufferFrontedFileStoreCache.initialize()
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TEIID-3422
>                 URL: https://issues.jboss.org/browse/TEIID-3422
>             Project: Teiid
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: 8.7
>         Environment: JBoss DV 6.0, 6.1
>            Reporter: hisao furuichi
>            Assignee: Steven Hawkins
>
> Setting more than 513MB for buffer-service-max-storage-object-size, OOME occurs at org.teiid.common.buffer.impl.BufferFrontedFileStoreCache.initialize().
> One of our user needs to set more than 512MB for buffer-service-max-storage-object-size to avoid TEIID30001[1], this limitation becomes a critical issue for them.
> Additional Information:
> By taking look at the source code[2], if the value of maxStorageObjectSize is more than equals with 536870912, the loop becomes infinit, and will cause OOME.
> [1]
> TEIID30001 Max block number exceeded by 233,144 21,422,155. Increase the maxStorageObjectSize to support larger storage objects.  Alternatively you could make the processor batch size smaller.
> [2]teiid/engine/src/main/java/org/teiid/common/buffer/impl/BufferFrontedFileStoreCache.java
> ===
> public static final long MAX_ADDRESSABLE_MEMORY = 1l<<(ADDRESS_BITS+LOG_BLOCK_SIZE);
> ~~
> static final int BLOCK_SIZE = 1 << LOG_BLOCK_SIZE;
> ~~
> public void initialize() throws TeiidComponentException {
> ~~
>   List<BlockStore> stores = new ArrayList<BlockStore>();
>   int size = BLOCK_SIZE;
>   int files = 32; //this allows us to have 64 terabytes of smaller block sizes
>   do {
>     stores.add(new BlockStore(this.storageManager, size, 30, files));
>     size <<=1;
>     if (files > 1) {
>       files >>= 1;
>     }
>   } while ((size>>1) < maxStorageObjectSize);
> ~~
> ===



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)