[teiid-issues] [JBoss JIRA] (TEIID-3106) BufferManager Cleaner consuming 90% of total CPU time

[Steven Hawkins (JIRA)]

     [ https://issues.jboss.org/browse/TEIID-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Hawkins updated TEIID-3106:
----------------------------------
    Attachment: LrfuEvictionQueue.patch


Based upon your log, a possibility is a prior race condition between an non-initial call to addMemoryEntry and a batch that has already been removed from the memory entries, but not the eviction queue.  However the only place that could happen is from a fastGet call that is not retaining the entry, which I'd have to look at more to see when that combination could occur.  

In any case this would update the cache key on the entry still in the queue, which would invalidate the queue ordering.  Attached is a patch against the queue that looks for invalid states ideally as soon as they occur.

> BufferManager Cleaner consuming 90% of total CPU time
> -----------------------------------------------------
>
>                 Key: TEIID-3106
>                 URL: https://issues.jboss.org/browse/TEIID-3106
>             Project: Teiid
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Query Engine
>    Affects Versions: 8.7
>         Environment: zOS
>            Reporter: Devesh Mishra
>            Assignee: Steven Hawkins
>              Labels: teiid-engine
>         Attachments: BufferManagerImpl.java, BufferManagerImpl.java, CacheKey_Loog.patch, cpu monitoring.docx, dump.zip, in_local.docx, LrfuEvictionQueue.patch, non_high_cpu.docx
>
>
>  BufferManager Cleaner thread is consuming almost all of the CPU utilized by the jboss process. Thread dump shows following information.
> 3XMTHREADINFO      "BufferManager Cleaner" J9VMThread:0x0000004C41CFEB00, j9thread_t:0x0000004C52B85AE0, java/lang/Thread:0x000000481C036E20, state:CW, prio=5
> 3XMJAVALTHREAD            (java/lang/Thread getId:0x76, isDaemon:true)
> 3XMTHREADINFO1            (native thread ID:0x3AEC2600, native priority:0x5, native policy:UNKNOWN)
> 3XMHEAPALLOC             Heap bytes allocated since last GC cycle=2609184 (0x27D020)
> 3XMTHREADINFO3           Java callstack:
> 4XESTACKTRACE                at java/util/concurrent/ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1070(Compiled Code))
> 4XESTACKTRACE                at java/util/concurrent/ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1659(Compiled Code))
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/LrfuEvictionQueue.remove(LrfuEvictionQueue.java:60(Compiled Code))
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/BufferManagerImpl.doEvictions(BufferManagerImpl.java:854(Compiled Code))
> 5XESTACKTRACE                   (entered lock: org/teiid/common/buffer/CacheEntry at 0x00000048393C2598, entry count: 1)
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/BufferManagerImpl$Cleaner.run(BufferManagerImpl.java:108)
> 4XESTACKTRACE                at java/util/TimerThread.mainLoop(Timer.java:555)
> 4XESTACKTRACE                at java/util/TimerThread.run(Timer.java:505)
> When we added log statements around the BufferManagerImpl.doEvictions() it loops through the remove and firstEntry loop. 
> Forum discussion link : https://community.jboss.org/message/901792



--
This message was sent by Atlassian JIRA
(v6.3.1#6329)