[teiid-issues] [JBoss JIRA] (TEIID-3106) BufferManager Cleaner consuming 90% of total CPU time

[Mark Addleman (JIRA)]

    [ https://issues.jboss.org/browse/TEIID-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998433#comment-12998433 ] 

Mark Addleman commented on TEIID-3106:
--------------------------------------

I'll take a wild stab at the problem:  I don't believe that the compareTo() definition is consistent with equals().  It appears that under some conditions equals() can return true where compareTo() does not return 0.  I can't create a complete story out of this yet, but I believe this inconsistency ends up driving ConcurrentSkipListMap.doRemove() mad as it tries to find the key to remove.  The failure of my theory is that I can't find where ConcurrentSkipListMap uses equals() when comparing keys.

I'm only mentioning it now because it seems to fit the profiling results that Devesh provided and on the off chance it gives you some insight into what's going on.

> BufferManager Cleaner consuming 90% of total CPU time
> -----------------------------------------------------
>
>                 Key: TEIID-3106
>                 URL: https://issues.jboss.org/browse/TEIID-3106
>             Project: Teiid
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Query Engine
>    Affects Versions: 8.7
>         Environment: zOS
>            Reporter: Devesh Mishra
>            Assignee: Steven Hawkins
>              Labels: teiid-engine
>         Attachments: BufferManagerImpl.java, CacheKey_Loog.patch, cpu monitoring.docx, in_local.docx, non_high_cpu.docx
>
>
>  BufferManager Cleaner thread is consuming almost all of the CPU utilized by the jboss process. Thread dump shows following information.
> 3XMTHREADINFO      "BufferManager Cleaner" J9VMThread:0x0000004C41CFEB00, j9thread_t:0x0000004C52B85AE0, java/lang/Thread:0x000000481C036E20, state:CW, prio=5
> 3XMJAVALTHREAD            (java/lang/Thread getId:0x76, isDaemon:true)
> 3XMTHREADINFO1            (native thread ID:0x3AEC2600, native priority:0x5, native policy:UNKNOWN)
> 3XMHEAPALLOC             Heap bytes allocated since last GC cycle=2609184 (0x27D020)
> 3XMTHREADINFO3           Java callstack:
> 4XESTACKTRACE                at java/util/concurrent/ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1070(Compiled Code))
> 4XESTACKTRACE                at java/util/concurrent/ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1659(Compiled Code))
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/LrfuEvictionQueue.remove(LrfuEvictionQueue.java:60(Compiled Code))
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/BufferManagerImpl.doEvictions(BufferManagerImpl.java:854(Compiled Code))
> 5XESTACKTRACE                   (entered lock: org/teiid/common/buffer/CacheEntry at 0x00000048393C2598, entry count: 1)
> 4XESTACKTRACE                at org/teiid/common/buffer/impl/BufferManagerImpl$Cleaner.run(BufferManagerImpl.java:108)
> 4XESTACKTRACE                at java/util/TimerThread.mainLoop(Timer.java:555)
> 4XESTACKTRACE                at java/util/TimerThread.run(Timer.java:505)
> When we added log statements around the BufferManagerImpl.doEvictions() it loops through the remove and firstEntry loop. 
> Forum discussion link : https://community.jboss.org/message/901792



--
This message was sent by Atlassian JIRA
(v6.3.1#6329)