BufferManager Cleaner consuming 90% of total CPU time ----------------------------------------------------- Key: TEIID-3106 URL: https://issues.jboss.org/browse/TEIID-3106 Project: Teiid Issue Type: Bug Security Level: Public(Everyone can see) Components: Query Engine Affects Versions: 8.7 Environment: zOS Reporter: Devesh Mishra Assignee: Steven Hawkins Labels: teiid-engine Attachments: BufferManagerImpl.java, BufferManagerImpl.java, CacheKey_Loog.patch, cpu monitoring.docx, dump.zip, in_local.docx, LrfuEvictionQueue.patch, non_high_cpu.docx BufferManager Cleaner thread is consuming almost all of the CPU utilized by the jboss process. Thread dump shows following information. 3XMTHREADINFO "BufferManager Cleaner" J9VMThread:0x0000004C41CFEB00, j9thread_t:0x0000004C52B85AE0, java/lang/Thread:0x000000481C036E20, state:CW, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x76, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x3AEC2600, native priority:0x5, native policy:UNKNOWN) 3XMHEAPALLOC Heap bytes allocated since last GC cycle=2609184 (0x27D020) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1070(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1659(Compiled Code)) 4XESTACKTRACE at org/teiid/common/buffer/impl/LrfuEvictionQueue.remove(LrfuEvictionQueue.java:60(Compiled Code)) 4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl.doEvictions(BufferManagerImpl.java:854(Compiled Code)) 5XESTACKTRACE (entered lock: org/teiid/common/buffer/CacheEntry@0x00000048393C2598, entry count: 1) 4XESTACKTRACE at org/teiid/common/buffer/impl/BufferManagerImpl$Cleaner.run(BufferManagerImpl.java:108) 4XESTACKTRACE at java/util/TimerThread.mainLoop(Timer.java:555) 4XESTACKTRACE at java/util/TimerThread.run(Timer.java:505) When we added log statements around the BufferManagerImpl.doEvictions() it loops through the remove and firstEntry loop. Forum discussion link : https://community.jboss.org/message/901792