[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8647:
----------------------------------
Fix Version/s: 9.2.0.CR1
9.1.5.Final
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 9.2.0.CR1, 9.1.5.Final
>
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by paragBorah Borah (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
paragBorah Borah edited comment on ISPN-8647 at 12/28/17 12:29 PM:
-------------------------------------------------------------------
Yes, that's correct.
Can u make sure this defect will be fixed in next release?
was (Author: parag2borah):
Yes, Can u make sure this defect will be fixed in next release?
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Assignee: Tristan Tarrant
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 11 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by paragBorah Borah (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
paragBorah Borah commented on ISPN-8647:
----------------------------------------
Yes, Can u make sure this defect will be fixed in next release?
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Assignee: Tristan Tarrant
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-8647:
-------------------------------------
Assignee: Tristan Tarrant
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Assignee: Tristan Tarrant
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-8647:
---------------------------------------
The logger for those classes should be static. Instead it is being instantiated for every construction of a stream.
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by paragBorah Borah (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
paragBorah Borah commented on ISPN-8647:
----------------------------------------
we are using log4j-1.2.15.
But this does not issue with log4j. Please check the stack trace. Its JBoss logger in infinispan which got blocked while classloader is trying to load class in getLog() method.
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.jboss.logging.Logger$1.run(Logger.java:2544)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
The resolution I have given in my previous comments works for me. I have made changes in infinispan code & push the compiled jar to my environment, no more DEADLOCK situation on thread dump. I wanted infinispan team to fix this in the upcoming version.
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-8647:
---------------------------------------
Jboss-logging is not the actual logger. Are you using log4j 1.x or 2.x or something else?
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months
[JBoss JIRA] (ISPN-8647) Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
by paragBorah Borah (JIRA)
[ https://issues.jboss.org/browse/ISPN-8647?page=com.atlassian.jira.plugin.... ]
paragBorah Borah commented on ISPN-8647:
----------------------------------------
we are using JBoss logger.
infinispan-core-9.1.1.Final.jar
jboss-logging-3.3.0.Final.jar
The issue has been occurred due to Log instance creation in AbstractLocalCacheStream & AbstractCacheStream is not static.
Resolution of the issue.
* Create a static Log instance in class AbstractLocalCacheStream & AbstractCacheStream
or
* Remove Log instance in class AbstractLocalCacheStream & AbstractCacheStream & create static log instance in sub classes.
> Log instance creation in class AbstractLocalCacheStream & AbstractCacheStream, cause system hang / hit performance
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-8647
> URL: https://issues.jboss.org/browse/ISPN-8647
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.1.Final, 9.1.2.Final, 9.1.3.Final, 9.1.4.Final
> Environment: On production environment
> Reporter: paragBorah Borah
> Priority: Critical
> Attachments: TD1(1).zip
>
>
> Log instance creation got blocked in class AbstractLocalCacheStream & AbstractCacheStream, while performing stream operations on infinispan.
> On each thread dump, getting below exception multiple times. Which hang the system & hit performance
> "Thread-93,EXECUTOR_POOL" #57030 daemon prio=5 os_prio=0 tid=0x00000000867ca000 nid=0x2a78 waiting for monitor entry [0x00000000d349d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.AbstractCacheStream.<init>(AbstractCacheStream.java:53)
> at org.infinispan.stream.impl.DistributedCacheStream.<init>(DistributedCacheStream.java:120)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet$1.<init>(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.stream(DistributionBulkInterceptor.java:147)
> at org.infinispan.interceptors.distribution.DistributionBulkInterceptor$BackingEntrySet.iterator(DistributionBulkInterceptor.java:102)
> at org.infinispan.stream.impl.local.ValueCacheCollection.iterator(ValueCacheCollection.java:38)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:873)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.iterator(EncoderCache.java:852)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getIntegrationServiceById(ServerDetailsCache.java:788)
> at com.bmc.unifiedadmin.persistence.service.server.ServerDetailsCache.getProxyDetails(ServerDetailsCache.java:419)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getIntegrationServiceDetails(IntegrationServicePersistenceServiceImpl.java:138)
> at com.bmc.unifiedadmin.persistence.service.server.IntegrationServicePersistenceServiceImpl.getAssociatedPAsWithISs(IntegrationServicePersistenceServiceImpl.java:828)
> at com.bmc.sas.ngp.service.unifiedadmin.integrationservice.IntegrationServiceImpl.getAllProxiesForServer(IntegrationServiceImpl.java:226)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.loadServerDetails(CAWSFacadePlatformImpl.java:1255)
> at com.proactivenet.api.aspen.ws.CAWSFacadePlatformImpl.getAllServerAndISDetails(CAWSFacadePlatformImpl.java:1283)
> at com.bmc.sms.cmaws.ngp.wsclient.CMAJServiceClient.getAllServerAndISDetails(CMAJServiceClient.java:452)
> at com.bmc.sms.cmaws.rsbuilder.CMATSIMServerDetailsResultBuilder.buildResult(CMATSIMServerDetailsResultBuilder.java:75)
> at com.bmc.truesight.rest.taskexecution.TaskExecutionWorker.run(TaskExecutionWorker.java:83)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> ==
> "GatewayEvtMessageProcessor" #185 prio=5 os_prio=0 tid=0x0000000074094000 nid=0x2454 waiting for monitor entry [0x000000007fa8d000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
> waiting to lock <0x00000002cdb18a78> (a java.lang.Object)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.jboss.logging.Logger$1.run(Logger.java:2544)
> at java.security.AccessController.doPrivileged(Native Method)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2529)
> at org.jboss.logging.Logger.getMessageLogger(Logger.java:2516)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:19)
> at org.infinispan.stream.impl.local.AbstractLocalCacheStream.<init>(AbstractLocalCacheStream.java:22)
> at org.infinispan.stream.impl.local.LocalCacheStream.<init>(LocalCacheStream.java:69)
> at org.infinispan.commands.read.EntrySetCommand$BackingEntrySet.stream(EntrySetCommand.java:145)
> at org.infinispan.stream.impl.local.ValueCacheCollection.stream(ValueCacheCollection.java:93)
> at org.infinispan.cache.impl.EncoderCache$EncoderValuesCollection.stream(EncoderCache.java:863)
> at com.bmc.truesight.api.extended.component.ComponentsDAO.getComponentsByType(ComponentsDAO.java:69)
> at com.bmc.truesight.api.extended.event.Events.getIdforComponentByCellName(Events.java:2414)
> at com.bmc.truesight.platform.components.gateway.CellGatewayServer.processNewEvent(CellGatewayServer.java:148)
> locked <0x00000002c7ad2a58> (a com.bmc.truesight.platform.components.gateway.CellGatewayServer)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.processMessage(GatewayServer.java:430)
> at com.bmc.sms.cellcomm.gw.GatewayServer$EventMessageProcessor.run(GatewayServer.java:340)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 12 months