[JBoss JIRA] (ISPN-5317) BlockingTaskAwareExecutorService.cherkForReadyTasks() may block CacheTopologyControlCommand
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5317?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5317:
------------------------------
Status: Open (was: New)
> BlockingTaskAwareExecutorService.cherkForReadyTasks() may block CacheTopologyControlCommand
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-5317
> URL: https://issues.jboss.org/browse/ISPN-5317
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.0.Beta1, 7.1.1.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> Current implementation, checkForReadyTasks() is invoked in the StateConsumer after the topology is updated.
> If the remote-thread-pool is full (i.e. no more threads available), the checkForReadyTasks() will start processing commands in the invoker thread. If it happens, the CacheTopologyControlCommand has not done processing and the cache topology updated is delayed.
> It may cause some problems!
> The solution, it would be to invoke the checkForReadyTasks() in the StateTransferManager, after the StateConsumer and StateProvider had finished they work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5317) BlockingTaskAwareExecutorService.cherkForReadyTasks() may block CacheTopologyControlCommand
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5317?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5317:
------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3434
> BlockingTaskAwareExecutorService.cherkForReadyTasks() may block CacheTopologyControlCommand
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-5317
> URL: https://issues.jboss.org/browse/ISPN-5317
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.0.Beta1, 7.1.1.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> Current implementation, checkForReadyTasks() is invoked in the StateConsumer after the topology is updated.
> If the remote-thread-pool is full (i.e. no more threads available), the checkForReadyTasks() will start processing commands in the invoker thread. If it happens, the CacheTopologyControlCommand has not done processing and the cache topology updated is delayed.
> It may cause some problems!
> The solution, it would be to invoke the checkForReadyTasks() in the StateTransferManager, after the StateConsumer and StateProvider had finished they work.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5420) Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-5420?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-5420:
------------------------------
Fix Version/s: 8.0.0.Alpha1
(was: 8.0.0.Final)
> Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-5420
> URL: https://issues.jboss.org/browse/ISPN-5420
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.2.Final, 7.1.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 8.0.0.Alpha1
>
>
> The join process was designed in the idea that a node would start its caches in sequential order, so {{ClusterTopologyManager.waitForView()}} would block at most once for each joining node. However, WildFly actually starts {{2 * Runtime.availableProcessors()}} caches in parallel, and this can be a problem when the machine has a lot of cores and multiple nodes.
> {{ClustertopologyManager.handleClusterView()}} only updates the {{viewId}} after it updated the cache topologies of each cache AND after it confirmed the availability of all the nodes with a {{POLICY_GET_STATUS}} RPC. This RPC can block, and it's very easy for the remote-executor thread pool on the coordinator to become overloades with threads like this:
> {noformat}
> "remote-thread-172" daemon prio=10 tid=0x00007f0cc48c0000 nid=0x28ca4 in Object.wait() [0x00007f0c5f25b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.infinispan.topology.ClusterTopologyManagerImpl.waitForView(ClusterTopologyManagerImpl.java:357)
> - locked <0x00000000ff3bd900> (a java.lang.Object)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:123)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5443) NPE in ReadCommittedEntry.getLifespan() during Cache.size()
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5443?page=com.atlassian.jira.plugin.... ]
Work on ISPN-5443 started by William Burns.
-------------------------------------------
> NPE in ReadCommittedEntry.getLifespan() during Cache.size()
> -----------------------------------------------------------
>
> Key: ISPN-5443
> URL: https://issues.jboss.org/browse/ISPN-5443
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.0.Final
> Reporter: Paul Ferraro
> Assignee: William Burns
> Priority: Blocker
> Fix For: 7.2.1.Final
>
>
> Infinispan 7.2.0 causes one of our 2LC cache tests to fail consistently. An entity is removed from the 2LC followed by printing of cache statistics, all within the context of a transaction. Here is the exception.
> {noformat}
> ERROR [org.jboss.as.ejb3.invocation] (default task-23) WFLYEJB0034: EJB Invocation failed on component StatefulBean for method public abstract void org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful.deleteEmployee(int): javax.ejb.EJBException: java.lang.NullPointerException
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful$$$view3.deleteEmployee(Unknown Source)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.servlet.StatefulServlet.doGet(StatefulServlet.java:155)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:274)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:253)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.infinispan.container.entries.ReadCommittedEntry.getLifespan(ReadCommittedEntry.java:99)
> at org.infinispan.container.InternalEntryFactoryImpl.create(InternalEntryFactoryImpl.java:53)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.getNextFromIterator(TransactionAwareCloseableIterator.java:119)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.<init>(TransactionAwareCloseableIterator.java:54)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:44)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:19)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:47)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:25)
> at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:124)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
> at org.infinispan.interceptors.TxInterceptor.visitSizeCommand(TxInterceptor.java:293)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:374)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:369)
> at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:267)
> at org.hibernate.cache.infinispan.impl.BaseRegion.getElementCountInMemory(BaseRegion.java:100)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.getElementCountInMemory(ConcurrentSecondLevelCacheStatisticsImpl.java:64)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.toString(ConcurrentSecondLevelCacheStatisticsImpl.java:93)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.logStats(StatefulBean.java:172)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.deleteEmployee(StatefulBean.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SFSBInvocationInterceptor.processInvocation(SFSBInvocationInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.processInvocation(StatefulSessionSynchronizationInterceptor.java:127)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:65)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> ... 63 more
> {noformat}
> Now I know that hibernate-infinispan should not be using a cluster-wide size() operation here (see https://hibernate.atlassian.net/browse/HHH-9776), but there is still an urgent issue here that need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5443) NPE in ReadCommittedEntry.getLifespan() during Cache.size()
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5443?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5443:
--------------------------------
Status: Open (was: New)
> NPE in ReadCommittedEntry.getLifespan() during Cache.size()
> -----------------------------------------------------------
>
> Key: ISPN-5443
> URL: https://issues.jboss.org/browse/ISPN-5443
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.0.Final
> Reporter: Paul Ferraro
> Assignee: William Burns
> Priority: Blocker
> Fix For: 7.2.1.Final
>
>
> Infinispan 7.2.0 causes one of our 2LC cache tests to fail consistently. An entity is removed from the 2LC followed by printing of cache statistics, all within the context of a transaction. Here is the exception.
> {noformat}
> ERROR [org.jboss.as.ejb3.invocation] (default task-23) WFLYEJB0034: EJB Invocation failed on component StatefulBean for method public abstract void org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful.deleteEmployee(int): javax.ejb.EJBException: java.lang.NullPointerException
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful$$$view3.deleteEmployee(Unknown Source)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.servlet.StatefulServlet.doGet(StatefulServlet.java:155)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:274)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:253)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.infinispan.container.entries.ReadCommittedEntry.getLifespan(ReadCommittedEntry.java:99)
> at org.infinispan.container.InternalEntryFactoryImpl.create(InternalEntryFactoryImpl.java:53)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.getNextFromIterator(TransactionAwareCloseableIterator.java:119)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.<init>(TransactionAwareCloseableIterator.java:54)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:44)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:19)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:47)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:25)
> at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:124)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
> at org.infinispan.interceptors.TxInterceptor.visitSizeCommand(TxInterceptor.java:293)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:374)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:369)
> at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:267)
> at org.hibernate.cache.infinispan.impl.BaseRegion.getElementCountInMemory(BaseRegion.java:100)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.getElementCountInMemory(ConcurrentSecondLevelCacheStatisticsImpl.java:64)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.toString(ConcurrentSecondLevelCacheStatisticsImpl.java:93)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.logStats(StatefulBean.java:172)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.deleteEmployee(StatefulBean.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SFSBInvocationInterceptor.processInvocation(SFSBInvocationInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.processInvocation(StatefulSessionSynchronizationInterceptor.java:127)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:65)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> ... 63 more
> {noformat}
> Now I know that hibernate-infinispan should not be using a cluster-wide size() operation here (see https://hibernate.atlassian.net/browse/HHH-9776), but there is still an urgent issue here that need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5443) NPE in ReadCommittedEntry.getLifespan() during Cache.size()
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5443?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-5443:
-------------------------------------
Assignee: William Burns
> NPE in ReadCommittedEntry.getLifespan() during Cache.size()
> -----------------------------------------------------------
>
> Key: ISPN-5443
> URL: https://issues.jboss.org/browse/ISPN-5443
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.0.Final
> Reporter: Paul Ferraro
> Assignee: William Burns
> Priority: Blocker
> Fix For: 7.2.1.Final
>
>
> Infinispan 7.2.0 causes one of our 2LC cache tests to fail consistently. An entity is removed from the 2LC followed by printing of cache statistics, all within the context of a transaction. Here is the exception.
> {noformat}
> ERROR [org.jboss.as.ejb3.invocation] (default task-23) WFLYEJB0034: EJB Invocation failed on component StatefulBean for method public abstract void org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful.deleteEmployee(int): javax.ejb.EJBException: java.lang.NullPointerException
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:634)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.Stateful$$$view3.deleteEmployee(Unknown Source)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.servlet.StatefulServlet.doGet(StatefulServlet.java:155)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:274)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:253)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.infinispan.container.entries.ReadCommittedEntry.getLifespan(ReadCommittedEntry.java:99)
> at org.infinispan.container.InternalEntryFactoryImpl.create(InternalEntryFactoryImpl.java:53)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.getNextFromIterator(TransactionAwareCloseableIterator.java:119)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterator.<init>(TransactionAwareCloseableIterator.java:54)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:44)
> at org.infinispan.iteration.impl.TransactionAwareCloseableIterable.iterator(TransactionAwareCloseableIterable.java:19)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:47)
> at org.infinispan.commands.read.SizeCommand.perform(SizeCommand.java:25)
> at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:124)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
> at org.infinispan.interceptors.TxInterceptor.visitSizeCommand(TxInterceptor.java:293)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitSizeCommand(AbstractVisitor.java:81)
> at org.infinispan.commands.read.SizeCommand.acceptVisitor(SizeCommand.java:35)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:374)
> at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:369)
> at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:267)
> at org.hibernate.cache.infinispan.impl.BaseRegion.getElementCountInMemory(BaseRegion.java:100)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.getElementCountInMemory(ConcurrentSecondLevelCacheStatisticsImpl.java:64)
> at org.hibernate.stat.internal.ConcurrentSecondLevelCacheStatisticsImpl.toString(ConcurrentSecondLevelCacheStatisticsImpl.java:93)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.logStats(StatefulBean.java:172)
> at org.jboss.as.test.clustering.cluster.ejb.xpc.bean.StatefulBean.deleteEmployee(StatefulBean.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.jpa.interceptor.SFSBInvocationInterceptor.processInvocation(SFSBInvocationInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.processInvocation(StatefulSessionSynchronizationInterceptor.java:127)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:65)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> ... 63 more
> {noformat}
> Now I know that hibernate-infinispan should not be using a cluster-wide size() operation here (see https://hibernate.atlassian.net/browse/HHH-9776), but there is still an urgent issue here that need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5281) XS replication - log exceptions on write when backup cache is unavailable
by Matej Čimbora (JIRA)
[ https://issues.jboss.org/browse/ISPN-5281?page=com.atlassian.jira.plugin.... ]
Matej Čimbora commented on ISPN-5281:
-------------------------------------
[~pruivo], Interesting enough, I'm not able to reproduce the issue outside of RadarGun using the same configuration. Therefore I think the issue can be closed ATM, as it's most likely RG-related (I'll eventually reopen it when it appears in the future).
> XS replication - log exceptions on write when backup cache is unavailable
> -------------------------------------------------------------------------
>
> Key: ISPN-5281
> URL: https://issues.jboss.org/browse/ISPN-5281
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Reporter: Matej Čimbora
> Assignee: Pedro Ruivo
>
> When backup site is down & we keep writing to the "main" site, the following exceptions appear in the log. Workaround - take the backup site offline first.
> {code}
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from localhost-64865, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:364)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:165)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:563)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:290)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:232)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:75)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:346)
> at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:434)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:168)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.xsite.NonTransactionalBackupInterceptor.handleSingleKeyWriteCommand(NonTransactionalBackupInterceptor.java:59)
> at org.infinispan.interceptors.xsite.NonTransactionalBackupInterceptor.visitPutKeyValueCommand(NonTransactionalBackupInterceptor.java:35)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:48)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:198)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:119)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1520)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:998)
> at org.infinispan.CacheImpl.put(CacheImpl.java:990)
> at org.infinispan.DecoratedCache.put(DecoratedCache.java:408)
> at org.radargun.service.InfinispanOperations$Cache.put(InfinispanOperations.java:51)
> at org.radargun.stages.cache.test.LoadDataStage$NonTxLoader.loadEntry(LoadDataStage.java:266)
> at org.radargun.stages.cache.test.LoadDataStage$Loader.run(LoadDataStage.java:228)
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
> at org.jgroups.JChannel.up(JChannel.java:739)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> at org.jgroups.protocols.relay.RELAY2.up(RELAY2.java:419)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1048)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:805)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:422)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:638)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:152)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:237)
> at org.jgroups.protocols.Discovery.up(Discovery.java:291)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1828)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Only XSiteReplicateCommand commands expected as a result of xsite calls but got org.infinispan.commands.remote.SingleRpcCommand
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromRemoteSite(CommandAwareRpcDispatcher.java:229)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:207)
> ... 25 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5420) Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5420?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5420:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3434
The fix for ISPN-5106 already fixed the waitForView() problem partially, but it wasn't enough. When the coordinator installed two views in quick succession, the thread updating the cache members list for the first view would block waiting for the {{CacheTopologyControlCommand(POLICY_GET_STATUS)}} response from the other members. Then, because the other members got the newer view before sending the join requests, all the remote-executor and OOB threads would block in {{waitForView()}}, and there would be no way to receive the {{POLICY_GET_STATUS}} responses (since processing a response also needs an OOB thread). The solution was to update the cache members asynchronously.
Testing with a limited amount of OOB/remote-executor threads also exposed some other deadlocks, and the pull request tries to plug as many of them as possible. However, because the caches will not start in the same order on every node, there is always the possibility of 2 nodes sending state transfer requests to each other (for different caches) and not being able to process the responses (because the OOB threads are all blocked, actually waiting for those responses). So a deadlock is still possible if {{remote-executor.max-threads + OOB.max_threads < number of caches}}.
> Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-5420
> URL: https://issues.jboss.org/browse/ISPN-5420
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.2.Final, 7.1.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 8.0.0.Final
>
>
> The join process was designed in the idea that a node would start its caches in sequential order, so {{ClusterTopologyManager.waitForView()}} would block at most once for each joining node. However, WildFly actually starts {{2 * Runtime.availableProcessors()}} caches in parallel, and this can be a problem when the machine has a lot of cores and multiple nodes.
> {{ClustertopologyManager.handleClusterView()}} only updates the {{viewId}} after it updated the cache topologies of each cache AND after it confirmed the availability of all the nodes with a {{POLICY_GET_STATUS}} RPC. This RPC can block, and it's very easy for the remote-executor thread pool on the coordinator to become overloades with threads like this:
> {noformat}
> "remote-thread-172" daemon prio=10 tid=0x00007f0cc48c0000 nid=0x28ca4 in Object.wait() [0x00007f0c5f25b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.infinispan.topology.ClusterTopologyManagerImpl.waitForView(ClusterTopologyManagerImpl.java:357)
> - locked <0x00000000ff3bd900> (a java.lang.Object)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:123)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5420) Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5420?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5420:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1208429|https://bugzilla.redhat.com/show_bug.cgi?id=1208429] from ASSIGNED to POST
> Thread pools are depleted by ClusterTopologyManagerImpl.waitForView() and causing deadlock
> ------------------------------------------------------------------------------------------
>
> Key: ISPN-5420
> URL: https://issues.jboss.org/browse/ISPN-5420
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 6.0.2.Final, 7.1.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 8.0.0.Final
>
>
> The join process was designed in the idea that a node would start its caches in sequential order, so {{ClusterTopologyManager.waitForView()}} would block at most once for each joining node. However, WildFly actually starts {{2 * Runtime.availableProcessors()}} caches in parallel, and this can be a problem when the machine has a lot of cores and multiple nodes.
> {{ClustertopologyManager.handleClusterView()}} only updates the {{viewId}} after it updated the cache topologies of each cache AND after it confirmed the availability of all the nodes with a {{POLICY_GET_STATUS}} RPC. This RPC can block, and it's very easy for the remote-executor thread pool on the coordinator to become overloades with threads like this:
> {noformat}
> "remote-thread-172" daemon prio=10 tid=0x00007f0cc48c0000 nid=0x28ca4 in Object.wait() [0x00007f0c5f25b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at org.infinispan.topology.ClusterTopologyManagerImpl.waitForView(ClusterTopologyManagerImpl.java:357)
> - locked <0x00000000ff3bd900> (a java.lang.Object)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleJoin(ClusterTopologyManagerImpl.java:123)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:162)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:144)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:276)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5221) Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5221?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5221:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1206590|https://bugzilla.redhat.com/show_bug.cgi?id=1206590] from ASSIGNED to POST
> Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5221
> URL: https://issues.jboss.org/browse/ISPN-5221
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Environment: Infinispan 7.1.0Final
> Reporter: Enrico Olivelli
> Assignee: Galder Zamarreño
> Fix For: 7.2.0.Beta1, 7.2.0.Final
>
>
> Using the LAZY nearcache (new in 7.1.0Final) when a java.net.SocketTimeoutException occurs against one of the given HotRod server the RemoteCache becames not usable any more
> This happens very frequently in our DEV environment, but I cannot reproduce it inside a test case
> This is the stacktrace
> {code}
> 15/02/13 14:51:39 ERROR event.ClientListenerNotifier: ISPN004043: Unrecoverable error reading event from server xxx.xxx.xxx/10.168.10.117:11222, exiting event reader thread
> org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
> at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months