[JBoss JIRA] (ISPN-2734) KeyAffinityServiceImpl.KeyGeneratorWorker hangs in loop
by Thomas Fromm (JIRA)
[ https://issues.jboss.org/browse/ISPN-2734?page=com.atlassian.jira.plugin.... ]
Thomas Fromm commented on ISPN-2734:
------------------------------------
Forgot to create Threaddump :-(
It was an endless ACTIVE/INACTIVE loop.
> KeyAffinityServiceImpl.KeyGeneratorWorker hangs in loop
> -------------------------------------------------------
>
> Key: ISPN-2734
> URL: https://issues.jboss.org/browse/ISPN-2734
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.0.CR1
> Reporter: Thomas Fromm
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
>
> I found a node in cluster which seems to hang in a loop because of not closed latch, maybe caused due an prior error error.
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as ACTIVE
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as INACTIVE
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as ACTIVE
> No further log available, since I've changed the loglevel of org.infinispan an runtime.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2766) Improve and adapt notification model to fit JSR107 notification model
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-2766?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-2766:
--------------------------------------
Parent: (was: ISPN-2369)
Issue Type: Feature Request (was: Sub-task)
> Improve and adapt notification model to fit JSR107 notification model
> ---------------------------------------------------------------------
>
> Key: ISPN-2766
> URL: https://issues.jboss.org/browse/ISPN-2766
> Project: Infinispan
> Issue Type: Feature Request
> Components: Listeners
> Affects Versions: 5.2.0.CR3
> Reporter: Vladimir Blagojevic
> Assignee: Mircea Markus
>
> Out current notification (listener) model does not fit JSR 107 notification model and thus needs a slight adaptation in order to implement JSR 107.
> More specifically the problem is with JSR107 CacheEntryCreatedListener and CacheEntryExpiredListener.
> The first one is not easy to implement because we need both key/value pair for jsr listener and our CacheEntryCreatedEvent does not provide value.
> The second is related to CacheEntryExpired event. True, spec is not rigorous that such an event has to be fired immediately after an entry has expired but eventually (which might be on access). However, we do not have such an event at all as of now.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2766) Improve and adapt notification model to fit JSR107 notification model
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-2766?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-2766:
--------------------------------------
Parent: ISPN-2639
Issue Type: Sub-task (was: Feature Request)
> Improve and adapt notification model to fit JSR107 notification model
> ---------------------------------------------------------------------
>
> Key: ISPN-2766
> URL: https://issues.jboss.org/browse/ISPN-2766
> Project: Infinispan
> Issue Type: Sub-task
> Components: Listeners
> Affects Versions: 5.2.0.CR3
> Reporter: Vladimir Blagojevic
> Assignee: Mircea Markus
>
> Out current notification (listener) model does not fit JSR 107 notification model and thus needs a slight adaptation in order to implement JSR 107.
> More specifically the problem is with JSR107 CacheEntryCreatedListener and CacheEntryExpiredListener.
> The first one is not easy to implement because we need both key/value pair for jsr listener and our CacheEntryCreatedEvent does not provide value.
> The second is related to CacheEntryExpired event. True, spec is not rigorous that such an event has to be fired immediately after an entry has expired but eventually (which might be on access). However, we do not have such an event at all as of now.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2766) Improve and adapt notification model to fit JSR107 notification model
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-2766:
-----------------------------------------
Summary: Improve and adapt notification model to fit JSR107 notification model
Key: ISPN-2766
URL: https://issues.jboss.org/browse/ISPN-2766
Project: Infinispan
Issue Type: Feature Request
Components: Listeners
Affects Versions: 5.2.0.CR3
Reporter: Vladimir Blagojevic
Assignee: Mircea Markus
Out current notification (listener) model does not fit JSR 107 notification model and thus needs a slight adaptation in order to implement JSR 107.
More specifically the problem is with JSR107 CacheEntryCreatedListener and CacheEntryExpiredListener.
The first one is not easy to implement because we need both key/value pair for jsr listener and our CacheEntryCreatedEvent does not provide value.
The second is related to CacheEntryExpired event. True, spec is not rigorous that such an event has to be fired immediately after an entry has expired but eventually (which might be on access). However, we do not have such an event at all as of now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2766) Improve and adapt notification model to fit JSR107 notification model
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-2766?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-2766:
--------------------------------------
Parent: ISPN-2369
Issue Type: Sub-task (was: Feature Request)
> Improve and adapt notification model to fit JSR107 notification model
> ---------------------------------------------------------------------
>
> Key: ISPN-2766
> URL: https://issues.jboss.org/browse/ISPN-2766
> Project: Infinispan
> Issue Type: Sub-task
> Components: Listeners
> Affects Versions: 5.2.0.CR3
> Reporter: Vladimir Blagojevic
> Assignee: Mircea Markus
>
> Out current notification (listener) model does not fit JSR 107 notification model and thus needs a slight adaptation in order to implement JSR 107.
> More specifically the problem is with JSR107 CacheEntryCreatedListener and CacheEntryExpiredListener.
> The first one is not easy to implement because we need both key/value pair for jsr listener and our CacheEntryCreatedEvent does not provide value.
> The second is related to CacheEntryExpired event. True, spec is not rigorous that such an event has to be fired immediately after an entry has expired but eventually (which might be on access). However, we do not have such an event at all as of now.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2734) KeyAffinityServiceImpl.KeyGeneratorWorker hangs in loop
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2734?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2734:
----------------------------------------
Hmmm, do you mean that after the ACTIVE message you don't see INACTIVE again?
Do you have a thread dump when the hang occurred?
> KeyAffinityServiceImpl.KeyGeneratorWorker hangs in loop
> -------------------------------------------------------
>
> Key: ISPN-2734
> URL: https://issues.jboss.org/browse/ISPN-2734
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.0.CR1
> Reporter: Thomas Fromm
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
>
> I found a node in cluster which seems to hang in a loop because of not closed latch, maybe caused due an prior error error.
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as ACTIVE
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as INACTIVE
> TRACE 21.01.13 09:43:04,653 [pool-44-thread-1] KeyAffinityServiceImpl KeyGeneratorWorker marked as ACTIVE
> No further log available, since I've changed the loglevel of org.infinispan an runtime.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2737) Thread naming anomaly when reporting lock timeout
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2737?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño resolved ISPN-2737.
------------------------------------
Fix Version/s: (was: 5.2.0.Final)
Resolution: Rejected
I don't think it's an issue. If a sender gets a remote lock timeout, the stacktrace will be printed in the log of the sender, cos it will show the stacktrace that happened remotely.
> Thread naming anomaly when reporting lock timeout
> -------------------------------------------------
>
> Key: ISPN-2737
> URL: https://issues.jboss.org/browse/ISPN-2737
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.2.0.CR1
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Priority: Minor
>
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EDG6/view/EDG-REPOR...
> {code}
> 11:47:30,859 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MemcachedServerWorker-277) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [memcachedCache#key763328] for requestor [Thread[OOB-127,null,5,Thread Pools]]! Lock held by [Thread[OOB-150,null,5,Thread Pools]]
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:217) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:200) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:114) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> ....
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2640) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1287) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1850) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1823) [jgroups-3.2.5.Final-redhat-1.jar:3.2.5.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38]
> {code}
> note the thread name "MemcachedServerWorker" in an operation coming from the JGroups stack...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2752) IllegalStateException on shutdown
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2752?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2752:
----------------------------------------
A way to silence this scenario is needed, which should be included as part of the fix for ISPN-2577.
> IllegalStateException on shutdown
> ---------------------------------
>
> Key: ISPN-2752
> URL: https://issues.jboss.org/browse/ISPN-2752
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.0.CR2
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Minor
> Fix For: 5.2.0.Final
>
>
> This happens on graceful shutdown of 8 or 6 nodes of 5.2.0.CR2:
> {code}
> 07:17:45,892 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Remote (node0005/default) failed unexpectedly
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_33]
> at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_33]
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:567) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:432) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:231) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.handleViewChange(ClusterTopologyManagerImpl.java:597) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_33]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_33]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_33]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_33]
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> Caused by: org.infinispan.CacheException: Remote (node0005/default) failed unexpectedly
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:96) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:549) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:546) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_33]
> at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
> Caused by: java.lang.IllegalStateException: transport was closed
> at org.jgroups.blocks.GroupRequest.transportClosed(GroupRequest.java:273) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.stop(RequestCorrelator.java:269) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher.stop(MessageDispatcher.java:152) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher.channelDisconnected(MessageDispatcher.java:455) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.Channel.notifyChannelDisconnected(Channel.java:507) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.jgroups.JChannel.disconnect(JChannel.java:363) [jgroups-3.2.6.Final-redhat-1.jar:3.2.6.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:258) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_33]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_33]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_33]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_33]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:690) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:568) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:260) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) [infinispan-core-5.2.0.CR2-redhat-1.jar:5.2.0.CR2-redhat-1]
> at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) [jboss-datagrid-infinispan-6.1.0.ER9-redhat-1.jar:6.1.0.ER9-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2]
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-2.jar:1.0.2.GA-redhat-2]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months