[JBoss JIRA] (ISPN-2538) Endless loop in TreeMap.remove at shutdown of query-enabled cache
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2538?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2538:
--------------------------------
Component/s: Querying
(was: remote query)
> Endless loop in TreeMap.remove at shutdown of query-enabled cache
> -----------------------------------------------------------------
>
> Key: ISPN-2538
> URL: https://issues.jboss.org/browse/ISPN-2538
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 5.2.0.Beta4
> Reporter: Marko Lukša
> Assignee: William Burns
>
> {code}
> "MSC service thread 1-6" prio=6 tid=0x000000000d5b9000 nid=0x10a0 runnable [0x000000000e45e000]
> java.lang.Thread.State: RUNNABLE
> at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2215)
> at java.util.TreeMap.deleteEntry(TreeMap.java:2173)
> at java.util.TreeMap.remove(TreeMap.java:600)
> at org.infinispan.query.impl.LifecycleManager.cacheStopped(LifecycleManager.java:181)
> at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:227)
> at org.infinispan.CacheImpl.stop(CacheImpl.java:540)
> at org.infinispan.CacheImpl.stop(CacheImpl.java:535)
> at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
> at org.jboss.as.capedwarf.services.CacheLifecycleService.stop(CacheLifecycleService.java:94)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-3450) Jon plugin - failing RebuildIndex operation on [MassIndexer] component
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3450?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3450:
--------------------------------
Component/s: JMX, reporting and management
> Jon plugin - failing RebuildIndex operation on [MassIndexer] component
> ----------------------------------------------------------------------
>
> Key: ISPN-3450
> URL: https://issues.jboss.org/browse/ISPN-3450
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Reporter: Tomas Sykora
> Assignee: Adrian Nistor
>
> RebuildIndex operation on [MassIndexer] component is failing while issuing in JON with this error:
> java.lang.NullPointerException
> at org.infinispan.rhq.CacheComponent.invokeOperation(CacheComponent.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> I reused configuration from ISPN test suite + I was able to issue this operation successfully via jconsole tool. That's why I suspect that problems are somewhere in JON-ISPN interaction or plugin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2710) Huge amount of OOB threads during performance test
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2710?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2710:
--------------------------------
Component/s: Core
> Huge amount of OOB threads during performance test
> --------------------------------------------------
>
> Key: ISPN-2710
> URL: https://issues.jboss.org/browse/ISPN-2710
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 5.2.0.CR1
> Reporter: Robert Stupp
> Assignee: Mircea Markus
>
> While running our performance test (as described in ISPN-2240), two of the four servers are running at 80 to 100% CPU - while the others just run at 10%.
> Before that phenomenom a huge amount (several 100s) of threads has been created (all called {{OOB-xxxx}}.
> The performance test just reads cached data - there was no cache put operation at that time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2710) Huge amount of OOB threads during performance test
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2710?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2710:
--------------------------------
Priority: Minor (was: Major)
> Huge amount of OOB threads during performance test
> --------------------------------------------------
>
> Key: ISPN-2710
> URL: https://issues.jboss.org/browse/ISPN-2710
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Affects Versions: 5.2.0.CR1
> Reporter: Robert Stupp
> Assignee: Mircea Markus
> Priority: Minor
>
> While running our performance test (as described in ISPN-2240), two of the four servers are running at 80 to 100% CPU - while the others just run at 10%.
> Before that phenomenom a huge amount (several 100s) of threads has been created (all called {{OOB-xxxx}}.
> The performance test just reads cached data - there was no cache put operation at that time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2853) Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2853?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2853:
--------------------------------
Component/s: Core
> Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2853
> URL: https://issues.jboss.org/browse/ISPN-2853
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.6.FINAL
> Reporter: William Burns
> Assignee: Adrian Nistor
> Priority: Minor
> Labels: onboard
> Attachments: Asymmetric.java
>
>
> We utilize Asymmetric clusters to prevent some unneeded communication between members that don't need to participate in the cluster cache. This works fine for the cache updates to not be sent to that node. However, I noticed that if you have this cache be transactional as well, then members that aren't clustered for this cache will get transaction prepare and commit messages which cause NullPointerExceptions since they don't have remote transactions configured for these nodes.
> Here is a sample test case that shows the error that is found.
> {code}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} [sender=wburns-1521]
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:1:local
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 15167 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} [sender=wburns-1521]
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric}
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:2:local
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 20173 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> {code}
> As a side note, these NPE appear to not be propagated to the client, since they are sent with a response mode of GET_NONE. However we have a site that will every once in a while get the NPE sent back to the updating member which then causes a CacheException to occur forcing the original nodes transaction to be rolled back forcing a retry of the operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-61) Dynamically switch between single and two phase commit based on cluster size
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-61?page=com.atlassian.jira.plugin.sy... ]
Mircea Markus resolved ISPN-61.
-------------------------------
Resolution: Rejected
total order is a better alternative
> Dynamically switch between single and two phase commit based on cluster size
> ----------------------------------------------------------------------------
>
> Key: ISPN-61
> URL: https://issues.jboss.org/browse/ISPN-61
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 4.0.0.ALPHA4
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> Following 2 optimizations might be implemented with respect to transactions.
> 1) From an email on infinispan-dev (horizon-dev actually):
> if there are only two members int the cluster always use an 1PC. If the 1st phase fails remotely, then also rollback locally. This would reduce one network roundtrip.
> While this is an interesting thought, it does raise the potential for race conditions - since this decision will have to be taken in the TxInterceptor in the beforeCompletion phase of a transaction, and by the time the call gets to the interceptor for replication, the topology may have changed such that you need to replicate to 2 instead of 1 other peer. Which would mean a 2PC again. So it does need some thought.
> 2) when asked to prepare, a participant might return a value indicating that no changes were made (read-only participant), so this one won't need an commit message, so less roundtrip.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months