[JBoss JIRA] (ISPN-2886) Using multiple DefaultCacheManager w/o jmx leads to JmxDomainConflictException
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2886?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2886:
--------------------------------
Priority: Minor (was: Major)
> Using multiple DefaultCacheManager w/o jmx leads to JmxDomainConflictException
> ------------------------------------------------------------------------------
>
> Key: ISPN-2886
> URL: https://issues.jboss.org/browse/ISPN-2886
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 5.2.1.Final
> Reporter: Thomas Fromm
> Assignee: Galder Zamarreño
> Priority: Minor
>
> Creating multiple instances of DefaultCacheManager leads to CacheManagerJmxRegistrationTest, even when JMX is disabled in configuration.
> org.infinispan.jmx.JmxDomainConflictException: Domain already registered org.infinispan when trying to register: type=CacheManager,name="DefaultCacheManager"
> at org.infinispan.jmx.JmxUtil.buildJmxDomain(JmxUtil.java:75)
> at org.infinispan.jmx.CacheManagerJmxRegistration.updateDomain(CacheManagerJmxRegistration.java:101)
> at org.infinispan.jmx.CacheManagerJmxRegistration.buildRegistrar(CacheManagerJmxRegistration.java:95)
> at org.infinispan.jmx.AbstractJmxRegistration.registerMBeans(AbstractJmxRegistration.java:59)
> at org.infinispan.jmx.CacheManagerJmxRegistration.start(CacheManagerJmxRegistration.java:63)
> at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:705)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:300)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:171)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2836:
--------------------------------
Labels: onboard (was: )
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Vladimir Blagojevic
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2876) AbstractComponentRegistry.getComponent can return an incomplete component
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2876?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-2876:
-------------------------------------
[~dan.berindei] so adding synchronised around get component should solve the problem then?
> AbstractComponentRegistry.getComponent can return an incomplete component
> -------------------------------------------------------------------------
>
> Key: ISPN-2876
> URL: https://issues.jboss.org/browse/ISPN-2876
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.2.2.Final
> Reporter: Dan Berindei
> Assignee: Mircea Markus
> Fix For: 6.0.0.Final
>
>
> AbstractComponentRegistry.getComponent doesn't have any locking, so it's possible for a thread to retrieve a component while another thread is injecting that component's dependencies.
> Normally this isn't a problem, because the global registry is locked during cache manager startup, and the cache registry is locked during cache startup. However, some global components are only created "on-demand", because a cache component depends on them, and this can happen if someone uses getComponent instead of getOrCreateComponent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2853) Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2853?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2853:
--------------------------------
Labels: onboard (was: )
> Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2853
> URL: https://issues.jboss.org/browse/ISPN-2853
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.1.6.FINAL
> Reporter: William Burns
> Assignee: Adrian Nistor
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: Asymmetric.java
>
>
> We utilize Asymmetric clusters to prevent some unneeded communication between members that don't need to participate in the cluster cache. This works fine for the cache updates to not be sent to that node. However, I noticed that if you have this cache be transactional as well, then members that aren't clustered for this cache will get transaction prepare and commit messages which cause NullPointerExceptions since they don't have remote transactions configured for these nodes.
> Here is a sample test case that shows the error that is found.
> {code}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} [sender=wburns-1521]
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:1:local
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 15167 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} [sender=wburns-1521]
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric}
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:2:local
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 20173 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> {code}
> As a side note, these NPE appear to not be propagated to the client, since they are sent with a response mode of GET_NONE. However we have a site that will every once in a while get the NPE sent back to the updating member which then causes a CacheException to occur forcing the original nodes transaction to be rolled back forcing a retry of the operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2853) Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2853?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2853:
--------------------------------
Priority: Minor (was: Major)
> Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2853
> URL: https://issues.jboss.org/browse/ISPN-2853
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.1.6.FINAL
> Reporter: William Burns
> Assignee: Adrian Nistor
> Priority: Minor
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: Asymmetric.java
>
>
> We utilize Asymmetric clusters to prevent some unneeded communication between members that don't need to participate in the cluster cache. This works fine for the cache updates to not be sent to that node. However, I noticed that if you have this cache be transactional as well, then members that aren't clustered for this cache will get transaction prepare and commit messages which cause NullPointerExceptions since they don't have remote transactions configured for these nodes.
> Here is a sample test case that shows the error that is found.
> {code}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} [sender=wburns-1521]
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:1:local
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 15167 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} [sender=wburns-1521]
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric}
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:2:local
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 20173 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> {code}
> As a side note, these NPE appear to not be propagated to the client, since they are sent with a response mode of GET_NONE. However we have a site that will every once in a while get the NPE sent back to the updating member which then causes a CacheException to occur forcing the original nodes transaction to be rolled back forcing a retry of the operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2853) Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2853?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-2853:
-------------------------------------
reducing priority as this only pollutes the logs
> Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2853
> URL: https://issues.jboss.org/browse/ISPN-2853
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.1.6.FINAL
> Reporter: William Burns
> Assignee: Adrian Nistor
> Priority: Minor
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: Asymmetric.java
>
>
> We utilize Asymmetric clusters to prevent some unneeded communication between members that don't need to participate in the cluster cache. This works fine for the cache updates to not be sent to that node. However, I noticed that if you have this cache be transactional as well, then members that aren't clustered for this cache will get transaction prepare and commit messages which cause NullPointerExceptions since they don't have remote transactions configured for these nodes.
> Here is a sample test case that shows the error that is found.
> {code}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} [sender=wburns-1521]
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric}
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:1:local
> 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 15167 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
> 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} [sender=wburns-1521]
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric}
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:2:local
> 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
> java.lang.NullPointerException
> at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340)
> at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)
> at org.jgroups.JChannel.up(JChannel.java:716)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
> at org.jgroups.protocols.RSVP.up(RSVP.java:192)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> 20173 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
> {code}
> As a side note, these NPE appear to not be propagated to the client, since they are sent with a response mode of GET_NONE. However we have a site that will every once in a while get the NPE sent back to the updating member which then causes a CacheException to occur forcing the original nodes transaction to be rolled back forcing a retry of the operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2851) add log.info when using backported concurrent hash map
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2851?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2851:
--------------------------------
Priority: Minor (was: Major)
> add log.info when using backported concurrent hash map
> ------------------------------------------------------
>
> Key: ISPN-2851
> URL: https://issues.jboss.org/browse/ISPN-2851
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core API
> Affects Versions: 5.2.2.Final
> Reporter: Mathieu Lachance
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 6.0.0.Final
>
>
> it would be nice if there was a bit more logging information when using backported concurrent hash map since it is activation is not only based on the :
> Boolean.getBoolean("infinispan.unsafe.allow_jdk8_chm");
> property as we could think.
> static {
> boolean sunIncompatibleJvm;
> boolean jdk8;
> boolean allowExperimentalMap = Boolean.getBoolean("infinispan.unsafe.allow_jdk8_chm");
> try {
> Class.forName("com.sun.unsafe.Unsafe");
> sunIncompatibleJvm = false;
> } catch (ClassNotFoundException e) {
> sunIncompatibleJvm = true;
> }
>
> try {
> Class.forName("java.util.concurrent.atomic.LongAdder");
> jdk8 = true;
> } catch (ClassNotFoundException e) {
> jdk8 = false;
> }
> if (jdk8 || sunIncompatibleJvm || !allowExperimentalMap){
> MAP_CREATOR = new JdkConcurrentMapCreator();
> }
> else{
> MAP_CREATOR = new BackportedV8ConcurrentMapCreator();
> logger.info("using backported jdk8 concurrent hash map implementation");
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2851) add log.info when using backported concurrent hash map
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2851?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2851:
--------------------------------
Assignee: Galder Zamarreño (was: Mircea Markus)
> add log.info when using backported concurrent hash map
> ------------------------------------------------------
>
> Key: ISPN-2851
> URL: https://issues.jboss.org/browse/ISPN-2851
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core API
> Affects Versions: 5.2.2.Final
> Reporter: Mathieu Lachance
> Assignee: Galder Zamarreño
> Priority: Minor
> Fix For: 6.0.0.Final
>
>
> it would be nice if there was a bit more logging information when using backported concurrent hash map since it is activation is not only based on the :
> Boolean.getBoolean("infinispan.unsafe.allow_jdk8_chm");
> property as we could think.
> static {
> boolean sunIncompatibleJvm;
> boolean jdk8;
> boolean allowExperimentalMap = Boolean.getBoolean("infinispan.unsafe.allow_jdk8_chm");
> try {
> Class.forName("com.sun.unsafe.Unsafe");
> sunIncompatibleJvm = false;
> } catch (ClassNotFoundException e) {
> sunIncompatibleJvm = true;
> }
>
> try {
> Class.forName("java.util.concurrent.atomic.LongAdder");
> jdk8 = true;
> } catch (ClassNotFoundException e) {
> jdk8 = false;
> }
> if (jdk8 || sunIncompatibleJvm || !allowExperimentalMap){
> MAP_CREATOR = new JdkConcurrentMapCreator();
> }
> else{
> MAP_CREATOR = new BackportedV8ConcurrentMapCreator();
> logger.info("using backported jdk8 concurrent hash map implementation");
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months
[JBoss JIRA] (ISPN-2835) Issues w/ M/R test cases if cache are not explicitly started on all nodes
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2835?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2835:
--------------------------------
Issue Type: Bug (was: Feature Request)
> Issues w/ M/R test cases if cache are not explicitly started on all nodes
> -------------------------------------------------------------------------
>
> Key: ISPN-2835
> URL: https://issues.jboss.org/browse/ISPN-2835
> Project: Infinispan
> Issue Type: Bug
> Components: Core API, Distributed Execution and Map/Reduce
> Reporter: Ray Tsang
> Assignee: Galder Zamarreño
> Labels: onboard
> Fix For: 5.3.0.Beta2, 5.3.0.Final
>
> Attachments: mr-test-src.zip
>
>
> I ran into some issues while using M/R. The gist of the issue I was seeing is that:
> Start a cluster of Embedded Caches, like 4 nodes
> Put in 100 elements
> Run a simple M/R job to count the number of keys
> If I run the M/R job using the node I'm inserting elements into as coordinator - the result is 100
> But if I run the M/R job using a different node as coordinator, the result is less than 100
> More interestingly, I can pause for 5 seconds and run the M/R jobs again, the results are always less than 100
> This behavior doens't occur if I explicitly run cacheManager.getCache() for each of the nodes...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 8 months