Asymmetric Transactional Clustered Cache causes NullPointerExceptions on non Clustered members ---------------------------------------------------------------------------------------------- Key: ISPN-2853 URL: https://issues.jboss.org/browse/ISPN-2853 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 5.1.6.FINAL Reporter: William Burns Assignee: Adrian Nistor Priority: Minor Labels: onboard Attachments: Asymmetric.java We utilize Asymmetric clusters to prevent some unneeded communication between members that don't need to participate in the cluster cache. This works fine for the cache updates to not be sent to that node. However, I noticed that if you have this cache be transactional as well, then members that aren't clustered for this cache will get transaction prepare and commit messages which cause NullPointerExceptions since they don't have remote transactions configured for these nodes. Here is a sample test case that shows the error that is found. {code} 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} [sender=wburns-1521] 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:1:local, cacheName=asymmetric} 15164 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:1:local 15164 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command java.lang.NullPointerException at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340) at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601) at org.jgroups.JChannel.up(JChannel.java:716) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) at org.jgroups.protocols.RSVP.up(RSVP.java:192) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 15167 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local 20170 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.VersionAwareMarshaller - Read version 510 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller 20171 [OOB-3,wburns-45269] TRACE org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} [sender=wburns-1521] 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on TxCompletionNotificationCommand{ xid=null, internalId=0, gtx=GlobalTransaction:<wburns-1521>:2:local, cacheName=asymmetric} 20171 [OOB-3,wburns-45269] TRACE org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand - Processing completed transaction GlobalTransaction:<wburns-1521>:2:local 20171 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command java.lang.NullPointerException at org.infinispan.transaction.TransactionTable.removeRemoteTransaction(TransactionTable.java:340) at org.infinispan.commands.remote.recovery.TxCompletionNotificationCommand.perform(TxCompletionNotificationCommand.java:92) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601) at org.jgroups.JChannel.up(JChannel.java:716) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) at org.jgroups.protocols.RSVP.up(RSVP.java:192) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:177) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 20173 [OOB-3,wburns-45269] TRACE org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse {code} As a side note, these NPE appear to not be propagated to the client, since they are sent with a response mode of GET_NONE. However we have a site that will every once in a while get the NPE sent back to the updating member which then causes a CacheException to occur forcing the original nodes transaction to be rolled back forcing a retry of the operation.