[infinispan-issues] [JBoss JIRA] (ISPN-3873) Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)

[Karl von Randow (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935430#comment-12935430 ] 

Karl von Randow commented on ISPN-3873:
---------------------------------------

I have implemented and confirmed the success of a workaround - basically I throw an IllegalStateException if the AbstractInvocationContextFactory.createRemoteInvocationContextForCommand(VisitableCommand, Address) method is called before the context factory has been started.

Because this fault resulted in a hung lock this has been a serious issue for us in production. I'm not sure if this is a permanent fix - it appears to work in production for us: there is no hung lock and no other observed ill effects. As you noted above, there is nothing to invalidate so the fact that we drop the message is probably okay.

I'll submit a pull request so it can be considered, but I'll understand if it doesn't make the cut as a proper solution! 
                
> Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ISPN-3873
>                 URL: https://issues.jboss.org/browse/ISPN-3873
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core API
>    Affects Versions: 6.0.1.Final
>         Environment: Ubuntu Linux 12.04_LTS, Oracle Java SE 1.7.0_45
>            Reporter: Karl von Randow
>            Assignee: Mircea Markus
>             Fix For: 7.0.0.Alpha1
>
>
> Sometimes when I start one server in a cluster I see the NullPointerException reproduced below. This only occurs if there is another server running. I run multiple caches with clustering using invalidation. JGroups using JDBC_PING and TCP. The caches are non-transactional.
> The NPE is because the keyEq (Equivalence instance for keys) is null. I believe this is because the command is processed before the start() method has been called on the AbstractInvocationContextFactory, as the start() method sets the keyEq ivar.
> At this point I'm not sure how to fix it!
> java.lang.NullPointerException
>         at org.infinispan.commons.equivalence.EquivalentHashMap.put(EquivalentHashMap.java:176)
>         at org.infinispan.commons.equivalence.EquivalentHashSet.add(EquivalentHashSet.java:109)
>         at org.infinispan.context.impl.NonTxInvocationContext.addLockedKey(NonTxInvocationContext.java:84)
>         at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:190)
>         at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:181)
>         at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
>         at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:85)
>         at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
>         at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
>         at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
>         at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
>         at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
>         at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
>         at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
>         at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
>         at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
>         at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
>         at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
>         at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
>         at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
>         at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
>         at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
>         at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
>         at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
>         at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
>         at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:186)
>         at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:84)
>         at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:259)
>         at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:211)
>         at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
>         at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
>         at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
>         at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
>         at org.jgroups.JChannel.up(JChannel.java:730)
>         at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019)
>         at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
>         at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
>         at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
>         at org.jgroups.stack.Protocol.up(Protocol.java:409)
>         at org.jgroups.stack.Protocol.up(Protocol.java:409)
>         at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
>         at org.jgroups.protocols.RSVP.up(RSVP.java:221)
>         at org.jgroups.protocols.UNICAST2.removeAndPassUp(UNICAST2.java:919)
>         at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:800)
>         at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:415)
>         at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
>         at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
>         at org.jgroups.protocols.FD.up(FD.java:255)
>         at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
>         at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
>         at org.jgroups.protocols.Discovery.up(Discovery.java:379)
>         at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)
>         at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira