[JBoss JIRA] (ISPN-3876) TcpTransportFactory stores failed SocketAddress in RequestBalancingStrategy
by Patrick Seeber (JIRA)
[ https://issues.jboss.org/browse/ISPN-3876?page=com.atlassian.jira.plugin.... ]
Patrick Seeber commented on ISPN-3876:
--------------------------------------
Thank you for your answer!
Indeed it is not self healing and we believe we found the problem.
The scenario is:
1. Start 2 servers in replicated mode
2. Start a Client which connects to both servers correctly
3. Shutdown Server 1 during maintenance
4. Perform a client operation, the new topology is now commited to the client, the balancer now knows only Server 2
5. Bring up server 1 again and shutdown server 2 during maintenance
6. Bring up server 2 again
Now the client is broken since the last server he knows (Server 2) wont send any topology changes to the client if the client performs operations now. There is no chance for the client balancer to receive both servers again, which is problematic if the communication to the only known server fails now.
To fix this, we would be forced to perform a client operation between step 5 and 6, but we have 10 clients and we do not want to trigger a getCache 10 times each time we shutdown a server.
Is there any way to ping the servers from the client in a specific intercal or to directly inform the client if the topology changes?
> TcpTransportFactory stores failed SocketAddress in RequestBalancingStrategy
> ---------------------------------------------------------------------------
>
> Key: ISPN-3876
> URL: https://issues.jboss.org/browse/ISPN-3876
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final, 5.3.0.Final, 6.0.0.Final
> Environment: Hotrod Client, Java
> Reporter: Patrick Seeber
> Assignee: Mircea Markus
>
> The "updateServers" Method in the TcpTransportFactory class iterates over all addedServers and adds them to the connection pool if no exceptions are thrown. Howerver, if an exception is thrown, the SocketAddress may not have been added to the conection pool but is added to the balancer afterwards. Therefore, the balancer may contain an invalid SocketAddress which is not contained in the connection pool.
> In our application with few distributed caches, we encounter situations where all servers (SocketAddresses) are corrupt and the application fails to load or store entries in/from the cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-2724) L1 is not used for write-operations (for node affine keys)
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-2724?page=com.atlassian.jira.plugin.... ]
William Burns resolved ISPN-2724.
---------------------------------
Resolution: Rejected
L1 is only updated from a non owner write when it is in tx mode. The reason for this is the transaction boundaries allow for atomic updates where as non tx doesn't have this since the atomicity is only guaranteed while updating on the owner node. Thus to guarantee consistency we err on the side of invalidating L1 values during periods when consistency is questionable.
More details can be found on ISPN-3197
> L1 is not used for write-operations (for node affine keys)
> ----------------------------------------------------------
>
> Key: ISPN-2724
> URL: https://issues.jboss.org/browse/ISPN-2724
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.0.CR1
> Reporter: Thomas Fromm
> Assignee: William Burns
> Priority: Minor
> Attachments: L1WriteTest.java
>
>
> I need to make sure, that a key written on a node is present in L1, even when the owner is a different node.
> Attached an example which shows the expected behaviour.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-3873) Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3873?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3873:
------------------------------------
Thanks Karl, I think I found the problem: in non-tx caches, replicated commands are wrapped in SingleRpcCommands, and the topology id of the SingleRpcCommands is always -1. So InboundInvocationHandlerImpl was waiting for the wrong topology id, and in fact it wasn't waiting at all.
You do have a point that AbstractInvocationContextFactory should not be able to create invalid context instances, but I think that would actually be simpler to do by initializing the key Equivalence in the init() method.
> Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3873
> URL: https://issues.jboss.org/browse/ISPN-3873
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.1.Final
> Environment: Ubuntu Linux 12.04_LTS, Oracle Java SE 1.7.0_45
> Reporter: Karl von Randow
> Assignee: Mircea Markus
> Fix For: 7.0.0.Alpha1
>
> Attachments: infinispan.trace.gz
>
>
> Sometimes when I start one server in a cluster I see the NullPointerException reproduced below. This only occurs if there is another server running. I run multiple caches with clustering using invalidation. JGroups using JDBC_PING and TCP. The caches are non-transactional.
> The NPE is because the keyEq (Equivalence instance for keys) is null. I believe this is because the command is processed before the start() method has been called on the AbstractInvocationContextFactory, as the start() method sets the keyEq ivar.
> At this point I'm not sure how to fix it!
> java.lang.NullPointerException
> at org.infinispan.commons.equivalence.EquivalentHashMap.put(EquivalentHashMap.java:176)
> at org.infinispan.commons.equivalence.EquivalentHashSet.add(EquivalentHashSet.java:109)
> at org.infinispan.context.impl.NonTxInvocationContext.addLockedKey(NonTxInvocationContext.java:84)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:190)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:181)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:85)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:186)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:84)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:259)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:211)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:730)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
> at org.jgroups.stack.Protocol.up(Protocol.java:409)
> at org.jgroups.stack.Protocol.up(Protocol.java:409)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
> at org.jgroups.protocols.RSVP.up(RSVP.java:221)
> at org.jgroups.protocols.UNICAST2.removeAndPassUp(UNICAST2.java:919)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:800)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:415)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD.up(FD.java:255)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-3873) Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)
by Karl von Randow (JIRA)
[ https://issues.jboss.org/browse/ISPN-3873?page=com.atlassian.jira.plugin.... ]
Karl von Randow updated ISPN-3873:
----------------------------------
Attachment: infinispan.trace.gz
Please find attached a trace log from our Tomcat server of the server startup. I have included the log from startup to a bit after the NullPointerException (search for that to find the failure point).
I note that the caches appear to be receiving InvalidateCommands prior to this point, and ignoring them because caches aren't defined (e.g. Silently ignoring that supermodel.findByCode cache is not defined). There are topology messages all through. Hopefully there's some sense in there for you!
I agree that the spot I've patched is definitely not the right one. However perhaps it's a useful safety measure, as the way the fault surfaces is quite detached from this class even though this is where the fault originates?
> Race condition in AbstractInvocationContextFactory (results in NullPointerException in EquivalentHashMap when receiving an invalidate command during startup)
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3873
> URL: https://issues.jboss.org/browse/ISPN-3873
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.1.Final
> Environment: Ubuntu Linux 12.04_LTS, Oracle Java SE 1.7.0_45
> Reporter: Karl von Randow
> Assignee: Mircea Markus
> Fix For: 7.0.0.Alpha1
>
> Attachments: infinispan.trace.gz
>
>
> Sometimes when I start one server in a cluster I see the NullPointerException reproduced below. This only occurs if there is another server running. I run multiple caches with clustering using invalidation. JGroups using JDBC_PING and TCP. The caches are non-transactional.
> The NPE is because the keyEq (Equivalence instance for keys) is null. I believe this is because the command is processed before the start() method has been called on the AbstractInvocationContextFactory, as the start() method sets the keyEq ivar.
> At this point I'm not sure how to fix it!
> java.lang.NullPointerException
> at org.infinispan.commons.equivalence.EquivalentHashMap.put(EquivalentHashMap.java:176)
> at org.infinispan.commons.equivalence.EquivalentHashSet.add(EquivalentHashSet.java:109)
> at org.infinispan.context.impl.NonTxInvocationContext.addLockedKey(NonTxInvocationContext.java:84)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:190)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:181)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateCommand(AbstractLockingInterceptor.java:85)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:111)
> at org.infinispan.commands.write.InvalidateCommand.acceptVisitor(InvalidateCommand.java:118)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)
> at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:186)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:84)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:259)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:211)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:730)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
> at org.jgroups.stack.Protocol.up(Protocol.java:409)
> at org.jgroups.stack.Protocol.up(Protocol.java:409)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
> at org.jgroups.protocols.RSVP.up(RSVP.java:221)
> at org.jgroups.protocols.UNICAST2.removeAndPassUp(UNICAST2.java:919)
> at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:800)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:415)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD.up(FD.java:255)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-3892) Tune for batching without transactions
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-3892?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero closed ISPN-3892.
---------------------------------
Resolution: Duplicate Issue
Created duplicate to ISPN-3891 - sorry network timeouts had me retry
> Tune for batching without transactions
> --------------------------------------
>
> Key: ISPN-3892
> URL: https://issues.jboss.org/browse/ISPN-3892
> Project: Infinispan
> Issue Type: Enhancement
> Components: Transactions
> Reporter: Sanne Grinovero
> Assignee: Mircea Markus
> Fix For: 7.0.0.Alpha1, 7.0.0.Final
>
>
> The usage of batching is a critically important feature, widely used to improve performance in many common scenarios, but while documentation makes it clear that it's built on transactions, it's unclear how these transactions should be configured.
> Also, I would challenge the decision of building batching on top of transactions and see if there are opportunities for further optimization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-3892) Tune for batching without transactions
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-3892:
-------------------------------------
Summary: Tune for batching without transactions
Key: ISPN-3892
URL: https://issues.jboss.org/browse/ISPN-3892
Project: Infinispan
Issue Type: Enhancement
Components: Transactions
Reporter: Sanne Grinovero
Assignee: Mircea Markus
Fix For: 7.0.0.Alpha1, 7.0.0.Final
The usage of batching is a critically important feature, widely used to improve performance in many common scenarios, but while documentation makes it clear that it's built on transactions, it's unclear how these transactions should be configured.
Also, I would challenge the decision of building batching on top of transactions and see if there are opportunities for further optimization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months
[JBoss JIRA] (ISPN-3891) Tune for batching without transactions
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-3891:
-------------------------------------
Summary: Tune for batching without transactions
Key: ISPN-3891
URL: https://issues.jboss.org/browse/ISPN-3891
Project: Infinispan
Issue Type: Enhancement
Components: Transactions
Reporter: Sanne Grinovero
Assignee: Mircea Markus
Fix For: 7.0.0.Alpha1, 7.0.0.Final
The usage of batching is a critically important feature, widely used to improve performance in many common scenarios, but while documentation makes it clear that it's built on transactions, it's unclear how these transactions should be configured.
Also, I would challenge the decision of building batching on top of transactions and see if there are opportunities for further optimization.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 2 months