[JBoss JIRA] (ISPN-2899) CLI stats command is broken for container
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2899?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-2899:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=915268
> CLI stats command is broken for container
> -----------------------------------------
>
> Key: ISPN-2899
> URL: https://issues.jboss.org/browse/ISPN-2899
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 5.2.3.Final
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Fix For: 5.2.4.Final, 5.3.0.Final
>
>
> Exception need to be solved with Intepreter error
> -------------------Client----------------------------
> [remoting://localhost:9999/local/]> stats --container
> d != java.lang.String
> ------------------Server-----------------------------
> 14:33:48,277 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.util.IllegalFormatConversionException: d != java.lang.String
> at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4045) [rt.jar:1.7.0_11]
> at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2748) [rt.jar:1.7.0_11]
> at java.util.Formatter$FormatSpecifier.print(Formatter.java:2702) [rt.jar:1.7.0_11]
> at java.util.Formatter.format(Formatter.java:2488) [rt.jar:1.7.0_11]
> at java.io.PrintWriter.format(PrintWriter.java:905) [rt.jar:1.7.0_11]
> at java.io.PrintWriter.printf(PrintWriter.java:804) [rt.jar:1.7.0_11]
> at org.infinispan.cli.interpreter.statement.StatsStatement.printContainerStats(StatsStatement.java:91) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.cli.interpreter.statement.StatsStatement.execute(StatsStatement.java:70) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:161) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) [:1.7.0_11]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_11]
> at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_11]
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:287) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_11]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) [rt.jar:1.7.0_11]
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2899) CLI stats command is broken for container
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-2899:
-------------------------------------
Summary: CLI stats command is broken for container
Key: ISPN-2899
URL: https://issues.jboss.org/browse/ISPN-2899
Project: Infinispan
Issue Type: Bug
Components: CLI
Affects Versions: 5.2.3.Final
Reporter: Tristan Tarrant
Assignee: Tristan Tarrant
Fix For: 5.2.4.Final, 5.3.0.Final
Exception need to be solved with Intepreter error
-------------------Client----------------------------
[remoting://localhost:9999/local/]> stats --container
d != java.lang.String
------------------Server-----------------------------
14:33:48,277 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.util.IllegalFormatConversionException: d != java.lang.String
at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4045) [rt.jar:1.7.0_11]
at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2748) [rt.jar:1.7.0_11]
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2702) [rt.jar:1.7.0_11]
at java.util.Formatter.format(Formatter.java:2488) [rt.jar:1.7.0_11]
at java.io.PrintWriter.format(PrintWriter.java:905) [rt.jar:1.7.0_11]
at java.io.PrintWriter.printf(PrintWriter.java:804) [rt.jar:1.7.0_11]
at org.infinispan.cli.interpreter.statement.StatsStatement.printContainerStats(StatsStatement.java:91) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
at org.infinispan.cli.interpreter.statement.StatsStatement.execute(StatsStatement.java:70) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:161) [infinispan-cli-server-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) [:1.7.0_11]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_11]
at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_11]
at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:287) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_11]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) [rt.jar:1.7.0_11]
at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2897) NPE in DefaultConsistentHash
by Michal Linhard (JIRA)
[ https://issues.jboss.org/browse/ISPN-2897?page=com.atlassian.jira.plugin.... ]
Michal Linhard commented on ISPN-2897:
--------------------------------------
This happened in the beginning of some local short test, during the startup of 4 nodes. I don't think I explicitly influenced the cluster to do something.
> NPE in DefaultConsistentHash
> ----------------------------
>
> Key: ISPN-2897
> URL: https://issues.jboss.org/browse/ISPN-2897
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.3.Final
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x
> Fix For: 5.2.4.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: serverlogs.zip
>
>
> Happened in local testing on my laptop with 4 nodes JDG 6.1.0.CR1 (ISPN 5.2.3.Final)
> For some reason there's a rebalance with pendingCH=null and this is not handled well by DefaultConsistentHash.union:
> {code}
> 15:11:04,782 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (OOB-5,shared=udp) Starting local rebalance for cache testCache, topology = CacheTopology{id=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null}
> 15:11:04,782 WARN [org.infinispan.topology.CacheTopologyControlCommand] (OOB-5,shared=udp) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=testCache, type=REBALANCE_START, sender=node04/default, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null, throwable=null, viewId=4}: java.lang.NullPointerException
> at org.infinispan.distribution.ch.DefaultConsistentHash.union(DefaultConsistentHash.java:243) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:120) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:168) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:253) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.RSVP.up(RSVP.java:172) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2897) NPE in DefaultConsistentHash
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2897?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2897:
------------------------------------
>From the logs, it appears the ClusterCacheStatus on node04 (the merge coordinator) already had a rebalance in progress, and when the recovery process that merges cache topologies from all the partitions tried to start a new rebalance it failed.
Then it tried to broadcast a REBALANCE_START command with the wrong topology, but the previous rebalance finished on the coordinator at the same time and because of that, {{pendingCH}} got changed to {{null}}.
Michal, what kind of test were you running? Is it using DISCARD to provoke the split/merge?
> NPE in DefaultConsistentHash
> ----------------------------
>
> Key: ISPN-2897
> URL: https://issues.jboss.org/browse/ISPN-2897
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.3.Final
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x
> Fix For: 5.2.4.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: serverlogs.zip
>
>
> Happened in local testing on my laptop with 4 nodes JDG 6.1.0.CR1 (ISPN 5.2.3.Final)
> For some reason there's a rebalance with pendingCH=null and this is not handled well by DefaultConsistentHash.union:
> {code}
> 15:11:04,782 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (OOB-5,shared=udp) Starting local rebalance for cache testCache, topology = CacheTopology{id=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null}
> 15:11:04,782 WARN [org.infinispan.topology.CacheTopologyControlCommand] (OOB-5,shared=udp) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=testCache, type=REBALANCE_START, sender=node04/default, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null, throwable=null, viewId=4}: java.lang.NullPointerException
> at org.infinispan.distribution.ch.DefaultConsistentHash.union(DefaultConsistentHash.java:243) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:120) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:168) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:253) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.RSVP.up(RSVP.java:172) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2825) ClusterTopologyManagerImpl should not hold a lock while invoking an RPC
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2825?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-2825:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=918887
> ClusterTopologyManagerImpl should not hold a lock while invoking an RPC
> -----------------------------------------------------------------------
>
> Key: ISPN-2825
> URL: https://issues.jboss.org/browse/ISPN-2825
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> On the coordinator, ClusterTopologyManagerImpl holds a lock on a cache's ClusterCacheStatus while it is invoking a synchronous REBALANCE_START or CH_UPDATE command. This helps ensure the ordering of the commands is the same on all the members.
> However, this has some downsides. On a joining node, it takes quite some time before replying to the coordinator (as it needs to request transactions from the other nodes). The nodes that don't need to request any data will send a REBALANCE_CONFIRM command to the coordinator right away, but that command will block on the ClusterCacheStatus lock. If the number of OOB threads is limited, this can even lead to a deadlock.
> Now that CH_UPDATE commands also increment the topology id, we don't really need to enforce the same ordering. If a CH_UPDATE command is sent after a REBALANCE_START command but arrives before it, LocalTopologyManagerImpl just needs to act as if the CH_UPDATE command was actually a REBALANCE_START. (It knows there should be a rebalance when a CH_UPDATE command has pendingCH != null.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2872) CH_UPDATE from new coord may crash rebalance from old coord
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2872?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-2872:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=918887
> CH_UPDATE from new coord may crash rebalance from old coord
> -----------------------------------------------------------
>
> Key: ISPN-2872
> URL: https://issues.jboss.org/browse/ISPN-2872
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.2.Final
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Fix For: 5.2.4.Final, 5.3.0.Alpha1
>
>
> This happened probably the first time, but the issue is here:
> When old coordinator leaves the cluster, it sends a REBALANCE_START as a goodbye. This will trigger rebalance process on some of the nodes. As we do sync GET_TRANSACTIONS, processing this command may take a while.
> However, new coordinator will send CH_UPDATE, which will change the current topologyId to a higher id. This command is processed in LocalTopologyManagerImpl synchronized on cacheStatus, but rebalance command has already left its synchronized block when it executes handler.rebalance.
> Then, as the old REBALANCE_START tries to call notifyTransactionDataReceived in its finally block, it finds out that the topologyId has increased and throws an exception. But the rebalance is left in inconsistent state (activeTopologyUpdates are non-zero, potentionally waitForState true, DataRehash listener notification not called...).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months
[JBoss JIRA] (ISPN-2897) NPE in DefaultConsistentHash
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-2897?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-2897:
--------------------------------
Fix Version/s: 5.2.4.Final
> NPE in DefaultConsistentHash
> ----------------------------
>
> Key: ISPN-2897
> URL: https://issues.jboss.org/browse/ISPN-2897
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.3.Final
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 5.2.x
> Fix For: 5.2.4.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: serverlogs.zip
>
>
> Happened in local testing on my laptop with 4 nodes JDG 6.1.0.CR1 (ISPN 5.2.3.Final)
> For some reason there's a rebalance with pendingCH=null and this is not handled well by DefaultConsistentHash.union:
> {code}
> 15:11:04,782 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (OOB-5,shared=udp) Starting local rebalance for cache testCache, topology = CacheTopology{id=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null}
> 15:11:04,782 WARN [org.infinispan.topology.CacheTopologyControlCommand] (OOB-5,shared=udp) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=testCache, type=REBALANCE_START, sender=node04/default, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[node04/default]}, pendingCH=null, throwable=null, viewId=4}: java.lang.NullPointerException
> at org.infinispan.distribution.ch.DefaultConsistentHash.union(DefaultConsistentHash.java:243) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:120) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.distribution.ch.DefaultConsistentHashFactory.union(DefaultConsistentHashFactory.java:45) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.LocalTopologyManagerImpl.handleRebalance(LocalTopologyManagerImpl.java:228) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:168) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:253) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) [infinispan-core-5.2.3.Final-redhat-1.jar:5.2.3.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.JChannel.up(JChannel.java:707) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.RSVP.up(RSVP.java:172) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) [jgroups-3.2.7.Final-redhat-1.jar:3.2.7.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 10 months