[JBoss JIRA] (ISPN-2827) rhq-plugin does not work anymore
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2827?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-2827:
---------------------------------------
The plugin has had a few slight changes (a couple of bugfixes regarding operations, and the addition of a few missing ops/attributes).
You should set the agent's log4j.xml to DEBUG and then issue a "discovery --full" command and check the logs there.
I notice you're using a RHQ snapshot, maybe the changes are triggering a bug in there. Can you try with a released version ?
> rhq-plugin does not work anymore
> --------------------------------
>
> Key: ISPN-2827
> URL: https://issues.jboss.org/browse/ISPN-2827
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 5.2.1.Final
> Environment: RHQ 4.6.0-SNAPSHOT Build-Nummer: 6917de4
> Linux
> Reporter: Stan Nowogrudski steam
> Assignee: Mircea Markus
> Fix For: 5.2.2.Final
>
>
> After update to ISPN 5.2.1 and RHQ plugin 5.2.1 final is RHQ plugin not working properly any more. By klicking on on of ISPN Cache MBeans on the JMX Servers tree in RHQ, the RHQ notifies about uncatched exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (ISPN-2802) Cache recovery fails due to missing responses
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2802?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-2802:
----------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
> Cache recovery fails due to missing responses
> ---------------------------------------------
>
> Key: ISPN-2802
> URL: https://issues.jboss.org/browse/ISPN-2802
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.CR3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
>
> When the cache recovery is started, the new coordinator sends CacheTopologyControlCommand.GET_STATUS to all nodes and waits for responses. However, I have a reproducible test-case where it always times out waiting for the responses.
> Here are the logs (TRACE is not doable here, but I added some byteman traces - see topology.btm in the archive): http://dl.dropbox.com/u/103079234/recovery.zip
> The problematic spot is on node3 at 05:37:57 receiving cluster view 34.
> All nodes (except the one which is killed, in this case node1) respond quickly to the GET_STATUS command (see BYTEMAN Receiving - Received pairs, these are bound to command execution in CommandAwareRpcDispatcher), but some responses are not received on node3 (look for Receiving rsp bound to GroupRequest).
> JGroups tracing could be useful here but it is not available (intensive logging often blocks on internal log4j locks and the node becomes unresponsive).
> As mentioned above, the case is reproducible, therefore if you can suggest any particular BYTEMAN hook, I can try it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (ISPN-2827) rhq-plugin does not work anymore
by Stan Nowogrudski steam (JIRA)
[ https://issues.jboss.org/browse/ISPN-2827?page=com.atlassian.jira.plugin.... ]
Stan Nowogrudski steam commented on ISPN-2827:
----------------------------------------------
I have already set the RHQ server into debug mode (rhq.server.log-level=DEBUG)
There is nothing suspicious in the RHQ server log, just this:
15:53:02,832 WARN [org.hibernate.hql.internal.ast.QueryTranslatorImpl] (http--0.0.0.0-7080-9) HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
15:53:03,077 WARN [org.hibernate.hql.internal.ast.QueryTranslatorImpl] (http--0.0.0.0-7080-9) HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
My suspicion is - structure of ISPN JMX MBeans was changed, or something like this.
How can I see some more debugging information?
> rhq-plugin does not work anymore
> --------------------------------
>
> Key: ISPN-2827
> URL: https://issues.jboss.org/browse/ISPN-2827
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 5.2.1.Final
> Environment: RHQ 4.6.0-SNAPSHOT Build-Nummer: 6917de4
> Linux
> Reporter: Stan Nowogrudski steam
> Assignee: Mircea Markus
> Fix For: 5.2.2.Final
>
>
> After update to ISPN 5.2.1 and RHQ plugin 5.2.1 final is RHQ plugin not working properly any more. By klicking on on of ISPN Cache MBeans on the JMX Servers tree in RHQ, the RHQ notifies about uncatched exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (ISPN-2827) rhq-plugin does not work anymore
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2827?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-2827:
---------------------------------------
We need a server-side log, that looks like a javascript error
> rhq-plugin does not work anymore
> --------------------------------
>
> Key: ISPN-2827
> URL: https://issues.jboss.org/browse/ISPN-2827
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Affects Versions: 5.2.1.Final
> Environment: RHQ 4.6.0-SNAPSHOT Build-Nummer: 6917de4
> Linux
> Reporter: Stan Nowogrudski steam
> Assignee: Mircea Markus
> Fix For: 5.2.2.Final
>
>
> After update to ISPN 5.2.1 and RHQ plugin 5.2.1 final is RHQ plugin not working properly any more. By klicking on on of ISPN Cache MBeans on the JMX Servers tree in RHQ, the RHQ notifies about uncatched exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (ISPN-2781) NPE in AbstractTopologyAwareEncoder1x
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2781?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2781:
------------------------------------
Galderz, yes, this is again a problem with the topology cache being modified while we are writing a topology response to the client.
Earlier versions of the code had the same problem - we don't hold any lock while generating the client response, so anyone can modify the topology cache while we are generating the response, and this was always the case. Note that a new client will always require a topology response, regardless of when the topology has last changed.
At one point I did add an extra check to skip a node's hash from the topology response if it wasn't in the topology cache any more, but I thought (wrongly) that the extra membership check that you pointed out made it impossible and I deleted it.
I'm going to take a snapshot of the topology cache before generating the response and I'm going to use that copy for everything. Topology changes are very rare anyway, so it shouldn't make a difference performance-wise, and it will make the code easier to reason about.
> NPE in AbstractTopologyAwareEncoder1x
> -------------------------------------
>
> Key: ISPN-2781
> URL: https://issues.jboss.org/browse/ISPN-2781
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 5.2.0.CR3
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Fix For: 5.2.2.Final, 5.3.0.Final
>
>
> got this in an elasticity test (32nodes)
> {code}
> 06:23:20,660 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-151) ISPN005009: Unexpected error before any request parameters read: java.lang.NullPointerException
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:102) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate11(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate(AbstractTopologyAwareEncoder1x.scala:52) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.AbstractEncoder1x.writeHeader(AbstractEncoder1x.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.1.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:179) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.hotrod.HotRodDecoder.customDecodeHeader(HotRodDecoder.scala:157) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:105) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:70) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.1.Final.jar:]
> at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:313) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.1.Final.jar:]
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.1.Final.jar:]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month
[JBoss JIRA] (ISPN-2402) Cache operations or transactions should never fail with SuspectException
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2402?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2402:
------------------------------------
If you're seeing the failures in MultiNodeReplicatedTest.testIndexingWorkDistribution(), ISPN-2648 is probably the most appropriate. If it's another test that doesn't have a JIRA yet, please create a new JIRA and post the log as well.
> Cache operations or transactions should never fail with SuspectException
> ------------------------------------------------------------------------
>
> Key: ISPN-2402
> URL: https://issues.jboss.org/browse/ISPN-2402
> Project: Infinispan
> Issue Type: Task
> Components: RPC, State transfer
> Affects Versions: 5.2.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.3.0.Alpha1
>
> Attachments: vrstt.log
>
>
> This is an extension of ISPN-1896 of sorts, but for all the cache operations that are visible to the user.
> After a node leaves, the other nodes that have sent commands to that node should either ignore SuspectExceptions or, if not possible, they should retry the operation (e.g. if they didn't get any response back).
> For example, VersionReplStateTransferTest quite often on my machine with a SuspectException, because the versioned prepare command expects a response from the coordinator and the coordinator has just left.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 1 month