[JBoss JIRA] (ISPN-2632) Uneven request balancing after node crash
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2632?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2632:
----------------------------------------
Michal, thx for attaching the logs. It only contains TRACE on client but it's Ok.
Taking one client as example (https://gist.github.com/4511311), the number of servers sent back looks right (missing 192.168.87.122), so it doesn't appear to be an issue with the way view is managed. IMO, it seems to be more of an issue with hashing itself.
Looking at the cached positions on the client, I don't see anything wrong at 1st glance: https://gist.github.com/4511569. The positions taken by .122 appear to have gone. Doesn't seem to be a problem with node4 taking more positions on the hash map.
I wonder if this issue has to do with ISPN-2670? I see org.infinispan.client.hotrod.impl.protocol.Codec12 in the logs, but ISPN-2670 seems to imply that Hot Rod client does not work with version 1.2 of the protocol.
I'm gonna tackle ISPN-2670 first and then we can see if the issue is still there.
> Uneven request balancing after node crash
> -----------------------------------------
>
> Key: ISPN-2632
> URL: https://issues.jboss.org/browse/ISPN-2632
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.CR1
> Reporter: Michal Linhard
> Assignee: Galder Zamarreño
> Priority: Critical
> Fix For: 5.2.0.CR2, 5.2.0.Final
>
>
> This is a new manifestation of ISPN-1995, but in this case this happens after killing only one node: the hot rod requests aren't very well balanced.
> these runs still manifest also ISPN-2550 and it may be cause of this bug.
> The uneven balancing of requests can be seen here:
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EDG6/view/EDG-REPOR...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2707) Trace log in EntryWrappingInterceptor could deteriorate performance
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-2707?page=com.atlassian.jira.plugin.... ]
Radim Vansa updated ISPN-2707:
------------------------------
Description:
In {{EntryWrappingInterceptor.visitInvalidateL1Command}} there is a trace log which does not check if the trace is enabled - with invalidate command with a lot of keys (such as in case of ST) this would construct for each of these keys really long strings (with all the keys...), and then throw it off.
Moreover, the InvalidateL1Command is still built even if I don't have L1 enabled, but that's another issue.
was:
In {{EntryWrappingInterceptor.visitInvalidateL1Command}} there is a trace log which does not check if the trace is enabled - with invalidate command with a lot of keys (such as in case of ST) this can construct many really long strings, for each of these keys and then throw it off.
Moreover, the InvalidateL1Command is still built even if I don't have L1 enabled, but that's another issue.
> Trace log in EntryWrappingInterceptor could deteriorate performance
> -------------------------------------------------------------------
>
> Key: ISPN-2707
> URL: https://issues.jboss.org/browse/ISPN-2707
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.0.CR1
> Reporter: Radim Vansa
> Assignee: Mircea Markus
>
> In {{EntryWrappingInterceptor.visitInvalidateL1Command}} there is a trace log which does not check if the trace is enabled - with invalidate command with a lot of keys (such as in case of ST) this would construct for each of these keys really long strings (with all the keys...), and then throw it off.
> Moreover, the InvalidateL1Command is still built even if I don't have L1 enabled, but that's another issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2707) Trace log in EntryWrappingInterceptor could deteriorate performance
by Radim Vansa (JIRA)
Radim Vansa created ISPN-2707:
---------------------------------
Summary: Trace log in EntryWrappingInterceptor could deteriorate performance
Key: ISPN-2707
URL: https://issues.jboss.org/browse/ISPN-2707
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.2.0.CR1
Reporter: Radim Vansa
Assignee: Mircea Markus
In {{EntryWrappingInterceptor.visitInvalidateL1Command}} there is a trace log which does not check if the trace is enabled - with invalidate command with a lot of keys (such as in case of ST) this can construct many really long strings, for each of these keys and then throw it off.
Moreover, the InvalidateL1Command is still built even if I don't have L1 enabled, but that's another issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2550:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> changed the Status of [bug 886565|https://bugzilla.redhat.com/show_bug.cgi?id=886565] from ON_QA to VERIFIED
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2550) NoSuchElementException in Hot Rod Encoder
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2550?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2550:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> made a comment on [bug 886565|https://bugzilla.redhat.com/show_bug.cgi?id=886565]
Verified for 6.1.0.ER8
> NoSuchElementException in Hot Rod Encoder
> -----------------------------------------
>
> Key: ISPN-2550
> URL: https://issues.jboss.org/browse/ISPN-2550
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta4
> Reporter: Michal Linhard
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> Tomas noticed this a while ago in a specific functional test:
> https://bugzilla.redhat.com/show_bug.cgi?id=875151
> I'm creating a more general JIRA, cause I'm having this in resilience test.
> What I found by quick debug, is that here:
> https://github.com/infinispan/infinispan/blob/master/server/hotrod/src/ma...
> {code}
> for (segmentIdx <- 0 until numSegments) {
> val denormalizedSegmentHashIds = allDenormalizedHashIds(segmentIdx)
> val segmentOwners = ch.locateOwnersForSegment(segmentIdx)
> for (ownerIdx <- 0 until segmentOwners.length) {
> val address = segmentOwners(ownerIdx % segmentOwners.size)
> val serverAddress = members(address)
> val hashId = denormalizedSegmentHashIds(ownerIdx)
> log.tracef("Writing hash id %d for %s:%s", hashId, serverAddress.host, serverAddress.port)
> writeString(serverAddress.host, buf)
> writeUnsignedShort(serverAddress.port, buf)
> buf.writeInt(hashId)
> }
> }
> {code}
> we're trying to obtain serverAddress for nonexistent address and NoSuchElementException is not handled properly.
> It hapens after I kill a node in a resilience test and the exception appears when querying for the node in the members cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2706) Minor mistakes in CLI help
by Martin Gencur (JIRA)
Martin Gencur created ISPN-2706:
-----------------------------------
Summary: Minor mistakes in CLI help
Key: ISPN-2706
URL: https://issues.jboss.org/browse/ISPN-2706
Project: Infinispan
Issue Type: Bug
Components: CLI
Affects Versions: 5.2.0.CR1
Reporter: Martin Gencur
Assignee: Tristan Tarrant
Priority: Trivial
Fix For: 5.2.0.CR2
Problems:
1) the help for version command contains incorrect synopsis (redundant square bracket)
2) the upgrade command's help contains on several places words concatenated together (e.g. The only migrator currently available ishotrodwhich migrates ...)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2705) site command does not work correctly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2705?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-2705:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=894321
> site command does not work correctly
> ------------------------------------
>
> Key: ISPN-2705
> URL: https://issues.jboss.org/browse/ISPN-2705
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 5.2.0.CR1
> Reporter: Martin Gencur
> Assignee: Tristan Tarrant
> Fix For: 5.2.0.CR2
>
>
> * Issuing "site --status" command results in NPE:
> {code}
> 13:11:10,278 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.lang.NullPointerException
> at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:56) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:161) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09]
> at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09]
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:270) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_09]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) [rt.jar:1.7.0_09]
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]
> {code}
> * "site --offline" has the same results
> * "site --status default.LON" results in "Incorrect site name: null"
> I'm pretty sure the active site is named LON as I was using an example configuration file shipped with JDG: standalone-xsite.xml. Also, the log shows Received new cluster view: [mgencur/LON|0] [mgencur/LON]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2705) site command does not work correctly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2705?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2705:
-----------------------------------------------
Martin Gencur <mgencur(a)redhat.com> made a comment on [bug 894321|https://bugzilla.redhat.com/show_bug.cgi?id=894321]
Please find the description of this bug in linked ISPN issue.
> site command does not work correctly
> ------------------------------------
>
> Key: ISPN-2705
> URL: https://issues.jboss.org/browse/ISPN-2705
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 5.2.0.CR1
> Reporter: Martin Gencur
> Assignee: Tristan Tarrant
> Fix For: 5.2.0.CR2
>
>
> * Issuing "site --status" command results in NPE:
> {code}
> 13:11:10,278 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.lang.NullPointerException
> at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:56) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:161) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09]
> at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09]
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:270) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_09]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) [rt.jar:1.7.0_09]
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09]
> at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]
> {code}
> * "site --offline" has the same results
> * "site --status default.LON" results in "Incorrect site name: null"
> I'm pretty sure the active site is named LON as I was using an example configuration file shipped with JDG: standalone-xsite.xml. Also, the log shows Received new cluster view: [mgencur/LON|0] [mgencur/LON]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
[JBoss JIRA] (ISPN-2705) site command does not work correctly
by Martin Gencur (JIRA)
Martin Gencur created ISPN-2705:
-----------------------------------
Summary: site command does not work correctly
Key: ISPN-2705
URL: https://issues.jboss.org/browse/ISPN-2705
Project: Infinispan
Issue Type: Bug
Components: CLI
Affects Versions: 5.2.0.CR1
Reporter: Martin Gencur
Assignee: Tristan Tarrant
Fix For: 5.2.0.CR2
* Issuing "site --status" command results in NPE:
{code}
13:11:10,278 ERROR [org.infinispan.cli.interpreter.Interpreter] (pool-1-thread-1) ISPN019003: Interpreter error: java.lang.NullPointerException
at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:56) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:161) [infinispan-cli-server-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09]
at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09]
at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:270) [infinispan-core-5.2.0.CR1-redhat-1.jar:5.2.0.CR1-redhat-1]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) [rt.jar:1.7.0_09]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791) [rt.jar:1.7.0_09]
at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:498) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:246) [jboss-as-jmx-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1034)
at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09]
{code}
* "site --offline" has the same results
* "site --status default.LON" results in "Incorrect site name: null"
I'm pretty sure the active site is named LON as I was using an example configuration file shipped with JDG: standalone-xsite.xml. Also, the log shows Received new cluster view: [mgencur/LON|0] [mgencur/LON]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 3 months
