[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5208:
-----------------------------------
Forum Reference: http://lists.jboss.org/pipermail/infinispan-dev/2015-March/015897.html
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Final
>
> Attachments: ISPN_5208.java
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5139) MissingFormatArgumentException in trace logging in ClusteredCacheLoaderInterceptor
by Vojtech Juranek (JIRA)
[ https://issues.jboss.org/browse/ISPN-5139?page=com.atlassian.jira.plugin.... ]
Vojtech Juranek commented on ISPN-5139:
---------------------------------------
Same happen with {{ClusteredActivationInterceptor}}:
{noformat}
LogManager error of type FORMAT_FAILURE: Formatting error
java.util.MissingFormatArgumentException: Format specifier 's'
at java.util.Formatter.format(Formatter.java:2487)
at java.util.Formatter.format(Formatter.java:2423)
at java.lang.String.format(String.java:2797)
at org.jboss.logmanager.ExtLogRecord.formatRecord(ExtLogRecord.java:434)
at org.jboss.logmanager.ExtLogRecord.getFormattedMessage(ExtLogRecord.java:397)
at org.jboss.logmanager.formatters.Formatters$10.renderRaw(Formatters.java:568)
at org.jboss.logmanager.formatters.Formatters$JustifyingFormatStep.render(Formatters.java:225)
at org.jboss.logmanager.formatters.MultistepFormatter.format(MultistepFormatter.java:86)
at org.jboss.logmanager.ExtFormatter.format(ExtFormatter.java:35)
at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:49)
at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:79)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:296)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:304)
at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
at org.jboss.logmanager.Logger.log(Logger.java:672)
at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.java:50)
at org.jboss.logging.Logger.logf(Logger.java:2096)
at org.jboss.logging.DelegatingBasicLogger.tracef(DelegatingBasicLogger.java:124)
at org.infinispan.interceptors.ClusteredActivationInterceptor.skipLoadForNonTxCommand(ClusteredActivationInterceptor.java:63)
{noformat}
> MissingFormatArgumentException in trace logging in ClusteredCacheLoaderInterceptor
> ----------------------------------------------------------------------------------
>
> Key: ISPN-5139
> URL: https://issues.jboss.org/browse/ISPN-5139
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.0.2.Final, 7.0.3.Final
> Reporter: Shawn Burgess
> Assignee: Vojtech Juranek
>
> Line 63 of org.infinispan.interceptors.ClusteredCacheLoaderInterceptor
> log.tracef("Skip load for command %s? %s", skip);
> throws
> java.util.MissingFormatArgumentException: Format specifier 's'
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by Matej Čimbora (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
Matej Čimbora updated ISPN-5208:
--------------------------------
Attachment: ISPN_5208.java
Reproducer
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Final
>
> Attachments: ISPN_5208.java
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by Matej Čimbora (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
Matej Čimbora commented on ISPN-5208:
-------------------------------------
Problem still persists, i.e. I'm getting exception as included in description
java.lang.ArrayIndexOutOfBoundsException: 0
at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
...
I managed to create reproducer for this (attached).
1. In clustered.xml, modify numOwners attribute for 'default' cache to 1.
2. Start 2 servers with clustered.xml configuration.
3. Start attached reproducer (client).
4. Kill one of the servers.
5. Start it again. Exception should appear in client log.
This probably relates to losing segments for a subset of keys.
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Final
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years
[JBoss JIRA] (ISPN-4919) Configuration templates
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-4919?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-4919:
---------------------------------------
Here is the latest proposal (from a short chat I've had with [~pferraro] on IRC:
- add a <[local|replicated|distributed|invalidation]-cache-configuration name="..." [extends="..."]?> for defining configurations
- keep the current <[local|replicated|distributed|invalidation]-cache> elements which keep a 1:1 relationship between caches and configurations, also allowing optional "configuration" attribute
- add a <caches names="name1 name2 ..." configuration="..." /> element for multi-cache declarations.
Constraints:
- make sure that the cache mode is "compatible" in the inheritance chain (i.e. a distributed-cache-configuration cannot extend a local-cache-configuration)
> Configuration templates
> -----------------------
>
> Key: ISPN-4919
> URL: https://issues.jboss.org/browse/ISPN-4919
> Project: Infinispan
> Issue Type: Feature Request
> Components: Configuration
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
>
> Currently there is a 1:1 relationship between configuration and named caches. While the programmatic API does have the ability to .read() an existing configuration to create a new one, the declarative config does not.
> We should introduce the concept of configuration inheritance, e.g.:
> {code}
> <local-cache name="eviction-cache">
> <eviction strategy="LIRS" maxEntries="10000"/>
> </local-cache>
> <local-cache name="mycache" template="eviction-cache" />
> {code}
> Possibly, cache templates should be made "abstract" so that they cannot be instantiated as named caches directly, e.g.:
> {code}
> <local-cache name="eviction-cache" abstract="true">
> ...
> </local-cache>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years