[JBoss JIRA] (ISPN-9345) TimeutException involving the org.infinispan.CONFIG cache
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-9345?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-9345:
--------------------------------
OK, thanks for the update!
All protocol headers have small (fixed) sizes and the headers added to a message below FRAG3 are typically retransmission (NAKACK2 or UNICAST3) and transport (UDP or TCP), so reducing max_bundle_size by this offset should work in most cases.
For TCP, max_bundle_size could be quite large, but I guess this also increases the burden on memory as message are collected until they're sent. If you use {{no-bundler}}, then max_bundle_size is moot, anyway.
> TimeutException involving the org.infinispan.CONFIG cache
> ---------------------------------------------------------
>
> Key: ISPN-9345
> URL: https://issues.jboss.org/browse/ISPN-9345
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Dan Berindei
>
> {noformat}
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on jedha-64980
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9345) TimeutException involving the org.infinispan.CONFIG cache
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9345?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-9345:
------------------------------------
[~belaban] after reading some more about IPv6 I realized I jumped to conclusions: the do-not-fragment rule is only for routers, and the network stack on the sender is free to fragment the packets:
{quote}
In order to send a packet that is too large to fit in the MTU of the
path to its destination, a source node may divide the packet into
fragments and send each fragment as a separate packet, to be
reassembled at the receiver.
{quote}
These logs from `tcpdump` actually show that a packet was fragmented, and the "bad length" message is logged by tcpdump every time the UDP datagram size is bigger than the IP packet size:
{noformat}
09:30:32.070507 IP6 (class 0x08, flowlabel 0xbb8fe, hlim 2, next-header Fragment (44) payload length: 1456) denulu-tp3 > ff0e::e406:708: frag (0xbbd85d01:0|1448) 56784 > 46655: UDP, bad length 1453 > 1440
09:30:32.070563 IP6 (class 0x08, flowlabel 0xbb8fe, hlim 2, next-header Fragment (44) payload length: 21) denulu-tp3 > ff0e::e406:708: frag (0xbbd85d01:1448|13)
{noformat}
This is a non-fragmented packet, with the length just below the threshold:
{noformat}
09:31:21.935633 IP6 (class 0x08, flowlabel 0x839b7, hlim 2, next-header UDP (17) payload length: 1460) denulu-tp3.57807 > ff0e::e406:708.46655: [udp sum ok] UDP, length 1452
{noformat}
Initially I thought it was a problem with the wireless router I was connected to at the time. But I was able to reproduce the problem by connecting only to an ethernet switch, so I'm now convinced the problem is in my machine's IPv6 stack.
{quote}
Sending a large packet at startup is too cumbersome, but this could be integrated into some configuration task perhaps?
{quote}
Well, it wouldn't be a simple configuration change, but we can surely modify {{FRAGx}} to send a message with {{frag_size}} bytes on startup/view change and log an error if there is no confirmation from the coordinator?
I'm not sure what to do with all the protocols below {{FRAGx}} that send data in the message body. Maybe they can check {{TP.max_bundle_size}} and do their own fragmentation based on that? The only inconvenience would be that {{TP.max_bundle_size}} is always set, even when using {{TCP}}, and we wouldn't need fragmentation there.
> TimeutException involving the org.infinispan.CONFIG cache
> ---------------------------------------------------------
>
> Key: ISPN-9345
> URL: https://issues.jboss.org/browse/ISPN-9345
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Dan Berindei
>
> {noformat}
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on jedha-64980
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9352) Report cache read/write/remote stats also as nanoseconds
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-9352:
-------------------------------------
Summary: Report cache read/write/remote stats also as nanoseconds
Key: ISPN-9352
URL: https://issues.jboss.org/browse/ISPN-9352
Project: Infinispan
Issue Type: Enhancement
Components: JMX, reporting and management
Affects Versions: 9.3.0.Final
Reporter: Tristan Tarrant
Assignee: Tristan Tarrant
Fix For: 9.4.0.Alpha1, 9.4.0.Final
Add three new stats that report read/write/remove averages in nanoseconds.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9345) TimeutException involving the org.infinispan.CONFIG cache
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9345?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-9345:
----------------------------------
Assignee: Dan Berindei
> TimeutException involving the org.infinispan.CONFIG cache
> ---------------------------------------------------------
>
> Key: ISPN-9345
> URL: https://issues.jboss.org/browse/ISPN-9345
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Dan Berindei
>
> {noformat}
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on jedha-64980
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9345) TimeutException involving the org.infinispan.CONFIG cache
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9345?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9345:
-------------------------------
Status: Open (was: New)
> TimeutException involving the org.infinispan.CONFIG cache
> ---------------------------------------------------------
>
> Key: ISPN-9345
> URL: https://issues.jboss.org/browse/ISPN-9345
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.3.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Dan Berindei
>
> {noformat}
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on jedha-64980
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-8905) Segment-aware non-shared cache stores
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8905?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-8905:
--------------------------------
Status: Open (was: New)
> Segment-aware non-shared cache stores
> -------------------------------------
>
> Key: ISPN-8905
> URL: https://issues.jboss.org/browse/ISPN-8905
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: William Burns
> Assignee: William Burns
> Fix For: 9.4.0.Final
>
>
> Non shared stores should allow for segment based store separation. This might involve creating a store per segment. But this allows for superior iteration performance over a subset of segments.
> Distributed stores benefit the most from this due to the fact that iteration is done at the segment level to help ensure that duplicate entries are not retrieved. It also would be beneficial for state transfer and other operations that operate only upon a given set of segments.
> This could be advantageous even for REPL and LOCAL caches as there is then a very clear separation of stores to process in parallel for given operations. This would have to be verified with tests to see if this is worth it though.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-8905) Segment-aware non-shared cache stores
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8905?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-8905:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6131
> Segment-aware non-shared cache stores
> -------------------------------------
>
> Key: ISPN-8905
> URL: https://issues.jboss.org/browse/ISPN-8905
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: William Burns
> Assignee: William Burns
> Fix For: 9.4.0.Final
>
>
> Non shared stores should allow for segment based store separation. This might involve creating a store per segment. But this allows for superior iteration performance over a subset of segments.
> Distributed stores benefit the most from this due to the fact that iteration is done at the segment level to help ensure that duplicate entries are not retrieved. It also would be beneficial for state transfer and other operations that operate only upon a given set of segments.
> This could be advantageous even for REPL and LOCAL caches as there is then a very clear separation of stores to process in parallel for given operations. This would have to be verified with tests to see if this is worth it though.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9351) nodes did not join to cluster because of timeoutException
by Robert Cernak (JIRA)
[ https://issues.jboss.org/browse/ISPN-9351?page=com.atlassian.jira.plugin.... ]
Robert Cernak updated ISPN-9351:
--------------------------------
Attachment: 15nodesTryingToJoin1ClusterHowever5DidNotJoin .zip
> nodes did not join to cluster because of timeoutException
> ---------------------------------------------------------
>
> Key: ISPN-9351
> URL: https://issues.jboss.org/browse/ISPN-9351
> Project: Infinispan
> Issue Type: Bug
> Reporter: Robert Cernak
> Attachments: 15nodesTryingToJoin1ClusterHowever5DidNotJoin .zip
>
>
> I was trying to connect 15nodes to 1 cluster, however nodes did not join.
> In logs, all nodes mostly had 2 kinds of exceptions, all caused by TimeoutException:
> 1:
> {noformat}
> 2018-07-03 07:35:31,670 DEBUG [Camel (camel-1) thread #0 - seda://systemInitializer] (LocalTopologyManagerImpl.java:169) - Error sending join request for cache org.infinispan.CONFIG to coordinator
> org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
> following by next exception 1 second after...
> 2018-07-03 07:35:31,968 INFO [Camel (camel-1) thread #0 - seda://systemInitializer] (JGroupsTransport.java:702) - ISPN000080: Disconnecting JGroups channel cloud11-15
> 2018-07-03 07:35:32,262 DEBUG [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:709) - Stopping cache manager cloud11-15 on null
> 2018-07-03 07:35:32,273 WARN [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:736) - ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
> java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at java.util.concurrent.CompletableFuture.reportJoin(Unknown Source) ~[?:1.8.0_131]
> at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:1.8.0_131]
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:688) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:734) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:711) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
> {noformat}
> 2:
> {noformat}
> 2018-07-03 09:12:55,788 WARN [Camel (camel-1) thread #2 - seda://northboundProvider] (TransactionImpl.java:429) - ISPN000927: exception while committing
> javax.transaction.xa.XAException: null
> at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.commons.tx.TransactionManagerImpl.rollback(TransactionManagerImpl.java:80) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.tryRollback(CacheImpl.java:1801) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1731) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1370) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:655) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:544) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:674) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> .....
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1043 from 0075f36a-2860-4107-810a-d087224c9105-32070
> {noformat}
> in attached zip including
> -infinispan logs from all nodes
> -cluster config
> -jgroups health status csv files from nodes(comma separated, time in csv is 2hours before time in logs)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months
[JBoss JIRA] (ISPN-9351) nodes did not join to cluster because of timeoutException
by Robert Cernak (JIRA)
Robert Cernak created ISPN-9351:
-----------------------------------
Summary: nodes did not join to cluster because of timeoutException
Key: ISPN-9351
URL: https://issues.jboss.org/browse/ISPN-9351
Project: Infinispan
Issue Type: Bug
Reporter: Robert Cernak
I was trying to connect 15nodes to 1 cluster, however nodes did not join.
In logs, all nodes mostly had 2 kinds of exceptions, all caused by TimeoutException:
1:
{noformat}
2018-07-03 07:35:31,670 DEBUG [Camel (camel-1) thread #0 - seda://systemInitializer] (LocalTopologyManagerImpl.java:169) - Error sending join request for cache org.infinispan.CONFIG to coordinator
org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
following by next exception 1 second after...
2018-07-03 07:35:31,968 INFO [Camel (camel-1) thread #0 - seda://systemInitializer] (JGroupsTransport.java:702) - ISPN000080: Disconnecting JGroups channel cloud11-15
2018-07-03 07:35:32,262 DEBUG [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:709) - Stopping cache manager cloud11-15 on null
2018-07-03 07:35:32,273 WARN [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:736) - ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at java.util.concurrent.CompletableFuture.reportJoin(Unknown Source) ~[?:1.8.0_131]
at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:1.8.0_131]
at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:688) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:734) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:711) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
{noformat}
2:
{noformat}
2018-07-03 09:12:55,788 WARN [Camel (camel-1) thread #2 - seda://northboundProvider] (TransactionImpl.java:429) - ISPN000927: exception while committing
javax.transaction.xa.XAException: null
at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.commons.tx.TransactionManagerImpl.rollback(TransactionManagerImpl.java:80) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.tryRollback(CacheImpl.java:1801) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1731) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1370) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:655) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:544) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:674) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
.....
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1043 from 0075f36a-2860-4107-810a-d087224c9105-32070
{noformat}
in attached zip including
-infinispan logs from all nodes
-cluster config
-jgroups health status csv files from nodes(comma separated, time in csv is 2hours before time in logs)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
5 years, 10 months