[Red Hat JIRA] (ISPN-12666) ReplicationIndexTest random failures
by Katia Aresti (Jira)
[ https://issues.redhat.com/browse/ISPN-12666?page=com.atlassian.jira.plugi... ]
Katia Aresti updated ISPN-12666:
--------------------------------
Fix Version/s: 12.0.1.Final
Resolution: Done
Status: Resolved (was: Pull Request Sent)
> ReplicationIndexTest random failures
> ------------------------------------
>
> Key: ISPN-12666
> URL: https://issues.redhat.com/browse/ISPN-12666
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 12.0.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
> Fix For: 12.0.1.Final
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<0>
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.assertIndexed(ReplicationIndexTest.java:149)
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.testIndexingDuringStateTransfer(ReplicationIndexTest.java:145)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> ... Removed 23 stack frames
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12669) Tests failing due to relying on published images
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-12669?page=com.atlassian.jira.plugi... ]
Ryan Emerson updated ISPN-12669:
--------------------------------
Component/s: Test Suite
> Tests failing due to relying on published images
> ------------------------------------------------
>
> Key: ISPN-12669
> URL: https://issues.redhat.com/browse/ISPN-12669
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> The following tests are pulling the server image from remote instead of utilising the local server build to create a container for testing. This results in the testsuite failing when we transition to a new major/minor that does not have a published image yet. We should update the relevant poms to ensure that the local server dist is always used.
> {code:java}
> org.infinispan.server.test.junit5.InfinispanServerExtensionContainerTest.(?)
> test.org.infinispan.spring.starter.remote.actuator.RemoteCacheMetricBinderTest.(?)
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12669) Tests failing due to relying on published images
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-12669?page=com.atlassian.jira.plugi... ]
Ryan Emerson updated ISPN-12669:
--------------------------------
Affects Version/s: 12.0.0.Final
> Tests failing due to relying on published images
> ------------------------------------------------
>
> Key: ISPN-12669
> URL: https://issues.redhat.com/browse/ISPN-12669
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 12.0.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> The following tests are pulling the server image from remote instead of utilising the local server build to create a container for testing. This results in the testsuite failing when we transition to a new major/minor that does not have a published image yet. We should update the relevant poms to ensure that the local server dist is always used.
> {code:java}
> org.infinispan.server.test.junit5.InfinispanServerExtensionContainerTest.(?)
> test.org.infinispan.spring.starter.remote.actuator.RemoteCacheMetricBinderTest.(?)
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12669) Tests failing due to relying on published images
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-12669?page=com.atlassian.jira.plugi... ]
Ryan Emerson updated ISPN-12669:
--------------------------------
Fix Version/s: 12.1.0.Final
12.0.1.Final
> Tests failing due to relying on published images
> ------------------------------------------------
>
> Key: ISPN-12669
> URL: https://issues.redhat.com/browse/ISPN-12669
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 12.0.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 12.1.0.Final, 12.0.1.Final
>
>
> The following tests are pulling the server image from remote instead of utilising the local server build to create a container for testing. This results in the testsuite failing when we transition to a new major/minor that does not have a published image yet. We should update the relevant poms to ensure that the local server dist is always used.
> {code:java}
> org.infinispan.server.test.junit5.InfinispanServerExtensionContainerTest.(?)
> test.org.infinispan.spring.starter.remote.actuator.RemoteCacheMetricBinderTest.(?)
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12669) Tests failing due to relying on published images
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-12669?page=com.atlassian.jira.plugi... ]
Ryan Emerson updated ISPN-12669:
--------------------------------
Status: Open (was: New)
> Tests failing due to relying on published images
> ------------------------------------------------
>
> Key: ISPN-12669
> URL: https://issues.redhat.com/browse/ISPN-12669
> Project: Infinispan
> Issue Type: Bug
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> The following tests are pulling the server image from remote instead of utilising the local server build to create a container for testing. This results in the testsuite failing when we transition to a new major/minor that does not have a published image yet. We should update the relevant poms to ensure that the local server dist is always used.
> {code:java}
> org.infinispan.server.test.junit5.InfinispanServerExtensionContainerTest.(?)
> test.org.infinispan.spring.starter.remote.actuator.RemoteCacheMetricBinderTest.(?)
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12652) Cluster is broken after one node is down
by Dmitry Kruglikov (Jira)
[ https://issues.redhat.com/browse/ISPN-12652?page=com.atlassian.jira.plugi... ]
Dmitry Kruglikov updated ISPN-12652:
------------------------------------
Description:
We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
{{{{ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1}}}}
{{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)}}}}
{{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)}}}}
{{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)}}}}
{{ \{{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}}}
{{ \{{ at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}}}
{{ \{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}}}
{{ \{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}}}
{{ \{{ at java.base/java.lang.Thread.run(Thread.java:834)}}}}
So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
was:
We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
{{ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)}}
{{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
{{ at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{ at java.base/java.lang.Thread.run(Thread.java:834)}}
So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
> Cluster is broken after one node is down
> ----------------------------------------
>
> Key: ISPN-12652
> URL: https://issues.redhat.com/browse/ISPN-12652
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.14.Final
> Reporter: Dmitry Kruglikov
> Priority: Blocker
>
> We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
>
> {{{{ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1}}}}
> {{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)}}}}
> {{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)}}}}
> {{ \{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)}}}}
> {{ \{{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}}}
> {{ \{{ at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}}}
> {{ \{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}}}
> {{ \{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}}}
> {{ \{{ at java.base/java.lang.Thread.run(Thread.java:834)}}}}
>
> So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
>
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12652) Cluster is broken after one node is down
by Dmitry Kruglikov (Jira)
[ https://issues.redhat.com/browse/ISPN-12652?page=com.atlassian.jira.plugi... ]
Dmitry Kruglikov updated ISPN-12652:
------------------------------------
Description:
We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
{{ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)}}
{{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)}}
{{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
{{ at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{ at java.base/java.lang.Thread.run(Thread.java:834)}}
So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
was:
We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
> Cluster is broken after one node is down
> ----------------------------------------
>
> Key: ISPN-12652
> URL: https://issues.redhat.com/browse/ISPN-12652
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.14.Final
> Reporter: Dmitry Kruglikov
> Priority: Blocker
>
> We have 3 nodes in cluster: app1, app2 and app3. App1 was shut down not gracefully because of some hardware issue. After that app2 and app3 started to fail with something like
>
> {{ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p23-t1) ISPN000136: Error executing command RemoveCommand on Cache 'fs.war', writing keys [SessionCreationMetaDataKey(PGARVVdjGKfifzrVfyd7HAllbrwaRG7wLhKha1On)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 422657 from app1}}
> {{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)}}
> {{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)}}
> {{ at org.infinispan@9.4.14.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)}}
> {{ at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
> {{ at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
> {{ at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
> {{ at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
> {{ at java.base/java.lang.Thread.run(Thread.java:834)}}
>
> So these 2 nodes (app2 and app3) could not serve user requests anymore until app1 recovered. My question is... Is it ok? Should not Infinispan identify that one of nodes is down, remove it from cluster and notify app2 and app3 about it? I know that there is something like VERIFY_SUSPECT but it didn't happen.
>
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12666) ReplicationIndexTest random failures
by Gustavo Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-12666?page=com.atlassian.jira.plugi... ]
Gustavo Fernandes updated ISPN-12666:
-------------------------------------
Affects Version/s: (was: 11.0.9.Final)
> ReplicationIndexTest random failures
> ------------------------------------
>
> Key: ISPN-12666
> URL: https://issues.redhat.com/browse/ISPN-12666
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 12.0.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<0>
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.assertIndexed(ReplicationIndexTest.java:149)
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.testIndexingDuringStateTransfer(ReplicationIndexTest.java:145)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> ... Removed 23 stack frames
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months
[Red Hat JIRA] (ISPN-12666) ReplicationIndexTest random failures
by Gustavo Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-12666?page=com.atlassian.jira.plugi... ]
Gustavo Fernandes updated ISPN-12666:
-------------------------------------
Affects Version/s: 12.0.0.Final
11.0.9.Final
> ReplicationIndexTest random failures
> ------------------------------------
>
> Key: ISPN-12666
> URL: https://issues.redhat.com/browse/ISPN-12666
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 11.0.9.Final, 12.0.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<0>
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.assertIndexed(ReplicationIndexTest.java:149)
> at org.infinispan.client.hotrod.query.ReplicationIndexTest.testIndexingDuringStateTransfer(ReplicationIndexTest.java:145)
> at org.infinispan.commons.test.TestNGLongTestsHook.run(TestNGLongTestsHook.java:24)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> ... Removed 23 stack frames
> {noformat}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
5 years, 2 months