[JBoss JIRA] (ISPN-10984) StackOverflowError following restart of scattered-cache with state-transfer awaitInitialTransfer disabled
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10984?page=com.atlassian.jira.plugin... ]
Tristan Tarrant reassigned ISPN-10984:
--------------------------------------
Assignee: Dan Berindei
> StackOverflowError following restart of scattered-cache with state-transfer awaitInitialTransfer disabled
> ---------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10984
> URL: https://issues.jboss.org/browse/ISPN-10984
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.16.Final
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Critical
>
> {noformat}
> 2019-11-24 18:30:00,837 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.clustering.web."clusterbench-ee8.ear.clusterbench-ee8-web.war": org.jboss.msc.service.StartException in service jboss.clustering.web."clusterbench-ee8.ear.clusterbench-ee8-web.war": org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at org.wildfly.clustering.service@18.0.1.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
> at org.jboss.msc@1.4.11.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
> at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at org.infinispan@9.4.16.Final//org.infinispan.interceptors.impl.PrefetchInterceptor$BackingIterator.hasNext(PrefetchInterceptor.java:651)
> at org.infinispan.commons@9.4.16.Final//org.infinispan.commons.util.IteratorMapper.hasNext(IteratorMapper.java:27)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactory.schedule(InfinispanSessionManagerFactory.java:232)
> at org.wildfly.clustering.web.infinispan(a)18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactory.<init>(InfinispanSessionManagerFactory.java:120)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactoryServiceConfigurator.get(InfinispanSessionManagerFactoryServiceConfigurator.java:92)
> at org.wildfly.clustering.web.infinispan@18.0.1.Final//org.wildfly.clustering.web.infinispan.session.InfinispanSessionManagerFactoryServiceConfigurator.get(InfinispanSessionManagerFactoryServiceConfigurator.java:69)
> at org.wildfly.clustering.service@18.0.1.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67)
> ... 8 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
> at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
> at org.infinispan@9.4.16.Final//org.infinispan.interceptors.impl.PrefetchInterceptor$BackingIterator.hasNext(PrefetchInterceptor.java:649)
> ... 14 more
> Caused by: java.lang.StackOverflowError
> at java.base/java.lang.Throwable.getMessage(Throwable.java:382)
> at java.base/java.lang.Throwable.getLocalizedMessage(Throwable.java:396)
> at java.base/java.lang.Throwable.toString(Throwable.java:485)
> at java.base/java.lang.Throwable.<init>(Throwable.java:316)
> at java.base/java.lang.Exception.<init>(Exception.java:102)
> at java.base/java.lang.RuntimeException.<init>(RuntimeException.java:96)
> at java.base/java.util.concurrent.CompletionException.<init>(CompletionException.java:88)
> at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1113)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.valuesFuture(ScatteredVersionManagerImpl.java:348)
> at org.infinispan@9.4.16.Final//org.infinispan.scattered.impl.ScatteredVersionManagerImpl.lambda$valuesFuture$3(ScatteredVersionManagerImpl.java:348)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> ...etc...
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11026) Cluster fails to operate after restart
by Jens Reimann (Jira)
[ https://issues.jboss.org/browse/ISPN-11026?page=com.atlassian.jira.plugin... ]
Jens Reimann updated ISPN-11026:
--------------------------------
Attachment: logs.tar.gz
> Cluster fails to operate after restart
> --------------------------------------
>
> Key: ISPN-11026
> URL: https://issues.jboss.org/browse/ISPN-11026
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 10.0.1.Final
> Reporter: Jens Reimann
> Priority: Blocker
> Attachments: deviceManagement.proto, infinispan.xml, logs.tar.gz
>
>
> Restarting all cluster nodes of an infinispan cluster (shut down all, start up all) doesn't bring back the cluster.
> All pods seem to be ready:
> {code:java}
> NAME READY STATUS RESTARTS AGE
> infinispan-0 1/1 Running 0 16m
> infinispan-1 1/1 Running 0 16m
> infinispan-10 1/1 Running 1 16m
> infinispan-11 1/1 Running 0 16m
> infinispan-2 1/1 Running 1 16m
> infinispan-3 1/1 Running 0 16m
> infinispan-4 1/1 Running 0 16m
> infinispan-5 1/1 Running 0 16m
> infinispan-6 1/1 Running 0 16m
> infinispan-7 1/1 Running 0 16m
> infinispan-8 1/1 Running 0 12m
> infinispan-9 1/1 Running 0 16m
> {code}
> However the clients logs are full of messages like:
> {code}
> 2019-12-03T13:58:06.191Z WARN [HOTROD] ISPN004015: Failed adding new server 10.128.6.15:11222
> org.infinispan.client.hotrod.exceptions.TransportException: java.net.SocketTimeoutException: AuthMechListOperation{(default), flags=0} timed out after 60000 ms
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11026) Cluster fails to operate after restart
by Jens Reimann (Jira)
[ https://issues.jboss.org/browse/ISPN-11026?page=com.atlassian.jira.plugin... ]
Jens Reimann updated ISPN-11026:
--------------------------------
Attachment: deviceManagement.proto
> Cluster fails to operate after restart
> --------------------------------------
>
> Key: ISPN-11026
> URL: https://issues.jboss.org/browse/ISPN-11026
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 10.0.1.Final
> Reporter: Jens Reimann
> Priority: Blocker
> Attachments: deviceManagement.proto, infinispan.xml
>
>
> Restarting all cluster nodes of an infinispan cluster (shut down all, start up all) doesn't bring back the cluster.
> All pods seem to be ready:
> {code:java}
> NAME READY STATUS RESTARTS AGE
> infinispan-0 1/1 Running 0 16m
> infinispan-1 1/1 Running 0 16m
> infinispan-10 1/1 Running 1 16m
> infinispan-11 1/1 Running 0 16m
> infinispan-2 1/1 Running 1 16m
> infinispan-3 1/1 Running 0 16m
> infinispan-4 1/1 Running 0 16m
> infinispan-5 1/1 Running 0 16m
> infinispan-6 1/1 Running 0 16m
> infinispan-7 1/1 Running 0 16m
> infinispan-8 1/1 Running 0 12m
> infinispan-9 1/1 Running 0 16m
> {code}
> However the clients logs are full of messages like:
> {code}
> 2019-12-03T13:58:06.191Z WARN [HOTROD] ISPN004015: Failed adding new server 10.128.6.15:11222
> org.infinispan.client.hotrod.exceptions.TransportException: java.net.SocketTimeoutException: AuthMechListOperation{(default), flags=0} timed out after 60000 ms
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11026) Cluster fails to operate after restart
by Jens Reimann (Jira)
[ https://issues.jboss.org/browse/ISPN-11026?page=com.atlassian.jira.plugin... ]
Jens Reimann updated ISPN-11026:
--------------------------------
Attachment: infinispan.xml
> Cluster fails to operate after restart
> --------------------------------------
>
> Key: ISPN-11026
> URL: https://issues.jboss.org/browse/ISPN-11026
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 10.0.1.Final
> Reporter: Jens Reimann
> Priority: Blocker
> Attachments: deviceManagement.proto, infinispan.xml
>
>
> Restarting all cluster nodes of an infinispan cluster (shut down all, start up all) doesn't bring back the cluster.
> All pods seem to be ready:
> {code:java}
> NAME READY STATUS RESTARTS AGE
> infinispan-0 1/1 Running 0 16m
> infinispan-1 1/1 Running 0 16m
> infinispan-10 1/1 Running 1 16m
> infinispan-11 1/1 Running 0 16m
> infinispan-2 1/1 Running 1 16m
> infinispan-3 1/1 Running 0 16m
> infinispan-4 1/1 Running 0 16m
> infinispan-5 1/1 Running 0 16m
> infinispan-6 1/1 Running 0 16m
> infinispan-7 1/1 Running 0 16m
> infinispan-8 1/1 Running 0 12m
> infinispan-9 1/1 Running 0 16m
> {code}
> However the clients logs are full of messages like:
> {code}
> 2019-12-03T13:58:06.191Z WARN [HOTROD] ISPN004015: Failed adding new server 10.128.6.15:11222
> org.infinispan.client.hotrod.exceptions.TransportException: java.net.SocketTimeoutException: AuthMechListOperation{(default), flags=0} timed out after 60000 ms
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11026) Cluster fails to operate after restart
by Jens Reimann (Jira)
Jens Reimann created ISPN-11026:
-----------------------------------
Summary: Cluster fails to operate after restart
Key: ISPN-11026
URL: https://issues.jboss.org/browse/ISPN-11026
Project: Infinispan
Issue Type: Bug
Components: Server
Affects Versions: 10.0.1.Final
Reporter: Jens Reimann
Attachments: deviceManagement.proto, infinispan.xml
Restarting all cluster nodes of an infinispan cluster (shut down all, start up all) doesn't bring back the cluster.
All pods seem to be ready:
{code:java}
NAME READY STATUS RESTARTS AGE
infinispan-0 1/1 Running 0 16m
infinispan-1 1/1 Running 0 16m
infinispan-10 1/1 Running 1 16m
infinispan-11 1/1 Running 0 16m
infinispan-2 1/1 Running 1 16m
infinispan-3 1/1 Running 0 16m
infinispan-4 1/1 Running 0 16m
infinispan-5 1/1 Running 0 16m
infinispan-6 1/1 Running 0 16m
infinispan-7 1/1 Running 0 16m
infinispan-8 1/1 Running 0 12m
infinispan-9 1/1 Running 0 16m
{code}
However the clients logs are full of messages like:
{code}
2019-12-03T13:58:06.191Z WARN [HOTROD] ISPN004015: Failed adding new server 10.128.6.15:11222
org.infinispan.client.hotrod.exceptions.TransportException: java.net.SocketTimeoutException: AuthMechListOperation{(default), flags=0} timed out after 60000 ms
{code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years
[JBoss JIRA] (ISPN-11013) Selecting a single field in a query returns wrong "total_count"
by Jens Reimann (Jira)
[ https://issues.jboss.org/browse/ISPN-11013?page=com.atlassian.jira.plugin... ]
Jens Reimann commented on ISPN-11013:
-------------------------------------
Yes. And I can also see the change in behavior when selecting a non-indexed-stored field vs an index-stored one:
{code}
oc -n enmasse-infra rsh infinispan-0 curl "http://app:test12@infinispan-0:11222/rest/v2/caches/devices?action=search..."
{
"total_results" : 10000,
"hits" : [ {
"hit" : {
"tenantId" : "jbtest.iot/2019-12-03T12:58:54Z"
}
} ]
}
{code}
vs:
{code}
oc -n enmasse-infra rsh infinispan-0 curl "http://app:test12@infinispan-0:11222/rest/v2/caches/devices?action=search..."
{
"total_results" : 1,
"hits" : [ {
"hit" : {
"deviceId" : "http-inserter-kgxhm0"
}
} ]
}
{code}
I re-checked the documentation about that, but couldn't find any hint about that. I guess it would be helpful to document that behavior.
> Selecting a single field in a query returns wrong "total_count"
> ---------------------------------------------------------------
>
> Key: ISPN-11013
> URL: https://issues.jboss.org/browse/ISPN-11013
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Querying
> Affects Versions: 10.0.1.Final
> Reporter: Jens Reimann
> Assignee: Gustavo Fernandes
> Priority: Major
> Attachments: deviceManagement.proto, infinispan.xml
>
>
> Adding a field selector to the query returns the wrong "total_count" in the result:
> {code:java}
> ➜ openshift oc -n enmasse-infra rsh infinispan-0 curl 'http://app:test12@infinispan-0:11222/rest/v2/caches/devices?action=search...'
> {
> "total_results" : 38847,
> "hits" : [ {
> "hit" : {
> "_type" : "io.enmasse.iot.registry.infinispan.data.DeviceInformation",
> "tenantId" : "jbtest.iot/2019-11-29T08:52:39Z",
> "deviceId" : "http-inserter-wh5xn103",
> "version" : "992899e8-c557-4a2c-8bc0-db950a803898",
> "registrationInformation" : "{}",
> "credentials" : [ {
> "authId" : "device-103",
> "type" : "hashed-password",
> "secrets" : [ "{\"hash-function\":\"bcrypt\",\"pwd-hash\":\"$2y$12$JELemetlJuc.6ZgQkCn5X..7UEPQm5iQV23lgno7/2sEKY2i.mPmS\"}" ]
> } ]
> }
> } ]
> }
> {code}
> {code:java}
> ➜ openshift oc -n enmasse-infra rsh infinispan-0 curl 'http://app:test12@infinispan-0:11222/rest/v2/caches/devices?action=search...'
> {
> "total_results" : 1,
> "hits" : [ {
> "hit" : {
> "deviceId" : "http-inserter-wh5xn103"
> }
> } ]
> }
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years