[JBoss JIRA] (WFLY-10073) Key requester node_X is not in current view Y; ignoring key request
by Radoslav Husar (Jira)
[ https://issues.jboss.org/browse/WFLY-10073?page=com.atlassian.jira.plugin... ]
Radoslav Husar commented on WFLY-10073:
---------------------------------------
[~tommaso-borgato] Please revalidate ^ with new jgroups, this logic is changed.
> Key requester node_X is not in current view Y; ignoring key request
> -------------------------------------------------------------------
>
> Key: WFLY-10073
> URL: https://issues.jboss.org/browse/WFLY-10073
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Daniel Cihak
> Assignee: Paul Ferraro
> Priority: Major
>
> Four errors were logged on *server* during failover scenario *eap-7x-failover-http-session-shutdown-dist-async-auth-asymEncrypt* (scenario with "AUTH" and "ASYM_ENCRYPT" protocols enabled) during new cluster view receiving:
> {code}
> 04:47:29,691 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-7,ee,perf21) ISPN000094: Received new cluster view for channel ejb: [perf21|8] (3) [perf21, perf18, perf19]
> [JBossINF] 04:47:29,692 WARN [org.jgroups.protocols.ASYM_ENCRYPT] (thread-1,ee,perf21) perf21: unrecognized cipher; discarding message from perf20
> [JBossINF] 04:47:29,692 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-9,ee,perf21) key requester perf20 is not in current view [perf21|8] (3) [perf21, perf18, perf19]; ignoring key request
> [JBossINF] 04:47:29,692 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-1,ee,perf21) key requester perf20 is not in current view [perf21|8] (3) [perf21, perf18, perf19]; ignoring key request
> [JBossINF] 04:47:29,694 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-13,ee,perf21) key requester perf20 is not in current view [perf21|8] (3) [perf21, perf18, perf19]; ignoring key request
> [JBossINF] 04:47:29,694 ERROR [org.jgroups.protocols.ASYM_ENCRYPT] (thread-13,ee,perf21) key requester perf20 is not in current view [perf21|8] (3) [perf21, perf18, perf19]; ignoring key request
> [JBossINF] 04:48:37,553 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,perf21) ISPN000094: Received new cluster view for channel server: [perf21|9] (4) [perf21, perf18, perf19, perf20]
> {code}
> WARN message was logged just before:
> {code}
> 04:47:29,692 WARN [org.jgroups.protocols.ASYM_ENCRYPT] (thread-1,ee,perf21) perf21: unrecognized cipher; discarding message from perf20
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failov...
> There are two WARN exceptions logged on client at the same time:
> {code}
> 04:47:28:988 EST [WARN ][Runner - 1066] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Link to client log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failov...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (WFLY-10074) Failed to deliver OOB message: channel is not connected
by Paul Ferraro (Jira)
[ https://issues.jboss.org/browse/WFLY-10074?page=com.atlassian.jira.plugin... ]
Paul Ferraro closed WFLY-10074.
-------------------------------
Resolution: Out of Date
> Failed to deliver OOB message: channel is not connected
> -------------------------------------------------------
>
> Key: WFLY-10074
> URL: https://issues.jboss.org/browse/WFLY-10074
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Daniel Cihak
> Assignee: Paul Ferraro
> Priority: Minor
> Labels: infinispan_clean_shutdown
>
> We saw an error in *server* log in failover scenario *shutdown-ha-ss* (1 occurence).
> Server log message:
> {code}
> 9:37:30,867 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.4.11.Final-redhat-1 stopping
> [JBossINF] [0m[31m09:37:30,921 ERROR [org.jboss.as.clustering.jgroups.protocol.NAKACK2] (thread-18) JGRP000039: perf18: failed to deliver OOB message [dst: <null>, src: perf19 (3 headers), size=32 bytes, flags=OOB|DONT_BUNDLE]: java.lang.IllegalStateException: channel is not connected
> [JBossINF] [0m[0m09:37:30,933 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: JBoss EAP 7.1.0.Alpha1 (WildFly Core 3.0.0.Beta6-redhat-1) stopped in 247ms
> {code}
> It occurs during server shutdown directly before server is finally stopped. WARN message is logged before followed by "Stopped deployment" info message:
> {code}
> 09:37:30,862 WARN [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (remote-thread--p8-t3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=client-mappings, type=REBALANCE_CONFIRM, sender=perf20, joinInfo=null, topologyId=9, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=3}: org.infinispan.commons.CacheConfigurationException: Unable to configure component (type: class org.infinispan.topology.CacheTopologyControlCommand, instance CacheTopologyControlCommand{cache=client-mappings, type=REBALANCE_CONFIRM, sender=perf20, joinInfo=null, topologyId=9, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=3})
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:151)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:157)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.IllegalLifecycleStateException: Trying to register a component after stopping: org.infinispan.topology.LocalTopologyManagerFactory
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:184)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:171)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:167)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.createComponentFactoryInternal(AbstractComponentRegistry.java:348)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.getFactory(AbstractComponentRegistry.java:327)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:293)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.wireDependencies(AbstractComponentRegistry.java:147)
> [JBossINF] ... 6 more
> 09:37:30,866 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment SStest-ejb-startup-ejb.jar (runtime-name: SStest-ejb-startup-ejb.jar) in 187ms
> {code}
> There is no exception logged on client at the same time.
> Link to server log:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-fa...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (WFLY-9958) CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed
by Paul Ferraro (Jira)
[ https://issues.jboss.org/browse/WFLY-9958?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-9958:
------------------------------------
[~tommaso-borgato] Can you revalidate this against a recent release?
> CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed
> --------------------------------------------------------------------------------------------
>
> Key: WFLY-9958
> URL: https://issues.jboss.org/browse/WFLY-9958
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Major
> Attachments: graph-throughput.png
>
>
> Seen in scenario: stress-granular-repl-sync
> While the test continues fine for 30 minutes (see [graph-throughput.png|https://issues.jboss.org/secure/attachment/12432242/...], number of clients: 4400), after that servers start logging:
> {code}
> [JBossINF] [0m[33m14:46:11,468 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t109) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=SessionAttributeKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er[1]), value=[B@45b6e38f, flags=[FORCE_SYNCHRONOUS], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=SessionAccessMetaDataKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er), value=org.wildfly.clustering.web.infinispan.session.SimpleSessionAccessMetaData@4b711549, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], onePhaseCommit=true, retried=false, gtx=GlobalTx:dev215:17647090, cacheName='clusterbench-ee7.ear.clusterbench-ee7-web-granular.war', topologyId=13}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:dev215:17647090, already completed
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:387)
> [JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:380)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:360)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Clients start logging Read timeout:
> {code}
> 2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
>
> 2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:266)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
> at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
> at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
> at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
> at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:147)
> ... 3 more
> {code}
> The test recovers after some time and then the issue occurs again in the cycles.
> Links
> client: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
> server: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (WFLY-9958) CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed
by Radoslav Husar (Jira)
[ https://issues.jboss.org/browse/WFLY-9958?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9958:
---------------------------------
Tester: Tommasso Borgato
> CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed
> --------------------------------------------------------------------------------------------
>
> Key: WFLY-9958
> URL: https://issues.jboss.org/browse/WFLY-9958
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Major
> Attachments: graph-throughput.png
>
>
> Seen in scenario: stress-granular-repl-sync
> While the test continues fine for 30 minutes (see [graph-throughput.png|https://issues.jboss.org/secure/attachment/12432242/...], number of clients: 4400), after that servers start logging:
> {code}
> [JBossINF] [0m[33m14:46:11,468 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t109) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=SessionAttributeKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er[1]), value=[B@45b6e38f, flags=[FORCE_SYNCHRONOUS], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=SessionAccessMetaDataKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er), value=org.wildfly.clustering.web.infinispan.session.SimpleSessionAccessMetaData@4b711549, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], onePhaseCommit=true, retried=false, gtx=GlobalTx:dev215:17647090, cacheName='clusterbench-ee7.ear.clusterbench-ee7-web-granular.war', topologyId=13}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:dev215:17647090, already completed
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:387)
> [JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:380)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:360)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Clients start logging Read timeout:
> {code}
> 2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
>
> 2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:266)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
> at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
> at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
> at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
> at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
> at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:147)
> ... 3 more
> {code}
> The test recovers after some time and then the issue occurs again in the cycles.
> Links
> client: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
> server: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (WFLY-9957) ISPN000481: Cannot create remote transaction GlobalTx:X, the originator is not in the cluster view
by Paul Ferraro (Jira)
[ https://issues.jboss.org/browse/WFLY-9957?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-9957:
------------------------------------
[~tommaso-borgato] Can you revalidate this against a recent release?
> ISPN000481: Cannot create remote transaction GlobalTx:X, the originator is not in the cluster view
> --------------------------------------------------------------------------------------------------
>
> Key: WFLY-9957
> URL: https://issues.jboss.org/browse/WFLY-9957
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Major
>
> Seen in scenario: stress-granular-dist-sync
> In the beggining of the test, all 4 servers form a cluster:
> {code}
> [JBossINF] [0m[0m09:38:07,301 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev213, dev214, dev215]
> [JBossINF] [0m[0m09:38:07,301 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN100000: Node dev215 joined the cluster
> [JBossINF] [0m[0m09:38:07,301 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev213, dev214, dev215]
> [JBossINF] [0m[0m09:38:07,302 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN100000: Node dev215 joined the cluster
> [JBossINF] [0m[0m09:38:07,303 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev213, dev214, dev215]
> [JBossINF] [0m[0m09:38:07,303 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN100000: Node dev215 joined the cluster
> [JBossINF] [0m[0m09:38:07,303 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN000094: Received new cluster view for channel ejb: [dev212|3] (4) [dev212, dev213, dev214, dev215]
> [JBossINF] [0m[0m09:38:07,303 INFO [org.infinispan.CLUSTER] (thread-10,ejb,dev212) ISPN100000: Node dev215 joined the cluster
> {code}
> After less than 1 minute of running test, servers start logging:
> {code}
> [JBossINF] [0m[31m10:36:11,844 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-259,ejb,dev212) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(gqatwr7cNgtvGW9Z60E3ug2v9sSoZZbGl40r1smV), SessionCreationMetaDataKey(gqatwr7cNgtvGW9Z60E3ug2v9sSoZZbGl40r1smV), SessionAttributeKey(gqatwr7cNgtvGW9Z60E3ug2v9sSoZZbGl40r1smV[1])]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from dev215, see cause for remote stack trace
> [JBossINF] at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:27)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:48)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.SyncMapResponseCollector.addException(SyncMapResponseCollector.java:26)
> [JBossINF] at org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29)
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:91)
> [JBossINF] at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1328)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1238)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:121)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1366)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:819)
> [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
> [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:340)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:134)
> [JBossINF] at org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
> [JBossINF] at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
> [JBossINF] at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:267)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
> [JBossINF] at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: ISPN000481: Cannot create remote transaction GlobalTx:dev212:63375657, the originator is not in the cluster view
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:375)
> [JBossINF] at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:360)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
> [JBossINF] at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> [JBossINF] ... 1 more
> {code}
> Also it is interesting, that clients in this test log either stale data:
> {code}
> 2018/02/16 10:36:16:596 EST [WARN ][Runner - 19] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 31196, received 31195, Runner: 19>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 31196, received 31195, Runner: 19
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> or 503 Service Temporarily Unavailable
> {code}
> 2018/02/16 10:37:24:837 EST [WARN ][Runner - 822] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 503 Content: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>503 Service Temporarily Unavailable</title>
> </head><body>
> <h1>Service Temporarily Unavailable</h1>
> <p>The server is temporarily unable to service your
> request due to maintenance downtime or capacity
> problems. Please try again later.</p>
> <hr>
> <address>Apache/2.2.26 (@VENDOR@) Server at dev221 Port 8080</address>
> </body></html>
> >
> {code}
> Logs:
> client http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
> server http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-gra...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
[JBoss JIRA] (WFLY-9952) TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey
by Paul Ferraro (Jira)
[ https://issues.jboss.org/browse/WFLY-9952?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-9952:
------------------------------------
[~tommaso-borgato] Can you revalidate this against a recent release? ISPN-9701 has since been resolved.
> TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey
> --------------------------------------------------------------------------------------------------------
>
> Key: WFLY-9952
> URL: https://issues.jboss.org/browse/WFLY-9952
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Beta1, 14.0.0.Beta2
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Minor
>
> The issue does not produce errors on the client side, only an ERROR entry in the server logs.
> Seen in: stress-session-repl-sync
> Other scenarios are likely affected as well.
> Besides servers logging WFLY-9890 constantly, thie issue has also been observed:
> {code}
> [JBossINF] [0m[31m13:15:02,901 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-32) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(Tp9qx2zySUrgdPpmDZV4HXbfUV_61KM5GZJR0fD4) and requestor GlobalTx:dev212:8863128. Lock is held by GlobalTx:dev212:8860838
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:253)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:269)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:161)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:83)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLock(PessimisticLockingInterceptor.java:93)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:110)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
> [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:374)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:191)
> [JBossINF] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:174)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:96)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:500)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:495)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:637)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:191)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:176)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:858)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:414)
> [JBossINF] at org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:241)
> [JBossINF] at org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:291)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1360)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Clients get HTTP 500 response:
> {code}
> 2018/03/04 13:15:02:903 EST [WARN ][Runner - 227] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:164)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 1 month
