[jboss-jira] [JBoss JIRA] (WFLY-9958) CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed

[Michal Vinkler (JIRA)]

Michal Vinkler created WFLY-9958:
------------------------------------

             Summary: CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed
                 Key: WFLY-9958
                 URL: https://issues.jboss.org/browse/WFLY-9958
             Project: WildFly
          Issue Type: Bug
          Components: Clustering
    Affects Versions: 12.0.0.Beta1
            Reporter: Michal Vinkler
            Assignee: Paul Ferraro
         Attachments: graph-throughput.png

Seen in scenario: stress-granular-repl-sync

While the test continues fine for 30 minutes (see [#graph-throughput.png], number of clients: 4400), after that servers start logging:

{code}
[JBossINF] [0m[33m14:46:11,468 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t109) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=SessionAttributeKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er[1]), value=[B at 45b6e38f, flags=[FORCE_SYNCHRONOUS], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=SessionAccessMetaDataKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er), value=org.wildfly.clustering.web.infinispan.session.SimpleSessionAccessMetaData at 4b711549, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], onePhaseCommit=true, retried=false, gtx=GlobalTx:dev215:17647090, cacheName='clusterbench-ee7.ear.clusterbench-ee7-web-granular.war', topologyId=13}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:dev215:17647090, already completed
[JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:387)
[JBossINF] 	at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
[JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:380)
[JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:360)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] 	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
[JBossINF] 	at java.lang.Thread.run(Thread.java:748)
{code}

Clients start logging Read timeout:
{code}
2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
        
2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] SFCORE_LOG - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
        org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:266)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:147)
	... 3 more
{code}

The test recovers after some time and then the issue occurs again in the cycles.

Links
client: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-granular-repl-sync/42/console-dev221/
server: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-granular-repl-sync/42/console-dev212/



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)