[JBoss JIRA] (WFLY-5835) Mean response times in soak test exceed 1000 milliseconds
by Michal Vinkler (JIRA)
Michal Vinkler created WFLY-5835:
------------------------------------
Summary: Mean response times in soak test exceed 1000 milliseconds
Key: WFLY-5835
URL: https://issues.jboss.org/browse/WFLY-5835
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Michal Vinkler
Assignee: Paul Ferraro
The responses in soak test reach really high values (hundreds of ms) compared to EAP 6 (max 10 ms). Also number of unhealthy samples (samples which arrive later than in 3 seconds) is really high in EAP7 run compared to 0 unhealthy samples in EAP6 run.
EAP7 run also logs few sampling errors in almost each iteration, EAP6 did not.
Compare these results:
EAP7: [performance.txt|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7...]
(example:
response min: 0 ms, mean: 965 ms, max: 16034 ms, sampling errors: 5, unhealthy samples: 86209)
EAP6: [performance.txt|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6...]
(example:
response min: 0 ms, mean: 6 ms, max: 1053 ms, sampling errors: 0, unhealthy samples: 0)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5835) Mean response times in soak test exceed 1000 milliseconds
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5835?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5835:
---------------------------------
Affects Version/s: 10.0.0.CR4
> Mean response times in soak test exceed 1000 milliseconds
> ---------------------------------------------------------
>
> Key: WFLY-5835
> URL: https://issues.jboss.org/browse/WFLY-5835
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> The responses in soak test reach really high values (hundreds of ms) compared to EAP 6 (max 10 ms). Also number of unhealthy samples (samples which arrive later than in 3 seconds) is really high in EAP7 run compared to 0 unhealthy samples in EAP6 run.
> EAP7 run also logs few sampling errors in almost each iteration, EAP6 did not.
> Compare these results:
> EAP7: [performance.txt|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7...]
> (example:
> response min: 0 ms, mean: 965 ms, max: 16034 ms, sampling errors: 5, unhealthy samples: 86209)
> EAP6: [performance.txt|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6...]
> (example:
> response min: 0 ms, mean: 6 ms, max: 1053 ms, sampling errors: 0, unhealthy samples: 0)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (JGRP-1989) Bundlers: reuse send buffer when transport == sync.
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/JGRP-1989?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on JGRP-1989:
---------------------------------------
That sound exciting!
Although it also sounds like you should reconsider wrapping them in some very light weight interface (rather than byte[]). I understand you want to keep the memory cost low, but this would be just an extremely small overhead compared to the size of the buffers itself.
The reason would be to make it explicit when things are safe to be reused, as you are already pointing out with various challenging questions. Not sure how the API would look like, but I'm assuming a {{release()}} or {{close()}} method would work.
> Bundlers: reuse send buffer when transport == sync.
> ---------------------------------------------------
>
> Key: JGRP-1989
> URL: https://issues.jboss.org/browse/JGRP-1989
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.6.7
>
>
> With the addition of {{TCP_NIO2}}, all bundlers now create new send buffers for every message (or message list). This generates a lot of memory allocations, perhaps it is better to revert this change for *synchronous transports* such as {{UDP}} and {{TCP}}, and still create new buffers for *asynchronous transports* such as {{TCP_NIO2}}.
> Synchronous transports guarantee a message has been put on the wire when {{TP.send()}} returns, whereas asynchronous transports may only have completed a partial write (so we cannot reuse the buffer).
> The code in the bundler should check for this, and copy if async or not copy if sync.
> Whether or not a transport is sync is determined by a new abstract method that needs to be overridden by every transport.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5834) NPE Exception handling request to /clusterbench/ejbservlet
by Michal Vinkler (JIRA)
Michal Vinkler created WFLY-5834:
------------------------------------
Summary: NPE Exception handling request to /clusterbench/ejbservlet
Key: WFLY-5834
URL: https://issues.jboss.org/browse/WFLY-5834
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Michal Vinkler
Assignee: Paul Ferraro
Seen in failover-ejb-ejbservlet-jvmkill-dist-sync scenario.
6 seconds after failing perf20 (jvmkill), perf18 logged ISPN000136: Execution error: java.lang.NullPointerException.
Timeline:
{code:title=client}
2015/11/27 08:23:25:206 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 2 (perf20) is down.
{code}
{code:title=perf18}
[JBossINF] [0m[31m08:23:31,049 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Execution error: java.lang.NullPointerException
[JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
[JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Another NPE followed right after:
{code}
[JBossINF] [0m[31m08:23:31,050 ERROR [io.undertow.request] (default task-70) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
[JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
[JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
This issue may be related:
https://issues.jboss.org/browse/WFLY-5208
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5834) NPE Exception handling request to /clusterbench/ejbservlet
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5834?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5834:
---------------------------------
Affects Version/s: 10.0.0.CR4
> NPE Exception handling request to /clusterbench/ejbservlet
> ----------------------------------------------------------
>
> Key: WFLY-5834
> URL: https://issues.jboss.org/browse/WFLY-5834
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> Seen in failover-ejb-ejbservlet-jvmkill-dist-sync scenario.
> 6 seconds after failing perf20 (jvmkill), perf18 logged ISPN000136: Execution error: java.lang.NullPointerException.
> Timeline:
> {code:title=client}
> 2015/11/27 08:23:25:206 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 2 (perf20) is down.
> {code}
> {code:title=perf18}
> [JBossINF] [0m[31m08:23:31,049 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Execution error: java.lang.NullPointerException
> [JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
> [JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
> [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
> [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Another NPE followed right after:
> {code}
> [JBossINF] [0m[31m08:23:31,050 ERROR [io.undertow.request] (default task-70) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
> [JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
> [JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
> [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
> [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
> This issue may be related:
> https://issues.jboss.org/browse/WFLY-5208
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5822) Clustering performance regression in ejbremote-dist-sync scenario
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5822?page=com.atlassian.jira.plugin.... ]
Michal Vinkler commented on WFLY-5822:
--------------------------------------
[~rachmato] Adding a byteman script to our testing framework is not a problem.
> Clustering performance regression in ejbremote-dist-sync scenario
> ------------------------------------------------------------------
>
> Key: WFLY-5822
> URL: https://issues.jboss.org/browse/WFLY-5822
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 10.0.0.CR5
> Reporter: Michal Vinkler
> Assignee: Richard Achmatowicz
>
> Compared to EAP 6, all SYNC scenarios have the same/better performance except of this one, wonder why?
> Compare these results:
> stress-ejbremote-dist-sync
> 7.0.0.ER2: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-str...]
> 6.4.0.GA: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-str...]
> ---------------------------------------
> Just for comparison: ejbremote REPL_SYNC scenario *performs well* on the other hand:
> stress-ejbremote-repl-sync
> 7.0.0.ER2: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-str...]
> 6.4.0.GA: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-str...]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5809) TimeoutException: Unable to acquire lock after 15 seconds. Lock is held by GlobalTransaction:local
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5809?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5809:
---------------------------------
Description:
Seen in these scenarios:
ejb-ejbservlet-shutdown-dist-async
http-session-shutdown-dist-sync
http-session-shutdown-dist-async
After perf18 received new view (perf19 left the cluster gracefully), it logged this error:
{code}
[JBossINF] [0m[31m15:25:11,994 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-50) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key perf18 and requestor GlobalTransaction:<perf18>:17121:local. Lock is held by GlobalTransaction:<perf18>:17120:local
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:189)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:121)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitPutKeyValueCommand(AbstractTxLockingInterceptor.java:61)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:366)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:220)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:304)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:286)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1634)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1094)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1083)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:447)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
[JBossINF] at org.wildfly.clustering.server.registry.CacheRegistry.getLocalEntry(CacheRegistry.java:139)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanRouteLocator.locate(InfinispanRouteLocator.java:62)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:48)
[JBossINF] at org.wildfly.extension.undertow.session.CodecSessionConfig.findSessionId(CodecSessionConfig.java:60)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl$IgnoreInvalidatedSessionConfig.findSessionId(ServletContextImpl.java:990)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:134)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Other nodes logged the error during the test as well - approx. 500 times in total.
Client log: as the perf19 is shut down, sampling error immediately start to occur (so the TimeoutException does seem to cause sampling errors):
{code}
2015/11/27 15:25:07:758 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 1 (perf19) is down.
2015/11/27 15:25:07:758 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Cluster state: [UP, DOWN, UP, UP].
2015/11/27 15:25:11:999 EST [WARN ][Runner - 421] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
at java.lang.Thread.run(Thread.java:745)
{code}
In the same run, these TimeoutExcpetions were also logged:
{code}
[JBossINF] [0m[31m15:25:22,306 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p8-t31) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on perf20 in behalf of transaction GlobalTransaction:<perf20>:79029:remote. Current owner GlobalTransaction:<perf20>:36201:remote.
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:224)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForAllKeys(DefaultPendingLockManager.java:131)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:197)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:239)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:103)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:76)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Are they related?
Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
was:
Seen in these scenarios:
ejb-ejbservlet-shutdown-dist-async
http-session-shutdown-dist-sync
http-session-shutdown-dist-async
After perf18 received new view (perf19 left the cluster gracefully), it logged this error:
{code}
[JBossINF] [0m[31m15:25:11,994 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-50) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key perf18 and requestor GlobalTransaction:<perf18>:17121:local. Lock is held by GlobalTransaction:<perf18>:17120:local
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:189)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:121)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitPutKeyValueCommand(AbstractTxLockingInterceptor.java:61)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:366)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:220)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:304)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:286)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1634)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1094)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1083)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:447)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
[JBossINF] at org.wildfly.clustering.server.registry.CacheRegistry.getLocalEntry(CacheRegistry.java:139)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanRouteLocator.locate(InfinispanRouteLocator.java:62)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:48)
[JBossINF] at org.wildfly.extension.undertow.session.CodecSessionConfig.findSessionId(CodecSessionConfig.java:60)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl$IgnoreInvalidatedSessionConfig.findSessionId(ServletContextImpl.java:990)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:134)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Other nodes logged the error during the test as well - approx. 500 times in total.
In the same run, these TimeoutExcpetions were also logged:
{code}
[JBossINF] [0m[31m15:25:22,306 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p8-t31) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on perf20 in behalf of transaction GlobalTransaction:<perf20>:79029:remote. Current owner GlobalTransaction:<perf20>:36201:remote.
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:224)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForAllKeys(DefaultPendingLockManager.java:131)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:197)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
[JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:239)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:103)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:76)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Are they related?
Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
> TimeoutException: Unable to acquire lock after 15 seconds. Lock is held by GlobalTransaction:local
> --------------------------------------------------------------------------------------------------
>
> Key: WFLY-5809
> URL: https://issues.jboss.org/browse/WFLY-5809
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR5
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> Seen in these scenarios:
> ejb-ejbservlet-shutdown-dist-async
> http-session-shutdown-dist-sync
> http-session-shutdown-dist-async
> After perf18 received new view (perf19 left the cluster gracefully), it logged this error:
> {code}
> [JBossINF] [0m[31m15:25:11,994 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-50) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key perf18 and requestor GlobalTransaction:<perf18>:17121:local. Lock is held by GlobalTransaction:<perf18>:17120:local
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:189)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:121)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitPutKeyValueCommand(AbstractTxLockingInterceptor.java:61)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:366)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:220)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:304)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:286)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:108)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1634)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1094)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1083)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:447)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:291)
> [JBossINF] at org.wildfly.clustering.server.registry.CacheRegistry.getLocalEntry(CacheRegistry.java:139)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanRouteLocator.locate(InfinispanRouteLocator.java:62)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:48)
> [JBossINF] at org.wildfly.extension.undertow.session.CodecSessionConfig.findSessionId(CodecSessionConfig.java:60)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl$IgnoreInvalidatedSessionConfig.findSessionId(ServletContextImpl.java:990)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:134)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
> [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
> [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Other nodes logged the error during the test as well - approx. 500 times in total.
> Client log: as the perf19 is shut down, sampling error immediately start to occur (so the TimeoutException does seem to cause sampling errors):
> {code}
> 2015/11/27 15:25:07:758 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 1 (perf19) is down.
> 2015/11/27 15:25:07:758 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Cluster state: [UP, DOWN, UP, UP].
> 2015/11/27 15:25:11:999 EST [WARN ][Runner - 421] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
> at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> In the same run, these TimeoutExcpetions were also logged:
> {code}
> [JBossINF] [0m[31m15:25:22,306 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p8-t31) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Could not acquire lock on perf20 in behalf of transaction GlobalTransaction:<perf20>:79029:remote. Current owner GlobalTransaction:<perf20>:36201:remote.
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.timeout(DefaultPendingLockManager.java:224)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForAllKeys(DefaultPendingLockManager.java:131)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:197)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:165)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:184)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163)
> [JBossINF] at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:75)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:239)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:103)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:76)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:122)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:92)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Are they related?
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (JGRP-1991) TCP_NIO2: copy on demand
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1991?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-1991.
----------------------------
Resolution: Done
> TCP_NIO2: copy on demand
> ------------------------
>
> Key: JGRP-1991
> URL: https://issues.jboss.org/browse/JGRP-1991
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.6.7
>
>
> When TCP_NIO2 writes data, we always pass a copy of data of it, because of JGRP-1961 [1].
> However, if we copied the data only on a *partial write* (full writes don't need a copy), then we could always pass data from reused buffers down to the transport.
> Example: we want to write 3 buffers of 100, 200 and 700 bytes in a gathering write (the buffers are allocated from reusable byte[] arrays).
> If the write returns 1000, we know that all 3 buffers have been written and there is no need to copy any of the buffers. However, if only 500 bytes were written, then we can trash the first 2 buffers, but need to make a copy of the 3rd buffer in range [201 .. 700].
> This allows the sender to reuse previously allocated buffers, as all transports (UDP, TCP, TCP_NIO2) now guarantee that, on return of the send(), either the data was written completely, or a copy was made.
> [1] https://issues.jboss.org/browse/JGRP-1961
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5833) WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
by Michal Vinkler (JIRA)
Michal Vinkler created WFLY-5833:
------------------------------------
Summary: WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
Key: WFLY-5833
URL: https://issues.jboss.org/browse/WFLY-5833
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Michal Vinkler
Assignee: Paul Ferraro
Priority: Minor
When using a singleton deployment and specifying quorum, the servers log ERROR message when they fail to reach quorum:
{code}
[JBossINF] [0m[31m06:09:25,064 ERROR [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0006: Failed to reach quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. No singleton master will be elected.
{code}
On the other hand, when reaching required quorum, WARNING message is logged:
{code}
[JBossINF] [0m[33m06:11:26,674 WARN [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0007: Just reached required quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. If this cluster loses another member, no node will be chosen to provide this service.
{code}
As this is expected behavior not to elect new singleton provider when the cluster fails to reach the quorum, this message imho should be logged as a WARN.
Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singl...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0
[JBoss JIRA] (WFLY-5833) WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5833?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5833:
---------------------------------
Affects Version/s: 10.0.0.CR4
> WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
> ------------------------------------------------------------------
>
> Key: WFLY-5833
> URL: https://issues.jboss.org/browse/WFLY-5833
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Minor
>
> When using a singleton deployment and specifying quorum, the servers log ERROR message when they fail to reach quorum:
> {code}
> [JBossINF] [0m[31m06:09:25,064 ERROR [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0006: Failed to reach quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. No singleton master will be elected.
> {code}
> On the other hand, when reaching required quorum, WARNING message is logged:
> {code}
> [JBossINF] [0m[33m06:11:26,674 WARN [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0007: Just reached required quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. If this cluster loses another member, no node will be chosen to provide this service.
> {code}
> As this is expected behavior not to elect new singleton provider when the cluster fails to reach the quorum, this message imho should be logged as a WARN.
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singl...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 7 months
- 1
- 0