[JBoss JIRA] (JGRP-1989) Bundlers: reuse send buffer when transport == sync.
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1989?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1989:
--------------------------------
You don't give up, do you? :-)
Even the most light-weight impl adds ~16 bytes to every message [1], and I'm not willing to add that cost.
Looking at my previous comment, that may not even be necessary, because it seems we can reuse buffers *for sure* for 75% of our use cases:
* Sending of message batches: OK
* Sending of single messages: OK
* Reception of message batches: OK
* Reception of single messages: MAYBE
The last 25% (reception of single messages) is (1) not a big use case as most messages are received as batches anyway and (2) we could come up with requirements for how the {{receive()}} callback needs to use the buffer in the message argument. For instance, we could say that a buffer needs to be read and data in it needs to be de-serialized before {{receive()}} returns, or a copy has to be made. I also need to look at how retransmission is affected in {{NAKACK2}} and {{UNICAST3}}.
Actually, most callbacks do perform de-serialization right away, with some exceptions (e.g. Async Invocation API), so they should be conformant.
[1] https://github.com/belaban/workshop/blob/master/slides/status2015.adoc#si...
> Bundlers: reuse send buffer when transport == sync.
> ---------------------------------------------------
>
> Key: JGRP-1989
> URL: https://issues.jboss.org/browse/JGRP-1989
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.6.7
>
>
> With the addition of {{TCP_NIO2}}, all bundlers now create new send buffers for every message (or message list). This generates a lot of memory allocations, perhaps it is better to revert this change for *synchronous transports* such as {{UDP}} and {{TCP}}, and still create new buffers for *asynchronous transports* such as {{TCP_NIO2}}.
> Synchronous transports guarantee a message has been put on the wire when {{TP.send()}} returns, whereas asynchronous transports may only have completed a partial write (so we cannot reuse the buffer).
> The code in the bundler should check for this, and copy if async or not copy if sync.
> Whether or not a transport is sync is determined by a new abstract method that needs to be overridden by every transport.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
[JBoss JIRA] (WFLY-5836) Stale session data received when using REPL SYNC on jvmkill
by Michal Vinkler (JIRA)
Michal Vinkler created WFLY-5836:
------------------------------------
Summary: Stale session data received when using REPL SYNC on jvmkill
Key: WFLY-5836
URL: https://issues.jboss.org/browse/WFLY-5836
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Michal Vinkler
Assignee: Paul Ferraro
Priority: Minor
Client received older data when using REPL SYNC right after failing perf21 (jvmkill).
-- seen only once, hardly reproducible
Timeline & stacktrace:
{code}
2015/11/27 15:05:40:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 3 (perf21)
2015/11/27 15:05:40:459 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 3 (perf21) is down.
2015/11/27 15:05:40:459 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Cluster state: [UP, UP, UP, DOWN].
2015/11/27 15:05:45:758 EST [INFO ][Runner - 725] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
2015/11/27 15:05:45:774 EST [INFO ][Runner - 1962] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
:
:
2015/11/27 15:05:49:999 EST [INFO ][Runner - 522] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
2015/11/27 15:05:50:533 EST [INFO ][Runner - 314] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
2015/11/27 15:05:53:908 EST [WARN ][Runner - 14] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 139, received 138, Runner: 14>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 139, received 138, Runner: 14
at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
at java.lang.Thread.run(Thread.java:745)
{code}
Link
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
[JBoss JIRA] (WFLY-5836) Stale session data received when using REPL SYNC on jvmkill
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5836?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5836:
---------------------------------
Affects Version/s: 10.0.0.CR4
> Stale session data received when using REPL SYNC on jvmkill
> -----------------------------------------------------------
>
> Key: WFLY-5836
> URL: https://issues.jboss.org/browse/WFLY-5836
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
> Priority: Minor
>
> Client received older data when using REPL SYNC right after failing perf21 (jvmkill).
> -- seen only once, hardly reproducible
> Timeline & stacktrace:
> {code}
> 2015/11/27 15:05:40:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 3 (perf21)
> 2015/11/27 15:05:40:459 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 3 (perf21) is down.
> 2015/11/27 15:05:40:459 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Cluster state: [UP, UP, UP, DOWN].
> 2015/11/27 15:05:45:758 EST [INFO ][Runner - 725] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
> 2015/11/27 15:05:45:774 EST [INFO ][Runner - 1962] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf19
> :
> :
> 2015/11/27 15:05:49:999 EST [INFO ][Runner - 522] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
> 2015/11/27 15:05:50:533 EST [INFO ][Runner - 314] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failover detected, JvmRoute changed. perf21 -> perf20
> 2015/11/27 15:05:53:908 EST [WARN ][Runner - 14] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 139, received 138, Runner: 14>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 139, received 138, Runner: 14
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Link
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http-...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
[JBoss JIRA] (JGRP-1989) Bundlers: reuse send buffer when transport == sync.
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/JGRP-1989?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on JGRP-1989:
---------------------------------------
That sound exciting!
Although it also sounds like you should reconsider wrapping them in some very light weight interface (rather than byte[]). I understand you want to keep the memory cost low, but this would be just an extremely small overhead compared to the size of the buffers itself.
The reason would be to make it explicit when things are safe to be reused, as you are already pointing out with various challenging questions. Not sure how the API would look like, but I'm assuming a {{release()}} or {{close()}} method would work.
> Bundlers: reuse send buffer when transport == sync.
> ---------------------------------------------------
>
> Key: JGRP-1989
> URL: https://issues.jboss.org/browse/JGRP-1989
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.6.7
>
>
> With the addition of {{TCP_NIO2}}, all bundlers now create new send buffers for every message (or message list). This generates a lot of memory allocations, perhaps it is better to revert this change for *synchronous transports* such as {{UDP}} and {{TCP}}, and still create new buffers for *asynchronous transports* such as {{TCP_NIO2}}.
> Synchronous transports guarantee a message has been put on the wire when {{TP.send()}} returns, whereas asynchronous transports may only have completed a partial write (so we cannot reuse the buffer).
> The code in the bundler should check for this, and copy if async or not copy if sync.
> Whether or not a transport is sync is determined by a new abstract method that needs to be overridden by every transport.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
[JBoss JIRA] (WFLY-5834) NPE Exception handling request to /clusterbench/ejbservlet
by Michal Vinkler (JIRA)
Michal Vinkler created WFLY-5834:
------------------------------------
Summary: NPE Exception handling request to /clusterbench/ejbservlet
Key: WFLY-5834
URL: https://issues.jboss.org/browse/WFLY-5834
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Michal Vinkler
Assignee: Paul Ferraro
Seen in failover-ejb-ejbservlet-jvmkill-dist-sync scenario.
6 seconds after failing perf20 (jvmkill), perf18 logged ISPN000136: Execution error: java.lang.NullPointerException.
Timeline:
{code:title=client}
2015/11/27 08:23:25:206 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 2 (perf20) is down.
{code}
{code:title=perf18}
[JBossINF] [0m[31m08:23:31,049 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Execution error: java.lang.NullPointerException
[JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
[JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Another NPE followed right after:
{code}
[JBossINF] [0m[31m08:23:31,050 ERROR [io.undertow.request] (default task-70) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
[JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
[JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
[JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
[JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
[JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
[JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
[JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
[JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
[JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
[JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
[JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
[JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
[JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
[JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
[JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
[JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
[JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
[JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
[JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
[JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
[JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
[JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
{code}
Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
This issue may be related:
https://issues.jboss.org/browse/WFLY-5208
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
[JBoss JIRA] (WFLY-5834) NPE Exception handling request to /clusterbench/ejbservlet
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-5834?page=com.atlassian.jira.plugin.... ]
Michal Vinkler updated WFLY-5834:
---------------------------------
Affects Version/s: 10.0.0.CR4
> NPE Exception handling request to /clusterbench/ejbservlet
> ----------------------------------------------------------
>
> Key: WFLY-5834
> URL: https://issues.jboss.org/browse/WFLY-5834
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> Seen in failover-ejb-ejbservlet-jvmkill-dist-sync scenario.
> 6 seconds after failing perf20 (jvmkill), perf18 logged ISPN000136: Execution error: java.lang.NullPointerException.
> Timeline:
> {code:title=client}
> 2015/11/27 08:23:25:206 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Node 2 (perf20) is down.
> {code}
> {code:title=perf18}
> [JBossINF] [0m[31m08:23:31,049 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-70) ISPN000136: Execution error: java.lang.NullPointerException
> [JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
> [JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
> [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
> [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Another NPE followed right after:
> {code}
> [JBossINF] [0m[31m08:23:31,050 ERROR [io.undertow.request] (default task-70) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
> [JBossINF] at org.infinispan.transaction.impl.AbstractCacheTransaction.containsLockOrBackupLock(AbstractCacheTransaction.java:254)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.lambda$getTransactionWithLockedKey$177(DefaultPendingLockManager.java:252)
> [JBossINF] at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ValuesView.forEach(EquivalentConcurrentHashMapV8.java:4604)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.forEachTransaction(DefaultPendingLockManager.java:284)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.getTransactionWithLockedKey(DefaultPendingLockManager.java:249)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.checkForPendingLock(DefaultPendingLockManager.java:187)
> [JBossINF] at org.infinispan.util.concurrent.locks.impl.DefaultPendingLockManager.awaitPendingTransactionsForKey(DefaultPendingLockManager.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:190)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:115)
> [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:69)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:76)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> [JBossINF] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:403)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:388)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> [JBossINF] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> [JBossINF] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:412)
> [JBossINF] at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:437)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:70)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:60)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
> [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:725)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:370)
> [JBossINF] at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
> [JBossINF] at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:231)
> [JBossINF] at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> [JBossINF] at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:245)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:283)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> {code}
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
> This issue may be related:
> https://issues.jboss.org/browse/WFLY-5208
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 11 months
