[JBoss JIRA] (WFLY-11007) Using OpenShift generated certificates and client auth cause TLS errors
by Martin Choma (JIRA)
[ https://issues.jboss.org/browse/WFLY-11007?page=com.atlassian.jira.plugin... ]
Martin Choma edited comment on WFLY-11007 at 9/14/18 3:55 AM:
--------------------------------------------------------------
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be jdk system property {{jsse.enableMFLNExtension}} to enable fragmenting of packets during SSLHandshake[1].
Also found request for parametrizing record length [2]. But could not find how they fix it.
And finally there is also system property {{jsse.SSLEngine.acceptLargeFragments}}, which seems worth trying.
Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
[2] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6388456
was (Author: mchoma):
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be jdk system property jsse.enableMFLNExtension to enable fragmenting of packets during SSLHandshake[1].
Also found request for parametrizing record length [2]. But could not find how they fix it.
And finally there is also system property jsse.SSLEngine.acceptLargeFragments, which seems worth trying.
Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
[2] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6388456
> Using OpenShift generated certificates and client auth cause TLS errors
> -----------------------------------------------------------------------
>
> Key: WFLY-11007
> URL: https://issues.jboss.org/browse/WFLY-11007
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web (Undertow)
> Affects Versions: 13.0.0.Final
> Reporter: Sebastian Łaskawiec
> Assignee: Stuart Douglas
>
> h2. Summary
> It seems that when using OpenShift generated certificates and client auth (with {{want-client-auth="true"}}) the TLS handshake fails with {{RECV TLSv1.2 ALERT: fatal, record_overflow}} message.
> h2. Explanation
> I'm using {{oc cluster up}} and deploying Keycloak (WF 13 based) on OpenShift local cluster using the (1) template. The service in the the template uses OpenShift generated certificates ({{"service.alpha.openshift.io/serving-cert-secret-name": "keycloak-x509-https-secret"}}). Both files are mounted in the Keycloak pod and translated into keystore and truststore (see the configuration after the transformation (2)). Once the pod is up and running, I'm issuing a {{curl}} command as shown in (3). {{curl}} fails saying that {{* error:1408F092:SSL routines:ssl3_get_record:data length too long}}. The server logs with TLS Handshake debugging turned on might be found here (4). As shown in the link, the server has written {{16384}} bytes.
> I also did a test with manually created certificates (5). The result might be found here (6). As shown in the link, we've written {{16050}} bytes instead of {{16384}} and the handshake was successful.
> h2. Possible solution
> Perhaps we should cut the list CAs transmitted by the server when asking for client auth when it exceeds certain number of bytes. It would be helpful to write a warn message too.
> Links:
> - (1) Keycloak OCP Template https://gist.github.com/slaskawi/57ed810a7109a02a9d884b61ce2e7f13
> - (2) Transformed configuration https://gist.github.com/slaskawi/92aead6c519b867621129b640b4a3c88
> - (3) curl command https://gist.github.com/slaskawi/3bc32b8e96c2499cb7b48c3c5cb28616
> - (4) https://gist.github.com/slaskawi/b6477fe3cd65890c879cfe6f95359450#file-lo...
> - (5) Keycloak and OpenShift integration demo https://github.com/keycloak/openshift-integration/blob/master/install-key...
> - (6) https://gist.github.com/slaskawi/7fd87e1f2e6c4faf657d9e8289ed3392#file-lo...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-11007) Using OpenShift generated certificates and client auth cause TLS errors
by Martin Choma (JIRA)
[ https://issues.jboss.org/browse/WFLY-11007?page=com.atlassian.jira.plugin... ]
Martin Choma edited comment on WFLY-11007 at 9/14/18 3:55 AM:
--------------------------------------------------------------
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be jdk system property jsse.enableMFLNExtension to enable fragmenting of packets during SSLHandshake[1].
Also found request for parametrizing record length [2]. But could not find how they fix it.
And finally there is also system property jsse.SSLEngine.acceptLargeFragments, which seems worth trying.
Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
[2] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6388456
was (Author: mchoma):
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be jdk system property jsse.enableMFLNExtension to enable fragmenting of packets during SSLHandshake[1]. Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
> Using OpenShift generated certificates and client auth cause TLS errors
> -----------------------------------------------------------------------
>
> Key: WFLY-11007
> URL: https://issues.jboss.org/browse/WFLY-11007
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web (Undertow)
> Affects Versions: 13.0.0.Final
> Reporter: Sebastian Łaskawiec
> Assignee: Stuart Douglas
>
> h2. Summary
> It seems that when using OpenShift generated certificates and client auth (with {{want-client-auth="true"}}) the TLS handshake fails with {{RECV TLSv1.2 ALERT: fatal, record_overflow}} message.
> h2. Explanation
> I'm using {{oc cluster up}} and deploying Keycloak (WF 13 based) on OpenShift local cluster using the (1) template. The service in the the template uses OpenShift generated certificates ({{"service.alpha.openshift.io/serving-cert-secret-name": "keycloak-x509-https-secret"}}). Both files are mounted in the Keycloak pod and translated into keystore and truststore (see the configuration after the transformation (2)). Once the pod is up and running, I'm issuing a {{curl}} command as shown in (3). {{curl}} fails saying that {{* error:1408F092:SSL routines:ssl3_get_record:data length too long}}. The server logs with TLS Handshake debugging turned on might be found here (4). As shown in the link, the server has written {{16384}} bytes.
> I also did a test with manually created certificates (5). The result might be found here (6). As shown in the link, we've written {{16050}} bytes instead of {{16384}} and the handshake was successful.
> h2. Possible solution
> Perhaps we should cut the list CAs transmitted by the server when asking for client auth when it exceeds certain number of bytes. It would be helpful to write a warn message too.
> Links:
> - (1) Keycloak OCP Template https://gist.github.com/slaskawi/57ed810a7109a02a9d884b61ce2e7f13
> - (2) Transformed configuration https://gist.github.com/slaskawi/92aead6c519b867621129b640b4a3c88
> - (3) curl command https://gist.github.com/slaskawi/3bc32b8e96c2499cb7b48c3c5cb28616
> - (4) https://gist.github.com/slaskawi/b6477fe3cd65890c879cfe6f95359450#file-lo...
> - (5) Keycloak and OpenShift integration demo https://github.com/keycloak/openshift-integration/blob/master/install-key...
> - (6) https://gist.github.com/slaskawi/7fd87e1f2e6c4faf657d9e8289ed3392#file-lo...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-11007) Using OpenShift generated certificates and client auth cause TLS errors
by Martin Choma (JIRA)
[ https://issues.jboss.org/browse/WFLY-11007?page=com.atlassian.jira.plugin... ]
Martin Choma edited comment on WFLY-11007 at 9/14/18 3:31 AM:
--------------------------------------------------------------
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be jdk system property jsse.enableMFLNExtension to enable fragmenting of packets during SSLHandshake[1]. Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
was (Author: mchoma):
Could you prepare some reproducer in environemnt without OpenShift? I mean especially provide server truststore as attachment? Also which java version are you using? Are we talking here about Legacy security or Elytron? It is not obvious to me.
Anyway in both cases WFLY delegates SSLHandshake handling to JDK. So only think WFLY can do is to propagate options to JDK properly.
So OpenShift is basically extending jdk cacert file with own set of CAs, so thats why we do not get this kind of reports on baremetal WFLY installations? If OpenShift has good reason for that we should raise the priority of this issue.
I have found there should be option to enable fragmenting of packets during SSLHandshake[1]. Could you try?
[1] https://docs.oracle.com/javase/10/security/java-secure-socket-extension-j...
> Using OpenShift generated certificates and client auth cause TLS errors
> -----------------------------------------------------------------------
>
> Key: WFLY-11007
> URL: https://issues.jboss.org/browse/WFLY-11007
> Project: WildFly
> Issue Type: Bug
> Components: Security, Web (Undertow)
> Affects Versions: 13.0.0.Final
> Reporter: Sebastian Łaskawiec
> Assignee: Stuart Douglas
>
> h2. Summary
> It seems that when using OpenShift generated certificates and client auth (with {{want-client-auth="true"}}) the TLS handshake fails with {{RECV TLSv1.2 ALERT: fatal, record_overflow}} message.
> h2. Explanation
> I'm using {{oc cluster up}} and deploying Keycloak (WF 13 based) on OpenShift local cluster using the (1) template. The service in the the template uses OpenShift generated certificates ({{"service.alpha.openshift.io/serving-cert-secret-name": "keycloak-x509-https-secret"}}). Both files are mounted in the Keycloak pod and translated into keystore and truststore (see the configuration after the transformation (2)). Once the pod is up and running, I'm issuing a {{curl}} command as shown in (3). {{curl}} fails saying that {{* error:1408F092:SSL routines:ssl3_get_record:data length too long}}. The server logs with TLS Handshake debugging turned on might be found here (4). As shown in the link, the server has written {{16384}} bytes.
> I also did a test with manually created certificates (5). The result might be found here (6). As shown in the link, we've written {{16050}} bytes instead of {{16384}} and the handshake was successful.
> h2. Possible solution
> Perhaps we should cut the list CAs transmitted by the server when asking for client auth when it exceeds certain number of bytes. It would be helpful to write a warn message too.
> Links:
> - (1) Keycloak OCP Template https://gist.github.com/slaskawi/57ed810a7109a02a9d884b61ce2e7f13
> - (2) Transformed configuration https://gist.github.com/slaskawi/92aead6c519b867621129b640b4a3c88
> - (3) curl command https://gist.github.com/slaskawi/3bc32b8e96c2499cb7b48c3c5cb28616
> - (4) https://gist.github.com/slaskawi/b6477fe3cd65890c879cfe6f95359450#file-lo...
> - (5) Keycloak and OpenShift integration demo https://github.com/keycloak/openshift-integration/blob/master/install-key...
> - (6) https://gist.github.com/slaskawi/7fd87e1f2e6c4faf657d9e8289ed3392#file-lo...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (ELY-1648) FIPS NoSuchAlgorithmException: JKS KeyStore not available when trustmanager SunX509
by Martin Choma (JIRA)
[ https://issues.jboss.org/browse/ELY-1648?page=com.atlassian.jira.plugin.s... ]
Martin Choma commented on ELY-1648:
-----------------------------------
[~jucook], we have documentation how to configure java in FIPS mode using PKCS11 [1]. Please follow these steps. However, last time when someone (I think it was Farah) was following the instructions she wasn't able to make it work. If it will be also your case please fill a documentation JIRA. I think it will be something with NSS default format change to sql.
[1] https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_ap...
> FIPS NoSuchAlgorithmException: JKS KeyStore not available when trustmanager SunX509
> -----------------------------------------------------------------------------------
>
> Key: ELY-1648
> URL: https://issues.jboss.org/browse/ELY-1648
> Project: WildFly Elytron
> Issue Type: Bug
> Components: SSL
> Affects Versions: 1.5.5.Final
> Reporter: Martin Choma
> Assignee: Justin Cook
> Attachments: java.security
>
>
> With SunX509 truststore algorithm I can succesfully connect with CLI.
> {code}
> <configuration>
> <authentication-client xmlns="urn:elytron:client:1.1">
> <key-stores>
> <key-store name="truststore" type="PKCS11">
> <key-store-clear-password password="${password}" />
> </key-store>
> </key-stores>
> <ssl-contexts>
> <ssl-context name="client-cli-context">
> <trust-manager algorithm="SunX509" />
> <trust-store key-store-name="truststore" />
> <cipher-suite selector="${cipher.suite.filter}" />
> <protocol names="${protocol}" />
> </ssl-context>
> </ssl-contexts>
> <ssl-context-rules>
> <rule use-ssl-context="client-cli-context" />
> </ssl-context-rules>
> </authentication-client>
> </configuration>
> {code}
> But there is a exception in log
> {code}
> 13:58:27,652 INFO [com.redhat.eap.qe.cli.CustomCLIExecutor] (main) java.security.KeyStoreException: JKS not found
> at java.security.KeyStore.getInstance(KeyStore.java:851)
> at sun.security.util.AnchorCertificates$1.run(AnchorCertificates.java:59)
> at sun.security.util.AnchorCertificates$1.run(AnchorCertificates.java:52)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.util.AnchorCertificates.<clinit>(AnchorCertificates.java:52)
> at sun.security.provider.certpath.AlgorithmChecker.checkFingerprint(AlgorithmChecker.java:214)
> at sun.security.provider.certpath.AlgorithmChecker.<init>(AlgorithmChecker.java:164)
> at sun.security.provider.certpath.AlgorithmChecker.<init>(AlgorithmChecker.java:118)
> at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:157)
> at sun.security.validator.Validator.validate(Validator.java:260)
> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at org.xnio.ssl.JsseSslConduitEngine.handleHandshake(JsseSslConduitEngine.java:543)
> at org.xnio.ssl.JsseSslConduitEngine.wrap(JsseSslConduitEngine.java:314)
> at org.xnio.ssl.JsseSslConduitEngine.wrap(JsseSslConduitEngine.java:204)
> at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:98)
> at org.xnio.ssl.JsseSslStreamSinkConduit.write(JsseSslStreamSinkConduit.java:72)
> at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
> at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385)
> at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372)
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
> at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:94)
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> Caused by: java.security.NoSuchAlgorithmException: JKS KeyStore not available
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.KeyStore.getInstance(KeyStore.java:848)
> ... 31 more
> {code}
> When I change SunX509 to PKIX exception does not occure anymore.
> Seems exception is thrown by code https://github.com/JetBrains/jdk8u_jdk/blob/master/src/share/classes/sun/...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Paul Ferraro commented on WFLY-10736:
-------------------------------------
[~mnovak] [~rhusar] [~kabirkhan] [~jamezp] Finally, some good news. Test looks to be passing in the CI now: https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
It's been running for about 3 hours so far, and I expect will complete all 50 runs without issue.
This run contains a custom build of JGroups containing the fix for https://issues.jboss.org/browse/JGRP-2286, as well as a custom build of Infinispan containing a small but critical fix.
This will require a new JGroups 4.0.14.Final release as well as a new Infinispan 9.3.4.Final release. I'll prepare a PR for Infinispan in the morning and ask both teams to expedite new releases.
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 15.0.0.Alpha1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-traces2.zip, logs-traces3.zip, logs-traces4.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Paul Ferraro updated WFLY-10736:
--------------------------------
Comment: was deleted
(was: [~mnovak] [~kabirkhan] [~jamezp] Finally, some good news. Test now passes 50 times in a row in the CI: https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
I'll trigger another run just to be sure.
This run contains a custom build of JGroups containing the fix for https://issues.jboss.org/browse/JGRP-2286, as well as a custom build of Infinispan containing a small but critical fix.
This will require a new JGroups 4.0.14.Final release as well as a new Infinispan 9.3.4.Final release. I'll prepare a PR for Infinispan tonight and ask both teams to expedite new releases.)
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 15.0.0.Alpha1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-traces2.zip, logs-traces3.zip, logs-traces4.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Paul Ferraro commented on WFLY-10736:
-------------------------------------
[~mnovak] [~kabirkhan] [~jamezp] Finally, some good news. Test now passes 50 times in a row in the CI: https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messag...
I'll trigger another run just to be sure.
This run contains a custom build of JGroups containing the fix for https://issues.jboss.org/browse/JGRP-2286, as well as a custom build of Infinispan containing a small but critical fix.
This will require a new JGroups 4.0.14.Final release as well as a new Infinispan 9.3.4.Final release. I'll prepare a PR for Infinispan tonight and ask both teams to expedite new releases.
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Fix For: 15.0.0.Alpha1
>
> Attachments: Lodh2TestCase.testRemoteJcaInboundOnly-traces.zip, Lodh2TestCase.testRemoteJcaInboundOnly.zip, Lodh2TestCase.testRemoteJcaInboundOnly2.zip, clusterKilTest.zip, logs-traces.zip, logs-traces2.zip, logs-traces3.zip, logs-traces4.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (DROOLS-2819) [DMN Designer] [IE11] Expression type change throws an Error
by Daniel José dos Santos (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2819?page=com.atlassian.jira.plugi... ]
Daniel José dos Santos edited comment on DROOLS-2819 at 9/13/18 5:52 PM:
-------------------------------------------------------------------------
[~manstis], the cause of both issues [2820] and this is the same and is fixed by the PR to [2820].
was (Author: dadossan):
The cause of both issues [2820] and this is the same and is fixed by the PR to [2820]. [~manstis]
> [DMN Designer] [IE11] Expression type change throws an Error
> ------------------------------------------------------------
>
> Key: DROOLS-2819
> URL: https://issues.jboss.org/browse/DROOLS-2819
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.10.0.Final
> Environment: IE11
> Reporter: Jozef Marko
> Assignee: Daniel José dos Santos
> Labels: drools-tools
>
> When user tries to change expression type the Unexpected error appears: *Uncaught exception: Exception caught: Exception caught: (TypeError) : Object doesn't support property or method 'startsWith' Caused by: Exception caught: (TypeError) : Object doesn't support property or method 'startsWith' Caused by: (TypeError) : Object doesn't support property or method 'startsWith'*
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months