May 2020 - jboss-jira - Jboss List Archives

[JBoss JIRA] (WFLY-13444) Observing High CPU in EPollArrayWrapper.epollCtl default I/O thread

by Srinivas ev (Jira)

[ https://issues.redhat.com/browse/WFLY-13444?page=com.atlassian.jira.plugi... ] Srinivas ev updated WFLY-13444: ------------------------------- Attachment: threaddump-1589514505334.tdump > Observing High CPU in EPollArrayWrapper.epollCtl default I/O thread > ------------------------------------------------------------------- > > Key: WFLY-13444 > URL: https://issues.redhat.com/browse/WFLY-13444 > Project: WildFly > Issue Type: Bug > Components: Remoting > Affects Versions: 10.1.0.Final > Reporter: Srinivas ev > Assignee: Flavia Rainone > Priority: Major > Attachments: IO thread memory.PNG, IO thread.PNG, Thread consuming high CPU.PNG, ThreadDump1set.zip, ThreadDump2set.zip, default IO thread 73.PNG, default io consuming snapshot.PNG, default io consuming.PNG, stack trace from snapshot.PNG, stack trace in thread dump.PNG, threaddump-1589514505334.tdump, top command output.PNG > > > Facing high CPU by one of the default I/O thread In Wildfly 10.1.0 Final. Looks none of our application component thread is consuming. > Can somebody check what is triggering this issue? I attached the thread stack to this Jira. This is blocking our releases. When the CPU cores are high in number, multiple default I/O threads will kick in consuming the CPU in 90~100% range continuously. > Wildfly 10.1.0 Final > Java - > openjdk version "1.8.0_232" > OpenJDK Runtime Environment (build 1.8.0_232-b09) > OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode > --------------------------------------------------------------------------------------------------- > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.fillInStackTrace(Throwable.java:784) > at java.lang.Throwable.<init>(Throwable.java:251) > at java.lang.Exception.<init>(Exception.java:54) > at java.io.IOException.<init>(IOException.java:47) > at java.nio.channels.ClosedChannelException.<init>(ClosedChannelException.java:52) > at org.xnio.ssl.JsseStreamConduit.write(JsseStreamConduit.java:1022) > at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) > at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385) > at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372) > at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) > at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65) > at org.xnio.ssl.JsseStreamConduit.run(JsseStreamConduit.java:393) > at org.xnio.ssl.JsseStreamConduit.readReady(JsseStreamConduit.java:547) > at org.xnio.ssl.JsseStreamConduit$2.readReady(JsseStreamConduit.java:319) > at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) > at org.xnio.nio.WorkerThread.run(WorkerThread.java:567) > ------------------------------------------------------------------------------------------- > at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method) > at sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:299) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:268) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > at org.xnio.nio.WorkerThread.run(WorkerThread.java:515) > ----------------------------------------------------------------------------------- -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills edited comment on WFCORE-4730 at 5/14/20 11:18 PM: ------------------------------------------------------------- Oh, and I forgot to mention, EPEL doesn't exist for RHEL / Centos 7 & s390x, so we'll just have to periodically rebuild newer versions of OpenSSL in /usr/local at least until we get a RHEL 8 s390x agent. was (Author: luck3y): Oh, and I forgot to mention, EPEL doesn't exist for RHEL / Centos 7 & s390x, so we'll just have to peridocally rebuild newer versions of OpenSSL in /usr/local at least until we get a RHEL 8 s390x agent. > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills edited comment on WFCORE-4730 at 5/14/20 8:34 PM: ------------------------------------------------------------ Oh, and I forgot to mention, EPEL doesn't exist for RHEL / Centos 7 & s390x, so we'll just have to peridocally rebuild newer versions of OpenSSL in /usr/local at least until we get a RHEL 8 s390x agent. was (Author: luck3y): Oh, and I forgot to mention, EPEL doesn't exist for RHEL / Centos 7 & s390x, so we'll just have to peridocally rebuild newer versions of OpenSSL in /usr/local > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills commented on WFCORE-4730: ----------------------------------- Oh, and I forgot to mention, EPEL doesn't exist for RHEL / Centos 7 & s390x, so we'll just have to peridocally rebuild newer versions of OpenSSL in /usr/local > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills commented on WFCORE-4730: ----------------------------------- Artifacts here: https://ci.wildfly.org/buildConfiguration/WildflyOpenSSL_S390build/204538... Build mysteriously started working :S Not a fan of mysteries :) > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills edited comment on WFCORE-4730 at 5/14/20 6:56 PM: ------------------------------------------------------------ I've installed 1.1.1g in /usr/local, and the job is a bit better, but there is some weirdness :( https://ci.wildfly.org/buildConfiguration/WildflyOpenSSL_S390build/204524? Failed tests: SslCiphersTest.testCipherSuiteConverter:53 TLS_AES_128_GCM_SHA256 I suspect what is happening here is that I've installed the updated openssl in /usr/local/lib64 etc, but the default RHEL one is still in /usr/lib64, along with the default headers, and it is finding these. I've tried setting CLFAGS etc, but so far that hasn't worked. (I'm assuming CFLAGS="-I/usr/local/include" should allow it to find the updated OpenSSL headers first.) Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... BasicOpenSSLSocketTest.basicOpenSSLTest1:40 » Bind Address already in use (Bin... BasicOpenSSLSocketTest.basicOpenSSLTest2:65 » Bind Address already in use (Bin... These fail reliably, due to the sockets being in TIME_WAIT. If i allow reuse, using: sysctl -w net.ipv4.tcp_timestamps=1 sysctl -w net.ipv4.tcp_tw_recycle=1 Then i can reduce it to just: Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... I'll add a config for the artifacts so you can download them. was (Author: luck3y): I've installed 1.1.1g in /usr/local, and the job is a bit better, but there is some weirdness :( https://ci.wildfly.org/buildConfiguration/WildflyOpenSSL_S390build/204524? Failed tests: SslCiphersTest.testCipherSuiteConverter:53 TLS_AES_128_GCM_SHA256 I suspect what is happening here is that I've installed the updated openssl in /usr/local/lib64 etc, but the default RHEL one is still in /usr/lib64, along with the default headers, and it is finding these. I've tried setting CLFAGS etc, but so far that hasn't worked. Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... BasicOpenSSLSocketTest.basicOpenSSLTest1:40 » Bind Address already in use (Bin... BasicOpenSSLSocketTest.basicOpenSSLTest2:65 » Bind Address already in use (Bin... These fail reliably, due to the sockets being in TIME_WAIT. If i allow reuse, using: sysctl -w net.ipv4.tcp_timestamps=1 sysctl -w net.ipv4.tcp_tw_recycle=1 Then i can reduce it to just: Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... I'll add a config for the artifacts so you can download them. > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills commented on WFCORE-4730: ----------------------------------- I've installed 1.1.1g in /usr/local, and the job is a bit better, but there is some weirdness :( https://ci.wildfly.org/buildConfiguration/WildflyOpenSSL_S390build/204524? Failed tests: SslCiphersTest.testCipherSuiteConverter:53 TLS_AES_128_GCM_SHA256 I suspect what is happening here is that I've installed the updated openssl in /usr/local/lib64 etc, but the default RHEL one is still in /usr/lib64, along with the default headers, and it is finding these. I've tried setting CLFAGS etc, but so far that hasn't worked. Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... BasicOpenSSLSocketTest.basicOpenSSLTest1:40 » Bind Address already in use (Bin... BasicOpenSSLSocketTest.basicOpenSSLTest2:65 » Bind Address already in use (Bin... These fail reliably, due to the sockets being in TIME_WAIT. If i allow reuse, using: sysctl -w net.ipv4.tcp_timestamps=1 sysctl -w net.ipv4.tcp_tw_recycle=1 Then i can reduce it to just: Tests in error: BasicOpenSSLEngineTest.openSslLotsOfDataTest:315 » Bind Address already in use... I'll add a config for the artifacts so you can download them. > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Farah Juma (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Farah Juma edited comment on WFCORE-4730 at 5/14/20 6:35 PM: ------------------------------------------------------------- That's great! Thanks very much, [~luck3y]. It would be good if we could use a newer OpenSSL, I'll check to see if I can find a newer version in a yum repo somewhere. Just to check, where will we be able to find the built linux-s390x artifact from? I saw that some tests failed on the job (I think because some assertions in the tests are assuming OpenSSL 1.1.0 or higher is in use) but the wildfly-openssl-linux-s390x portion of the build did succeed. I don't see the resulting artifact under the [Artifacts|https://ci.wildfly.org/viewLog.html?buildId=204514&buildTypeId=...] tab though. I might be looking in the wrong place. was (Author: fjuma): That's great! Thanks very much, [~luck3y]. It would be good if we could use a newer OpenSSL, I'll check to see if I can find a newer version in a yum repo somewhere. Just to check, where will we be able to find the built linux-s390x artifact from? I saw that some tests failed on the job (I think because some assertions in the tests are assuming OpenSSL 1.1.0 or higher is in use) but the wildfly-openssl-linux-s390x portion of the build did succeed. I don't see the resulting artifact under the [Artifacts|https://ci.wildfly.org/viewLog.html?buildId=204514&buildTypeId=...] tab though. > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Farah Juma (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Farah Juma commented on WFCORE-4730: ------------------------------------ That's great! Thanks very much, [~luck3y]. It would be good if we could use a newer OpenSSL, I'll check to see if I can find a newer version in a yum repo somewhere. Just to check, where will we be able to find the built linux-s390x artifact from? I saw that some tests failed on the job (I think because some assertions in the tests are assuming OpenSSL 1.1.0 or higher is in use) but the wildfly-openssl-linux-s390x portion of the build did succeed. I don't see the resulting artifact under the [Artifacts|https://ci.wildfly.org/viewLog.html?buildId=204514&buildTypeId=...] tab though. > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

[JBoss JIRA] (WFCORE-4730) Include the wildfly-openssl s390 binding

by Ken Wills (Jira)

[ https://issues.redhat.com/browse/WFCORE-4730?page=com.atlassian.jira.plug... ] Ken Wills commented on WFCORE-4730: ----------------------------------- [~fjuma] OK, I've got a test job setup. The host is running RHEL 7.8, and it has the below installed for OpenSSL (from the default yum repos): openssl-libs-1.0.2k-19.el7.s390x openssl-1.0.2k-19.el7.s390x I can install a newer OpenSSL that we can use, but ideally it would be in a yum repo somewhere, so we don't have to manually update it (these agents generally auto-update themselves running yum from cron). > Include the wildfly-openssl s390 binding > ---------------------------------------- > > Key: WFCORE-4730 > URL: https://issues.redhat.com/browse/WFCORE-4730 > Project: WildFly Core > Issue Type: Enhancement > Components: Build System, Security > Reporter: Brian Stansberry > Assignee: Farah Juma > Priority: Major > > The wildfly-openssl project include s390 bindings; we should include them. > https://github.com/wildfly/wildfly-openssl/tree/1.0.8.Final/linux-s390x > The code was there for the 1.0.8 release but it doesn't look like that variant was built and pushed to nexus: > https://repository.jboss.org/org/wildfly/openssl/ -- This message was sent by Atlassian Jira (v7.13.8#713008)

6 years, 2 months

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

jboss-jira May 2020