9:57 a.m.
Hello,
Our deployment of Aerogear in production has stopped responding to requests
on the ag-push.war or auth.war applications (wildfly admin console works
fine). I can only see one type of stack trace when i do a kill -3 (see
bellow). So far i've tried restarting wildfly, undeploying/deploying both
WARs, rebooting the server and nothing has had any effect. There has been
no change in configuration or code for a few weeks now and i've had a
couple of service restarts without a issue. I'm using UPS 1.1.3-final on a
CentOS server.
Also i'm not sure if its relevant but since the start we've been using a
slightly modified version of UPS which we built with maven. The only change
is using the push_message_info.client_identifier for our own purpose. Its
basically a one line code change.
"default task-62" #221 prio=5 os_prio=0 tid=0x00007fc7901a9000 nid=0x32c4
runnable [0x00007fc6eb7f5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x00000006c5abff60> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(
SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000006c5ac0020> (a sun.security.ssl.AppInputStream)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(
AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(
SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(
AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(
DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(
AbstractMessageParser.java:260)
at org.apache.http.impl.AbstractHttpClientConnection.
receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.
receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.AbstractClientConnAdapter.
receiveResponseHeader(AbstractClientConnAdapter.java:223)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(
HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(
HttpRequestExecutor.java:123)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(
DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(
DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(
AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(
CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(
CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(
CloseableHttpClient.java:57)
at org.keycloak.adapters.AdapterDeploymentContext.resolveRealmKey(
AdapterDeploymentContext.java:105)
at org.keycloak.adapters.AdapterDeploymentContext.resolveDeployment(
AdapterDeploymentContext.java:83)
at org.keycloak.adapters.PreAuthActionsHandler.preflightCors(
PreAuthActionsHandler.java:71)
at org.keycloak.adapters.PreAuthActionsHandler.handleRequest(
PreAuthActionsHandler.java:47)
at org.keycloak.adapters.undertow.ServletPreAuthActionsHandler.
handleRequest(ServletPreAuthActionsHandler.java:68)
at io.undertow.server.handlers.PredicateHandler.handleRequest(
PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.
handleFirstRequest(ServletInitialHandler.java:282)
at io.undertow.servlet.handlers.ServletInitialHandler.
dispatchRequest(ServletInitialHandler.java:261)
at io.undertow.servlet.handlers.ServletInitialHandler.access$
000(ServletInitialHandler.java:80)
at io.undertow.servlet.handlers.ServletInitialHandler$1.
handleRequest(ServletInitialHandler.java:172)
at io.undertow.server.Connectors.executeRootHandler(Connectors.
java:199)
at io.undertow.server.HttpServerExchange$1.run(
HttpServerExchange.java:774)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I've been trying to solve this for 2 days without success, so at this point
any help would be greatly appreciated!
--
Yoann
2:37 p.m.
What version of the JDK are you running Wildfly on? A quick Googling of
some of the issues in your stack trace point to this thread (
https://github.com/notnoop/java-apns/issues/116) which claims a similar
issue is a JVM bug.
It sounds like you might be hitting a (similar) race condition.
A few things that come to mind to check:
Do you notice if the server is using a lot of CPU or memory?
Are you possibly out of disk space?
Do your system logs have anything interesting which might shed some light
on the issue?
Is selinux blocking something?
Sorry you are having troubles,
Summers
On Thu, Sep 1, 2016 at 10:57 AM, Yoann Archambault <ya(a)sherpa-solutions.com>
wrote:
Hello,
Our deployment of Aerogear in production has stopped responding to
requests on the ag-push.war or auth.war applications (wildfly admin console
works fine). I can only see one type of stack trace when i do a kill -3
(see bellow). So far i've tried restarting wildfly, undeploying/deploying
both WARs, rebooting the server and nothing has had any effect. There has
been no change in configuration or code for a few weeks now and i've had a
couple of service restarts without a issue. I'm using UPS 1.1.3-final on a
CentOS server.
Also i'm not sure if its relevant but since the start we've been using a
slightly modified version of UPS which we built with maven. The only change
is using the push_message_info.client_identifier for our own purpose. Its
basically a one line code change.
"default task-62" #221 prio=5 os_prio=0 tid=0x00007fc7901a9000 nid=0x32c4
runnable [0x00007fc6eb7f5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java
:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.
java:973)
- locked <0x00000006c5abff60> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.
java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked <0x00000006c5ac0020> (a sun.security.ssl.AppInputStream)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffe
r(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketI
nputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(
AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
d(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
d(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(Abstract
MessageParser.java:260)
at org.apache.http.impl.AbstractHttpClientConnection.receiveRes
ponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveRes
ponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveR
esponseHeader(AbstractClientConnAdapter.java:223)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveRespon
se(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpReq
uestExecutor.java:123)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecut
e(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(D
efaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(Abs
tractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
eableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
eableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
eableHttpClient.java:57)
at org.keycloak.adapters.AdapterDeploymentContext.resolveRealmK
ey(AdapterDeploymentContext.java:105)
at org.keycloak.adapters.AdapterDeploymentContext.resolveDeploy
ment(AdapterDeploymentContext.java:83)
at org.keycloak.adapters.PreAuthActionsHandler.preflightCors(Pr
eAuthActionsHandler.java:71)
at org.keycloak.adapters.PreAuthActionsHandler.handleRequest(Pr
eAuthActionsHandler.java:47)
at org.keycloak.adapters.undertow.ServletPreAuthActionsHandler.
handleRequest(ServletPreAuthActionsHandler.java:68)
at io.undertow.server.handlers.PredicateHandler.handleRequest(P
redicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFir
stRequest(ServletInitialHandler.java:282)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchR
equest(ServletInitialHandler.java:261)
at io.undertow.servlet.handlers.ServletInitialHandler.access$00
0(ServletInitialHandler.java:80)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleR
equest(ServletInitialHandler.java:172)
at io.undertow.server.Connectors.executeRootHandler(Connectors.
java:199)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchan
ge.java:774)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
Executor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
lExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I've been trying to solve this for 2 days without success, so at this
point any help would be greatly appreciated!
--
Yoann
_______________________________________________
Aerogear-users mailing list
Aerogear-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/aerogear-users
3026
days inactive
3032
days old
aerogear-users@lists.jboss.org
1 comments
2 participants
participants (2)
-
Summers Pittman -
Yoann Archambault