[JBoss JIRA] (WFCORE-40) Deadlock while logging
by David Lloyd (JIRA)
[ https://issues.jboss.org/browse/WFCORE-40?page=com.atlassian.jira.plugin.... ]
David Lloyd commented on WFCORE-40:
-----------------------------------
Right, this is not a deadlock. For example, make sure that your log directory is not an NFS mount, as NFS can hang in some cases. If this is the console handler, make sure that your console output is not suspended (for example by running wildfly in the foreground and then stopping the process with ctrl+s (SIGTSTP)).
It's possible that you should not be using a console handler.
> Deadlock while logging
> ----------------------
>
> Key: WFCORE-40
> URL: https://issues.jboss.org/browse/WFCORE-40
> Project: WildFly Core
> Issue Type: Bug
> Components: Logging
> Environment: CentOS 6.5 64bit, java7u45 64bit (and 32 bit, the same behavior)
> Reporter: Stefan Schueffler
> Assignee: James Perkins
> Priority: Critical
> Attachments: stack_140901_2254
>
>
> We hit really often a deadlock? in org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> Even if the "StdioContext" belongs to Jboss-Logging, the problem occurs in our production wildfly installation twice to 4 times a day - all threads are deadlocking while trying to log.debug, log.error, or (sometimes) System.out.println from our application code, and wildfly does not respond anymore...
> The partial stacktrace always is similar to this one:
> {code}
> "default task-64" prio=10 tid=0x4c539c00 nid=0x5ef9 waiting for monitor entry [0x495e0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.io.PrintStream.println(PrintStream.java:806)
> - waiting to lock <0x5ee0adf8> (a java.io.PrintStream)
> at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> at jsp.communications.statuschange.selectStatus_jsp._jspService(selectStatus_jsp.java:413)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:82)
> {code}
> While investigating the StdioContext - class, i really wondered whether the used "locking/checking by using a threadlocal" could have worked in a multi-threaded environment (it should have the very same problems as every "double checking" algorithm without proper synchronization).
> If all threads are hanging in this particular lock, only a full wildfly-restart recovers in our case.
> My preferred solution would be a rework of the used org.jboss.stdio. classes, as the used idioms of ThreadLocals for reentrant-checks are at least highly unusual?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFCORE-3738) Upgrade jboss-logmanager from 2.1.0.Final to 2.1.1.Final
by James Perkins (JIRA)
James Perkins created WFCORE-3738:
-------------------------------------
Summary: Upgrade jboss-logmanager from 2.1.0.Final to 2.1.1.Final
Key: WFCORE-3738
URL: https://issues.jboss.org/browse/WFCORE-3738
Project: WildFly Core
Issue Type: Component Upgrade
Components: Logging
Reporter: James Perkins
Assignee: James Perkins
Please note in this upgrade an unused method was removed. See LOGMGR-193 for details as to why. The 2.1.0.Final is not any in WildFly release which should make the method removal safe as it is a bug.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFCORE-40) Deadlock while logging
by Harshvardhan Joshi (JIRA)
[ https://issues.jboss.org/browse/WFCORE-40?page=com.atlassian.jira.plugin.... ]
Harshvardhan Joshi commented on WFCORE-40:
------------------------------------------
I am also not fully aware of any other approaches to debug or identify this problem, it would be great if you can tell me what other things i need to do that can help identify or resolve this.
> Deadlock while logging
> ----------------------
>
> Key: WFCORE-40
> URL: https://issues.jboss.org/browse/WFCORE-40
> Project: WildFly Core
> Issue Type: Bug
> Components: Logging
> Environment: CentOS 6.5 64bit, java7u45 64bit (and 32 bit, the same behavior)
> Reporter: Stefan Schueffler
> Assignee: James Perkins
> Priority: Critical
> Attachments: stack_140901_2254
>
>
> We hit really often a deadlock? in org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> Even if the "StdioContext" belongs to Jboss-Logging, the problem occurs in our production wildfly installation twice to 4 times a day - all threads are deadlocking while trying to log.debug, log.error, or (sometimes) System.out.println from our application code, and wildfly does not respond anymore...
> The partial stacktrace always is similar to this one:
> {code}
> "default task-64" prio=10 tid=0x4c539c00 nid=0x5ef9 waiting for monitor entry [0x495e0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.io.PrintStream.println(PrintStream.java:806)
> - waiting to lock <0x5ee0adf8> (a java.io.PrintStream)
> at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> at jsp.communications.statuschange.selectStatus_jsp._jspService(selectStatus_jsp.java:413)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:82)
> {code}
> While investigating the StdioContext - class, i really wondered whether the used "locking/checking by using a threadlocal" could have worked in a multi-threaded environment (it should have the very same problems as every "double checking" algorithm without proper synchronization).
> If all threads are hanging in this particular lock, only a full wildfly-restart recovers in our case.
> My preferred solution would be a rework of the used org.jboss.stdio. classes, as the used idioms of ThreadLocals for reentrant-checks are at least highly unusual?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFCORE-40) Deadlock while logging
by Harshvardhan Joshi (JIRA)
[ https://issues.jboss.org/browse/WFCORE-40?page=com.atlassian.jira.plugin.... ]
Harshvardhan Joshi commented on WFCORE-40:
------------------------------------------
Yes I do. But I am just using SOP s and I also know that those are also printed in server.log file. But file is not locked and whenever this happens I restart the wildfly and it is resolved.
And also I see some SQL resources not available exception when ever this happens.
> Deadlock while logging
> ----------------------
>
> Key: WFCORE-40
> URL: https://issues.jboss.org/browse/WFCORE-40
> Project: WildFly Core
> Issue Type: Bug
> Components: Logging
> Environment: CentOS 6.5 64bit, java7u45 64bit (and 32 bit, the same behavior)
> Reporter: Stefan Schueffler
> Assignee: James Perkins
> Priority: Critical
> Attachments: stack_140901_2254
>
>
> We hit really often a deadlock? in org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> Even if the "StdioContext" belongs to Jboss-Logging, the problem occurs in our production wildfly installation twice to 4 times a day - all threads are deadlocking while trying to log.debug, log.error, or (sometimes) System.out.println from our application code, and wildfly does not respond anymore...
> The partial stacktrace always is similar to this one:
> {code}
> "default task-64" prio=10 tid=0x4c539c00 nid=0x5ef9 waiting for monitor entry [0x495e0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.io.PrintStream.println(PrintStream.java:806)
> - waiting to lock <0x5ee0adf8> (a java.io.PrintStream)
> at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> at jsp.communications.statuschange.selectStatus_jsp._jspService(selectStatus_jsp.java:413)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:82)
> {code}
> While investigating the StdioContext - class, i really wondered whether the used "locking/checking by using a threadlocal" could have worked in a multi-threaded environment (it should have the very same problems as every "double checking" algorithm without proper synchronization).
> If all threads are hanging in this particular lock, only a full wildfly-restart recovers in our case.
> My preferred solution would be a rework of the used org.jboss.stdio. classes, as the used idioms of ThreadLocals for reentrant-checks are at least highly unusual?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months
[JBoss JIRA] (WFCORE-40) Deadlock while logging
by James Perkins (JIRA)
[ https://issues.jboss.org/browse/WFCORE-40?page=com.atlassian.jira.plugin.... ]
James Perkins commented on WFCORE-40:
-------------------------------------
You'd want to try to figure out why it's stuck on {{java.io.FileOutputStream.writeBytes(Native Method)}}. Maybe there's some kind of error while writing.
> Deadlock while logging
> ----------------------
>
> Key: WFCORE-40
> URL: https://issues.jboss.org/browse/WFCORE-40
> Project: WildFly Core
> Issue Type: Bug
> Components: Logging
> Environment: CentOS 6.5 64bit, java7u45 64bit (and 32 bit, the same behavior)
> Reporter: Stefan Schueffler
> Assignee: James Perkins
> Priority: Critical
> Attachments: stack_140901_2254
>
>
> We hit really often a deadlock? in org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> Even if the "StdioContext" belongs to Jboss-Logging, the problem occurs in our production wildfly installation twice to 4 times a day - all threads are deadlocking while trying to log.debug, log.error, or (sometimes) System.out.println from our application code, and wildfly does not respond anymore...
> The partial stacktrace always is similar to this one:
> {code}
> "default task-64" prio=10 tid=0x4c539c00 nid=0x5ef9 waiting for monitor entry [0x495e0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.io.PrintStream.println(PrintStream.java:806)
> - waiting to lock <0x5ee0adf8> (a java.io.PrintStream)
> at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
> at jsp.communications.statuschange.selectStatus_jsp._jspService(selectStatus_jsp.java:413)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:82)
> {code}
> While investigating the StdioContext - class, i really wondered whether the used "locking/checking by using a threadlocal" could have worked in a multi-threaded environment (it should have the very same problems as every "double checking" algorithm without proper synchronization).
> If all threads are hanging in this particular lock, only a full wildfly-restart recovers in our case.
> My preferred solution would be a rework of the used org.jboss.stdio. classes, as the used idioms of ThreadLocals for reentrant-checks are at least highly unusual?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 3 months