12:17 p.m.
[
https://issues.jboss.org/browse/RF-12899?page=com.atlassian.jira.plugin.s...
]
Kurt Mansel commented on RF-12899:
----------------------------------
Hi, I'm experiencing the same issue. Our environment is:
-> apache-tomcat-6.0.18.3
-> richfaces-4.2.3.Final
We are using the a4j:pusf component to make push notifications (the avarage is 1 push each
5 seconds and 60 concurrent logged in users to be notified) and in production environment
I've locks appearing each 3-4 days that forces me to restart the tomcat, beacuse the
application become slow and at certain point the push notifications stop to work.
I've been able to reproduce the issue in our test environment having 80 concurrent
users logged in on the portal (just logged in without making any other operation) and
having 1 push notification per second and in this scenario I'm able to reproduce a
deadlock in 10-15 minutes.
I've tried to upgrade the version to 4.3.0 and with the same test case I have the same
issue. Do you have any idea of why this deadlock appears? Is there something I can do as
workaround or is there a version/patch that clearly solves that bug? I ask if someone can
help me because it is a serious issue in production environment and considering that our
load is growing each day I don't want to end up in a situation of restarting the
tomcat each hour and already now we have an embarrassing situation of an unstable
application and customers complaining .
Below i report some of the deadlocks that I've been able to reproduce in our test
environment (I've been able to produce 8 deadlocks in 1 hour):
Java stack information for the threads listed above:
===================================================
"Atmosphere-AsyncWrite-0":
at
org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:125)
- waiting to lock <0xe7385318> (a
org.richfaces.application.push.impl.SessionImpl)
at
org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:127)
at org.richfaces.application.push.impl.RequestImpl.onResume(RequestImpl.java:135)
at
org.atmosphere.cpr.AtmosphereResourceImpl.onResume(AtmosphereResourceImpl.java:804)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:752)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:734)
at
org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:771)
at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:801)
- locked <0xeaf05330> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"http-8081-144":
at
org.atmosphere.cpr.AtmosphereResourceImpl.resume(AtmosphereResourceImpl.java:274)
- waiting to lock <0xeaf05330> (a
org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.Meteor.resume(Meteor.java:275)
at org.richfaces.application.push.impl.RequestImpl.resume(RequestImpl.java:70)
at
org.richfaces.application.push.impl.SessionImpl.releaseRequest(SessionImpl.java:120)
at
org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:125)
- locked <0xe7385318> (a org.richfaces.application.push.impl.SessionImpl)
at
org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:127)
at org.richfaces.application.push.impl.RequestImpl.onResume(RequestImpl.java:135)
at
org.atmosphere.cpr.AtmosphereResourceImpl.onResume(AtmosphereResourceImpl.java:804)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:752)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:734)
at
org.atmosphere.cpr.AsynchronousProcessor.timedout(AsynchronousProcessor.java:457)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:157)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1293)
at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:293)
at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:279)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
Java stack information for the threads listed above:
===================================================
"Atmosphere-AsyncWrite-1":
at
org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:125)
- waiting to lock <0xe80ab530> (a
org.richfaces.application.push.impl.SessionImpl)
at
org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:127)
at org.richfaces.application.push.impl.RequestImpl.onResume(RequestImpl.java:135)
at
org.atmosphere.cpr.AtmosphereResourceImpl.onResume(AtmosphereResourceImpl.java:804)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:752)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:734)
at
org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:771)
at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:801)
- locked <0xea6adc88> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"http-8081-54":
at
org.atmosphere.cpr.AtmosphereResourceImpl.resume(AtmosphereResourceImpl.java:274)
- waiting to lock <0xea6adc88> (a
org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.Meteor.resume(Meteor.java:275)
at org.richfaces.application.push.impl.RequestImpl.resume(RequestImpl.java:70)
at
org.richfaces.application.push.impl.SessionImpl.releaseRequest(SessionImpl.java:120)
at
org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:125)
- locked <0xe80ab530> (a org.richfaces.application.push.impl.SessionImpl)
at
org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:127)
at org.richfaces.application.push.impl.RequestImpl.onResume(RequestImpl.java:135)
at
org.atmosphere.cpr.AtmosphereResourceImpl.onResume(AtmosphereResourceImpl.java:804)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:752)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:734)
at
org.atmosphere.cpr.AsynchronousProcessor.timedout(AsynchronousProcessor.java:457)
at
org.atmosphere.container.BlockingIOCometSupport.suspend(BlockingIOCometSupport.java:157)
at
org.atmosphere.container.BlockingIOCometSupport.service(BlockingIOCometSupport.java:103)
at
org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1293)
at org.atmosphere.cpr.AtmosphereServlet.doPost(AtmosphereServlet.java:293)
at org.atmosphere.cpr.AtmosphereServlet.doGet(AtmosphereServlet.java:279)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:343)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
I report also some configuration on the web.xml file made for atmosphere:
<context-param>
<param-name>org.richfaces.push.handlerMapping</param-name>
<param-value>/__richfaces_push</param-value>
</context-param>
<servlet>
<servlet-name>Push Servlet</servlet-name>
<servlet-class>org.richfaces.webapp.PushServlet</servlet-class>
<async-supported>true</async-supported>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>Push Servlet</servlet-name>
<url-pattern>/__richfaces_push</url-pattern>
</servlet-mapping>
<context-param>
<param-name>org.atmosphere.useBlocking</param-name>
<param-value>true</param-value>
</context-param>
<context-param>
<param-name>org.atmosphere.cpr.CometSupport.maxInactiveActivity</param-name>
<param-value>35000</param-value>
</context-param>
<context-param>
<param-name>org.richfaces.push.session.maxInactiveInterval</param-name>
<param-value>40000</param-value>
</context-param>
Thank you in advanced for your quick feedback
Deadlock appears in push component
----------------------------------
Key: RF-12899
URL: https://issues.jboss.org/browse/RF-12899
Project: RichFaces
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: component-push/poll
Affects Versions: 4.2.2.Final
Reporter: Jiří Mikulášek
Assignee: Lukáš Fryč
Fix For: 5-Tracking
Our application freezes occasionally. We used Jstack and have seen the deadlock below. It
seems to be very simillar to RF-12013, but the stack is slightly different
Found one Java-level deadlock:
=============================
{code}
"localhost-startStop-2":
waiting to lock monitor 0x000000000782a368 (object 0x000000078da957e0, a
org.richfaces.application.push.impl.SessionImpl),
which is held by "http-nio-30085-exec-4"
"http-nio-30085-exec-4":
waiting to lock monitor 0x00000000085cbe00 (object 0x000000078e8ecf90, a
org.atmosphere.cpr.AtmosphereResourceImpl),
which is held by "Atmosphere-AsyncWrite-0"
"Atmosphere-AsyncWrite-0":
waiting to lock monitor 0x000000000782a368 (object 0x000000078da957e0, a
org.richfaces.application.push.impl.SessionImpl),
which is held by "http-nio-30085-exec-4"
{code}
Java stack information for the threads listed above:
===================================================
{code}
"localhost-startStop-2":
at org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:117)
- waiting to lock <0x000000078da957e0> (a
org.richfaces.application.push.impl.SessionImpl)
at org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:128)
at org.richfaces.application.push.impl.RequestImpl.onResume(RequestImpl.java:136)
at org.atmosphere.cpr.AtmosphereResourceImpl.onResume(AtmosphereResourceImpl.java:658)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:607)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:591)
at org.atmosphere.cpr.AtmosphereResourceImpl.resume(AtmosphereResourceImpl.java:187)
- locked <0x000000078e8b64a8> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.DefaultBroadcaster.resumeAll(DefaultBroadcaster.java:300)
- locked <0x000000078e8d1ac8> (a java.util.concurrent.ConcurrentLinkedQueue)
at
org.atmosphere.cpr.DefaultBroadcasterFactory.destroy(DefaultBroadcasterFactory.java:274)
- locked <0x000000078cb112d8> (a org.atmosphere.cpr.DefaultBroadcasterFactory)
at org.atmosphere.cpr.AtmosphereServlet.destroy(AtmosphereServlet.java:867)
at org.atmosphere.cpr.MeteorServlet.destroy(MeteorServlet.java:88)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1481)
- locked <0x0000000784b6a168> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1842)
- locked <0x0000000784b6a168> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
- locked <0x0000000784b6a168> (a org.apache.catalina.core.StandardWrapper)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5471)
- locked <0x000000078312e828> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
- locked <0x000000078312e828> (a org.apache.catalina.core.StandardContext)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
"http-nio-30085-exec-4":
at org.atmosphere.cpr.AtmosphereResourceImpl.resume(AtmosphereResourceImpl.java:168)
- waiting to lock <0x000000078e8ecf90> (a
org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.Meteor.resume(Meteor.java:240)
at org.richfaces.application.push.impl.RequestImpl.resume(RequestImpl.java:71)
at org.richfaces.application.push.impl.SessionImpl.releaseRequest(SessionImpl.java:112)
at org.richfaces.application.push.impl.SessionImpl.connect(SessionImpl.java:85)
- locked <0x000000078da957e0> (a org.richfaces.application.push.impl.SessionImpl)
at org.richfaces.application.push.impl.RequestImpl.onSuspend(RequestImpl.java:119)
at org.atmosphere.cpr.AtmosphereResourceImpl.onSuspend(AtmosphereResourceImpl.java:652)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:611)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:591)
at org.atmosphere.cpr.AtmosphereResourceImpl.suspend(AtmosphereResourceImpl.java:347)
at org.atmosphere.cpr.Meteor.suspend(Meteor.java:213)
at org.richfaces.application.push.impl.RequestImpl.suspend(RequestImpl.java:67)
at org.richfaces.webapp.PushHandlerFilter.doFilter(PushHandlerFilter.java:107)
at org.atmosphere.util.AtmosphereFilterChain.doFilter(AtmosphereFilterChain.java:139)
at
org.atmosphere.util.AtmosphereFilterChain.invokeFilterChain(AtmosphereFilterChain.java:116)
at
org.atmosphere.handler.ReflectorServletProcessor$FilterChainServletWrapper.service(ReflectorServletProcessor.java:293)
at
org.atmosphere.handler.ReflectorServletProcessor.onRequest(ReflectorServletProcessor.java:151)
at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:219)
at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:154)
at org.atmosphere.container.Tomcat7CometSupport.service(Tomcat7CometSupport.java:85)
at org.atmosphere.cpr.AtmosphereServlet.doCometSupport(AtmosphereServlet.java:1218)
at org.atmosphere.cpr.AtmosphereServlet.event(AtmosphereServlet.java:1286)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilterEvent(ApplicationFilterChain.java:484)
at
org.apache.catalina.core.ApplicationFilterChain.doFilterEvent(ApplicationFilterChain.java:377)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
- locked <0x000000078e7fcc28> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
"Atmosphere-AsyncWrite-0":
at org.richfaces.application.push.impl.SessionImpl.disconnect(SessionImpl.java:117)
- waiting to lock <0x000000078da957e0> (a
org.richfaces.application.push.impl.SessionImpl)
at org.richfaces.application.push.impl.RequestImpl.disconnect(RequestImpl.java:128)
at org.richfaces.application.push.impl.RequestImpl.onDisconnect(RequestImpl.java:140)
at
org.atmosphere.cpr.AtmosphereResourceImpl.onDisconnect(AtmosphereResourceImpl.java:664)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:609)
at
org.atmosphere.cpr.AtmosphereResourceImpl.notifyListeners(AtmosphereResourceImpl.java:591)
at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:719)
at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:749)
- locked <0x000000078e8ecf90> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
{code}
Found 1 deadlock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
4173
days inactive
4173
days old
richfaces-issues@lists.jboss.org
0 comments
1 participants
participants (1)
-
Kurt Mansel (JIRA)