[jboss-user] [Clustering/JBoss] - Serious production problems

[Marlig]

Hi,

we are using JBoss 4.0.4 under Solaris 5.8 and JDK 1.5 in production. JBoss is fronted by Apache (2.0.43) with mod_jk (1.2.19) for load-balancing the web-frontend. We have two JBoss cluster nodes running.

Every couple of days (3-5 days) we run into serious problems which ultimately require the restart of the whole JBoss cluster. The first sign is that the number of Apache processes increases rapidly (from around 30 to well over 200), and then we get an OutOfMemory error on one of the JBoss nodes. However, we are not sure here what is cause and what is effect. 

But we see one behaviour where we believe it might be a part of the problem. We were able to recover some information while the server was having the OutOfMemory erros, and noticed that some threads seem to be stuck while doing some sort of socket access.

The Java stack dump shows entries like this:

  | Thread t at 125: (state = IN_NATIVE)
  | Error occurred during stack walking:
  | 
(nothing after the colon)

for which jstack -m shows this:

  | ----------------- t at 125 -----------------
  | 0xff29eccc	_read + 0x8
  | 0xfacdc1a4	Java_java_net_SocketInputStream_socketRead0 + 0x1fc
  | 
(just that)

Has anyone of you seen this kind of problem before? Or maybe knows that it is the normal behaviour and definitely not the source of our OutOfMemorys?

Btw, we also noticed that during the OutOfMemory situation, the memory consumed by byte[]-arrays is much higher (magnitude of 10) than during normal operation.

Maybe any other idea on the problem?

Thanks a lot
Martin

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4037606#4037606

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4037606