[Installation, Configuration & DEPLOYMENT] - Running out of file descriptors when redeploying EAR - jboss-user

Monday, 5 January 2009

I'm trying out JBossAS 4.3.0 GA CP03.  I've discovered that I run out of
deployment descriptors when redeploying things in the container.  I've tested this out
on a base installation of EAP.  I deployed a single ear in the container.  The ear
contains some MDB's.  I then used the following to redeploy the ear every 6 seconds up
to 1000 times:

for i in `seq 1 1000`; do touch /var/lib/jbossas/server/production/deploy/MY-EAR.ear;
sleep 6; echo "lsof count ($i) = `ps aux | grep "^jboss" | grep Main | sed
's/^[^ ]*\s*\([^ ]*\).*/lsof -p \1 | wc -l/g' | sh`"; done

This redeploys the ear, waits 6 seconds, then prints the line count from lsof for the
jboss process along with the redeployment count.  In my most test I run out of file
descriptors after 315 redeployments.  I've seen it fail a lot more.  If I add in the
full complement of stuff normally deployed for my scenario I see failure after as few as 7
redeployments.

Checking the lsof count (lsof -p 9999 | wc -l) shows the count goes up for a while, then
drops down.  This drop occurs roughly every 50 redeploys, but never drops down
completely.

Here is my original post:
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4199195

Looking at lsof changes over time, it looks like the container is keeping a handle on the
jar files contained within the ear, even after the ear is undeployed completely.

Any idea why this is happening?  I'm currently using JBossAS 4.3.0 GA CP01 and have
never seen this problem in that version nor any previous versions.  Note that I was unable
to use CP02 due to some bugs that have been fixed in CP03.

Here is the exception I got in my initial testing, as noted on the linked post:

2008-12-19 14:11:36,744 ERROR [org.jboss.deployment.MainDeployer] Could not initialise
deployment: file:/var/lib/jbossas/server/production/deploy/esb-ESB-COMPONENT-NAME.esb
  | org.jboss.deployment.DeploymentException: exception in init of
file:/var/lib/jbossas/server/production/tmp/deploy/tmp61014esb-ESB-COMPONENT-NAME.esb-contents/lib/commons-pool-1.4.jar;
- nested throwable: (java.lang.RuntimeException: java.util.zip.ZipException: error in
opening zip file)
  | 	at
org.jboss.deployment.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:53)
  | 	at org.jboss.deployment.MainDeployer.init(MainDeployer.java:901)
  | 	at org.jboss.deployment.MainDeployer.init(MainDeployer.java:895)
  | 	at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:809)
  | 	at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
  | 	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
  | 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:597)
  | 	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
  | 	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
  | 	at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
  | 	at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
  | 	at
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
  | 	at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
  | 	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
  | 	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
  | 	at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
  | 	at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
  | 	at $Proxy9.deploy(Unknown Source)
  | 	at
org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
  | 	at
org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:610)
  | 	at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
  | 	at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:274)
  | 	at
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
  | Caused by: java.lang.RuntimeException: java.util.zip.ZipException: error in opening
zip file
  | 	at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:74)
  | 	at
org.jboss.util.file.FileProtocolArchiveBrowserFactory.create(FileProtocolArchiveBrowserFactory.java:48)
  | 	at org.jboss.util.file.ArchiveBrowser.getBrowser(ArchiveBrowser.java:57)
  | 	at org.jboss.ejb3.EJB3Deployer.hasEjbAnnotation(EJB3Deployer.java:213)
  | 	at org.jboss.ejb3.EJB3Deployer.accepts(EJB3Deployer.java:279)
  | 	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
  | 	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  | 	at java.lang.reflect.Method.invoke(Method.java:597)
  | 	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
  | 	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
  | 	at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
  | 	at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
  | 	at
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
  | 	at org.jboss.mx.interceptor.DynamicInterceptor.invoke(DynamicInterceptor.java:97)
  | 	at
org.jboss.deployment.SubDeployerInterceptor.invokeNext(SubDeployerInterceptor.java:124)
  | 	at
org.jboss.deployment.SubDeployerInterceptor.invoke(SubDeployerInterceptor.java:109)
  | 	at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
  | 	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
  | 	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
  | 	at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)
  | 	at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
  | 	at $Proxy34.accepts(Unknown Source)
  | 	at org.jboss.deployment.MainDeployer.findDeployer(MainDeployer.java:1078)
  | 	at org.jboss.deployment.MainDeployer.init(MainDeployer.java:857)
  | 	... 22 more
  | Caused by: java.util.zip.ZipException: error in opening zip file
  | 	at java.util.zip.ZipFile.open(Native Method)
  | 	at java.util.zip.ZipFile.<init>(ZipFile.java:114)
  | 	at java.util.jar.JarFile.<init>(JarFile.java:133)
  | 	at java.util.jar.JarFile.<init>(JarFile.java:97)
  | 	at org.jboss.util.file.JarArchiveBrowser.<init>(JarArchiveBrowser.java:69)
  | 	... 45 more
  | 2008-12-19 14:11:41,763 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:11:46,767 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:11:51,771 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:11:56,775 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:01,780 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:06,784 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:11,788 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:16,792 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:21,804 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:26,809 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:31,814 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:36,818 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:41,822 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown
  | 2008-12-19 14:12:45,307 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in
job executor thread. waiting 5000 milliseconds
  | org.jbpm.JbpmException: couldn't start JTA transaction
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17)
  | 	at org.jbpm.svc.Services.getService(Services.java:144)
  | 	at org.jbpm.svc.Services.getPersistenceService(Services.java:183)
  | 	at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628)
  | 	at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561)
  | 	at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112)
  | 	at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)
  | Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with
name UserTransaction
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49)
  | 	... 8 more
  | Caused by: javax.naming.CommunicationException [Root exception is
java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS; nested
exception is: 
  | 	java.net.SocketException: Too many open files]
  | 	at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
  | 	at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
  | 	at javax.naming.InitialContext.lookup(InitialContext.java:392)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88)
  | 	... 9 more
  | Caused by: java.rmi.ConnectIOException: Exception creating connection to: IP_ADDRESS;
nested exception is: 
  | 	java.net.SocketException: Too many open files
  | 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:614)
  | 	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
  | 	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
  | 	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
  | 	at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
  | 	at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
  | 	... 12 more
  | Caused by: java.net.SocketException: Too many open files
  | 	at java.net.Socket.createImpl(Socket.java:388)
  | 	at java.net.Socket.<init>(Socket.java:362)
  | 	at java.net.Socket.<init>(Socket.java:180)
  | 	at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
  | 	at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
  | 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
  | 	... 17 more
  | 2008-12-19 14:12:50,314 ERROR [org.jbpm.job.executor.JobExecutorThread] exception in
job executor thread. waiting 10000 milliseconds
  | org.jbpm.JbpmException: couldn't start JTA transaction
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:51)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.<init>(JtaDbPersistenceService.java:28)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceServiceFactory.openService(JtaDbPersistenceServiceFactory.java:17)
  | 	at org.jbpm.svc.Services.getService(Services.java:144)
  | 	at org.jbpm.svc.Services.getPersistenceService(Services.java:183)
  | 	at org.jbpm.JbpmContext.getPersistenceService(JbpmContext.java:628)
  | 	at org.jbpm.JbpmContext.getJobSession(JbpmContext.java:561)
  | 	at org.jbpm.job.executor.JobExecutorThread.acquireJobs(JobExecutorThread.java:112)
  | 	at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:58)
  | Caused by: org.jbpm.JbpmException: couldn't lookup UserTransaction in JNDI with
name UserTransaction
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:90)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.beginJtaTransaction(JtaDbPersistenceService.java:49)
  | 	... 8 more
  | Caused by: javax.naming.CommunicationException: Could not obtain connection to any of
these urls: IP_ADDRESS:1099 and discovery failed with error:
javax.naming.CommunicationException: Too many open files [Root exception is
java.net.SocketException: Too many open files] [Root exception is
javax.naming.CommunicationException: Failed to connect to server IP_ADDRESS:1099 [Root
exception is javax.naming.ServiceUnavailableException: Failed to connect to server
IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]]]
  | 	at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1562)
  | 	at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634)
  | 	at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
  | 	at javax.naming.InitialContext.lookup(InitialContext.java:392)
  | 	at
org.jbpm.persistence.jta.JtaDbPersistenceService.getUserTransaction(JtaDbPersistenceService.java:88)
  | 	... 9 more
  | Caused by: javax.naming.CommunicationException: Failed to connect to server
IP_ADDRESS:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to
connect to server IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many
open files]]
  | 	at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:274)
  | 	at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533)
  | 	... 13 more
  | Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server
IP_ADDRESS:1099 [Root exception is java.net.SocketException: Too many open files]
  | 	at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:248)
  | 	... 14 more
  | Caused by: java.net.SocketException: Too many open files
  | 	at java.net.Socket.createImpl(Socket.java:388)
  | 	at java.net.Socket.<init>(Socket.java:362)
  | 	at java.net.Socket.<init>(Socket.java:267)
  | 	at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:84)
  | 	at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:77)
  | 	at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:244)
  | 	... 14 more
  | 2008-12-19 14:12:51,842 WARN  [org.jboss.deployment.scanner.URLDeploymentScanner] Scan
URL, caught java.io.IOException: Could not list directory
'/var/lib/jbossas/server/production/deploy', reason unknown

View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4199560#...

Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[Installation, Configuration & DEPLOYMENT] - Running out of file descriptors when redeploying EAR