JBoss Embedded AS - JVM keeps running
by Jaikiran Pai
While experimenting with the latest JBoss Embedded AS (against AS trunk)
from within a standalone Java client we have been seeing that the JVM
never shuts down. The relevant code looks like:
public class EmbeddedJBossASClient
{
private JBossASEmbeddedServer jbossServer;
public static void main(String[] args) throws Exception
{
EmbeddedJBossASClient embeddedASClient = new EmbeddedJBossASClient();
try
{
embeddedASClient.startServer(JBOSS_HOME);
embeddedASClient.deployEjb();
// call ejb
int result = embeddedASClient.invokeSumOnEjb(1, 2);
System.out.println("Result is " + result);
}
finally
{
embeddedASClient.shutdownServer();
}
}
...
}
Not much code - it just starts the embedded server within the current VM
deploys an EJB and shuts down the server. So after
embeddedASClient.shutdownServer() returns, it would be expected that the
JVM shuts down. However, the JVM keeps running even after the call
returns successfully. jstack util on this JVM process shows the
following threads [2] which are still active. Most of the threads are
"daemon" which means that they won't prevent the JVM from shutting down
[1].
However, there are some non-daemon threads still active even after a
server shutdown:
"pool-15-thread-1" prio=10 tid=0x8cb1ec00 nid=0x4d13 waiting on
condition [0x8e7a6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xb20d6258> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
...
"pool-5-thread-1" prio=10 tid=0x8cfc2800 nid=0x4ce0 waiting on condition
[0x8cdad000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x97ece7e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
These pool-x-thread-y threads prevent the JVM from shutting down. The
thread names aren't very informative to figure out which service started
those threads and why were those not shutdown. Anyone knows what these
threads are for and why they are hanging around after a server shutdown?
Note that this issue isn't observed in a normal JBoss AS server
environment because:
a) When the server is shutdown through a Ctrl+C, it leads to JVM
termination and effectively these threads are nuked.
b) When the server is shutdown through shutdown.sh (or through
jmx-console), it leads to
org.jboss.system.server.jmx.JMXKernel.shutdown() which does a
Runtime.exit() leading to JVM shutdown:
public void shutdown()
{
...
// JBAS-6916:
//TODO This must be removed, VM exit is not a feature of server
shutdown.
// Now exit the Runtime, because AS has non-daemon Threads still
// chugging away.
new Thread()
{
public void run()
{
log.info("Server exit called, exiting the JVM now!");
Runtime.getRuntime().exit(1);
}
}.start();
}
[1]
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Runtime.html#addShutdow...
-
"The Java virtual machine shuts down in response to two kinds of events:
* The program exits normally, when the last non-daemon thread exits
or when the exit (equivalently, System.exit) method is invoked, or
* The virtual machine is terminated in response to a user interrupt,
such as typing ^C, or a system-wide event, such as user logoff or system
shutdown."
[2] Entire thread dump from jstack:
2010-04-13 23:15:11
Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x08da4000 nid=0x4d2a waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" prio=10 tid=0x8cf17000 nid=0x4cc8 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"ConnectionValidator" daemon prio=10 tid=0x8cb7a000 nid=0x4d17 in
Object.wait() [0x8b338000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb2108f88> (a java.util.concurrent.CopyOnWriteArrayList)
at
org.jboss.resource.connectionmanager.ConnectionValidator$ConnectionValidatorRunnable.run(ConnectionValidator.java:156)
- locked <0xb2108f88> (a java.util.concurrent.CopyOnWriteArrayList)
at java.lang.Thread.run(Thread.java:619)
"IdleRemover" daemon prio=10 tid=0x8c924800 nid=0x4d16 in Object.wait()
[0x8b2e7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb2108d90> (a java.util.concurrent.CopyOnWriteArrayList)
at
org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:167)
- locked <0xb2108d90> (a java.util.concurrent.CopyOnWriteArrayList)
at java.lang.Thread.run(Thread.java:619)
"pool-15-thread-1" prio=10 tid=0x8cb1ec00 nid=0x4d13 waiting on
condition [0x8e7a6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xb20d6258> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"ajp-127.0.0.1-8009-Poller" daemon prio=10 tid=0x8c92d000 nid=0x4d11 in
Object.wait() [0x8b1f4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb210f200> (a
org.apache.tomcat.util.net.JIoEndpoint$Poller)
at
org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:764)
- locked <0xb210f200> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-127.0.0.1-8080-Acceptor-0" daemon prio=10 tid=0x8c935000
nid=0x4d10 waiting on condition [0x8b245000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:335)
at java.lang.Thread.run(Thread.java:619)
"http-127.0.0.1-8080-Poller" daemon prio=10 tid=0x8c936800 nid=0x4d0f in
Object.wait() [0x8b296000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb210f650> (a
org.apache.tomcat.util.net.JIoEndpoint$Poller)
at
org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:764)
- locked <0xb210f650> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"JCA PoolFiller" daemon prio=10 tid=0x8cbb1800 nid=0x4d09 in
Object.wait() [0x8b47c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xb2040a00> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:485)
at
org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:88)
- locked <0xb2040a00> (a java.util.LinkedList)
at java.lang.Thread.run(Thread.java:619)
"Timer-1" daemon prio=10 tid=0x8cb6fc00 nid=0x4d07 in Object.wait()
[0x8b51e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x989f04e8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x989f04e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Thread-9" daemon prio=10 tid=0x8cf3c400 nid=0x4cf8 in Object.wait()
[0x8b9dd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x98847c88> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:485)
at
com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:421)
- locked <0x98847c88> (a java.util.LinkedList)
at
com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:74)
"Thread-8" daemon prio=10 tid=0x8cf3b800 nid=0x4cf7 in Object.wait()
[0x8ba2e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x988471f8> (a
com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at
com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:104)
- locked <0x988471f8> (a
com.arjuna.ats.arjuna.coordinator.TransactionReaper)
"Listener:4713" daemon prio=10 tid=0x8cbc8000 nid=0x4cf2 runnable
[0x8bb72000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x98824280> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:127)
"Timer-0" daemon prio=10 tid=0x8cbfbc00 nid=0x4ce4 in Object.wait()
[0x8ccba000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x98449890> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x98449890> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"pool-5-thread-1" prio=10 tid=0x8cfc2800 nid=0x4ce0 waiting on condition
[0x8cdad000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x97ece7e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x08e3f000 nid=0x4cd1 runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x08e3d000 nid=0x4cd0 waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x08e3a400 nid=0x4ccf waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08e38c00 nid=0x4cce runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x08e29000 nid=0x4ccd in Object.wait()
[0x90545000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x94be4528> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x94be4528> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x08e24400 nid=0x4ccc in
Object.wait() [0x90596000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x94b398d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x94b398d8> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x08e20400 nid=0x4ccb runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x08daac00 nid=0x4cc9 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x08dac000 nid=0x4cca runnable
"VM Periodic Task Thread" prio=10 tid=0x08e9cc00 nid=0x4cd2 waiting on
condition
JNI global references: 678
regards,
-Jaikiran
15 years, 12 months
Re: [jboss-dev] JBossOSGi 1.0.0.Beta7 released
by Heiko Braun
great job. congrats!
On Apr 15, 2010, at 5:30 PM, Tom Fennelly wrote:
> Well done guys!
>
> Thomas Diesler wrote:
>>
>> Folks,
>>
>> I am happy to announce the release of JBossOSGi-1.0.0.Beta7.
>>
>> You can download the binary here: jboss-osgi-installer-1.0.0.Beta7.jar <http://sourceforge.net/projects/jboss/files/JBossOSGi/1.0.0.Beta7/jboss-o...>
>>
>> The release comes with improvements in the following areas
>>
>> * Enterprise OSGi bundle management (JMX)
>> * JSR160 compliant JMXConnector
>> * Integrate JBoss OSGi with AS 6.0.0.M3
>> * Framework service hooks
>> * OSGi core TCK compliance 40%
>>
>> For details please have a look at the latest version of our User Guide <http://jbmuc.dyndns.org/jboss-osgi-1.0.0.Beta7/userguide/html/>.
>>
>> cheers
>> -thomas
>>
>> --
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> Thomas Diesler
>> JBoss OSGi Lead
>> JBoss, a division of Red Hat
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
15 years, 12 months
fix AS trunk testing on windows
by Alexey Loubyansky
It currently isn't working (e.g. one-test target) since the classpath is
too long. To work around the issue, a manifest-only jar has to be
created which would reference all the jars and dirs. Can we fix testing
this way?
This is what I am doing on my laptop at the moment.
Thanks,
Alexey
16 years
OSGi removed from 'all' profile
by Thomas Diesler
Hi Jason,
I see that you removed OSGi from the 'all' profile. Were there any
specific issues with it? I couldn't find pointers to why this was necessary.
I reopened https://jira.jboss.org/jira/browse/JBAS-7661 where I make a
comment on why I think osgi should be part of the 'all' profile. I am
now working on osgi smoke tests that I ultimate want to run against
'all' as part of the smoke-tests target.
cheers
-thomas
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
16 years
Including testsuite in Eclipse
by Thomas Diesler
I am having a hard time including jbossas/trunk/testsuite into eclipse.
What is the reason that eclpise project files (that do not contain local
settings) are not checked in any more?
More importantly, how do you work with the testsuite in eclipse?
cheers
-thomas
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
16 years
as trunk build error this morning
by Scott Marlow
I updated as trunk source and tried a clean build and get the following
error:
[INFO] Building JBoss Application Server Testsuite
[INFO] task-segment: [clean, install, idea:idea]
[INFO]
------------------------------------------------------------------------
[INFO] [clean:clean {execution: default-clean}]
[INFO] [buildnumber:create {execution: set-build-properties}]
[INFO] [build-helper:maven-version {execution: default}]
[INFO] [enforcer:enforce {execution: enforce-plugin-versions}]
[INFO] [enforcer:enforce {execution: enforce-java-version}]
[INFO] [enforcer:enforce {execution: enforce-maven-version}]
[INFO] [enforcer:enforce {execution: ban-bad-dependencies}]
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
failed with message:
Found Banned Dependency:
org.jboss.logging:jboss-logging-spi:jar:2.1.0.GA
I'm looking for the offender now.
Scott
16 years
New AS module: "distribution"
by Andrew Lee Rubinger
https://jira.jboss.org/jira/browse/JBAS-7919
The final build-based change introduced this week is the split of the
"build" module into both "build" and "dist".
"jboss-as-build" is tasked with creating the $JBOSS_HOME assembly.
"jboss-as-distribution" ZIPs this up and installs into the Maven repo.
Inbetween we can run a series of checks to ensure the integrity before
we install/deploy the distribution. For instance, the EmbeddedAS
integration testsuite runs between these modules, using
build/target/$JBOSS_HOME.
I've locally tested the split of build/dist using the "release" profile,
but I'm not intimately familiar with the formal release process.
Perhaps someone from QA would like to verify that everything is really
in check before we cut M3?
S,
ALR
--
Andrew Lee Rubinger
Sr. Software Engineer
JBoss by Red Hat
http://exitcondition.alrubinger.com
http://twitter.com/ALRubinger
16 years
New AS Module: "embedded"
by Andrew Lee Rubinger
https://jira.jboss.org/jira/browse/JBAS-7916
https://jira.jboss.org/jira/browse/JBAS-7771
The EmbeddedAS libraries are now a part of the AS distribution, along
with ShrinkWrap. All are available under $JBOSS_HOME/common/lib.
Users wishing to take advantage of JBossAS via these APIs in an existing
runtime are now free to do so without any additional downloads.
In addition, I've added the "embedded" module to AS; its purpose is to
provide the build with an Integration TestSuite running in Embedded
mode. This acts as a guard; if it fails the Hudson build job for AS6
will not continue, and the distribution will not be installed into any
local or remote Maven repos.
To run the EmbeddedAS testsuite, use "mvn install -Pembedded". The
"embedded" profile is disabled by default so we don't impact regular
build time too adversely, but IMO any incoming commits should run this
*and* smoke-tests before making it upstream. The Hudson QA job has been
altered to use this profile when building.
In addition, this now opens the door to us using Embedded testing as a
tool for ourselves. For instance we can take advantage of sharing
memory between the test and the application server by using locks,
barriers, latches, and pass-by-reference.
The existing integration test passes through some JEE techs:
http://anonsvn.jboss.org/repos/jbossas/trunk/embedded/src/test/java/org/j...
S,
ALR
--
Andrew Lee Rubinger
Sr. Software Engineer
JBoss by Red Hat
http://exitcondition.alrubinger.com
http://twitter.com/ALRubinger
16 years
