[Beginner's Corner] - JBoss 5.1.0.GA Severe Latency Problem
by cdegeiso
cdegeiso [https://community.jboss.org/people/cdegeiso] created the discussion
"JBoss 5.1.0.GA Severe Latency Problem"
To view the discussion, visit: https://community.jboss.org/message/832794#832794
--------------------------------------------------------------
Hi all,
I have an issue that I have not been able to determine whether it is a symptom or a cause of a severe latency problem I am experiencing with a deployment of a COTS package using JBoss 5.1.0.GA. We use 5.1 because that is what is currently supported with the application. I have heard the next release will have JBoss 7 support and am anxious to move on to something more solid.
We've been running 5.1 for a couple of years now (JBoss 4.2.1 years before that) and it used to run fine in JBoss 4 and for a little over a year on 5.1, then performance started degrading and has become unusable at times. Although we have been using JBoss for a long time, I am by no means a JBoss expert. I know enough to get it running and processing data for our application, that's about it. Unfortunately, due to apathy of system admins (don't ask), the task of figuring out what is going on has been assigned to me (an application administrator), so I will have to get spun up quick on it.
First some symptoms. After a few days of the application running, latency starts to drift in. It will get so bad that transaction times are clocked at roughly 75s per transaction. CPU and memory usage are good on windows boxes and HP/UX database box, I can access JMX console just fine and monitor activity there, but some anomalies exist. One, JCA connections Available and connections in use do not equal our total connection pool size. For example, our connection pool is set to max 300 connections and I will seel something like 164 connections in use and 52 avaliable. When we aren't having issues, these two numbers always equal 300 if added together. The only way we have found to recover is to completely shut off the JBoss servers, bump the database, and restart the JBoss servers. Stopping JBoss services doesn't fix it and neither does restarting the JBoss servers. The database has to clear out all sessions on it's end before JBoss will connect again. DBA states that the database is fully functioning during this time and has not provided any anomalies from logs on the oracle end.
The second thing that occurs when we get slammed with latency is when I run a thread dump to see what tasks are currently processing during the latency, I see a lot of threads that look like this:
*Thread: Thread-6293325* : priority:5, demon:false, threadId:6370353, threadState:RUNNABLE
> java.net.SocketInputStream.socketRead0(Native Method)
> java.net.SocketInputStream.__AW_read(Unknown Source)
> java.net.SocketInputStream.read(Unknown Source)
> oracle.net.ns.Packet.receive(Packet.java:300)
> oracle.net.ns.DataPacket.receive(DataPacket.java:106)
> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
> oracle.net.ns.NetInputStream.__AW_read(NetInputStream.java:260)
> oracle.net.ns.NetInputStream.read(NetInputStream.java)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> oracle.jdbc.driver.T4CTTIoping.doOPING(T4CTTIoping.java:52)
> oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:4008)
> - locked <0x14948d6e> (a oracle.jdbc.driver.T4CConnection)
> oracle.jdbc.driver.PhysicalConnection$3.run(PhysicalConnection.java:7868)
> java.lang.Thread.run(Unknown Source)
>
*Thread: Thread-6293326* : priority:5, demon:true, threadId:6370354, threadState:RUNNABLE
*Thread: Thread-6293327* : priority:5, demon:true, threadId:6370355, threadState:RUNNABLE
> java.net.SocketInputStream.socketRead0(Native Method)
> java.net.SocketInputStream.__AW_read(Unknown Source)
> java.net.SocketInputStream.read(Unknown Source)
> oracle.net.ns.Packet.receive(Packet.java:300)
> oracle.net.ns.DataPacket.receive(DataPacket.java:106)
> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
> oracle.net.ns.NetInputStream.__AW_read(NetInputStream.java:260)
> oracle.net.ns.NetInputStream.read(NetInputStream.java)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> oracle.jdbc.driver.T4CTTIoping.doOPING(T4CTTIoping.java:52)
> oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:4008)
> - locked <0x65f90b32> (a oracle.jdbc.driver.T4CConnection)
> oracle.jdbc.driver.PhysicalConnection$3.run(PhysicalConnection.java:7868)
> java.lang.Thread.run(Unknown Source)
>
*Thread: Thread-6293328* : priority:5, demon:true, threadId:6370356, threadState:RUNNABLE
*Thread: Thread-6293329* : priority:5, demon:true, threadId:6370357, threadState:RUNNABLE
*Thread: Thread-6293330* : priority:5, demon:true, threadId:6370358, threadState:RUNNABLE
*Thread: Thread-6293331* : priority:5, demon:true, threadId:6370359, threadState:RUNNABLE
*Thread: Thread-6293332* : priority:5, demon:true, threadId:6370360, threadState:RUNNABLE
*Thread: Thread-6293323* : priority:5, demon:true, threadId:6370361, threadState:RUNNABLE
*Thread: Thread-6293333* : priority:5, demon:true, threadId:6370362, threadState:RUNNABLE
*Thread: Thread-6293334* : priority:5, demon:true, threadId:6370363, threadState:RUNNABLE
All of the other threads have some kind of name that I can determine what they are used for, like:
*Thread: ajp-0.0.0.0-8005-197*
**Thread: ReportQueueAgent**
***Thread: HeartbeatThread***
All of these threads have stack traces that I can see what they are doing and can be reasonably sure that they are doing something productive. The generic threads (what I call them) above don't really seem to do much other than open connections and hold onto them. When things are running smoothly, the generic threads are non-existent. They gradually build up over time though as latency kicks in. These also seem to correlate with higher user connections (the more users that are in, the more likely performance will tank).
As I said earlier, I am no expert, but I feel like there is a direct correlation to these threads and our latency issue. While they may not be the cause, I would like to know how I go about getting JBoss to tell me why these threads are being created so I can find a way to stop it or, in the case of a code defect, submit it to the developer for review.
More info pertaining to our environment. I am not sure it matters, but in case it does, here are some specs:
User Base: Total 4,000 users
Concurrent Users: 175 (peak), 110 (avg)
No Load balancing - App and process servers serve two completely separate functions, although they both run JBoss and TRIRIGA.
(All are virtual with the exception of HP/UX)
Web Server: AMD64 Dual Core processor, 4GB Ram, Windows 2003, IIS 6.0, ISAPI Redirector 1.2.32
App Server: AMD64 Dual Core processor, 12GB Ram, Windows 2003, JBoss 5.1.0.GA (hosting Tririga) - Java Heap 8GB
Process Server: AMD64 Dual Core processor, 10GB Ram, Windows 2003, JBoss 5.1.0.GA (hosting Tririga) - Java Heap 6GB
Report Server: AMD64 Dual Core processor, 4GB Ram, Windows 2003, Crystal RAS 2008 SP3
Database Server: 6 processors, 18GB Ram, Oracle 11gR2
Windows boxes will be upgraded to Windows 2008R2 within the next couple of months.
Thanks,
Chris
--------------------------------------------------------------
Reply to this message by going to Community
[https://community.jboss.org/message/832794#832794]
Start a new discussion in Beginner's Corner at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&con...]
12 years, 3 months
[jBPM] - Having issues getting a KnowledgeSession when using JPA persistence
by Marshall Lucas
Marshall Lucas [https://community.jboss.org/people/mlucas67] created the discussion
"Having issues getting a KnowledgeSession when using JPA persistence"
To view the discussion, visit: https://community.jboss.org/message/831230#831230
--------------------------------------------------------------
Seems that Fuse is having issues finding classes across bundles. Any ideas? Marshall 2013-08-02 16:41:11,740 | ERROR | NAPSHOT-thread-1 | JbpmEngine | ? ? | 239 - gov.faa.apws.workflow.workflow-impl - 0.0.1.SNAPSHOT | java.lang.IllegalStateException: java.lang.reflect.InvocationTargetExceptio n java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException at org.drools.persistence.jpa.KnowledgeStoreServiceImpl.buildCommandService(KnowledgeStoreServiceImpl.java:142)[228:org.droo ls.persistence-jpa:5.5.0.Final] at org.drools.persistence.jpa.KnowledgeStoreServiceImpl.newStatefulKnowledgeSession(KnowledgeStoreServiceImpl.java:66)[228:o rg.drools.persistence-jpa:5.5.0.Final] at gov.faa.apws.workflow.server.impl.JbpmEngine.startTheEngine(JbpmEngine.java:414)[239:gov.faa.apws.workflow.workflow-impl: 0.0.1.SNAPSHOT] at gov.faa.apws.workflow.server.impl.JbpmEngine.setResourceFactoryService(JbpmEngine.java:324)[239:gov.faa.apws.workflow.wor kflow-impl:0.0.1.SNAPSHOT] at gov.faa.apws.workflow.server.osgi.ResourceFactoryServiceTracker.addingService(ResourceFactoryServiceTracker.java:42)[239: gov.faa.apws.workflow.workflow-impl:0.0.1.SNAPSHOT] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:932)[karaf.jar:2.3.0.redhat-60024] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:864)[karaf.jar:2.3.0.redhat-60024] at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)[karaf.jar:2.3.0.redhat-60024] at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)[karaf.jar:2.3.0.redhat-60024] at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:317)[karaf.jar:2.3.0.redhat-60024] at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)[karaf.jar:2.3.0.redhat-60024] at gov.faa.apws.workflow.server.impl.JbpmMonitor.init(JbpmMonitor.java:52)[239:gov.faa.apws.workflow.workflow-impl:0.0.1.SNA PSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.6.0_38] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)[:1.6.0_38] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[:1.6.0_38] at java.lang.reflect.Method.invoke(Method.java:597)[:1.6.0_38] at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:297)[14:org.apache.aries.blueprint.core:1.0. 1.redhat-60024] at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:958)[14:org.apache.aries.blueprint.core:1.0.1.redh at-60024] at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:712)[14:org.apache.aries.blueprint.core:1 .0.1.redhat-60024] at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:824)[14:org.apache.aries.blueprint.core:1 .0.1.redhat-60024] at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:787)[14:org.apache.aries.blueprint.core:1. 0.1.redhat-60024] at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[14:org.apache.aries.blueprint.core:1.0.1.redh at-60024] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)[:1.6.0_38] at java.util.concurrent.FutureTask.run(FutureTask.java:138)[:1.6.0_38] at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[14:org.apache.aries.blueprint.core:1.0.1.redh at-60024] at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245)[14:org.apache.arie s.blueprint.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183)[14:org.apache.aries.blue print.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:667)[1 4:org.apache.aries.blueprint.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:370)[14:org.apache.aries.bl ueprint.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)[14:org.apache.aries.blue print.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:270)[14:org.apache.aries.bl ueprint.core:1.0.1.redhat-60024] at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:233)[14:org.apache.aries.blu eprint.core:1.0.1.redhat-60024] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[1 3:org.apache.aries.util:1.0.1.redhat-60024] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[1 3:org.apache.aries.util:1.0.1.redhat-60024] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[13:org .apache.aries.util:1.0.1.redhat-60024] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[13:org .apache.aries.util:1.0.1.redhat-60024] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[13:org .apache.aries.util:1.0.1.redhat-60024] at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)[org.apache.felix.framework-4.0 .3.redhat-60024.jar:] at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:696)[org.apache.felix.frame work-4.0.3.redhat-60024.jar:] at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:484)[org.apache.felix.framework-4.0. 3.redhat-60024.jar:] at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4580)[org.apache.felix.framework-4.0.3.redhat-60024.jar:] at org.apache.felix.framework.Felix$4.run(Felix.java:2115)[org.apache.felix.framework-4.0.3.redhat-60024.jar:] at org.apache.felix.framework.Felix$5.run(Felix.java:2159)[org.apache.felix.framework-4.0.3.redhat-60024.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)[:1.6.0_38] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)[:1.6.0_38] at java.util.concurrent.FutureTask.run(FutureTask.java:138)[:1.6.0_38] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[:1.6.0_38] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[:1.6.0_38] at java.lang.Thread.run(Thread.java:662)[:1.6.0_38] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.6.0_38] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)[:1.6.0_38] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)[:1.6.0_38] at java.lang.reflect.Constructor.newInstance(Constructor.java:513)[:1.6.0_38] at org.drools.persistence.jpa.KnowledgeStoreServiceImpl.buildCommandService(KnowledgeStoreServiceImpl.java:128)[228:org.droo ls.persistence-jpa:5.5.0.Final] ... 48 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.drools.persistence.jpa.JpaTimeJobFactoryManager not fou nd by org.drools.core [226] at org.drools.TimerJobFactoryType$3.createInstance(TimerJobFactoryType.java:26)[226:org.drools.core:5.5.0.Final] at org.drools.SessionConfiguration.getTimerJobFactoryManager(SessionConfiguration.java:274)[226:org.drools.core:5.5.0.Final] at org.drools.common.AbstractWorkingMemory.(AbstractWorkingMemory.java:316) at org.drools.common.AbstractWorkingMemory.(AbstractWorkingMemory.java:246) at org.drools.common.AbstractWorkingMemory.(AbstractWorkingMemory.java:212) at org.drools.reteoo.ReteooWorkingMemory.(ReteooWorkingMemory.java:103) at org.drools.reteoo.ReteooStatefulSession.(ReteooStatefulSession.java:81) at org.drools.reteoo.ReteooRuleBase.newStatefulSession(ReteooRuleBase.java:402) at org.drools.reteoo.ReteooRuleBase.newStatefulSession(ReteooRuleBase.java:387) at org.drools.impl.KnowledgeBaseImpl.newStatefulKnowledgeSession(KnowledgeBaseImpl.java:175) at org.drools.persistence.SingleSessionCommandService.(SingleSessionCommandService.java:120)[228:org.drools.persistenc e-jpa:5.5.0.Final] ... 53 more Caused by: java.lang.ClassNotFoundException: org.drools.persistence.jpa.JpaTimeJobFactoryManager not found by org.drools.core [226] at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1532) at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75) at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955) at java.lang.ClassLoader.loadClass(ClassLoader.java:247)[:1.6.0_38] at java.lang.Class.forName0(Native Method)[:1.6.0_38] at java.lang.Class.forName(Class.java:169)[:1.6.0_38] at org.drools.TimerJobFactoryType$3.createInstance(TimerJobFactoryType.java:24)[226:org.drools.core:5.5.0.Final] ... 63 more
--------------------------------------------------------------
Reply to this message by going to Community
[https://community.jboss.org/message/831230#831230]
Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&con...]
12 years, 3 months
[IronJacamar] - mem leak in TransactionSynchronizer on afterCompletion() phase
by Andrey Smirnov
Andrey Smirnov [https://community.jboss.org/people/cheezzario] created the discussion
"mem leak in TransactionSynchronizer on afterCompletion() phase"
To view the discussion, visit: https://community.jboss.org/message/831728#831728
--------------------------------------------------------------
Hello.
Let me introduce the problem
App server ( jboss-as-7.1.1.Final ) which is a part of complex distributed app using CORBA for communication fails with OutOfMemoryError
I've investigated the problem and seems like root cause is inability to clean up txSynchs ( and locks ) maps resided in org.jboss.jca.core.connectionmanager.transaction.TransactionSynchronizer class.
More detailed:
Collections mentioned above should be cleaned up in afterCompletion() method but TX is unavailable at that stage, so Transaction.hashCode() returns -1 which leads to dangled TX associations.
Right now I don't understand why javax.transaction.Transaction is used as a key in this maps? For me it seems like this code wil always fail.
I've attached modified source code of TransactionSynchronizer which I used during investigation and partial app log with records specific to one TX.
Regards, Andrey.
--------------------------------------------------------------
Reply to this message by going to Community
[https://community.jboss.org/message/831728#831728]
Start a new discussion in IronJacamar at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&con...]
12 years, 3 months
[jBPM] - jbpm-5.4.0.Final-installer-full.zip fails to install
by Al Koch
Al Koch [https://community.jboss.org/people/alkoch] created the discussion
"jbpm-5.4.0.Final-installer-full.zip fails to install"
To view the discussion, visit: https://community.jboss.org/message/832171#832171
--------------------------------------------------------------
Hello,
I'm new to jBMP and need to evaluate it. I have purchased the text jBPM 5 Developer Guide by Mauricio Salatino and Esteban Aliverti and am following the instructions in Chapter 4 for installation using jbpm-5.4.0.Final-installer-full.zip:
- I have a fresh Windows 7 machine.
- I have installed the latest Jave EE SDK.
- I have installed the latest Ant.
- I launch the installtion with *+ant install.demo > InstallLog.txt+*.
The installation runs for a while but then terminates with the following:
C:\jbpm-installer>ant install.demo > InstallLog.txt
[copy] Warning: Could not find file C:\jbpm-installer\db\jbpm-persistence.xml to copy.
[copy] Warning: Could not find file C:\jbpm-installer\db\task-persistence.xml to copy.
[copy] Warning: Could not find file C:\jbpm-installer\db\Taskorm.xml to copy.
BUILD FAILED
C:\jbpm-installer\build.xml:254: Can't get http://download.eclipse.org/technology/epp/downloads/release/helios/SR2/e... http://download.eclipse.org/technology/epp/downloads/release/helios/SR2/e... to C:\jbpm-installer\lib\eclipse-java-helios-SR2-win32.zip
Total time: 2 minutes 5 seconds
C:\jbpm-installer>
I'm not sure how serious the Warnings are but the failure to install Eclipse is fatal. I assume this error is because Helios is several versions back and is no longer at the supplied URL. Why would the latest installer still reference an out-of-date version of Eclipse?
Can someone please help me with the following:
- How do I get a successful installation? I'd really prefer to have the current version of Eclipse (Kepler) installed so is it possible to alter the installation to do that?
- How do I resolve the Warnings?
- Once I can get the demo setup and get familar wirth the system I will need to incorporate jBPM into an existing code base managed by Eclipse Kepler. How do I locate documentation describing how to install the various components (the jBMP Eclipse plugin, the jBMP system itself, the GWT console code and so forth)?
Thank you for your help.
Al
--------------------------------------------------------------
Reply to this message by going to Community
[https://community.jboss.org/message/832171#832171]
Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&con...]
12 years, 3 months