[Design of JBoss Portal] - Re: portal clustering problem when using optimistic locking
by bstansberry＠jboss.com
>From perf11.log, a lot of threads waiting on what I assume is the DB connection pool:
"ajp-10.18.1.129-8009-523" daemon prio=1 tid=0x0000002b6d668680 nid=0x5b1b in Object.wait() [0x000000006d724000..0x000000006d725db0]
| [JBoss] at java.lang.Object.wait(Native Method)
| [JBoss] - waiting on <0x0000002b37969c98> (a EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore$WaitQueue$WaitNode)
| [JBoss] at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore$WaitQueue$WaitNode.doTimedWait(QueuedSemaphore.java:123)
| [JBoss] - locked <0x0000002b37969c98> (a EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore$WaitQueue$WaitNode)
| [JBoss] at EDU.oswego.cs.dl.util.concurrent.QueuedSemaphore.attempt(QueuedSemaphore.java:47)
| [JBoss] at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:181)
| [JBoss] at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:538)
| [JBoss] at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:347)
| [JBoss] at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:330)
| [JBoss] at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:402)
| [JBoss] at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:849)
| [JBoss] at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:90)
| [JBoss] at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69)
| [JBoss] at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
| [JBoss] at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
| [JBoss] at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
| [JBoss] at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
| [JBoss] at org.hibernate.loader.Loader.doQuery(Loader.java:673)
| [JBoss] at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
| [JBoss] at org.hibernate.loader.Loader.loadEntity(Loader.java:1907)
| [JBoss] at org.hibernate.loader.entity.CollectionElementLoader.loadElement(CollectionElementLoader.java:72)
| [JBoss] at org.hibernate.persister.collection.OneToManyPersister.getElementByIndex(OneToManyPersister.java:360)
| [JBoss] at org.hibernate.collection.AbstractPersistentCollection.readElementByIndex(AbstractPersistentCollection.java:158)
| [JBoss] at org.hibernate.collection.PersistentMap.get(PersistentMap.java:146)
| [JBoss] at org.jboss.portal.core.impl.model.portal.PortalObjectImpl.getChild(PortalObjectImpl.java:427)
| [JBoss] at org.jboss.portal.core.model.portal.command.mapping.DefaultPortalObjectPathMapper$1.getChild(DefaultPortalObjectPathMapper.java:91)
| [JBoss] at org.jboss.portal.server.servlet.PathParser.map(PathParser.java:86)
| [JBoss] at org.jboss.portal.core.model.portal.command.mapping.DefaultPortalObjectPathMapper.getTarget(DefaultPortalObjectPathMapper.java:109)
| [JBoss] at org.jboss.portal.core.model.portal.PortalObjectCommandFactory.doMapping(PortalObjectCommandFactory.java:77)
| [JBoss] at org.jboss.portal.core.controller.command.mapper.CommandFactoryDelegate.doMapping(CommandFactoryDelegate.java:87)
| [JBoss] at org.jboss.portal.core.controller.command.mapper.DelegatingCommandFactoryService.doMapping(DelegatingCommandFactoryService.java:142)
| [JBoss] at org.jboss.portal.core.model.portal.DefaultPortalCommandFactory.doMapping(DefaultPortalCommandFactory.java:69)
| [JBoss] at org.jboss.portal.core.controller.Controller.handle(Controller.java:208)
| [JBoss] at org.jboss.portal.server.RequestControllerDispatcher.invoke(RequestControllerDispatcher.java:51)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:131)
| [JBoss] at org.jboss.portal.core.cms.aspect.IdentityBindingInterceptor.invoke(IdentityBindingInterceptor.java:47)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.server.aspects.server.ContentTypeInterceptor.invoke(ContentTypeInterceptor.java:68)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.core.aspects.server.PortalContextPathInterceptor.invoke(PortalContextPathInterceptor.java:45)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.core.aspects.server.LocaleInterceptor.invoke(LocaleInterceptor.java:96)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.core.aspects.server.UserInterceptor.invoke(UserInterceptor.java:246)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.server.aspects.server.SignOutInterceptor.invoke(SignOutInterceptor.java:98)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.core.impl.api.user.UserEventBridgeTriggerInterceptor.invoke(UserEventBridgeTriggerInterceptor.java:65)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.core.aspects.server.TransactionInterceptor.org$jboss$portal$core$aspects$server$TransactionInterceptor$invoke$aop(TransactionInterceptor.java:49)
| [JBoss] at org.jboss.portal.core.aspects.server.TransactionInterceptor$invoke_N5143606530999904530.invokeNext(TransactionInterceptor$invoke_N5143606530999904530.java)
| [JBoss] at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
| [JBoss] at org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:262)
| [JBoss] at org.jboss.portal.core.aspects.server.TransactionInterceptor$invoke_N5143606530999904530.invokeNext(TransactionInterceptor$invoke_N5143606530999904530.java)
| [JBoss] at org.jboss.portal.core.aspects.server.TransactionInterceptor.invoke(TransactionInterceptor.java)
| [JBoss] at org.jboss.portal.server.ServerInterceptor.invoke(ServerInterceptor.java:38)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.server.aspects.LockInterceptor$InternalLock.invoke(LockInterceptor.java:69)
| [JBoss] at org.jboss.portal.server.aspects.LockInterceptor.invoke(LockInterceptor.java:130)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invokeNext(Invocation.java:115)
| [JBoss] at org.jboss.portal.common.invocation.Invocation.invoke(Invocation.java:157)
| [JBoss] at org.jboss.portal.server.servlet.PortalServlet.service(PortalServlet.java:250)
Should whatever data that stack is getting from the DB be coming from the cache?
On perf13 I just noticed that there a ton of "Timer-123" threads.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4169421#4169421
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4169421
17 years, 8 months
[Design of JBoss Portal] - Re: portal clustering problem when using optimistic locking
by prabhat.jha＠jboss.com
anonymous wrote :
| In none of the thread dumps is it doing anything, which itself is informative -- there's no intra-cluster messages being handled by any of these servers in any of these thread dumps. That in and of itself is interesting. Makes me question whether whatever is going on here has anything to do w/ clustering.
|
The only reason I started on clustering configuration modification is because after 3-nodes, portal cluster was not able to handle more requests Upto 3 node, I do not see any problem with scalability but with 4 and 5 I do.
Other potential bottleneck is database but given that 3-node cluster can handle around 4K users while with 5-node cluster, it starts crawling with 2.5K users, I am not suspecting database at this stage. I hope I am not overlooking something here.
Brian, what else do you suspect based on what you saw in thread dumps?
anonymous wrote :
| If you run these tests with a set of non-clustered portal servers, what kind of #s do you see?
|
I have not run tests with more than 2 server in a non-clustered environment. With 2-nodes, it could handle twice more load than that of 1-node which is also the case with 2-node cluster.
I would anyday prefer there is nothing wrong with jboss cache and hibernate integrations and it's something in Portal itself. ;-)
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4169414#4169414
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4169414
17 years, 8 months
[Design of JBoss Remoting, Unified Invokers] - Re: Remoting and remote classloading
by ron.sigal＠jboss.com
"scott.stark(a)jboss.org" wrote :
| A potential problem I see with the MarshallerLoaderHandler is that is calling the ServerInvoker for a class loader.
|
Right, pre-JBREM-962, that's all it did, which restricted the facility to Remoting's classloader. Now, if that doesn't work, it will try all the configured classloader repositories. E.g., if the Connector MBean is configured like
| <mbean code="org.jboss.remoting.transport.Connector"
| name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
| <depends>jboss.aop:service=AspectDeployer</depends>
| <attribute name="Configuration">
| <config>
| ...
|
| <repositories>
| <repository>jboss.remoting:loader=titan1.ear</repository>
| <repository>jboss.remoting:loader=titan2.ear</repository>
| </repositories>
|
| ...
| </config>
| </attribute>
| </mbean>
|
where, for example, jboss.remoting:loader=titan1.ear comes from
| <jboss-app>
| <loader-repository>jboss.remoting:loader=titan1.ear</loader-repository>
| </jboss-app>
|
then MarshallerLoaderHandler will also try jboss.remoting:loader=titan1.ear and jboss.remoting:loader=titan2.ear.
What I'd like to do is replace the "repositories" element with a list of classloaders.
"scott.stark(a)jboss.org" wrote :
| Yes, if we can have an application level bean register its class loader with the remoting layer to enable remote class loading, that is what we want.
|
I was thinking more in terms of declarative injection by way of *-beans.xml files. I'm not sure I see how an application level bean would get access to Remoting ... . Actually, that raises a question I've been meaning to figure out: What is the microcontainter/POJO analog to grabbing an MBeanServer and accessing an MBean by way of its object name?
"scott.stark(a)jboss.org" wrote :
| if we can have an application level bean register its class loader ... Registering a class loader with remoting should be a priviledged operation with a security check to ensure only application framework code can perform the registration.
|
I'm missing something. If "application framework code" means the Application Server, then these statements sound contradictory ... . Also, is there a way for a Connector method to check that it's being called by JBoss code?
"scott.stark(a)jboss.org" wrote :
| simply a feature, not a crippling issue in my view.
|
I won't let this hold up the 2.4.0.SP1 release unnecessarily.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4169409#4169409
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4169409
17 years, 8 months
[Design of JBoss Remoting, Unified Invokers] - Re: Remoting and remote classloading
by scott.stark＠jboss.org
"ron.sigal(a)jboss.com" wrote :
| Without access to classloaders other than its own, Remoting's remote classloading facility would be crippled, which was the genesis of JBREM-962. A user had EJB A and EJB B in two different scoped EARs, and he wanted EJB A to call a method on EJB B that returns an instance of a class that was unavailable to EJB A. With the fix from JBREM-962, the unmarshaller in EJB A can download the missing class by contacting Remoting's MarshallerLoaderHandler running in a Connector "affiliated" with the EJB3 Connector that serves EJB B. But that only works if the user configured the EJB3 Connector that serves EJB B with the name of the right classloader repository.
|
This is a form of rmi remote class loading, but that is simply a feature, not a crippling issue in my view. In general that usage is not possible because if the caller does have a class from ejb B that conflicts with it due to a local version difference, it has to be marshalled into the class loader space of ejb A.
"ron.sigal(a)jboss.com" wrote :
| I would argue that in this situation, Remoting's MarshallerLoaderHandler *is* the handler responsible for setting the TCL, though it's outside of the usual channels for mapping applications to classloaders.
|
| I guess the basic question is, do we want to continue to support Remoting's independent remote classloader facility? Or should Remoting just depend on the Application Server's RMI classloader facility?
I'm not saying we should not have a remote class loading feature to pull classes across the wire. I'm saying that its a pure delegation facility that requires the actual class loader to be provided to the remoting layer. A potential problem I see with the MarshallerLoaderHandler is that is calling the ServerInvoker for a class loader. A protocol specific handler is not going to know how to get class loaders for all of the invocation targets in use. Ejb3 is complete different from ejb2.
"ron.sigal(a)jboss.com" wrote :
| I guess injecting a list of classloaders would be pretty general.
|
Yes, if we can have an application level bean register its class loader with the remoting layer to enable remote class loading, that is what we want. This would also allow the application layer to wrap its class loader in a filtered class loader with a policy that restricts what content can actually be remoted. Registering a class loader with remoting should be a priviledged operation with a security check to ensure only application framework code can perform the registration.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4169390#4169390
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4169390
17 years, 8 months