[JBoss JIRA] (JBREM-1308) MicroSocketClientInvoker hangs on SocketInputStream.read causing deadlock in MessageSucker.setConsuming
by Giovanni Pelosi (JIRA)
Giovanni Pelosi created JBREM-1308:
--------------------------------------
Summary: MicroSocketClientInvoker hangs on SocketInputStream.read causing deadlock in MessageSucker.setConsuming
Key: JBREM-1308
URL: https://issues.jboss.org/browse/JBREM-1308
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: transport
Affects Versions: 2.2.4
Environment: jboss-4.2.3.GA with jboss-messaging-1.4.5.GA and latest jboss-remoting-2_2_4,
3 node cluster, jms clustered queues
Reporter: Giovanni Pelosi
In production enviroment, our jboss line of 3 clustered servers randomly (once a week) seemes to hang, refusing to reply to remote jms clients.
Taking stack trace of all jboss instances (jstack) we always see the same pattern of deadlock:
A non-pooled thread, started internally by remoting, is trying to recover a "connectionExeption" condition
(ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)),
but the task hangs (without timeout) on a socket read:
(MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077))
This non-terminating network operation holds a lock, acquired in
MessageSucker.suspend(MessageSucker.java:225)
that in turns cause deadlock in messaging layer:
org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566)
causing blocks in all jms traffic.
Offending thread stack:
-------------------------------------------------
"Thread-5163" daemon prio=10 tid=0x00007f1bc4d10800 nid=0x2b70 runnable [0x00007f1af61e0000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00000006d23445f8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682)
at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:458)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
at org.jboss.remoting.Client.invoke(Client.java:1925)
at org.jboss.remoting.Client.invoke(Client.java:786)
at org.jboss.remoting.Client.invoke(Client.java:774)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
at org.jboss.jms.client.delegate.ClientConsumerDelegate.org$jboss$jms$client$delegate$ClientConsumerDelegate$close$aop(ClientConsumerDelegate.java:122)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
at org.jboss.jms.client.delegate.ClientConsumerDelegate.close(ClientConsumerDelegate.java)
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.suspend(MessageSucker.java:225)
- locked <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.cleanupConnection(ClusterConnectionManager.java:804)
- locked <0x0000000605960ca8> (a org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo)
at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager$ConnectionInfo.onException(ClusterConnectionManager.java:785)
at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:498)
-------------------------------------------------
JMS Layer main lock
-------------------------------------------------
"Thread-5167" daemon prio=10 tid=0x00007f1be0069800 nid=0x2b82 waiting for monitor entry [0x00007f1afac2b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:288)
- waiting to lock <0x0000000605960788> (a org.jboss.messaging.core.impl.clusterconnection.MessageSucker)
at org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:566)
at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:516)
at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:375)
- locked <0x00000006012cdd50> (a java.lang.Object)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1574)
at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
-------------------------------------------------
Sample of deadlocked JMS Client Thread:
-------------------------------------------
"WorkManager(2)-42959" daemon prio=10 tid=0x00007f1bb806c000 nid=0x2b6c waiting for monitor entry [0x00007f1adf230000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:242)
- waiting to lock <0x00000006012cdd50> (a java.lang.Object)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2345)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:769)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)
at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1901)
at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1912)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:832)
at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
at org.jboss.remoting.Client.invoke(Client.java:1925)
at org.jboss.remoting.Client.invoke(Client.java:786)
at org.jboss.remoting.Client.invokeOneway(Client.java:837)
at org.jboss.remoting.Client.invokeOneway(Client.java:1023)
at org.jboss.remoting.Client.invokeOneway(Client.java:1014)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)
at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)
at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)
at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:457)
at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:606)
at com.pirelli.pmp.jms.JmsClient.send(JmsClient.java:655)
at com.pirelli.pmp.jms.GuiResponseQueueHelper.sendReplyMessage(GuiResponseQueueHelper.java:31)
at com.pirelli.pmp.jms.GuiResponseQueueHelper.sendReplyMessage(GuiResponseQueueHelper.java:26)
at com.pirelli.pmp.mdb.ActivityStarterMdb.replyToGUI(ActivityStarterMdb.java:153)
at com.pirelli.pmp.mdb.ActivityStarterMdb.onMessage(ActivityStarterMdb.java:95)
at sun.reflect.GeneratedMethodAccessor1335.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:195)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:249)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
at $Proxy117.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
at java.lang.Thread.run(Thread.java:679)
-------------------------------------------
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 9 months
[JBoss JIRA] (JBREM-1305) Maximum number of client sockets in MicroSocketClientInvoker increases after timeout waiting for a free socket
by Torsten Neise (JIRA)
Torsten Neise created JBREM-1305:
------------------------------------
Summary: Maximum number of client sockets in MicroSocketClientInvoker increases after timeout waiting for a free socket
Key: JBREM-1305
URL: https://issues.jboss.org/browse/JBREM-1305
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: transport
Affects Versions: 2.5.4.SP3, 2.2.4, 2.5.2.SP3 (Flounder), 2.5.0.SP2 (Flounder) , 2.4.0.SP1 (Pinto)
Environment: all
Reporter: Torsten Neise
Maximum number of client sockets in MicroSocketClientInvoker increases after timeout waiting for a free socket. This happens because of an illegal call of semaphore.release() in MicroSocketClientInvoker.transport(...) after semaphore.attempt(timeToWait) returned with false (no lock aquired) in MicroSocketClientInvoker.getConnection(...).
public class MicroSocketClientInvoker extends RemoteClientInvoker
{
...
protected Object transport(String sessionID, Object invocation, Map metadata,
Marshaller marshaller, UnMarshaller unmarshaller)
throws IOException, ConnectionFailedException, ClassNotFoundException
{
...
try
{
socketWrapper = getConnection(marshaller, unmarshaller, timeLeft);
}
...
catch (Exception e)
{
// if (bailOut)
// return null;
semaphore.release();
...
}
...
}
protected SocketWrapper getConnection(Marshaller marshaller,
UnMarshaller unmarshaller,
int timeAllowed)
throws Exception
{
long start = System.currentTimeMillis();
long timeToWait = (timeAllowed > 0) ? timeAllowed : 30000;
boolean timedout = !semaphore.attempt(timeToWait);
if (trace) log.trace(this + " obtained semaphore: " + semaphore.permits());
if (timedout)
{
throw new IllegalStateException("Timeout waiting for a free socket");
}
...
}
...
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 9 months
[JBoss JIRA] Created: (JBREM-1301) WebUtil.isBinary fails when charSet appended
by Tom Cunningham (JIRA)
WebUtil.isBinary fails when charSet appended
--------------------------------------------
Key: JBREM-1301
URL: https://issues.jboss.org/browse/JBREM-1301
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: jmx remoting, marshall
Affects Versions: 2.5.4.SP2
Reporter: Tom Cunningham
WebUtil.isBinary fails when the contentType equals
application/octet-stream?charSet=UTF-8
It only accepts "application/octet-stream" as valid. On 1.6.0_21, I'm receiving "application/octet-stream?charSet=UTF-8" as my ContentType from HttpUrlConnection.getHeaderFields() within HttpClientInvoker, which overwrites any ContentType I try to send in a map. Because of this, my responses are InvocationResponses, but they are treated as a string and not unmarshalled.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 9 months
[JBoss JIRA] (AS7-4441) JPA hibernate.cache.region_prefix property fails JConsole to connect to JBoss
by Osten Forshed (JIRA)
Osten Forshed created AS7-4441:
----------------------------------
Summary: JPA hibernate.cache.region_prefix property fails JConsole to connect to JBoss
Key: AS7-4441
URL: https://issues.jboss.org/browse/AS7-4441
Project: Application Server 7
Issue Type: Bug
Components: JMX, JPA / Hibernate
Affects Versions: 7.1.1.Final
Environment: Windows Vista 64 bit, JDK 1.6.0_18
Reporter: Osten Forshed
Assignee: John Bailey
Priority: Minor
If a web app is deployed (standalone) with a persistence.xml containing:
<property name="hibernate.cache.region_prefix" value=""/>
Then it is not possible to connect to the MBean Server with JConsole using instructions here:
https://community.jboss.org/wiki/UsingJconsoleToConnectToJMXOnAS7
Following stack trace is written to JBoss log:
12:27:03,406 WARN [org.jboss.remotingjmx.protocol.v1.ServerProxy] (pool-5-thread-16) Unexpected internal error: java.lang.NullPointerException
at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:49)
at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:55)
at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:55)
at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:55)
at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:55)
at org.jboss.as.jmx.model.RootResourceIterator.iterate(RootResourceIterator.java:39)
at org.jboss.as.jmx.model.ModelControllerMBeanHelper.queryNames(ModelControllerMBeanHelper.java:136)
at org.jboss.as.jmx.model.ModelControllerMBeanServerPlugin.queryNames(ModelControllerMBeanServerPlugin.java:124)
at org.jboss.as.jmx.PluggableMBeanServerImpl.queryNames(PluggableMBeanServerImpl.java:280)
at org.jboss.remotingjmx.protocol.v1.ServerProxy$QueryNamesHandler.handle(ServerProxy.java:1197)
at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:215)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_18]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_18]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_18]
If property is removed from persistence.xml it is possible to connect using JConsole and browse the MBeans.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 9 months