[JBoss Messaging] - Bridge error because of network problems also influences oth - jboss-user

Thursday, 5 March 2009

Hi Tim, Hi Howard.

Its me again and I a remoting problem.

I am using JBM 1.4.2.GA.SP1 with remoting SP11Brew.

I have 4 Jboss 4.2.3GA instances A B C X (non clustered)

A C and X are running on the same machine (OVSPC186) using different ports (development)
B is running on a different one. (NVC1218)

On X I have configured 2 Bridges to each other instance where one is sending data to the
instance and the other one receives data from the instance. So I have 6 Bridges deployed
on X.
Bridge QOS level set to 2 means XA. 

The whole system works fine sending messages around.

Now a strange thing happens:
I kill the remote instance B or unplug network cable to simulate network errors. Then the
expected thing happens. The bridges to B detect the erros and retry in my defined time of
50s. 
BUT somehow also the other bridges crashing now. I would assume that they are NOT
influenced on that B instance problem.
When I kill the local instance A or C I never saw this behavior.

This is reproduceable and I thing I have noticed that while I have tested the upgrade from
1.4.0.SP3  to 1.4.0.SP3.CP04 ( in production I am using 1.4.0.SP3 )

The stacktraces and output of the logger:

Detection of instance B problems 

  | 09:09:43,546 WARN  [Bridge] jboss.messaging:name=OutgoingBridge_AeosB,service=Bridge
Detected failure on connection
  | javax.jms.JMSException: Failure on underlying remoting connection
  |         at
org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:102)
  |         at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:641)
  | 09:09:49,515 ERROR [SocketClientInvoker] Got marshalling exception, exiting
  | java.net.SocketException: Connection reset
  |         at java.net.SocketInputStream.read(SocketInputStream.java:168)
  |         at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
  |         at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
  |         at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
  |         at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
  |         at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
  |         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  |         at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  |         at java.io.FilterInputStream.read(FilterInputStream.java:66)
  |         at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:988)
  |         at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:621)
  |         at
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
  |         at
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133)
  |         at
org.jboss.remoting.ConnectionValidator.doCheckConnectionWithLease(ConnectionValidator.java:548)
  |         at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:328)
  |         at java.util.TimerThread.mainLoop(Timer.java:512)
  |         at java.util.TimerThread.run(Timer.java:462)
  | 09:09:49,578 ERROR [SocketClientInvoker] Got marshalling exception, exiting
  | java.net.SocketException: Connection reset
  |         at java.net.SocketInputStream.read(SocketInputStream.java:168)
  |         at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
  |         at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
  |         at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
  |         at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
  |         at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
  |         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  |         at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
  |         at java.io.FilterInputStream.read(FilterInputStream.java:66)
  |         at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:988)
  |         at
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:621)
  |         at
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
  |         at
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:133)
  |         at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:93)
  |         at
org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:353)
  |         at
org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:693)
  |         at java.util.TimerThread.mainLoop(Timer.java:512)
  |         at java.util.TimerThread.run(Timer.java:462)
  | 09:09:49,578 WARN  [Client] unable to remove remote callback handler: Can not get
connection to server. Problem establishing socket connection for InvokerLocator
[sslbisocket://nvc1218:44457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=10&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat]
  | 09:09:49,593 ERROR [ClosedInterceptor]
ClosedInterceptor.ClientSessionDelegate[g2-rgxo5xrf-1-dqio5xrf-qtzfw4-2227p4]: method
getXAResource() did not go through, the interceptor is CLOSED
  | 09:09:49,640 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:49,640 WARN  [Bridge] jboss.messaging:name=IncommingBridge_AeosB,service=Bridge
Detected failure on connection
  | javax.jms.JMSException: Failure on underlying remoting connection
  |         at
org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:102)
  |         at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:641)
  | 09:09:49,640 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:47 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:49,703 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:49,718 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:47 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:49,718 WARN  [Client] unable to remove remote callback handler: Can not get
connection to server. Problem establishing socket connection for InvokerLocator
[sslbisocket://nvc1218:44457//?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=1&numberOfRetries=10&pingFrequency=214748364&pingWindowFactor=10&socket.check_connection=false&stopLeaseOnFailure=true&timeout=0&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat]
  | 09:09:49,718 WARN  [Bridge] jboss.messaging:name=OutgoingBridge_AeosB,service=Bridge
Will retry after a pause of 50000 ms
  | 09:09:49,718 ERROR [ClosedInterceptor]
ClosedInterceptor.ClientSessionDelegate[t2-w1cj5xrf-1-59rc5xrf-zn9jwb-26z24a]: method
getXAResource() did not go through, the interceptor is CLOSED
  | 09:09:49,734 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:49,734 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:26 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:49,734 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:49,734 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:26 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:49,734 WARN  [Bridge] jboss.messaging:name=IncommingBridge_AeosB,service=Bridge
Will retry after a pause of 50000 ms
  | 

But then 3,5 seconds later the Bridge to C has problems which is the strange thing to
investigate. C is on OVSPC186 which is the same machine as this instance X:

  | 09:09:53,312 WARN  [Bridge] jboss.messaging:name=OutgoingBridge_AEOSC,service=Bridge
Detected failure on connection
  | javax.jms.JMSException: Failure on underlying remoting connection
  |         at
org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:102)
  |         at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:641)
  | 09:09:56,406 ERROR [ClosedInterceptor]
ClosedInterceptor.ClientSessionDelegate[w2-iixo5xrf-1-dqio5xrf-qtzfw4-2227p4]: method
getXAResource() did not go through, the interceptor is CLOSED
  | 09:09:56,406 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:56,406 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:51 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:56,406 WARN  [loggerI18N]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
[com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror]
XAResourceRecord.rollback - xa error XAException.XAER_NOTA
  | 09:09:56,406 WARN  [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_54]
- Top-level abort of action -5601fdfe:6db:49af887e:51 received
TwoPhaseOutcome.FINISH_ERROR from <ClassName:RecordType.JTA_RECORD>
  | 09:09:56,406 WARN  [Bridge] jboss.messaging:name=OutgoingBridge_AEOSC,service=Bridge
Will retry after a pause of 50000 ms
  | 

here also the debug output. It seems that the connectionValidator detects problems.

  | 09:09:53,312 DEBUG [org.jboss.remoting.ConnectionValidator]
ConnectionValidator[SocketClientInvoker[1ac6103, sslbisocket://OVSPC186:34457],
pingPeriod=2000 ms]'s connections is invalid
  | 09:09:53,312 DEBUG [org.jboss.remoting.InvokerRegistry] decremented
SocketClientInvoker[1ac6103, sslbisocket://OVSPC186:34457]'s count, current count 1
  | 09:09:53,312 DEBUG [org.jboss.remoting.ConnectionValidator]
ConnectionValidator[SocketClientInvoker[1ac6103, sslbisocket://OVSPC186:34457],
pingPeriod=2000 ms] stopped, returning true
  | 09:09:53,312 DEBUG [org.jboss.remoting.ConnectionValidator]
org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask@1106f61 detected
connection failure: stopping LeasePinger
  | 09:09:53,312 WARN  [org.jboss.jms.server.bridge.Bridge]
jboss.messaging:name=OutgoingBridge_AEOSC,service=Bridge Detected failure on connection
  | javax.jms.JMSException: Failure on underlying remoting connection
  | 	at
org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:102)
  | 	at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:641)
  | 09:09:53,312 DEBUG [org.jboss.remoting.LeasePinger]
LeasePinger[SocketClientInvoker[1333004,
sslbisocket://OVSPC186:34457](4p7222-4wfztq-frx5oiqd-1-frx5ovr4-18)] setting disconnect
timeout to: -1
  | 09:09:53,312 DEBUG [org.jboss.jms.server.connectionmanager.SimpleConnectionManager]
unregistered connection ConnectionEndpoint[q2-mhxo5xrf-1-dqio5xrf-qtzfw4-2227p4] with
remoting session ID ...-frx5oxh7-2k
  | 09:09:53,312 DEBUG [org.jboss.jms.server.remoting.JMSServerInvocationHandler] removing
callback handler
ServerInvokerCallbackHandler[4p7222-4wfztq-frx5oiqd-1-frx5oxh7-2k+4p7222-4wfztq-frx5oiqd-1-frx5oxh7-2n]
  | 09:09:53,312 DEBUG [org.jboss.remoting.InvokerRegistry] removed
org.jboss.remoting.transport.local.LocalClientInvoker@1933cc from registry
  | 09:09:53,312 DEBUG [org.jboss.remoting.ServerInvoker] ServerInvoker
(SocketServerInvoker[UNINITIALIZED]) removing client callback handler with session id of
4p7222-4wfztq-frx5oiqd-1-frx5oxh7-2k+4p7222-4wfztq-frx5oiqd-1-frx5oxh7-2n.
  | 09:09:53,312 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker]
unrecognized listener ID: 4p7222-4wfztq-frx5oiqd-1-frx5oxh7-2n
  | 09:09:53,312 DEBUG [org.jboss.remoting.InvokerRegistry] removed
org.jboss.remoting.transport.local.LocalClientInvoker@97a376 from registry
  | 09:09:53,312 DEBUG [org.jboss.remoting.ServerInvoker]
SocketServerInvoker[UNINITIALIZED] stopped
  | 09:09:53,312 DEBUG [org.jboss.remoting.InvokerRegistry] decremented
org.jboss.remoting.transport.local.LocalClientInvoker@1b147ab's count, current count
4
  | 09:09:53,312 DEBUG [org.jboss.remoting.InvokerRegistry] decremented
org.jboss.remoting.transport.local.LocalClientInvoker@1d83ac7's count, current count
4
  | 09:09:54,890 DEBUG [org.jboss.remoting.ConnectionValidator]
ConnectionValidator[SocketClientInvoker[1ac6103, sslbisocket://OVSPC186:34457],
pingPeriod=2000 ms] shut down lease pinger
  | 09:09:56,406 DEBUG [org.jboss.remoting.ServerInvoker] ServerInvoker
(SocketServerInvoker[UNINITIALIZED]) removing client callback handler with session id of
4p7222-4wfztq-frx5oiqd-1-frx5oxhm-2r+4p7222-4wfztq-frx5oiqd-1-frx5oxi2-2u.
  | 09:09:56,406 DEBUG [org.jboss.remoting.transport.bisocket.BisocketServerInvoker]
SocketServerInvoker[UNINITIALIZED] shutting down control connection:
4p7222-4wfztq-frx5oiqd-1-frx5oxi2-2u
  | 09:09:56,406 DEBUG [org.jboss.remoting.InvokerRegistry] removed
org.jboss.remoting.transport.local.LocalClientInvoker@8ce172 from registry
  | 09:09:56,406 DEBUG [org.jboss.remoting.ServerInvoker]
SocketServerInvoker[UNINITIALIZED] stopped
  | 09:09:56,406 DEBUG [org.jboss.remoting.InvokerRegistry] decremented
SocketClientInvoker[1333004, sslbisocket://OVSPC186:34457]'s count, current count 1
  | 09:09:56,406 DEBUG [org.jboss.remoting.InvokerRegistry] decremented
SocketClientInvoker[1b52a38, sslbisocket://OVSPC186:34457]'s count, current count 1
  | 09:09:56,406 ERROR [org.jboss.jms.client.container.ClosedInterceptor]
ClosedInterceptor.ClientSessionDelegate[w2-iixo5xrf-1-dqio5xrf-qtzfw4-2227p4]: method
getXAResource() did not go through, the interceptor is CLOSED
  | 

View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215130#...

Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&a...

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[JBoss Messaging] - Bridge error because of network problems also influences oth