[jboss-dev] Remoting behavior on error (Re: Possible blocker with jboss-cl 2.2.0.Alpha1)

[Jaikiran Pai]

The testcase has been hung since more than 3 days, now :) So looks like 
that default client timeout of 30 minutes isn't being honoured. By the 
way, EJB3 uses its own config file for remoting JBOSS_HOME/server/< 
servername>/deploy/ejb3-connectors-jboss-beans.xml. But that's 
irrelevant since the defaults should apply to any remoting config 
irrespective of which file it's configured in.

So I looked at this in a bit more detail and it now appears to be a bug 
in JBoss Remoting. Here's what's happening:

- A EJB3 client interceptor 
(org.jboss.aspects.remoting.InvokeRemoteInterceptor) responsible for 
invoking the remote EJB container does a remoting call:

     InvokerLocator locator = 
(InvokerLocator)invocation.getMetaData(REMOTING, INVOKER_LOCATOR);
     ...
     Client client = new Client(locator, subsystem);
     
      try
      {
         client.connect();
     ...
         client.invoke(invocation, null);

- Remoting internally creates a 
org.jboss.remoting.transport.socket.MicroSocketClientInvoker which 
creates a ServerAddress:

protected ServerAddress createServerAddress(InetAddress addr, int port)
   {
      return new ServerAddress(addr.getHostAddress(), port, 
enableTcpNoDelay, -1, maxPoolSize);
   }

Notice the hardcoded -1 being passed to the ServerAddress. That's the 
"timeout". The ServerAddress internally ignores negative timeout values 
and sets the timeout to 0 (resulting in infinite timeout).

- The MicroSocketClientInvoker then ultimately ends up creating a client 
socket with timeout = 0.

- So when the server side remoting thread crashes (for any unrelated 
reason - like the classloading issue), the client ends up with this 
infinite block.

The issues i see here:

1) I don't see a way, how i can pass a timeout for the client socket. 
Contrary to the comments in remoting-jboss-beans.xml,  based on what i 
see in the code, it's only the server side timeout which has a 
corresponding param. Did i miss something?

2) Even if there was some way of passing the client socket timeout, 
currently the ServerAddress creation through MicroSocketClientInvoker 
hardcodes the timeout to -1.

 regards,
-Jaikiran  
Adrian Brock wrote:
> On Tue, 2010-02-16 at 18:14 +0530, Jaikiran Pai wrote:
>> Ron Sigal wrote:
>>> Yeah, Carlo also pointed out that NCDFE is a java.lang.Error, which, 
>>> as you note, is not caught.  I agree that if ServerThread is going to 
>>> terminate due to an error, then it should certainly close its socket, 
>>> so that's a bug (JBREM-1183 "ServerThread should catch 
>>> java.lang.Error").  But I have mixed feelings about terminating 
>>> ServerThread in this case.  Unlike, IOException, SocketException, 
>>> etc., the NoClassDefFoundError doesn't suggest that the socket is no 
>>> longer usable.  Maybe a better solution (along with fixing the 
>>> classloader problem, of course) is to use a positive timeout on the 
>>> client side.
>> I think some (sensible) default timeout needs to be added so that the 
>> client does not end up hung. Is this something that can be added to some 
>> *-jboss-beans.xml in the AS?
>>
>
> It already exists, but whether 30 minutes is a "sensible" default
> is a different issue. ;-)
>
> See remoting-jboss-beans.xml
>
>             
>             <!-- Socket read timeout.  Defaults to 60000 ms (1 minute)
> -->
>             <!-- on the server, 1800000 ms (30 minutes) on the client.
> -->
>             <!--entry><key>timeout</key> <value>120000</value></entry-->
>
>