[jboss-jira] [JBoss JIRA] Commented: (JBREM-786) stale sockets can be gotten from pool even with current rety logic

[james ahlborn (JIRA)]

    [ http://jira.jboss.com/jira/browse/JBREM-786?page=comments#action_12387781 ] 
            
james ahlborn commented on JBREM-786:
-------------------------------------

ah, my bad.  i forgot that our library already includes patches backported from the 2.0.0 release, one of which is the flushConnectionPool() addition to the SocketClientInvoker (ported from the MicroSocketClientInvoker).  

anyway, a little more background on our use case.  we are running a service which is getting hit by multiple threads on many different boxes, easily maxing out the connection pool under peak load.  this reveals a number of problems in the socket pooling algorithm, which led us to backport code which fixed some of the issues.  this patch applies to that backported code, so probably really applies to the 2.0.0 release series.  what i think is happening under load, however, is that a socket which is near the end of its useful life is added to the pool after the flush has been called (there's heavy turnover in the pool because of the high load).  the socket did not fail for the previous caller, so is not closed, but fails when retrieved from the pool post-flush.  this causes the client to run out of retries and fail the call with an EOFException.  the changes i made force the client to ignore any pooled connections on the final retry, which solved the problem in our application.

> stale sockets can be gotten from pool even with current rety logic
> ------------------------------------------------------------------
>
>                 Key: JBREM-786
>                 URL: http://jira.jboss.com/jira/browse/JBREM-786
>             Project: JBoss Remoting
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: transport
>    Affects Versions: 1.4.6.GA
>            Reporter: james ahlborn
>         Assigned To: Ron Sigal
>             Fix For: 2.4.0.Beta1 (Pinto)
>
>         Attachments: remoting-fixes.txt
>
>
> The retry logic in SocketClientInvoker.transport attempts to ensure that the invoker does not get a stale socket on the last attempt by flushing the pool.  however, this only discards any currently unused scokets.  a second invoker could return a stale socket after the flush, causing the first invoker to fail again on the last retry (with an EOFException).  while this seems like an edge case, we can hit it reliably under load with many connections to the same destination.  i've patched this class so that the last call to getConnection will never return a pooled connection.  we never got the EOFException after applying this fix.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira