Potential race condition attempting to prevent OutOfMemoryError writing messages from client to service and visa versa using a proxy server

[cgrogan]

Summary
-----------

Writing large volumes of large messages from a client via a proxy to a slow
end service using a solution for preventing OutOfMemoryError based on the
Netty Discard example appears to be resulting in a race condition.  

Attached is sample code which reproduces the problem together with a sample
log output file. (Note that the log has been trimmed to keep the size down)

Description of setup
-------------------------

The attached LoadTestDemo class sets up the following:

Client -------------->    Proxy     ----------------> Service
                       (Server on                    (Server on
		        Port 5680)                    Port 5683)

* Proxy server
* Service server
* Client connection / bootstrap between the client and proxy server
* Client connection / bootstrap between the proxy server and service


Messages are written from the client to the service and back via the proxy
as follows:

Client  ---(1)------->  Proxy                       Service
Client                  Proxy ------(2)-----------> Service
Client                  Proxy <------(3)----------  Service
Client  <----(4)------- Proxy                       Service


(1) Client writes a message to the Proxy server
(2) Proxy receives the message from the client & forwards the message onto
the service, i.e. writes to the ‘serviceClient’ channel.
(3) Service receives the message from the proxy, sleeps for 100 milliseconds
to simulate a slow end service & then writes a response (the same message in
this instance) back to the proxy.
(4) Service Client receives the messages from the service & forwards the
message back to the client, i.e. writes to the ‘proxyServer’ channel.

Attached is the code for the above scenario - the main class is LoadTestDemo
& all supporting classes are contained within the same file for convenience.

Description of the Problem
------------------------------

The code is specifically testing for issues under load when sending a large
number of large messages from a client through to a slow end service. 
(Note that the same situation in reverse will need to be also catered for
but this is not yet incorporated into the test, i.e. dealing with a slow
client and a quick service).

As expected OutOfMemoryErrors occur when no solution for managing this was
in place.  To fix this the solution described in the ‘Discard’ Netty example
was used as a starting point/basis for the attached code. The attached code
is more involved than the discard example due to the additional proxy layer.   
* The test is initiated and managed by the LoadTestDemo.execute() method. 
This starts writes messages from the client to the proxy within a loop; it
does this only when the client channel is writeable.  As this is a test it
uses a simple Thread mechanism to achieve this.  
* The "Service Client" handler (see startServiceClient() method) checks
whether the channel to the service is writeable in channelInterestChanged().
If not writeable then the proxy channel is set to not readable until the
service client channel is writeable again, i.e. we block further reads on
the inbound channel until the outbound channel is writeable again. 

Note: the main areas of code which aim to prevent OutOfMemoryErrors are
highlighted by the following comment [// <<<------]


When executing LoadTestDemo, it does appear to have fixed the OutOfMemory
error.  However it eventually locks with no further output... the number of
messages written varies across executions... sometimes reaching around 900 -
1000 messages before locking out.
Attached is a sample log output - this example includes information on the
threads executing which helps document the execution and the problem.


Looking at the log, the channelInterestChanged() log items are of particular
interest, e.g.
[New I/O server worker #2-1] LoadTestDemo Service Client. New interest Ops:
5
....
[New I/O client worker #1-1] LoadTestDemo Service Client. New interest Ops:
1
...

Throughout the log the server worker thread #2-1 consistently sees the
service client channel is not writeable - this sets the inbound channel to
not readable blocking further reads.

Throughout the log the client worker thread #1-1 consistently sees the
service client channel as writeable - this sets the inbound channel back to
readable again. 

The timestamps of these logs are generally at the same time or extremely
close so both threads are working in opposition to each other.  This appears
to result in some sort of race condition.

Are we using the channelInterestChanged() correctly in this scenario?  Is
there another way in which we should be using this?

Thanks in advance for your assistance.

http://netty-forums-and-mailing-lists.685743.n2.nabble.com/file/n5392146/LoadTestDemo.java.zip
LoadTestDemo.java.zip 
http://netty-forums-and-mailing-lists.685743.n2.nabble.com/file/n5392146/LoadTest-Log_Trimmed.zip
LoadTest-Log_Trimmed.zip 
-- 
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Potential-race-condition-attempting-to-prevent-OutOfMemoryError-writing-messages-from-client-to-servr-tp5392146p5392146.html
Sent from the Netty User Group mailing list archive at Nabble.com.