OutOfMemoryError: when with large number of Worker Threads in NioClientSocketChannelFactory

[Wesly smith]

Hello,

After re-think about the "problem" I posted, probably there is nothing wrong
with the content buffer is cached. If the buffer is not cached, it could
result in allocation and deallocation at each request. It is not very
efficient.

I specified too much worker thread in the channel factory:
  ChannelFactory factory =
    new NioClientSocketChannelFactory(
        Executors.newCachedThreadPool(),
         Executors.newCachedThreadPool(), 160);

Thanks



Wesly smith wrote:
> 
> Hello Trustin, 
> 
> I got OutOfMemoryError. The profile showed me that the
> NioClientSockeetChannelFactory holds the buffer too long.
> 
> The message:
> =============
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> 	at test.netty.http.HttpClient.main(HttpClient.java:78)
> 
> A sample code:
> ==============
> I have a long-live http client case. In the code below, I demoed as a loop
> to connect the server, and create a ChannelBuffer with 1M byte, and send
> it as content to server. The Handler will close the channel when
> messageReceived.
> 
> FYI, I had NioClientSocketChannelFactory with a big number of workerCount,
> such as 160. 
>   ChannelFactory factory =
>     new NioClientSocketChannelFactory(
>         Executors.newCachedThreadPool(),
>          Executors.newCachedThreadPool(), 160);
> 
> I noticed for each request (could be only the first 160. On my try, it got
> OutOfMemory less than 160 request.), the java memory use will jump 1M. It
> seems those losts are the 1M byte buffers I allocated as content. For some
> reason, they are always referred by those Executors, and so it can not be
> released.
> 
> Any suggestions?
> 
> ======== The code snip. It is modified from the http sample =======
> public class HttpClient {
> 
>     public static void main(String[] args) throws Exception {
>         if (args.length != 1) {
>             System.err.println(
>                     "Usage: " + HttpClient.class.getSimpleName() +
>                     " <URL>");
>             return;
>         }
> 
>         URI uri = new URI(args[0]);
>         String scheme = uri.getScheme() == null? "http" : uri.getScheme();
>         String host = uri.getHost() == null? "localhost" : uri.getHost();
>         int port = uri.getPort() == -1? 80 : uri.getPort();
> 
>         if (!scheme.equals("http")) {
>             // We can actually support HTTPS fairly easily by inserting
>             // an SslHandler to the pipeline - left as an exercise.
>             System.err.println("Only HTTP is supported.");
>             return;
>         }
> 
>         // Configure the client.
>         ChannelFactory factory =
>             new NioClientSocketChannelFactory(
>                     Executors.newCachedThreadPool(),
>                     Executors.newCachedThreadPool(), 160);
> 
>         ClientBootstrap bootstrap = new ClientBootstrap(factory);
>         HttpClientPipelineFactory handler = new
> HttpClientPipelineFactory(new HttpResponseHandler());
>         bootstrap.setPipelineFactory(handler);
>         bootstrap.setOption("keepAlive", true);
> 
>         // Send 10000 HTTP requests, each with 1M content
>         for ( int loop = 0; loop < 10000; loop++)
>         {       
>         	// Start the connection attempt.
> 			ChannelFuture future = bootstrap.connect(new InetSocketAddress(
> 					host, port));
> 
> 			// Wait until the connection attempt succeeds or fails.
> 			Channel channel = future.awaitUninterruptibly().getChannel();
> 			if (!future.isSuccess()) {
> 				future.getCause().printStackTrace();
> 				factory.releaseExternalResources();
> 				return;
> 			}
> 
> 			HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1,
> 					HttpMethod.POST, uri.toASCIIString());
> 			request.addHeader(HttpHeaders.HOST, host);
> 
> 			ChannelBuffer content = ChannelBuffers.buffer(1024*1024);
> 			byte[] b = new byte[1024*1024];
> 			for (int i = 0; i < 1024*1024; i++) {
> 				b[i] = 100;
> 			}
> 			content.writeBytes(b, 0, 1024*1024);
> 			request.setContent(content);
>             // set headers
> 			request.addHeader("Connection", "Keep-Alive");
> 			request.addHeader(HttpHeaders.CONTENT_LENGTH,
> String.valueOf(content.readableBytes()));
>             
> 			channel.write(request);
> 			Thread.sleep(1000);
>         }
>         
>         // Wait for the server to close the connection.
>         //channel.getCloseFuture().awaitUninterruptibly();
> 
>         // Shut down executor threads to exit.
>         factory.releaseExternalResources();
>     }
> }
> 

-- 
View this message in context: http://n2.nabble.com/OutOfMemoryError%3A-when-with-large-number-of-Worker-Threads-in-NioClientSocketChannelFactory-tp2140700p2146135.html
Sent from the Netty User Group mailing list archive at Nabble.com.