Netty is Freezing on Load Testing

[mlaverd]

Hello list,

I'm a Netty newbie and I decided to port my application's custom-made IO
layer to Netty. I thought it would be a piece of cake. It was more or less
the case, and things work fine when users are clicking their way to make
requests.

But the problem is when I'm load-testing it... the server will take all the
CPU and everything is so slow that the load-testing client threads keep on
timing out or get disconnect by the server (I set a timeout of 20 seconds).
I don't understand what I did wrong...

Here are some anonymized code snippets:

//First, the code that initializes the ServerBootStrap

ThreadPoolExecutor basePool = new
MemoryAwareThreadPoolExecutor(NUM_THREADS/10, 0, MAX_MEMORY_PER_POOL_MB *
FileUtils.ONE_MB/4);
//We need twice the number of threads, since we have the timers too
WORKER_THREAD_POOL = new MemoryAwareThreadPoolExecutor(NUM_THREADS*2, 0,
MAX_MEMORY_PER_POOL_MB * FileUtils.ONE_MB);
channelFactory = new NioServerSocketChannelFactory(basePool,
WORKER_THREAD_POOL);
clientServerBootStrap = new ServerBootstrap(channelFactory);

SSLContext context = CryptoUtils.initTlsContext(KEYSTORE_PATH, KS_PASS,
TRUSTSTORE_PATH, TS_PASS, TLS_SESSION_CACHE_SIZE);
clientServerBootStrap.setPipelineFactory(new MyPipelineFactory(context));
Channel clientChannel = clientServerBootStrap.bind(new
InetSocketAddress(BINDING_PORT));
allChannels.add(clientChannel);

// What the pipeline looks like
class MyPipelineFactory implements ChannelPipelineFactory{

	final private SSLContext context;
	
	public MyPipelineFactory(SSLContext context){
		this.context = context;
	}
	
	@Override
	public ChannelPipeline getPipeline() throws Exception {
		
		
		//Init the TLS for that channel		
		SSLEngine engine = context.createSSLEngine();
		engine.setUseClientMode(false);
		engine.setWantClientAuth(false);
		engine.setEnableSessionCreation(true);

		ChannelPipeline pipeline = Channels.pipeline();
		pipeline.addLast("TLS", new SslHandler(engine));
		pipeline.addLast("Timeout", new Disconnector(new
HashedWheelTimer(WORKER_THREAD_POOL.getThreadFactory()), 
				SESSION_TIMEOUT,SESSION_TIMEOUT,SESSION_TIMEOUT,
TimeUnit.MILLISECONDS));

		//Decode requests: first get the protobuf message length, then convert to
protobuf, then convert to our own objects
		pipeline.addLast("frameDecoder",
                  new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0, 4));
		pipeline.addLast("protobufDecoder",
                  new ProtobufDecoder(MyMessage.getDefaultInstance()));
		pipeline.addLast("requestDecoder", new MessageDecoder());
		
		//Encode responses: first convert from our objects to protobuf, then
protobuf to binary, then add the length field
		pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
		pipeline.addLast("protobufEncoder", new ProtobufEncoder());
		pipeline.addLast("responseEncoder", new MessageEncoder());

		pipeline.addLast("Logging", new ChannelLogger()); //logs using our own API
		pipeline.addLast("handler", new RequestHandler()); //actually processes
the request. Normal processing time: ~5-10 seconds
		pipeline.addLast("disconnectionNotifier", new DisconnectionNotifier());
//Informs the observers that a channel is disconnected
		
		return pipeline;
	}
	
}

// How the disconnections are handled
class Disconnector extends IdleStateHandler{
	public Disconnector(Timer timer, int readerIdleTimeSeconds,
			int writerIdleTimeSeconds, int allIdleTimeSeconds) {
		super(timer, readerIdleTimeSeconds, writerIdleTimeSeconds,
allIdleTimeSeconds);
	}

	public Disconnector(Timer timer, long readerIdleTime,
			long writerIdleTime, long allIdleTime, TimeUnit unit) {
		super(timer, readerIdleTime, writerIdleTime, allIdleTime, unit);
	}

	@Override
	protected void channelIdle(ChannelHandlerContext ctx, IdleState state,
			long lastActivityTimeMillis) throws Exception {
		super.channelIdle(ctx, state, lastActivityTimeMillis);
		ctx.getChannel().close();
		releaseExternalResources();
		
	}
	
	@Override
	protected void finalize() throws Throwable {
		super.finalize();
		releaseExternalResources();
	}
	
}

Can anyone point out what I'm doing wrong??? 

Thanks in advance!
-- 
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Netty-is-Freezing-on-Load-Testing-tp5138397p5138397.html
Sent from the Netty User Group mailing list archive at Nabble.com.