Netty is Freezing on Load Testing
Nicholas Hagen
nicholas.hagen at znetdevelopment.com
Sat Jun 5 15:34:36 EDT 2010
I'm just guessing here but would calling setReuseAddr on both client
impl and servers netty impl help? I had similar issues where I ran
out of sockets until I enabled reuse addr, especially for load testing
short lived sockets. May be worth a quick try. Just make sure you
enable the client side in the netty socket options.
Nicholas Hagen
Z|NET Development, LLC
www.znetdevelopment.com
* sent from my iPhone *
On Jun 5, 2010, at 8:33 AM, Marc-André Laverdière <marcandre.laverdiere at gmail.c
om> wrote:
> It seems like this is something in the framework... When putting
> serious stress on Netty, it breaks apart...
> This sample code talks to the sample Echo Server
>
>
> import java.io.DataInputStream;
> import java.io.DataOutputStream;
> import java.io.IOException;
> import java.io.StringWriter;
> import java.net.InetSocketAddress;
> import java.net.Socket;
> import java.util.concurrent.Callable;
> import java.util.concurrent.Executors;
> import java.util.concurrent.ScheduledExecutorService;
> import java.util.concurrent.TimeUnit;
>
> import javax.net.SocketFactory;
>
> import lombok.Data;
>
>
> public class Client {
>
> private final static int TIMEOUT_MS = 45*1000;
> private final static int NUM_THREADS = 750;
> private final static int INTERVAL_MS = 1* 1000;
> @Data
> static class Worker implements Runnable{
>
> protected final InetSocketAddress server;
> @Override
> public void run(){
> try{
> Socket sock = SocketFactory.getDefault().createSocket();
> sock.connect(server);
>
> DataOutputStream dOut = new DataOutputStream(sock.getOutputStream
> ());
> dOut.writeUTF
> ("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa");
> dOut.flush();
> sock.setSoTimeout(TIMEOUT_MS);
>
> DataInputStream dIn = new DataInputStream(sock.getInputStream());
> String got = dIn.readUTF();
> sock.close();
> } catch (IOException e){
> e.printStackTrace();
> }
> }
>
>
> }
>
> public static void main(String[] args){
>
> ScheduledExecutorService service = Executors.newScheduledThreadPool
> (NUM_THREADS);
> InetSocketAddress serverAddr = new InetSocketAddress("localhost",
> 8080);
> //Set the requesters
> for (int i = 0; i < NUM_THREADS; i++){
> Worker W = new Worker(serverAddr);
> service.scheduleAtFixedRate(W,INTERVAL_MS, INTERVAL_MS,
> TimeUnit.MILLISECONDS);
> }
>
>
> }
> }
>
> After a little bit of time, the client starts timing out, and
> eventually, it throws java.net.NoRouteToHostException: Cannot assign
> requested address
>
> This shouldn't be the case: the server uses a cached thread pool,
> the data goes on the loopback interface, the interval is every
> second, which is should be more than enough time to grab a thread
> from the pool, fetch the data from the interface, read it and write
> it back...
>
> What's going on here?
>
> Marc-André LAVERDIÈRE
> "Perseverance must finish its work so that you may be mature and
> complete, not lacking anything." -James 1:4
> mlaverd.theunixplace.com/blog
>
> /"\
> \ / ASCII Ribbon Campaign
> X against HTML e-mail
> / \
>
>
> 2010/6/4 mlaverd <marcandre.laverdiere at gmail.com>
>
> Hello list,
>
> I'm a Netty newbie and I decided to port my application's custom-
> made IO
> layer to Netty. I thought it would be a piece of cake. It was more
> or less
> the case, and things work fine when users are clicking their way to
> make
> requests.
>
> But the problem is when I'm load-testing it... the server will take
> all the
> CPU and everything is so slow that the load-testing client threads
> keep on
> timing out or get disconnect by the server (I set a timeout of 20
> seconds).
> I don't understand what I did wrong...
>
> Here are some anonymized code snippets:
>
> //First, the code that initializes the ServerBootStrap
>
> ThreadPoolExecutor basePool = new
> MemoryAwareThreadPoolExecutor(NUM_THREADS/10, 0,
> MAX_MEMORY_PER_POOL_MB *
> FileUtils.ONE_MB/4);
> //We need twice the number of threads, since we have the timers too
> WORKER_THREAD_POOL = new MemoryAwareThreadPoolExecutor
> (NUM_THREADS*2, 0,
> MAX_MEMORY_PER_POOL_MB * FileUtils.ONE_MB);
> channelFactory = new NioServerSocketChannelFactory(basePool,
> WORKER_THREAD_POOL);
> clientServerBootStrap = new ServerBootstrap(channelFactory);
>
> SSLContext context = CryptoUtils.initTlsContext(KEYSTORE_PATH,
> KS_PASS,
> TRUSTSTORE_PATH, TS_PASS, TLS_SESSION_CACHE_SIZE);
> clientServerBootStrap.setPipelineFactory(new MyPipelineFactory
> (context));
> Channel clientChannel = clientServerBootStrap.bind(new
> InetSocketAddress(BINDING_PORT));
> allChannels.add(clientChannel);
>
> // What the pipeline looks like
> class MyPipelineFactory implements ChannelPipelineFactory{
>
> final private SSLContext context;
>
> public MyPipelineFactory(SSLContext context){
> this.context = context;
> }
>
> @Override
> public ChannelPipeline getPipeline() throws Exception {
>
>
> //Init the TLS for that channel
> SSLEngine engine = context.createSSLEngine();
> engine.setUseClientMode(false);
> engine.setWantClientAuth(false);
> engine.setEnableSessionCreation(true);
>
> ChannelPipeline pipeline = Channels.pipeline();
> pipeline.addLast("TLS", new SslHandler(engine));
> pipeline.addLast("Timeout", new Disconnector(new
> HashedWheelTimer(WORKER_THREAD_POOL.getThreadFactory()),
>
> SESSION_TIMEOUT,SESSION_TIMEOUT,SESSION_TIMEOUT,
> TimeUnit.MILLISECONDS));
>
> //Decode requests: first get the protobuf message
> length, then convert to
> protobuf, then convert to our own objects
> pipeline.addLast("frameDecoder",
> new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0,
> 4));
> pipeline.addLast("protobufDecoder",
> new ProtobufDecoder(MyMessage.getDefaultInstance()));
> pipeline.addLast("requestDecoder", new MessageDecoder
> ());
>
> //Encode responses: first convert from our objects to
> protobuf, then
> protobuf to binary, then add the length field
> pipeline.addLast("frameEncoder", new
> LengthFieldPrepender(4));
> pipeline.addLast("protobufEncoder", new
> ProtobufEncoder());
> pipeline.addLast("responseEncoder", new MessageEncoder
> ());
>
> pipeline.addLast("Logging", new ChannelLogger()); //
> logs using our own API
> pipeline.addLast("handler", new RequestHandler()); //
> actually processes
> the request. Normal processing time: ~5-10 seconds
> pipeline.addLast("disconnectionNotifier", new
> DisconnectionNotifier());
> //Informs the observers that a channel is disconnected
>
> return pipeline;
> }
>
> }
>
> // How the disconnections are handled
> class Disconnector extends IdleStateHandler{
> public Disconnector(Timer timer, int readerIdleTimeSeconds,
> int writerIdleTimeSeconds, int
> allIdleTimeSeconds) {
> super(timer, readerIdleTimeSeconds,
> writerIdleTimeSeconds,
> allIdleTimeSeconds);
> }
>
> public Disconnector(Timer timer, long readerIdleTime,
> long writerIdleTime, long allIdleTime,
> TimeUnit unit) {
> super(timer, readerIdleTime, writerIdleTime,
> allIdleTime, unit);
> }
>
> @Override
> protected void channelIdle(ChannelHandlerContext ctx,
> IdleState state,
> long lastActivityTimeMillis) throws Exception {
> super.channelIdle(ctx, state, lastActivityTimeMillis);
> ctx.getChannel().close();
> releaseExternalResources();
>
> }
>
> @Override
> protected void finalize() throws Throwable {
> super.finalize();
> releaseExternalResources();
> }
>
> }
>
> Can anyone point out what I'm doing wrong???
>
> Thanks in advance!
> --
> View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Netty-is-Freezing-on-Load-Testing-tp5138397p5138397.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20100605/1f358e1f/attachment.html
More information about the netty-users
mailing list