[jboss-user] [JBoss Cache: Core Edition] - Re: TcpDelegatingCacheLoader

[lovelyliatroim]

Reason im asking is this, i didnt have time yesterday to go into detail.


I ran some load tests on my app in a 3 node cluster. One node in particular takes more of the load than the others and this node also hosts the TcpCacheServer. After a while lets call it node3 ran into difficulties with GC. The whole node froze for quite a while. Now the problem is this, after a while the other 2 nodes froze as well which is not what i expected. When node3 recovered from its GC problems the other 2 nodes recovered as well.

Now there is 2 ways the other 2 nodes are connected to node3. One way is that it communicates via UDP, so i dont see this as a problem. The 2nd way is via the TcpDelegatingLoader. Now unfortunately these tests where running on our production or will be production system and i dont have write access and no-one was there from operations to take a dump for me on this so i cant say for sure what exactly happened.

I am using just the _get method of the delegator


  |  protected Map<Object, Object> _get(Fqn name) throws Exception
  |    {
  |       synchronized (out)
  |       {
  |          out.reset();
  | 
  |          out.writeByte(TcpCacheOperations.GET);
  |          out.writeObject(name);
  |          out.flush();
  |          Object retval = in.readObject();
  |          if (retval instanceof Exception)
  |          {
  |             throw (Exception) retval;
  |          }
  |          return (Map) retval;
  |       }
  |    }
  | 

However my theory on it is this, node 1 and node2 after a period of time after node3 became totally unresponsive, it couldnt take any more requests. Only after node3 recovered did the other 2 recover so it looked like the complete thread pool of node1 and node2 got exhausted, node1 and node2 were still receiving requests for data that is located on node3, after a while the thread pool gets exhausted because every thread is waiting for a lock on the outputstream, this will never happen because we have one thread who is asking node3 for data but is blocked forever because it doesnt timeout on the read.  When node1 and node2 keep receiving requests for data on node3 then eventually every thread will end up waiting for a lock that will never come.

Like I said, its a theory, very hard to say what happened for sure without a dump but my original question stands, why isnt there a timeout on the read??

Thanks guys,
LL


View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4196813#4196813

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4196813