[jboss-user] [JBossCache] - Reading threads are blocked

[FredrikJ]

We are currently testing the 2.0 CR2 in a load intensive environment. I started to notice a lot of blocked threads that were blocked by the cache.

In order to investigate further I constructed a simple load test on a cache. The cache uses the standard replAsync-service.xml

This is what I do:
1. Create a cache
  | 2. Initialize the cache with some nodes and data
  | 3. Start a large number of concurrent readers 
  | 4. Observe the threads and monitors in the system using jprofiler
  | 
  | 
  | Contents of the created and initialized cache:
  | 
  |   | DEBUG CacheLockGetTest - null  {}
  |   |   /d  {}
  |   |     /g  {1=Data}
  |   |     /h  {1=Data}
  |   |     /f  {1=Data}
  |   |     /e  {1=Data}
  |   |   /b  {}
  |   |     /g  {1=Data}
  |   |     /h  {1=Data}
  |   |     /f  {1=Data}
  |   |     /e  {1=Data}
  |   |   /a  {}
  |   |     /g  {1=Data}
  |   |     /h  {1=Data}
  |   |     /f  {1=Data}
  |   |     /e  {1=Data}
  |   |   /c  {}
  |   |     /g  {1=Data}
  |   |     /h  {1=Data}
  |   |     /f  {1=Data}
  |   |     /e  {1=Data}
  |   | 
  | 
  | The task used to read from the cache:
  | 
  | 
  |   |         private class CacheExecutor implements Runnable {
  |   | 		private final Fqn fqn;
  |   | 		public CacheExecutor(Fqn fqn) { this.fqn = fqn;	}
  |   | 		
  |   | 		public void run() {
  |   | 			cache.get(fqn, 1);
  |   | 		}
  |   | 	}
  |   | 
  | 
  | 
  | The cache has been initialized before I start the readers, so that when the readers are running no write occurs on the cache. Now, I thought that read operations on the cache would be concurrent but when inspecting the threads in jprofiler I got this:
  | 
  | http://www.robotsociety.com/cache/threads.GIF
  | 
  | If we look at the contending threads' stack traces we see that they all block against UnversionedNode (See: http://www.robotsociety.com/cache/monitors.GIF)
  | 
  | I have looked at unversioned node and there is much synchronization going on in there. I am not sure why the blocking is so frequent and I also saw threads working in different regions blocking each other.
  | 
  | One question is, if I access a/b/c, will a read lock be acquired for all those nodes,  if so then we will ultimately have synchronization on the common root node for all read operations in the cache, correct?
  | 
  | I understand that the test above is a bit artificial since we do not run any other type of logic that would permit pauses for access, but I also run into this blocking issues in our server where we do execute other functionality.
  | 
  | Now, our cache is pretty large so I will try and divide the regions into separate caches to minimize the synchronization issues or try to find some other workaround, but shouldn't a cache solution permit more concurrent access rather then having what looks like a global sync lock?
  | 
  | On a side note, after running a while the UnversionedNode sync lock is replaced by the readOwnerList_ in LockMap (see: http://www.robotsociety.com/cache/monitors_copyarray.GIF). 

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4053977#4053977

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4053977