[jboss-user] [JBoss Cache: Core Edition] - Re: Deadlock in JDBCCacheLoader

[gpklimek]

Actually the 2 fqns' hashcodes do not map to the same shared lock.
Let's say that "/a/m" maps to lock X and "/l/p" maps to lock Y. But then "/a" maps to lock Y and "/l" maps to lock X.
JDBCCacheLoader when executing put() acquires at least two locks at a time. First it is lock for the fqn itself (this one is in exclusive mode) and then it tries to acquire lock for the fqn's parent when it calls exist().
Generally when you have a scenario where you attempt to acquire 2 locks you have two be sure that there is no concurrent scenario where the locks are acquired in reverse order. Otherwise you might experience a deadlock. Once it happens you will have rest of your locks in deadlock quickly.

I have to admit chances of this are small but I have experienced a production server going down because of this. Since the first deadlock happened the server was not responding to certain requests and then in the next 2 hours was not responding at all. The number of deadlocked threads was growing like a snow ball and eventually all of them were locked.

A solution I see for this issue would be to maintain one StripedLock instance per fqn size and review the code to make sure that whenever you acquire 2 locks you start from child and then go to parent (you also have to be careful about the exclusive mode).

Regards,
Grzegorz


View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4200636#4200636

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4200636