[jboss-user] [JBossCache] - org.jboss.cache.lock.TimeoutException thrown in second insta

When we startup another one of our web applications on a second instance in a cluster we get an org.jboss.cache.lock.TimeoutException thrown by org.jboss.cache.lock.IdentityLock. It seems that IdentityLock is attempting to acquire a WL when attempting to cache the query below but there is already a lock on the node /org/hibernate/cache/StandardQueryCache/sql, probably in the first instance. This application is running on Resin 3.1.2. We are using Hibernate 3.2.3 and JBoss cache 1.4.0.SP1. We have Hibernate configured to use a query and second level cache which is org.hibernate.cache.TreeCacheProvider, the JBoss TreeCache. We have the JBoss TreeCache to use the pessimistic locking scheme, the node isolation level is REPEATABLE_READ and the cache mode is LOCAL. How do we resolve this problem? Please let me know if you need further information from me. Stack trace: [14:10:41,302] ERROR [pool-9-thread-8] IdentityLock : write lock for /org/hibernate/cache/StandardQueryCache/sql: select searchro ut0_.ID as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat as hotel3_28_, searchrout0_.hotel_long as hotel4_28_ , searchrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat as destinat6_28_, searchrout0_.destination_long as de stinat7_28_, searchrout0_.distance_miles as distance8_28_, searchrout0_.distance_km as distance9_28_, searchrout0_.route_type as rou te10_28_ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout 0_.hotel_long=? and searchrout0_.destination_lat=? and searchrout0_.destination_long=? and searchrout0_.route_type=?; parameters: ; named parameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2, hotelLat=40.3444, destinationLong=-75.164199, destinationLat=39 .952201} could not be acquired after 0 ms. Locks: Read lock owners: [] Write lock owner: null (caller=Thread[pool-9-thread-8,5,main], lock info: (activeReaders=0, activeWriter=null, waitingReaders=0, waitingWriters =0, waitingUpgrader=0)) [14:10:41,305] INFO [pool-9-thread-8] TxInterceptor : There was a problem handling this request org.jboss.cache.lock.TimeoutException: failure acquiring lock: fqn=/org/hibernate/cache/StandardQueryCache/sql: select searchrout0_. ID as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat as hotel3_28_, searchrout0_.hotel_long as hotel4_28_, sea rchrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat as destinat6_28_, searchrout0_.destination_long as destina t7_28_, searchrout0_.distance_miles as distance8_28_, searchrout0_.distance_km as distance9_28_, searchrout0_.route_type as route10_ 28_ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout0_.ho tel_long=? and searchrout0_.destination_lat=? and searchrout0_.destination_long=? and searchrout0_.route_type=?; parameters: ; named parameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2, hotelLat=40.3444, destinationLong=-75.164199, destinationLat=39.9522 01}, caller=Thread[pool-9-thread-8,5,main], lock= (activeReaders=0, activeWriter=null, waitingReaders=0, waitingWriters=0, waitingUpgrader=0) at org.jboss.cache.Node.acquire(Node.java:407) at org.jboss.cache.interceptors.PessimisticLockInterceptor.lock(PessimisticLockInterceptor.java:228) at org.jboss.cache.interceptors.PessimisticLockInterceptor.invoke(PessimisticLockInterceptor.java:160) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.UnlockInterceptor.invoke(UnlockInterceptor.java:32) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:345) at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:156) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:157) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5520) at org.jboss.cache.TreeCache.putFailFast(TreeCache.java:3643) at org.hibernate.cache.TreeCache.put(TreeCache.java:74) at org.hibernate.cache.StandardQueryCache.put(StandardQueryCache.java:87) at org.hibernate.loader.Loader.putResultInQueryCache(Loader.java:2194) at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2138) at org.hibernate.loader.Loader.list(Loader.java:2096) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79) at com.ihg.dec.framework.dataAccessServices.hibernate.HibernateOperation.execGetListNamedQuery(HibernateOperation.java:208) at com.ihg.dec.framework.dataAccessServices.hibernate.destination.search.HibernateSearchResultDAO.getSearchRoute(HibernateSe archResultDAO.java:219) at com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO$4.retreiveCachedObject(Cac heableSearchResultDAO.java:156) at com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO$4.retreiveCachedObject(Cac heableSearchResultDAO.java:153) at com.ihg.dec.framework.dataAccessServices.cacheable.AbstractCacheableDAO.getCachedObject(AbstractCacheableDAO.java:429) at com.ihg.dec.framework.dataAccessServices.cacheable.AbstractProxiedCacheableDAO.getProxiedCachedObject(AbstractProxiedCach eableDAO.java:56) at com.ihg.dec.framework.dataAccessServices.cacheable.destination.search.CacheableSearchResultDAO.getSearchRoute(CacheableSe archResultDAO.java:150) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillDistanceFromSearchResultDAO(SearchDelegate.java :975) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillDistanceInformation(SearchDelegate.java:904) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.fillHotelInformation(SearchDelegate.java:792) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.constructHotelSearchResultSet(SearchDelegate.java:1 250) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.search(SearchDelegate.java:159) at com.ihg.dec.framework.businessServices.hotel.delegates.SearchDelegate.searchWithFilterAndSort(SearchDelegate.java:1448) at com.ihg.dec.framework.businessServices.hotel.HotelSearchBO.findHotels(HotelSearchBO.java:65) at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.ihg.dec.framework.businessServices.BusinessObjectProxy.invoke_aroundBody0(BusinessObjectProxy.java:111) at com.ihg.dec.framework.businessServices.BusinessObjectProxy$AjcClosure1.run(BusinessObjectProxy.java:1) at com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector.ajc$around$com_ihg_dec_aop_statistics_b usinessServices_BusinessObjectProxyStatisticCollector$1$640f34fproceed(BusinessObjectProxyStatisticCollector.aj:1) at com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector$1.callProceed(BusinessObjectProxyStatis ticCollector.aj:39) at com.ihg.dec.aop.statistics.AbstractStatisticCollector.callObjectProxy(AbstractStatisticCollector.aj:49) at com.ihg.dec.aop.statistics.businessServices.BusinessObjectProxyStatisticCollector.ajc$around$com_ihg_dec_aop_statistics_b usinessServices_BusinessObjectProxyStatisticCollector$1$640f34f(BusinessObjectProxyStatisticCollector.aj:36) at com.ihg.dec.framework.businessServices.BusinessObjectProxy.invoke(BusinessObjectProxy.java:94) at $Proxy15.findHotels(Unknown Source) at com.ihg.dec.framework.runtime.cacheWarmer.HotelSearchCacheWarmer$HotelSearchWarmTask.call(HotelSearchCacheWarmer.java:168 ) at com.ihg.dec.framework.runtime.cacheWarmer.HotelSearchCacheWarmer$HotelSearchWarmTask.call(HotelSearchCacheWarmer.java:128 ) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: org.jboss.cache.lock.TimeoutException: write lock for /org/hibernate/cache/StandardQueryCache/sql: select searchrout0_.ID as ID28_, searchrout0_.hotel_code as hotel2_28_, searchrout0_.hotel_lat as hotel3_28_, searchrout0_.hotel_long as hotel4_28_, searc hrout0_.destination_name as destinat5_28_, searchrout0_.destination_lat as destinat6_28_, searchrout0_.destination_long as destinat7 _28_, searchrout0_.distance_miles as distance8_28_, searchrout0_.distance_km as distance9_28_, searchrout0_.route_type as route10_28 _ from UA_MQ_HOTEL_DESTINATION_ROUTE searchrout0_ where searchrout0_.hotel_code=? and searchrout0_.hotel_lat=? and searchrout0_.hote l_long=? and searchrout0_.destination_lat=? and searchrout0_.destination_long=? and searchrout0_.route_type=?; parameters: ; named p arameters: {hotelCode=JMBNJ, hotelLong=-74.4783, routeType=2, hotelLat=40.3444, destinationLong=-75.164199, destinationLat=39.952201 } could not be acquired after 0 ms. Locks: Read lock owners: [] Write lock owner: null (caller=Thread[pool-9-thread-8,5,main], lock info: (activeReaders=0, activeWriter=null, waitingReaders=0, waitingWriters =0, waitingUpgrader=0)) at org.jboss.cache.lock.IdentityLock.acquireWriteLock(IdentityLock.java:202) at org.jboss.cache.Node.acquireWriteLock(Node.java:431) at org.jboss.cache.Node.acquire(Node.java:386) Hibernate configuration: | <!DOCTYPE hibernate-configuration PUBLIC | "-//Hibernate/Hibernate Configuration DTD 3.0//EN" | "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd"> | | <hibernate-configuration> | <session-factory> | <property name="show_sql">false</property> | <property name="connection.datasource">java:comp/env/jdbc/oracle</property> | <property name="dialect"> | org.hibernate.dialect.Oracle9Dialect | </property> | <property name="cache.provider_class"> | org.hibernate.cache.TreeCacheProvider | </property> | <property name="cache.use_query_cache">true</property> | <property name="cache.use_second_level_cache">true</property> | <property name="transaction.manager_lookup_class"> | org.hibernate.transaction.ResinTransactionManagerLookup | </property> | <property name="transaction.factory_class"> | org.hibernate.transaction.JTATransactionFactory | </property> | </session-factory> | </hibernate-configuration> | JBoss TreeCache configuration: | | <?xml version="1.0" encoding="UTF-8"?> | | <!-- ===================================================================== --> | <!-- --> | <!-- TreeCache Service Configuration --> | <!-- --> | <!-- ===================================================================== --> | | <server> | | <classpath codebase="./lib" archives="jboss-cache.jar" /> | | | <!-- ==================================================================== --> | <!-- Defines TreeCache configuration --> | <!-- ==================================================================== --> | | <mbean code="org.jboss.cache.TreeCache" | name="jboss.cache:service=TreeCache"> | | <depends>jboss:service=Naming</depends> | | <!-- | Configure the TransactionManager | --> | <!-- | Node locking scheme: | OPTIMISTIC | PESSIMISTIC (default) | --> | <attribute name="NodeLockingScheme">PESSIMISTIC</attribute> | | <attribute name="FetchInMemoryState">false</attribute> | <attribute name="FetchPersistentState">false</attribute> | | <!-- | Node isolation level : SERIALIZABLE | REPEATABLE_READ (default) | READ_COMMITTED | READ_UNCOMMITTED | NONE | --> | <attribute name="IsolationLevel">REPEATABLE_READ</attribute> | <!-- | Valid modes are LOCAL | REPL_ASYNC | REPL_SYNC | --> | <attribute name="CacheMode">LOCAL</attribute> | | <!-- Whether each interceptor should have an mbean | registered to capture and display its statistics. --> | <attribute name="UseInterceptorMbeans">true</attribute> | | <!-- | The max amount of time (in milliseconds) we wait until the | initial state (ie. the contents of the cache) are retrieved from | existing members in a clustered environment | --> | <attribute name="InitialStateRetrievalTimeout"> | 100000 | </attribute> | | <!-- | Number of milliseconds to wait until all responses for a | synchronous call have been received. | --> | <attribute name="SyncReplTimeout">10000</attribute> | | <!-- Max number of milliseconds to wait for a lock acquisition --> | <attribute name="LockAcquisitionTimeout">60000</attribute> | | <!-- Name of the eviction policy class. --> | <attribute name="EvictionPolicyClass"> | org.jboss.cache.eviction.LRUPolicy | </attribute> | | <!-- Specific eviction policy configurations. This is LRU --> | <attribute name="EvictionPolicyConfig"> | <config> | <attribute name="wakeUpIntervalSeconds">5</attribute> | <!-- Cache wide default --> | <region name="/_default_"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds">3600</attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <region name="/cro/numbers"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds">3600</attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <region name="/hotels/detail"> | <attribute name="maxNodes">2000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <!-- Vales based on that we are not properly clearing the cache after a data load --> | <region name="/hotels/sales/features"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">21600</attribute> | </region> | | <region name="/hotels/sales/help"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">21600</attribute> | </region> | | <region name="/hotels/geocode"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds">86400</attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <region name="/hotels/search/addresses"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <region name="/destination/searchResult"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <region name="/destination/searchRoute"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <region name="/hotels/search/hotels"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <region name="/lookup/countryCollections"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <region name="/lookup/stateCollections"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <region name="/lookup/pointsOfInterests"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <region name="/teasers"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds">3600</attribute> | <attribute name="maxAgeSeconds">259200</attribute> | </region> | | <region name="/urlmanager"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds">3600</attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | | <region name="/rate/RateCategory"> | <attribute name="maxNodes">1500</attribute> | <attribute name="timeToLiveSeconds"> | 604800 | </attribute> | <attribute name="maxAgeSeconds">604800</attribute> | </region> | <region name="/b2b/displayableText"> | <attribute name="maxNodes">1000</attribute> | <attribute name="timeToLiveSeconds"> | 604800 | </attribute> | <attribute name="maxAgeSeconds">604800</attribute> | </region> | <region name="/b2b/client"> | <attribute name="maxNodes">1000</attribute> | <attribute name="timeToLiveSeconds"> | 604800 | </attribute> | <attribute name="maxAgeSeconds">604800</attribute> | </region> | <region name="/b2b/codes"> | <attribute name="maxNodes">1000</attribute> | <attribute name="timeToLiveSeconds"> | 604800 | </attribute> | <attribute name="maxAgeSeconds">604800</attribute> | </region> | <region name="/b2b/errors"> | <attribute name="maxNodes">1000</attribute> | <attribute name="timeToLiveSeconds"> | 604800 | </attribute> | <attribute name="maxAgeSeconds">604800</attribute> | </region> | <!-- | This region should be used for caching transaction errors from | external systems such as 500 errors from Content servres. Nodes should | not be stored directly to this region but should be stored in sub-regions | such as /transactionErrors/ | --> | <region name="/transactionErrors"> | <attribute name="maxNodes">1000</attribute> | <attribute name="timeToLiveSeconds">60</attribute> | <attribute name="maxAgeSeconds">60</attribute> | </region> | | <!-- | This region is used as an in-memory cache for avail cache. Items in avail cache should | not live beyond 15 minutes. | --> | <region name="/availCache"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds">900</attribute> | <attribute name="maxAgeSeconds">900</attribute> | </region> | | <!-- Region to store the collection of Brand objects --> | <region name="/lookup/brand"> | <attribute name="maxNodes">50</attribute> | <attribute name="timeToLiveSeconds">0</attribute> | <attribute name="maxAgeSeconds">0</attribute> | </region> | | <!-- Region to store the collection of FeatureType objects --> | <region name="/lookup/featureType"> | <attribute name="maxNodes">2000</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store geo destination objects --> | <region name="/hotels/search/geoDestinations"> | <attribute name="maxNodes">5000</attribute> | <attribute name="timeToLiveSeconds"> | 14400 | </attribute> | <attribute name="maxAgeSeconds">43200</attribute> | </region> | | <!-- Region to store the RoomType objects --> | <region name="/lookup/roomType"> | <attribute name="maxNodes">100</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store hotel name objects, the maxAge is 24 hours and timeToLive is 12 hours --> | <region name="/hotels/search/hotelName"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 43200 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store city nick name, the maxAge is 24 hours and timeToLive is 12 hours --> | <region name="/hotels/search/cityNickname"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds"> | 43200 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store the Currency objects --> | <region name="/lookup/currency"> | <attribute name="maxNodes">200</attribute> | <attribute name="timeToLiveSeconds"> | 21600 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store the CurrencyDetail objects --> | <region name="/lookup/currencyDetail"> | <attribute name="maxNodes">200</attribute> | <attribute name="timeToLiveSeconds">3600</attribute> | <attribute name="maxAgeSeconds">7200</attribute> | </region> | | <!-- Region to store credit card identifiers --> | <region name="/lookup/creditCard"> | <attribute name="maxNodes">10</attribute> | <attribute name="timeToLiveSeconds"> | 86400 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store contact centers --> | <region name="/lookup/contactCenter"> | <attribute name="maxNodes">20</attribute> | <attribute name="timeToLiveSeconds"> | 86400 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | <!-- Region to store contact centers --> | <region name="/proxy/proxyClasses"> | <attribute name="maxNodes">2000</attribute> | <attribute name="timeToLiveSeconds"> | 86400 | </attribute> | <attribute name="maxAgeSeconds">86400</attribute> | </region> | | </config> | </attribute> | | </mbean> | </server> | View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4114227#... Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...