[jbosscache-issues] [JBoss JIRA] Issue Comment Edited: (JBCACHE-1592) org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn is very frequently in cluster
Sebastian Gruber (JIRA)
jira-events at lists.jboss.org
Fri Dec 3 09:23:48 EST 2010
[ https://jira.jboss.org/browse/JBCACHE-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567477#comment-12567477 ]
Sebastian Gruber edited comment on JBCACHE-1592 at 12/3/10 9:22 AM:
--------------------------------------------------------------------
We were experiencing the same problem (even in an unclustered environment) under heavy load. An in-depth analysis of the problem yielded the following two approaches to solve or work around the issue:
- increase the concurrency level (see http://jbosscache.blogspot.com/2008/07/mvcc-has-landed.html )
- disable lock striping (see http://community.jboss.org/wiki/LockingandConcurrency#Lock_striping )
For us, increasing the concurrency level (even up to 1000000) did not resolve the problem. We needed to switch to disabling lock striping.
Details: See class {{org.jboss.cache.lock.MVCCLockManager}}. If lock striping is enabled (default), an {{OwnableReentrantSharedLockContainer}} used, otherwise a {{PerElementReentrantLockContainer}}.
was (Author: grubers):
We were experiencing the same problem (even in an unclustered environment) under heavy load. An in-depth analysis of the problem yielded the following two approaches to solve or work around the issue:
* increase the concurrency level (see http://jbosscache.blogspot.com/2008/07/mvcc-has-landed.html)
* disable lock striping (see http://community.jboss.org/wiki/LockingandConcurrency#Lock_striping)
For us, increasing the concurrency level (even up to 1000000) did not resolve the problem. We needed to switch to disabling lock striping.
Details: See class {{org.jboss.cache.lock.MVCCLockManager}}. If lock striping is enabled (default), an {{OwnableReentrantSharedLockContainer}} used, otherwise a {{PerElementReentrantLockContainer}}.
> org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn is very frequently in cluster
> --------------------------------------------------------------------------------------------------
>
> Key: JBCACHE-1592
> URL: https://jira.jboss.org/browse/JBCACHE-1592
> Project: JBoss Cache
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Locking
> Affects Versions: 3.2.5.GA
> Environment: windows os,cluster node in one machine.
> Reporter: l jw
> Assignee: Manik Surtani
> Priority: Blocker
>
> When used in cluster,the following exception is throw frequently:
> org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn [/courseClass/360/male] after [5000] milliseconds for requestor [GlobalTransaction:<172.18.73.162:3430>:1843]! Lock held by [GlobalTransaction:<172.18.73.162:3428>:6416]
> at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:159)
> at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:217)
> at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:186)
> at org.jboss.cache.interceptors.MVCCLockingInterceptor.handlePutKeyValueCommand(MVCCLockingInterceptor.java:101)
> at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitPutKeyValueCommand(PrePostProcessingCommandInterceptor.java:88)
> at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
> at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
> at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:150)
> at org.jboss.cache.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:107)
> at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
> at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
> at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
> at org.jboss.cache.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65)
> at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
> at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
> at org.jboss.cache.interceptors.TxInterceptor.replayModifications(TxInterceptor.java:501)
> at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:388)
> at org.jboss.cache.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:134)
> at org.jboss.cache.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:68)
> at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
> at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
> at org.jboss.cache.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:140)
> at org.jboss.cache.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:68)
> at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
> at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
> at org.jboss.cache.interceptors.InvocationContextInterceptor.visitPrepareCommand(InvocationContextInterceptor.java:106)
> at org.jboss.cache.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:68)
> at org.jboss.cache.interceptors.InterceptorChain.invokeRemote(InterceptorChain.java:316)
> at org.jboss.cache.commands.remote.ReplicateCommand.processSingleCommand(ReplicateCommand.java:139)
> at org.jboss.cache.commands.remote.ReplicateCommand.perform(ReplicateCommand.java:115)
> at org.jboss.cache.marshall.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:319)
> at org.jboss.cache.marshall.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:246)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
> at org.jgroups.JChannel.up(JChannel.java:1336)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
> at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:486)
> at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:319)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:820)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
> at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
> at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
> at org.jgroups.protocols.FD.up(FD.java:284)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:307)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
> at org.jgroups.protocols.Discovery.up(Discovery.java:264)
> at org.jgroups.protocols.PING.up(PING.java:273)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1273)
> at org.jgroups.protocols.TP.access$100(TP.java:49)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1826)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1805)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> and following is my config:
> <jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:jboss:jbosscache-core:config:3.2">
> <locking isolationLevel="REPEATABLE_READ"
> lockParentForChildInsertRemove="false"
> lockAcquisitionTimeout="5000"
> nodeLockingScheme="mvcc"
> writeSkewCheck="true"
> useLockStriping="true"
> concurrencyLevel="10000"/>
> <clustering mode="replication" clusterName="JBossCache-cluster">
> <stateRetrieval timeout="5000" fetchInMemoryState="false" nonBlocking="false"/>
> <sync replTimeout="5000"/>
> <jgroupsConfig>
> <UDP discard_incompatible_packets="true" enable_bundling="false" enable_diagnostics="false" ip_ttl="2"
> loopback="false" max_bundle_size="64000" max_bundle_timeout="30" mcast_addr="224.0.0.1"
> mcast_port="45588" mcast_recv_buf_size="25000000" mcast_send_buf_size="640000"
> oob_thread_pool.enabled="true" oob_thread_pool.keep_alive_time="10000" oob_thread_pool.max_threads="4"
> oob_thread_pool.min_threads="1" oob_thread_pool.queue_enabled="true"
> oob_thread_pool.queue_max_size="10"
> oob_thread_pool.rejection_policy="Run" thread_naming_pattern="pl" thread_pool.enabled="true"
> thread_pool.keep_alive_time="30000" thread_pool.max_threads="25" thread_pool.min_threads="1"
> thread_pool.queue_enabled="true" thread_pool.queue_max_size="10" thread_pool.rejection_policy="Run"
> tos="8" ucast_recv_buf_size="20000000" ucast_send_buf_size="640000" use_concurrent_stack="true"
> use_incoming_packet_handler="true"/>
> <PING num_initial_members="3" timeout="2000"/>
> <MERGE2 max_interval="30000" min_interval="10000"/>
> <FD_SOCK/>
> <FD max_tries="5" shun="true" timeout="10000"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <pbcast.NAKACK discard_delivered_msgs="true" gc_lag="0" retransmit_timeout="300,600,1200,2400,4800"
> use_mcast_xmit="false"/>
> <UNICAST timeout="300,600,1200,2400,3600"/>
> <pbcast.STABLE desired_avg_gossip="50000" max_bytes="400000" stability_delay="1000"/>
> <pbcast.GMS join_timeout="5000" print_local_addr="true" shun="false" view_ack_collection_timeout="5000"
> view_bundling="true"/>
> <FRAG2 frag_size="60000"/>
> <pbcast.STREAMING_STATE_TRANSFER/>
> <pbcast.FLUSH timeout="0"/>
> </jgroupsConfig>
> </clustering>
> </jbosscache>
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jbosscache-issues
mailing list