[jbosscache-issues] [JBoss JIRA] Commented: (JBCACHE-1528) Data gravitation prefers exception response over valid gravitation

Brian Stansberry (JIRA) jira-events at lists.jboss.org
Wed Aug 5 18:14:29 EDT 2009 

    [ https://jira.jboss.org/jira/browse/JBCACHE-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12479147#action_12479147 ] 

Brian Stansberry commented on JBCACHE-1528:
-------------------------------------------

Hmm this is tricky to deal with on the caller side, as RpcManagerImpl intentionally throws any received exceptions, so the caller doesn't get the opportunity to ignore the exception and consume other responses.

On the callee side the RequestIgnoredResponse is used to try to communicate the node is unavailable, but it seems there's a race and invocations are getting through to where an ISE can be thrown. Here's a full stack trace:

[JBoss] java.lang.IllegalStateException: Cache not in STARTED state!
[JBoss] 	at org.jboss.cache.invocation.CacheInvocationDelegate.cacheStatusCheck(CacheInvocationDelegate.java:695)
[JBoss] 	at org.jboss.cache.invocation.CacheInvocationDelegate.getNode(CacheInvocationDelegate.java:492)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.perform(GravitateDataCommand.java:153)
[JBoss] 	at org.jboss.cache.interceptors.CallInterceptor.invokeCommand(CallInterceptor.java:108)
[JBoss] 	at org.jboss.cache.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:99)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.handleGravitateDataCommand(PrePostProcessingCommandInterceptor.java:375)
[JBoss] 	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitGravitateDataCommand(PrePostProcessingCommandInterceptor.java:365)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
[JBoss] 	at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBoss] 	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
[JBoss] 	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
[JBoss] 	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
[JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:211)
[JBoss] 	at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
[JBoss] 	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:297)
[JBoss] 	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:245)
[JBoss] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
[JBoss] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
[JBoss] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
[JBoss] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
[JBoss] 	at org.jgroups.JChannel.up(JChannel.java:1339)
[JBoss] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
[JBoss] 	at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:432)
[JBoss] 	at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:144)
[JBoss] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
[JBoss] 	at org.jgroups.protocols.FC.up(FC.java:473)
[JBoss] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:824)
[JBoss] 	at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
[JBoss] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
[JBoss] 	at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
[JBoss] 	at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
[JBoss] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:758)
[JBoss] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
[JBoss] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
[JBoss] 	at org.jgroups.protocols.FD.up(FD.java:284)
[JBoss] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:309)
[JBoss] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
[JBoss] 	at org.jgroups.protocols.Discovery.up(Discovery.java:264)
[JBoss] 	at org.jgroups.protocols.PING.up(PING.java:273)
[JBoss] 	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2327)
[JBoss] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1261)
[JBoss] 	at org.jgroups.protocols.TP.access$100(TP.java:49)
[JBoss] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1838)
[JBoss] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1817)
[JBoss] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[JBoss] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[JBoss] 	at java.lang.Thread.run(Thread.java:619)

> Data gravitation prefers exception response over valid gravitation
> ------------------------------------------------------------------
>
>                 Key: JBCACHE-1528
>                 URL: https://jira.jboss.org/jira/browse/JBCACHE-1528
>             Project: JBoss Cache
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Buddy Replication
>    Affects Versions: 3.1.0.GA
>            Reporter: Brian Stansberry
>            Assignee: Manik Surtani
>
> A web request that performed a data gravitation for the session failed because one node returned an ISE while another returned the data. The ISE should have been ignored and the returned data used.
> The first log message below shows what was returned from the gravitation RPC; the second indicates that the ISE propagated.
> [JBoss] 16:48:56,696 TRACE [org.jboss.cache.marshall.CommandAwareRpcDispatcher] (ajp-jawa04%2F10.34.32.156-8009-7) responses: [sender=10.34.32.154:59373, retval=GravitateResult dataFound=false nodeData=null fqn=null, received=true, suspected=false]
> [JBoss] [sender=10.34.32.153:40508, retval=GravitateResult dataFound=true nodeData=[NodeData {fqn: /JSESSION/st_localhost/BrvolmfTMDqZAzEL1quXCA__, attrs={0=192, 1=1249483734634, 2=org.jboss.web.tomcat.service.session.distributedcache.spi.DistributableSessionMetadata at 500942cd, 3=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=nullserialized=true}}}] fqn=/_BUDDY_BACKUP_/10.34.32.155_47742/JSESSION/st_localhost/BrvolmfTMDqZAzEL1quXCA__, received=true, suspected=false]
> [JBoss] [sender=10.34.32.155:47742, retval=java.lang.IllegalStateException: Cache not in STARTED state!, received=true, suspected=false]
> [JBoss] 
> [JBoss] 16:48:56,698 ERROR [org.apache.catalina.connector.CoyoteAdapter] (ajp-jawa04%2F10.34.32.156-8009-7) An exception or error occurred in the container during the request processing
> [JBoss] java.lang.IllegalStateException: Cache not in STARTED state!
> [JBoss] 	at org.jboss.cache.invocation.CacheInvocationDelegate.cacheStatusCheck(CacheInvocationDelegate.java:695)
> [JBoss] 	at org.jboss.cache.invocation.CacheInvocationDelegate.getChildrenNamesInternal(CacheInvocationDelegate.java:652)
> [JBoss] 	at org.jboss.cache.invocation.CacheInvocationDelegate.getChildrenNames(CacheInvocationDelegate.java:639)
> [JBoss] 	at org.jboss.cache.commands.read.GravitateDataCommand.perform(GravitateDataCommand.java:130)
> ....
> The stack trace below is what was returned from the remote node; it's not the caller-side stack.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

More information about the jbosscache-issues mailing list