[JBoss JIRA] (ISPN-2903) Manual eviction should not delete entry from cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2903?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2903:
-----------------------------------------------
Jitka Kudrnacova <jkudrnac(a)redhat.com> made a comment on [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549]
Rado,
thanks for the quick update. So here is what I will do:
1. will try the test with mod_cluster to look at the 500 codes
2. will run the following performance tests:
- write heavy, scenario session, repl async
- write heavy, scenario session, repl sync
- read heavy, sceanario session, repl async
I'll update this BZ once the runs are ready.
> Manual eviction should not delete entry from cache store
> --------------------------------------------------------
>
> Key: ISPN-2903
> URL: https://issues.jboss.org/browse/ISPN-2903
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.3.Final
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.5.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: AtomicMapServlet.java, AtomicMapTestCase.java, server.log, server.log
>
>
> Here's the scenario:
> Given 2 nodes with REPL_SYNC cache with passivating cache store (e.g. default web cache in AS7).
> 1. Create cache entry containing atomic map with 2 map entries on node1.
> 2. Passivate that cache entry on node2 via manual evict.
> 3. Modify 1 of the atomic map entries within the cache entry on node1.
> 4. Lookup atomic map on node2. It only contains 1 map entry - the map entry modified in step 3. The other map entry is lost.
> It's a side effect of ISPN-2384, where some changes were made to tighten the passivation/activation scenarios, but it did not cover manual eviction calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-2402) Cache operations or transactions should never fail with SuspectException
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-2402?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-2402:
---------------------------------------
up :)
> Cache operations or transactions should never fail with SuspectException
> ------------------------------------------------------------------------
>
> Key: ISPN-2402
> URL: https://issues.jboss.org/browse/ISPN-2402
> Project: Infinispan
> Issue Type: Task
> Components: RPC, State transfer
> Affects Versions: 5.2.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: 5.2.x
> Fix For: 6.0.0.Final
>
> Attachments: vrstt.log
>
>
> This is an extension of ISPN-1896 of sorts, but for all the cache operations that are visible to the user.
> After a node leaves, the other nodes that have sent commands to that node should either ignore SuspectExceptions or, if not possible, they should retry the operation (e.g. if they didn't get any response back).
> For example, VersionReplStateTransferTest quite often on my machine with a SuspectException, because the versioned prepare command expects a response from the coordinator and the coordinator has just left.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-2903) Manual eviction should not delete entry from cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2903?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2903:
-----------------------------------------------
Radoslav Husar <rhusar(a)redhat.com> made a comment on [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549]
Great to see we have one solution to the problem, thanks for the runs.
@Jitka Could we also get performance stress tests for both write-heavy and read-heavy scenarios to see what is the actual performance implication?
I was looking at the logs and there are no relevant exceptions in the server logs. The SC 500 are due to non-graceful shutdown, i.e. the AJP connector shuts down and the LB tries to connect resulting in SC 500. This is normally solved by session draining or using mod_cluster which this test does not do, moreover we do not retry in these situations as to not mask other problems (we currently retry on 404 and 503 only). The corresponding load balancer log message is as follow:
[Wed Jul 17 02:13:12.288 2013] [7603:139679944914688] [error] ajp_send_request::jk_ajp_common.c (1585): (perf18) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111)
Just to clear up some confusion from the previous assumtions and comments: this issue happens *on failover* and is not related passivation, nor dist/repl mode, nor to CDI/Weld. The stacktrace for non-CDI application is as follows (if anybody searches for the BZ):
17:15:27,682 ERROR [org.apache.catalina.connector] (http-/127.0.0.1:8080-2) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.NullPointerException
at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:973) [jboss-as-web-7.2.0.Final-redhat-X.jar:7.2.0.Final-redhat-X]
at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1397) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:681) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.connector.Request.doGetSession(Request.java:2605) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.connector.Request.getSession(Request.java:2357) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Updating BZ summary, setting flags, fields..
> Manual eviction should not delete entry from cache store
> --------------------------------------------------------
>
> Key: ISPN-2903
> URL: https://issues.jboss.org/browse/ISPN-2903
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.3.Final
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.5.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: AtomicMapServlet.java, AtomicMapTestCase.java, server.log, server.log
>
>
> Here's the scenario:
> Given 2 nodes with REPL_SYNC cache with passivating cache store (e.g. default web cache in AS7).
> 1. Create cache entry containing atomic map with 2 map entries on node1.
> 2. Passivate that cache entry on node2 via manual evict.
> 3. Modify 1 of the atomic map entries within the cache entry on node1.
> 4. Lookup atomic map on node2. It only contains 1 map entry - the map entry modified in step 3. The other map entry is lost.
> It's a side effect of ISPN-2384, where some changes were made to tighten the passivation/activation scenarios, but it did not cover manual eviction calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (ISPN-2903) Manual eviction should not delete entry from cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2903?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2903:
-----------------------------------------------
Jitka Kudrnacova <jkudrnac(a)redhat.com> made a comment on [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549]
@Rado, it works! no NPE in my test run:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http...
See the parsed logs:
client:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http...
server:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http...
However, I have 2 concerns:
1. I am getting quite a few 500 error code. Please take a look.
2. Please elaborate on "maintains session correctness but has performance implications."
Thank you.
> Manual eviction should not delete entry from cache store
> --------------------------------------------------------
>
> Key: ISPN-2903
> URL: https://issues.jboss.org/browse/ISPN-2903
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.3.Final
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.5.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: AtomicMapServlet.java, AtomicMapTestCase.java, server.log, server.log
>
>
> Here's the scenario:
> Given 2 nodes with REPL_SYNC cache with passivating cache store (e.g. default web cache in AS7).
> 1. Create cache entry containing atomic map with 2 map entries on node1.
> 2. Passivate that cache entry on node2 via manual evict.
> 3. Modify 1 of the atomic map entries within the cache entry on node1.
> 4. Lookup atomic map on node2. It only contains 1 map entry - the map entry modified in step 3. The other map entry is lost.
> It's a side effect of ISPN-2384, where some changes were made to tighten the passivation/activation scenarios, but it did not cover manual eviction calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months