[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- And it also affects default SFSB cache. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani updated ISPN-835: ------------------------------- Description: Enabling state transfer is forcing even asynchronous caches to become synchronous. A better way is needed to make sure state transfer works correctly (sync calls needed here) while normal replication calls remain asynchronous. See https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o... where the response mode is overridden based on whether replay is supported. And replay support is always on when state transfer is enabled: https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o... Need to explore whether a joiner flag can be maintained based on a view change, and replay only supported when a joiner is still "joining", and not otherwise. was: Enabling state transfer is forcing even asynchronous caches to become synchronous. A better way is needed to make sure state transfer works correctly (sync calls needed here) while normal replication calls remain asynchronous/ -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Sanne Grinovero edited comment on ISPN-835 at 12/15/10 12:02 PM: ----------------------------------------------------------------- not to be fixed for 4.2.0.Final ? was (Author: sannegrinovero): no to be fixed for 4.2.0.Final ? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-835: ------------------------------------ Will depend on how complex/involved the fix is. I have a feeling it won't be trivial. But it doesn't break functionality and correctness, and there is a valid workaround. If it doesn't make 4.2.0.Final, it is more likely to make a 4.2.1.Final rather than all the way to 5.0. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño updated ISPN-835: ---------------------------------- Fix Version/s: 4.2.1.Final This should be fixed for 4.2.1. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-835: ------------------------------------ This is definitely non-trivial. One approach would be to scrap senders replaying messages altogether. Let joiners receive - and enqueue - messages it receives but cannot process due to state transfer not being completed. Then at the end of the ST process, apply enqueued writes, and then throw open the latch. This way the whole sync approach to deal with replays goes away. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-835: ------------------------------------ No, senders still track pending prepares and writes. That is a whole different story. The replay mechanism has nothing to do with senders; it is to ensure modifications by third-parties (other nodes - not just the state transfer sender and receiver) who send updates to the joiner replay the message to the joiner after the joiner applies its state. It is to support this replay that we force sync repl. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani updated ISPN-835: ------------------------------- Fix Version/s: (was: 4.2.1.Final) Workaround Description: Workaround #1: Disable state transfer - and use a ClusteredCacheLoader to lazily lookup remote state as and when needed. It might require some code changes to make the code efficient, for example: If put() method's return value is not needed, pass Flag.SKIP_CACHE_LOAD so that clustered cache loader is not queried for previous value. Workaround #2: Enable state transfer and REPL_SYNC. Use the Async API (e.g., cache.putAsync(k, v)) to actieve 'fire-and-forget' capabilities. Workaround #3: Enable state transfer and REPL_ASYNC. All RPCs end up becoming sync, but client threads *will not* be held up if you enable a replication queue (which is recommended for async mode anyway). was: Disable state transfer - and use a ClusteredCacheLoader to lazily lookup remote state as and when needed. It might require some code changes to make the code efficient, for example: If put() method's return value is not needed, pass Flag.SKIP_CACHE_LOAD so that clustered cache loader is not queried for previous value. Priority: Major (was: Critical) De-prioritising due to the large number of options as workarounds. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-835: ------------------------------------ Yes, you'd need an adequately sized threadpool for the async executor. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- As I already said, #1 might require some code reworking to be fully performant. Otherwise you're see repl timeouts like the ones reported by Cyrille in http://community.jboss.org/message/581825#581825 #2 requires code changes. #3 is probably the only one that you can use safely modifying AS 6's config. Try it out and let us know how it goes! -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- Manik, not sure you understood what I meant: If you use a repl queue, you'll use a separate thread to send the replication, but the replication message underneath will still be sync due to the replay transformation which happens further down the code, correct? The only difference is that instead of blocking the user thread and getting the exception to bubble up to the client thread, you'd see the exception in the repl queue thread. Clearly, as you said, cos these threads will block, you'll need to size the pool correctly to avoid starvation. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] John Ament edited comment on ISPN-835 at 1/20/11 11:14 AM: ----------------------------------------------------------- Also, I don't have access to http://community.jboss.org/message/581825#581825 it seems, but if I had to guess it's the same error I'm getting. I agree w/ your assessment that this would simply move the exception to another thread. Was I wrong in understanding that #1 is enabled already then? BTW, in my case the exception occurs when attempting to commit a transaction for a JPA Entity. was (Author: meetoblivion): Also, I don't have access to http://community.jboss.org/message/581825#581825 it seems, but if I had to guess it's the same error I'm getting. I agree w/ your assessment that this would simply move the exception to another thread. Was I wrong in understanding that #1 is enabled already then? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- Oh bugger, Cyrille started a private discussion. I'll check with the JBoss.org guys to move this to Infinispan community. I'm attaching the logs he attached so that you can compare the timeouts. @John, with regards to #1, Paul Ferraro and Scott Marlow would be the right persons to see if they made the code changes as well. I discovered the need for these code changes afterwards, and so there's a chance they might not have added them (i.e. adding Flag.SKIP_CACHE_LOAD for those situations where previous values are not needed). With this code I could solve the timeouts Cyrille reported. For #3, you need to enable the replication queue which is just a matter of turning <async useReplQueue="false"/> to <async useReplQueue="true"/>. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- @John, just saw your latest edit wrt JPA entity. This is the same issue Cyrille had. You'd need a new Hibernate version so that the Flag.SKIP_CACHE_LOAD is used to avoid unnecessary remote calls. If you grab one of the latest Hibernate 3.6.1 snapshots, the code should be there. There's a JIRA to track in the HHH side: http://opensource.atlassian.com/projects/hibernate/browse/HHH-5793 If 3.6.1 snapshots don't work, try building your own Hibernate Infinispan 2LC from the latest of https://github.com/hibernate/hibernate-core/tree/3.6 branch. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-835: --------------------------------------- Having talked to Steve Ebersole, they're planning a Hibernate 3.6.1 release next week which should fix the issue. This release will be included in AS 6.0.1. Re: File loader? Hmmm, not sure but I really think you'd be better off waiting for a fix than trying out diff configurations. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-835: ------------------------------------ @Paul/John - what are your thoughts around Workaround #2? This would give users more flexibility in that they could tune the async executor thread pool. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-835?page=com.atlassian.jira.plugin.s... ] Manik Surtani resolved ISPN-835. -------------------------------- Fix Version/s: (was: 5.1.0.BETA1) (was: 5.1.0.FINAL) Resolution: Rejected This is out of date. It is superceded by ISPN-1194. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira