Re: [infinispan-dev] Threadpools in a large cluster

Hi all, Recently I came up with a solution that can help with the thread pool problem motivated by the following: In one of the first implementation of Total Order based commit protocol (TO), I had the requirement to move the PrepareCommand to another thread pool. In resume, the TO protocol delivers the PrepareCommand in a deterministic order in all the nodes, by a single deliver thread. To ensure consistency, if it delivers two conflicting transactions, the second transaction must wait until the first transaction finishes. However, blocking single deliver thread is not a good solution, because no more transaction can be validated, even if they don't conflict, while the thread is blocked. So, after creating a dependency graph (i.e. the second transaction knows that it must wait for the first transaction to finish) I move the PrepareCommand to another thread pool. Initially, I implemented a new command, called PrepareResponseCommand, that sends back the reply of the PrepareCommand. This solution has one disadvantage: I had to implement an ack collector in ISPN, while JGroups already offers me that with a synchronous communication. Recently (2 or 3 months ago) I implemented a simple modification in JGroups. In a more generic approach, it allows other threads to reply to a RPC request (such as the PrepareCommand). In the previous scenario, I replaced the PrepareResponseCommand and the ack collector implementation with a synchronous RPC invocation. I've used this solution in other issues in the Cloud-TM's ISPN fork. This solution is quite simple to implement and may help you to move the commands to ISPN internal thread pools. The modifications I've made are the following: 1) I added a new interface (see [1]) that is sent to the application instead of the Message object (see [4]). This interface contains the Message and it has a method to allow the application send the reply to that particular request. 2) I added a new object in [4] with the meaning: this return value is not the reply to the RPC request. This is the returned value that I return when I want to release the thread, because ISPN should return some object in the handle() method. Of course, I know that ISPN will invoke the sendReply() in some other place, otherwise, I will get a TimeoutException in the sender side. 3) Also I've changed the RequestCorrelator implementation to support the previous modifications (see [2] and [3]) In the Cloud-TM's ISPN fork I added a reference in the BaseCacheRpcCommand to [1] and added the method sendReply() [5]. In addition, I have the following uses cases working perfectly with this: 1) Total Order The scenario described in the beginning. The ParallelTotalOrderManager returns the DO_NOT_REPLY object when it receives a remote PrepareCommand (see [6] line 77). When the PrepareCommand is finally processed by the rest of the interceptor chain, it invokes the PreapreCommand.sendReply() (see [6] line 230). 2) GMU remote get GMU ensures SERIALIZABLE Isolation Level and the remote gets must ensure that the node that is processing the request has a minimum version available to ensure data consistency. The problem in ours initial implementation in large cluster, is the number of remote gets are very high and all the OOB are being blocked because of this condition. Same thing I've done with the ClusteredRemoteGet as you can in see [7], line 93 and 105. 3) GMU CommitCommand In GMU, the CommitCommand cannot be processed by any order. If T1 is serialized before T2, the commit command of T1 must be processed before the commit command of T2, even if the transactions do not have conflicts. This generates the same problem above and the same solution was adopted. I know that you have discussed some solutions and I would like to know what it is your opinion about what I've described. If you have questions, please let me know. Cheers, Pedro [1] https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/M... <https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/M... [2] https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... <https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... [3] https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... <https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... [4] https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... <https://github.com/pruivo/JGroups/blob/t_cloudtm/src/org/jgroups/blocks/R... [5] https://github.com/pruivo/infinispan/blob/cloudtm_v1/core/src/main/java/o... <https://github.com/pruivo/infinispan/blob/cloudtm_v1/core/src/main/java/o... [6] https://github.com/pruivo/infinispan/blob/cloudtm_v1/core/src/main/java/o... [7] https://github.com/pruivo/infinispan/blob/cloudtm_v1/core/src/main/java/o... On 2/3/13 11:35 AM, Bela Ban wrote: