We currently use JGroups' partial state transfer to transfer individual caches from one Infinispan instance to another. Since I got rid of partial state transfer in JGroups 3.0, and don't like to add it back, I'd like to know whether this is still needed. I thought that we currently require the same set of caches to be available in all Infinispan instances, and the reason (IIRC) was that distribution wouldn't work if we have caches 1 and 2 available on instances A and B, but not on C, because consistent hashing distributes the data based on views, and we didn't want to have to keep track of individual caches...

Why are we actually using JGroups' state transfer with replication, but use our own state transfer with distribution ?

Opinions ? -- Bela Ban Lead JGroups / Clustering Team JBoss _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

>> Why are we actually using JGroups' state transfer with replication, but >> use our own state transfer with distribution ? > > I don't know, but guess it's because each node has a different set of > keys so no node has the same state as another ? You could still use JGroups state transfer; getState() would list the state provider as target node. In general, partial state transfer involves transferring (1) the partial state and (2) the digest, which is a vector of highest seqnos seen for every member. When we get a partial state, we always overwrite our own digest with the one received, and update our state accordingly. However, when this is done a couple of times, for different partial states, I'm not sure that we won't receive a few messages multiple times, due to the digest overwriting... I think the cleanest solution would be for you guys to reuse the state transfer you already use for state transfer in distribution mode.

On 6/1/11 4:21 PM, Sanne Grinovero wrote: > Hi Bela, > > 2011/6/1 Bela Ban<bban(a)redhat.com>: >> We currently use JGroups' partial state transfer to transfer individual >> caches from one Infinispan instance to another. >> >> Since I got rid of partial state transfer in JGroups 3.0, and don't like >> to add it back, I'd like to know whether this is still needed. >> >> I thought that we currently require the same set of caches to be >> available in all Infinispan instances, and the reason (IIRC) was that >> distribution wouldn't work if we have caches 1 and 2 available on >> instances A and B, but not on C, because consistent hashing distributes >> the data based on views, and we didn't want to have to keep track of >> individual caches... > > Well I really don't like this limitation in Infinispan and was > actually hoping that we could remove it at some point. > Imagine the scenario in which you have a running cluster, and at some > point the new release of your application needs an additional cache: > there's no way to start a new node having this new cache. Yes, I fully agree. Another example is HTTP web sessions: currently 1 webapp == 1 cache, so we currently require the same webapps to be deployed in all JBoss instances *if* we use replication (distribution is different) !

> Also right now when an application starts, it's possible with the > proper timing that it joins the cluster before having defined and > started all caches (starting the cachemanager and the caches is not an > atomic operation), basically failing to start because of this > limitation. Yep > Maybe it's still possible to build such a thing on top of non-partial > state transfer? As it doesn't exist, we didn't design it. Yes. Well, Infinispan already uses its own state transfer for distribution, I wonder why this isn't the case for replication. >> Why are we actually using JGroups' state transfer with replication, but >> use our own state transfer with distribution ? > > I don't know, but guess it's because each node has a different set of > keys so no node has the same state as another ? You could still use JGroups state transfer; getState() would list the state provider as target node. In general, partial state transfer involves transferring (1) the partial state and (2) the digest, which is a vector of highest seqnos seen for every member. When we get a partial state, we always overwrite our own digest with the one received, and update our state accordingly. However, when this is done a couple of times, for different partial states, I'm not sure that we won't receive a few messages multiple times, due to the digest overwriting... I think the cleanest solution would be for you guys to reuse the state transfer you already use for state transfer in distribution mode.

We use partial state transfer not to generate partial state per cache, but the entire state per cache, but since we have> 1 cache sharing a given JGroups channel, as far as JGroups in concerned this *is* partial state of a node. I.e., the state of just 1 cache on a channel, not all the caches. So we actually use cacheName as the state identifier (in JGroups' ExtendedMessageListener). But yes, there is no reason why we can't replace this with RPC as per Distribution, however I think we do need a streaming solution - not just for replication but distribution as well. As such I'd only want to re-implement this bit once, rather than a temp RPC based solution first. So we need a mechanism to either: (1) open a separate TCP socket for the sake of streaming state, or (2) reuse the sockets JGroups opens. They both have their pros and cons. (1) is more configuration, firewall setup, and a spiderweb of connections in a large grid (2) would mean multiplexing with JGroups' use of the socket. Any thoughts and suggestions? Cheers Manik On 1 Jun 2011, at 15:14, Bela Ban wrote: > We currently use JGroups' partial state transfer to transfer individual > caches from one Infinispan instance to another. > > Since I got rid of partial state transfer in JGroups 3.0, and don't like > to add it back, I'd like to know whether this is still needed. > > I thought that we currently require the same set of caches to be > available in all Infinispan instances, and the reason (IIRC) was that > distribution wouldn't work if we have caches 1 and 2 available on > instances A and B, but not on C, because consistent hashing distributes > the data based on views, and we didn't want to have to keep track of > individual caches... > > Why are we actually using JGroups' state transfer with replication, but > use our own state transfer with distribution ? > > Opinions ? > > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Now this might sound a bit too radical but do we really need REPLICATED mode? This is not fully brewed, but if e.g. we set numOwners = Integer.MAX_INTEGER the cluster is effectively in replicated mode, so can't we just drop the REPLICATION entirely? This would reduce the code size significantly...

> But yes, there is no reason why we can't replace this with RPC as per Distribution, however I think we do need a streaming solution - not just for replication but distribution as well. As such I'd only want to re-implement this bit once, rather than a temp RPC based solution first. So we need a mechanism to either: Now this might sound a bit too radical but do we really need REPLICATED mode?

This is not fully brewed, but if e.g. we set numOwners = Integer.MAX_INTEGER the cluster is effectively in replicated mode, so can't we just drop the REPLICATION entirely? This would reduce the code size significantly...

> > (1) open a separate TCP socket for the sake of streaming state, or > (2) reuse the sockets JGroups opens. > They both have their pros and cons. > > (1) is more configuration, firewall setup, and a spiderweb of connections in a large grid > (2) would mean multiplexing with JGroups' use of the socket. Having our own sockets might cause an administration complications. Also borrowing sockets from jgroups doesn't seems nice...I'm not a fan of either solution really: I think this should be transport's responsibility and we should enhance jgroups to offer the streaming service.