Looking through JGroups merge code; there is a possibility of n-way merge so it is not always the case that we have a two way merge as suggested interface indicates. How about something like: public interface MergeHandler { void handleMerge(Cache currentInstance, List<Address> preMergeView, List<Address> postMergeView); } Manik Surtani wrote: > This was on 2.2.0's task list: > > http://jira.jboss.com/jira/browse/JBCACHE-471 > > The last time we spoke about this was in Neuchatel, and we decided > we'd expose an interface, a MergeHandler, that people could > implement. We'd also provide a few standard implementations, like > EvictEntireClusterStateHandler and FatalStopMergeHandler? I think a > SingleWinnerHandler and DisjointedSubtreeMergeHandler can come > later, as they would both involve a proxying mechanism to talk to > remote caches being in place - something which will be developed as a > part of Partitioning [1]. > > We didn't discuss the design of this interface - how about a single > callback with: > > public interface MergeHandler > { > void handleMerge(Cache currentInstance, List<Address> > winningSubPartition, List<Address> losingSubPartition); > }

> I suggest you come up with a number of scenarios you'd like to cover > in JBC and then we see what information is needed. Maybe implementing > this will make the interface clearer... As far as JBC is concerned, I think we can only handle 2 scenarios effectively. 1. EvictEntireClusterStateHandler Each instance evicts it's local tree when it gets this. A variation of this could do a remove instead of an evict (to clear local cache loader state as well). Useful if the data exists elsewhere - e.g., when used with Hibernate. 2. FatalStopMergeHandler Logs a fatal error and shuts down the cache when a merge is detected

Either way though, this discussion is to design the handler interface - as far as solutions 1 and 2 above are concerned, all you need is a reference to the current cache instance (to (1) evict all state or (2) stop the cache).

Bela Ban wrote: <snip/> > > I don't think we have thought hard enough about all use cases we > might implement, or at least consider for the design of the > interface. Maybe we need to brainstorm about this at our next > clustering conf call ? > Where is this on our priority list? Is this something we are going to actually do over the next 3-4 months? Or are we discussing something that's not going to happen for a long time?

Bela Ban wrote: <snip/> > I don't think we have thought hard enough about all use cases we > might implement, or at least consider for the design of the > interface. Maybe we need to brainstorm about this at our next > clustering conf call ? Where is this on our priority list? Is this something we are going to actually do over the next 3-4 months? Or are we discussing something that's not going to happen for a long time?

Manik Surtani wrote: > On 19 Mar 2008, at 17:26, Brian Stansberry wrote: >> Bela Ban wrote: >> >> <snip/> >> >>> I don't think we have thought hard enough about all use cases we >>> might implement, or at least consider for the design of the >>> interface. Maybe we need to brainstorm about this at our next >>> clustering conf call ? >> >> Where is this on our priority list? Is this something we are going >> to actually do over the next 3-4 months? Or are we discussing >> something that's not going to happen for a long time? > I'm thinking if we can come up with a valid interface quick enough > (over the next week or two) I'd put it in 2.2.0 along with the > "easy" implementations (evict, shutdown). The more interesting > implementations of MergeHandlers can (a) come later, or (b) be > implemented by the client if they are in a rush for it. > If we can't come up with a satisfactory interface in the next 2 > weeks I'll defer this for 3.x. OK, I'll start thinking more about this then. I've been just vaguely following this thread until my name started popping up. What's your gut feel on whether this will happen for 2.2?

Manik Surtani wrote: > On 19 Mar 2008, at 19:26, Brian Stansberry wrote: <snip/> >> >> What's your gut feel on whether this will happen for 2.2? > I think it's possible if that's where we set our sights. At the > end of the day, we'd include an interface and an adapter with no-op > methods, and if people stick to using the adapter, the interface in > question can later be enhanced for 3.x if need be with more > sophisticated overloaded callbacks, to create more sophisticated > handlers. > In terms of my preference, I'd *like* to get it out for 2.2 to give > the community more time to play with it, write handlers, etc., and > help shape a more formal interface in 3.x. Perhaps it could be > marked as a "preview" feature, prone to change/enhancements in 3.x? That certainly takes some of the pressure off the interface discussion. :-) As long as it's "preview" in the sense of "we might change this a lot in 3.x" and not in the sense of "these 2 impls we provide are not quite production ready." If it's the latter, probably not worth doing.

>> I suggest you come up with a number of scenarios you'd like to cover >> in JBC and then we see what information is needed. Maybe implementing >> this will make the interface clearer... > > As far as JBC is concerned, I think we can only handle 2 scenarios > effectively. > > 1. EvictEntireClusterStateHandler > Each instance evicts it's local tree when it gets this. A variation > of this could do a remove instead of an evict (to clear local cache > loader state as well). Useful if the data exists elsewhere - e.g., > when used with Hibernate. > > 2. FatalStopMergeHandler > Logs a fatal error and shuts down the cache when a merge is detected These are simple policies, let's try to find a hard one. For example, 3. HTTP session replication (Brian): here, we could provide a policy which merges the (usually disjoint) data sets. If there are conflicts, we could use the number of updates to a session as arbiter. Or the time stamp of the last update, or something deterministic.

4. Untion of all substates. Again, if the user provides an ID of some sort, we might be able to take advantage of that. > Either way though, this discussion is to design the handler interface > - as far as solutions 1 and 2 above are concerned, all you need is a > reference to the current cache instance (to (1) evict all state or (2) > stop the cache). Yes, but as soon as you start looking into 3 and 4, this interface might change. My point is that an interface is a public API, once you've provided it, it gets hard to change it.

I don't think we have thought hard enough about all use cases we might implement, or at least consider for the design of the interface. Maybe we need to brainstorm about this at our next clustering conf call ?

On 20 Mar 2008, at 03:46, Brian Stansberry wrote: > Bela Ban wrote: >>>> I suggest you come up with a number of scenarios you'd like to >>>> cover in JBC and then we see what information is needed. Maybe >>>> implementing this will make the interface clearer... >>> >>> As far as JBC is concerned, I think we can only handle 2 scenarios >>> effectively. >>> >>> 1. EvictEntireClusterStateHandler >>> Each instance evicts it's local tree when it gets this. A variation >>> of this could do a remove instead of an evict (to clear local cache >>> loader state as well). Useful if the data exists elsewhere - e.g., >>> when used with Hibernate. >>> >>> 2. FatalStopMergeHandler >>> Logs a fatal error and shuts down the cache when a merge is detected >> These are simple policies, let's try to find a hard one. For example, >> 3. HTTP session replication (Brian): here, we could provide a policy >> which merges the (usually disjoint) data sets. If there are >> conflicts, we could use the number of updates to a session as >> arbiter. Or the time stamp of the last update, or something >> deterministic. > > The following applies to non-FIELD based sessions. Once POJO Cache is > involved its a whole different subject. > > A handler that merged sessions would need to be able to: > > a) Find out where in the tree the root of interesting data is. That is > the /JSESSION/localhost/webapp Fqn. A session cache might have > several such points (different webapps) and not all would necessarily > follow the same structure. e.g. you may also have things like > /SFSB/ear=foo,jar=bar,bean=fubar in the same cache if you are caching > SFSBs as well. With BR this includes the buddy backup subtrees > associated with any "dead" data owners. > > b) Identify what nodes are the "data owners". With total replication > this could just be the former coordinator nodes of the subpartitions. > With BR it's every node. > > c) Provide the names of all the children of the interesting root > across the data owners. > > d) Provide the data from all the data owners for a given Fqn so the > winner can be selected. > > e) Compare the contents of nodes to determine a winner. What attribute > map key holds the timestamp; what method on the sfsb holds the timestamp. > > f) Merge a given subtree once a winner has been picked for a given > Fqn. Get the winner data stored in the right place (which buddy > groups) and the loser data removed. So in terms of what you'd need from JBC's merge handler callback: 1) List of each subpartition's members (in order, so you can determine coordinator) 2) Remote (proxy) access to a given cache. Something like: Cache c = new CacheProxy(currentInstance, addressToProxyTo); so that you could query any caches (coordinators, or all caches in the case of BR) you care about, about their "interesting" roots and children. I'm guessing everything you need in c, d and f can be achieved with proxy access described above?

So I'm now the callback hasn't changed, and still looks like: void handleMerge(Cache currentInstance, List<Address> mergedMembership, List<Address>... subpartitionMemberships); NB: CacheProxy does not exist as yet. It is something we need for Partitioning (JBCACHE-60), so I thought it may make sense here.

Cheers, Manik -- Manik Surtani Lead, JBoss Cache manik(a)jboss.org

Manik Surtani wrote: > 2) Remote (proxy) access to a given cache. Something like: > > Cache c = new CacheProxy(currentInstance, addressToProxyTo); > > so that you could query any caches (coordinators, or all caches > in the case of BR) you care about, about their "interesting" roots > and children. > > I'm guessing everything you need in c, d and f can be achieved with > proxy access described above? > > So I'm now the callback hasn't changed, and still looks like: > > > void handleMerge(Cache currentInstance, List<Address> > mergedMembership, List<Address>... subpartitionMemberships); > > NB: CacheProxy does not exist as yet. It is something we need for > Partitioning (JBCACHE-60), so I thought it may make sense here. Are proxies something we could do before Data Partitioning *in case we decide to do merging first* ?

In what respect are those proxies different from a remote cache loader ? Do you envisage RCLs to use proxies once proxies are in place ?