[infinispan-dev] Cross-site clustering

[Bela Ban]

Hi Mircea,

I'm in the process of setting up the Infinispan GUI demo across 3 data 
centers, and I ran into an issue with the Infinispan config file. Not 
really a bug, but something that could be improved.

Currently, the Infinispan config (let's call it infinispan.xml) defines 
the sites and their backup sites and refers to the local JGroups cluster 
config file (jgroups.xml). The latter has a RELAY2 protocol at the top 
of its stack which points to the relay configuration (relay2.xml), which 
defines the sites and how they connect to each other. Finally, there's a 
jgroups-relay2.xml file (referred to from relay2.xml) which defines the 
global bridge, connecting all 3 sites.

OK, so that 4 config files and I don't see how to reduce them. 
However... currently infinispan.xml is *site-specific*, ie. we need 1 
infinispan.xml for LON, 1 for SFO and 1 for NYC.

The 3 JGroups config files (jgroups.xml, relay2.xml and 
relay2-bridge.xml) are *not* site-specific, ie. they can be used in any 
site, provided that we set a few system properties, such as 
cluster-specific mcast address and port, and site name (relay2.xml).

I believe cross-site clustering would benefit from making infinispan.xml 
symmetric as well, that is, we could use it in any site.

To do this, I suggest the following:

#1 Sites config:
  <sites local="LON">
             <site name="SFO"/>
             <site name="NYC"/>
             <site name="LON"/>
         </sites>


Nothing needs to be changed here, except to parameterize local: <sites 
local="${SITE:LON}"...>. A node in a given site can then be started 
using -DSITE=LON. This could also be used for relay2.xml.

- Question-1: why do we need to list the sites above ? We're not 
defining any config for those sites, so I don't see why this is needed...

#2 Backups config:
  <namedCache name="users">
         <sites>
             <backups>
                 <backup site="NYC" backupFailurePolicy="WARN" 
strategy="SYNC" timeout="12000"/>
                 <backup site="SFO" backupFailurePolicy="IGNORE" 
strategy="ASYNC"/>
             </backups>
         </sites>
     </namedCache>


This is *asymmetric*. IMO, a better config would be:

<global>
   <sites>
        <localSite="${site:LON}" backupSites="${backup-sites:SFO,NYC}" />
        <backups>
                 <backup site="NYC" backupFailurePolicy="WARN" 
strategy="SYNC" timeout="12000"/>
                 <backup site="SFO" backupFailurePolicy="IGNORE" 
strategy="ASYNC"/>
                 <backup site="LON" backupFailurePolicy="IGNORE" 
strategy="ASYNC"/>
        </backups>
    </sites>
</global>


Here, we define global backup strategies and the local site. Both of 
them can be parameterized, e.g. we start up nodes in
- LON: -Dsite=LON -DbackupSites="SFO,NYC"
- SFO: -Dsite=SFO -DbackupSites=NYC
- NYC: -Dsite=NYC -DbackupSites=     // no backups


This would allow us to have only 1 infinispan.xml, regardless of the 
site in which it is used.

Of course, since backup strageties are globally defined, if people 
wanted different backup strategies per site (or per named cache), they 
could still copy infinispan.xml and modify it...

WDYT ?




-- 
Bela Ban, JGroups lead (http://www.jgroups.org)