[jboss-user] [JBoss Cache]

Rick Feldmann work at rf-consulting.com
Mon Jan 25 11:10:21 EST 2010


Everyone,

I have the following configuration and have been unable to have the 
logons work properly.  Once properly logged in, I can terminate app2 on 
either server and it will bounce to the other server.  It will even 
bounce during normal execution between the 2 servers.  What am I missing?

Jboss - 4.0.2
Jboss cache - 1.4.0

app1 is using a custom written sso module and is configured to have it's 
own session cache that is shared with other servers running app1.
app2 is configured to have it's on session cache that is shared with 
other servers running app2.

both app1 and app2 have a shared credential cache.

our shared credential cache configuration file looks like and is the 
same for both applications on all servers in the deploy directory:

<?xml version="1.0" encoding="UTF-8" ?>
<server>
  <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />

  <!--  
====================================================================  -->
  <!--  Defines TreeCache configuration for the Credential 
Cache.             -->
  <!--  
====================================================================  -->


 <mbean code="org.jboss.cache.TreeCache" 
name="jboss.cache:service=BrivoTreeCache">
    <depends>jboss:service=Naming</depends>
    <depends>jboss:service=TransactionManager</depends>


    <!-- Configure the TransactionManager -->
    <!-- org.jboss.cache.DummyTransactionManagerLookup -->
    <attribute 
name="TransactionManagerLookupClass">org.jboss.cache.BatchModeTransactionManagerLookup</attribute>

    <!--
      Node locking scheme :
      PESSIMISTIC (default)
      OPTIMISTIC
    -->
    <attribute name="NodeLockingScheme">PESSIMISTIC</attribute>       

    <!--
      Node locking isolation level :
      SERIALIZABLE
      REPEATABLE_READ (default)
      READ_COMMITTED
      READ_UNCOMMITTED
      NONE
     (ignored if NodeLockingScheme is OPTIMISTIC)
     -->
    <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

    <!--    
      Valid modes are LOCAL
      REPL_ASYNC
      REPL_SYNC
      INVALIDATION_ASYNC
      INVALIDATION_SYNC
    -->
    <attribute name="CacheMode">REPL_SYNC</attribute>
   
    <!-- 
      Whether each interceptor should have an mbean
      registered to capture and display its statistics. 
      <attribute name="UseInterceptorMbeans">true</attribute>
    -->

    <!--
      Name of cluster. Needs to be the same for all clusters,
      in order to find each other
    -->
    <attribute 
name="ClusterName">Credential-${jboss.partition.name:DefaultPartition}</attribute>

    <attribute name="ClusterConfig">
      <config>

          <UDP mcast_addr="${jboss.partition.udpGroup:224.0.0.10}" 
mcast_port="45568"
               ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
               mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
               ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
               enable_bundling="false"
               loopback="false" />
            <PING timeout="2000" num_initial_members="4" />
            <MERGE2 min_interval="10000" max_interval="20000" />
            <FD_SOCK  />
            <FD shun="true" timeout="10000" max_tries="5"/>
            <VERIFY_SUSPECT timeout="3000" num_msgs="3" />
            <pbcast.NAKACK gc_lag="50" 
retransmit_timeout="300,600,1200,2400,4800"
               max_xmit_size="8192" />
            <UNICAST timeout="300,600,1200,2400,4800" window_size="100" 
min_threshold="10" />
            <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000" />
            <FRAG frag_size="8192" />
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
               shun="true" print_local_addr="true"/>
            <pbcast.STATE_TRANSFER />


      </config>
    </attribute>

    <!--   
      The max amount of time (in milliseconds) we wait until the
      initial state (ie. the contents of the cache) are retrieved from
      existing members in a clustered environment
    -->
    <attribute name="InitialStateRetrievalTimeout">5000</attribute>

    <!--   
      Number of milliseconds to wait until all responses for a
      synchronous call have been received.
    -->
    <attribute name="SyncReplTimeout">10000</attribute>

    <!--  Max number of milliseconds to wait for a lock acquisition -->
    <attribute name="LockAcquisitionTimeout">15000</attribute>

  </mbean>
</server>    



the jboss-service.xml file in tc5-cluster-sar/META-INF directory for 
app1 is:

<?xml version="1.0" encoding="UTF-8"?>

<!-- 
===================================================================== -->
<!--                                                                       
-->
<!--  Customized TreeCache Service Configuration for Tomcat 5 
Clustering   -->
<!--                                                                       
-->
<!-- 
===================================================================== -->

<server>

    <!-- 
==================================================================== -->
    <!-- Defines TreeCache 
configuration                                      -->
    <!-- 
==================================================================== -->

     <!-- Note we are using TreeCacheAop -->
    <mbean code="org.jboss.cache.aop.TreeCacheAop"
        name="jboss.cache:service=TomcatClusteringCache">

        <depends>jboss:service=Naming</depends>
        <depends>jboss:service=TransactionManager</depends>
        <!-- We need the AspectDeployer to deploy our FIELD granularity 
aspects -->
        <depends>jboss.aop:service=AspectDeployer</depends>

        <!-- Name of cluster. Needs to be the same for all nodes in the
             cluster, in order to find each other
        -->
        <attribute 
name="ClusterName">APP1-Session-${jboss.partition.name:Cluster}</attribute>
       
        <!--
            Isolation level : SERIALIZABLE
                              REPEATABLE_READ (default)
                              READ_COMMITTED
                              READ_UNCOMMITTED
                              NONE
        -->
        <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

        <!-- Valid modes are LOCAL, REPL_ASYNC and REPL_SYNC
            
             If you use REPL_SYNC and a UDP-based ClusterConfig
             we recommend you comment out the FC (flow control)
             protocol in the ClusterConfig section below.
        -->
        <attribute name="CacheMode">REPL_SYNC</attribute>

        <!--
          Indicates whether to the cache should unmarshall objects 
replicated
          from other cluster nodes, or store them internally as a byte[]
          until a web app requests them.  Must be "true" if session 
replication
          granularity "FIELD" is used in any webapp, otherwise "false" is
          recommended.
        -->
        <attribute name="UseRegionBasedMarshalling">false</attribute>
       
        <!--  Whether or not the entire tree is inactive upon startup, only
          responding to replication messages after activateRegion() is
          called to activate one or more parts of the tree when a webapp is
          deployed.  Must have the same value as 
"UseRegionBasedMarshalling".
        -->
        <attribute name="InactiveOnStartup">false</attribute>
         
        <!--  Make sure to specify BatchModeTransactionManager only! -->
        <attribute 
name="TransactionManagerLookupClass">org.jboss.cache.BatchModeTransactionManagerLookup</attribute>

        <!-- Configures binary format of messages sent between cluster 
nodes.
             Changing this allows a later version of JBoss Cache to 
interoperate
             with an earlier version. You might, for example, change this
             if you are integrating a 4.0.4 server into a cluster with
             servers running an earlier AS version.
            
             Possible values:
            
             1.2.3     JBC 1.2.3 or earlier; bundled with AS 4.0.3.SP1 
and earlier
             1.2.4     JBC 1.2.4
             1.2.4.SP1 JBC 1.2.4.SP1
             1.2.4.SP2 JBC 1.2.4.SP2; bundled with AS 4.0.4
            
             For version 1.3.0.GA and later, use the version name.
       
             If left blank or commented out, JBoss Cache will use the 
default
             for that release (e.g. 1.4.0 for releases in the 1.4.0 series.
             
             The binary format of replication version 1.4.0 is much more 
efficient
             than earlier releases, so there is a significant 
performance penalty
             to trying to interoperate 1.4.0 with earlier releases vs. a 
pure
             1.4.0 cluster.
        -->
       
        <attribute name="ReplicationVersion">1.4.0.GA</attribute>
       
        <!-- JGroups protocol stack properties. Can also be a URL,
             e.g. file:/home/bela/default.xml
        <attribute name="ClusterProperties"></attribute>
        -->

        <attribute name="ClusterConfig">
            <!--
            The default UDP stack:
            - If you have a multihomed machine, set the UDP protocol's 
bind_addr attribute to the
            appropriate NIC IP address, e.g bind_addr="192.168.0.2".
            - On Windows machines, because of the media sense feature 
being broken with multicast
            (even after disabling media sense) set the UDP protocol's 
loopback attribute to true
           
            - If your CacheMode is set to REPL_SYNC we recommend you 
comment
            out the FC (flow control) protocol
            -->
            <config>
                <UDP mcast_addr="${jboss.partition.udpGroup:224.0.0.30}"
                     mcast_port="45577"
                     ucast_recv_buf_size="20000000"
                     ucast_send_buf_size="640000"
                     mcast_recv_buf_size="25000000"
                     mcast_send_buf_size="640000"
                     loopback="false"
                     max_bundle_size="64000"
                     max_bundle_timeout="30"
                     use_incoming_packet_handler="true"
                     use_outgoing_packet_handler="true"
                     ip_ttl="${jgroups.mcast.ip_ttl:2}"
                     enable_bundling="false"/>
                <PING timeout="2000" num_initial_members="3"/>
                <MERGE2 max_interval="100000" min_interval="20000"/>
                <FD_SOCK />
                <FD shun="true" timeout="20000" max_tries="5"/>
                <VERIFY_SUSPECT timeout="1500" />
                <pbcast.NAKACK max_xmit_size="60000"
                               use_mcast_xmit="false" gc_lag="50"
                               retransmit_timeout="300,600,1200,2400,4800"
                               discard_delivered_msgs="true"/>
                <UNICAST timeout="300,600,1200,2400,3600" />
                <pbcast.STABLE stability_delay="1000" 
desired_avg_gossip="50000"
                               max_bytes="400000"/>
                <pbcast.GMS print_local_addr="true" join_timeout="3000"
                            join_retry_timeout="2000" shun="true"/>
              <!--  commented out based on recommendation above about 
using REPL_SYNC instead of REPL_ASYNC <FC max_credits="2000000" 
min_threshold="0.10"/> -->
                <FRAG2 frag_size="60000" />
                <pbcast.STATE_TRANSFER />
           </config>

           <!-- Alternate TCP stack: customize it for your environment, 
change bind_addr and initial_hosts -->
           <!--
           <config>
              <TCP bind_addr="thishost" start_port="7810" loopback="true"
                   tcp_nodelay="false" down_thread="false" 
up_thread="false"/>
              <TCPPING initial_hosts="thishost[7810],otherhost[7810]" 
port_range="3" timeout="3500"
                 num_initial_members="3" up_thread="false" 
down_thread="false"/>
              <MERGE2 min_interval="5000" max_interval="10000"
                 up_thread="false" down_thread="false"/>
              <FD_SOCK down_thread="false" up_thread="false"/>
              <FD shun="true" up_thread="false" down_thread="false"
                 timeout="10000" max_tries="5"/>
              <VERIFY_SUSPECT timeout="1500" down_thread="false" 
up_thread="false" />
              <pbcast.NAKACK down_thread="false" up_thread="false" 
gc_lag="100"
                 retransmit_timeout="3000"/>
              <pbcast.STABLE desired_avg_gossip="20000" 
down_thread="false" up_thread="false" />
              <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" 
shun="true"
                 print_local_addr="true" down_thread="false" 
up_thread="false"/>
              <FC max_credits="2000000" down_thread="false" 
up_thread="false"
                 min_threshold="0.10"/>
              <FRAG2 frag_size="60000" down_thread="false" 
up_thread="false"/>
              <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
           </config>
           -->

        </attribute>

        <!--
            Number of milliseconds to wait until all responses for a
            synchronous call have been received.
        -->
        <attribute name="SyncReplTimeout">20000</attribute>

        <!-- Max number of milliseconds to wait for a lock acquisition -->
        <attribute name="LockAcquisitionTimeout">15000</attribute>

        <!-- Buddy Replication config.
       
             See 
http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheBuddyReplicationDesign
             and the JBoss Cache docs for more on buddy replication.
            
             By default, buddy replication is disabled.
            
             Following are the configuration elements likely to be changed:
            
             buddyReplicationEnabled  true if you want buddy 
replication; false if data
                                      should be replicated to all nodes 
in the cluster
                                     
             numBuddies               to how many backup nodes should 
each node replicate
                                      its state
                                     
             buddyPoolName            allows logical subgrouping of 
nodes within the cluster;
                                      if possible, buddies will be 
chosen from nodes in the
                                      same buddy pool
                                     
             Do not change the data gravitation related 
options.            
        -->
        <attribute name="BuddyReplicationConfig">
            <config>
                <buddyReplicationEnabled>false</buddyReplicationEnabled>
                
<buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
                <buddyLocatorProperties>
                    numBuddies = 1
                    ignoreColocatedBuddies = true
                </buddyLocatorProperties>

                <buddyPoolName>default</buddyPoolName>
                <buddyCommunicationTimeout>2000</buddyCommunicationTimeout>

                <autoDataGravitation>false</autoDataGravitation>
                
<dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>
                
<dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>

            </config>
        </attribute>
          
    </mbean>

</server>




the jboss-service.xml file in tc5-cluster-sar/META-INF directory for 
app2 is:

<?xml version="1.0" encoding="UTF-8"?>

<!-- 
===================================================================== -->
<!--                                                                       
-->
<!--  Customized TreeCache Service Configuration for Tomcat 5 
Clustering   -->
<!--                                                                       
-->
<!-- 
===================================================================== -->

<server>

    <!-- 
==================================================================== -->
    <!-- Defines TreeCache 
configuration                                      -->
    <!-- 
==================================================================== -->

     <!-- Note we are using TreeCacheAop -->
    <mbean code="org.jboss.cache.aop.TreeCacheAop"
        name="jboss.cache:service=TomcatClusteringCache">

        <depends>jboss:service=Naming</depends>
        <depends>jboss:service=TransactionManager</depends>
        <!-- We need the AspectDeployer to deploy our FIELD granularity 
aspects -->
        <depends>jboss.aop:service=AspectDeployer</depends>

        <!-- Name of cluster. Needs to be the same for all nodes in the
             cluster, in order to find each other
        -->
        <attribute 
name="ClusterName">APP2-Session-${jboss.partition.name:Cluster}</attribute>
       
        <!--
            Isolation level : SERIALIZABLE
                              REPEATABLE_READ (default)
                              READ_COMMITTED
                              READ_UNCOMMITTED
                              NONE
        -->
        <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

        <!-- Valid modes are LOCAL, REPL_ASYNC and REPL_SYNC
            
             If you use REPL_SYNC and a UDP-based ClusterConfig
             we recommend you comment out the FC (flow control)
             protocol in the ClusterConfig section below.
        -->
        <attribute name="CacheMode">REPL_SYNC</attribute>

        <!--
          Indicates whether to the cache should unmarshall objects 
replicated
          from other cluster nodes, or store them internally as a byte[]
          until a web app requests them.  Must be "true" if session 
replication
          granularity "FIELD" is used in any webapp, otherwise "false" is
          recommended.
        -->
        <attribute name="UseRegionBasedMarshalling">false</attribute>
       
        <!--  Whether or not the entire tree is inactive upon startup, only
          responding to replication messages after activateRegion() is
          called to activate one or more parts of the tree when a webapp is
          deployed.  Must have the same value as 
"UseRegionBasedMarshalling".
        -->
        <attribute name="InactiveOnStartup">false</attribute>
         
        <!--  Make sure to specify BatchModeTransactionManager only! -->
        <attribute 
name="TransactionManagerLookupClass">org.jboss.cache.BatchModeTransactionManagerLookup</attribute>

        <!-- Configures binary format of messages sent between cluster 
nodes.
             Changing this allows a later version of JBoss Cache to 
interoperate
             with an earlier version. You might, for example, change this
             if you are integrating a 4.0.4 server into a cluster with
             servers running an earlier AS version.
            
             Possible values:
            
             1.2.3     JBC 1.2.3 or earlier; bundled with AS 4.0.3.SP1 
and earlier
             1.2.4     JBC 1.2.4
             1.2.4.SP1 JBC 1.2.4.SP1
             1.2.4.SP2 JBC 1.2.4.SP2; bundled with AS 4.0.4
            
             For version 1.3.0.GA and later, use the version name.
       
             If left blank or commented out, JBoss Cache will use the 
default
             for that release (e.g. 1.4.0 for releases in the 1.4.0 series.
             
             The binary format of replication version 1.4.0 is much more 
efficient
             than earlier releases, so there is a significant 
performance penalty
             to trying to interoperate 1.4.0 with earlier releases vs. a 
pure
             1.4.0 cluster.
        -->
       
        <attribute name="ReplicationVersion">1.4.0.GA</attribute>
       
        <!-- JGroups protocol stack properties. Can also be a URL,
             e.g. file:/home/bela/default.xml
        <attribute name="ClusterProperties"></attribute>
        -->

        <attribute name="ClusterConfig">
            <!--
            The default UDP stack:
            - If you have a multihomed machine, set the UDP protocol's 
bind_addr attribute to the
            appropriate NIC IP address, e.g bind_addr="192.168.0.2".
            - On Windows machines, because of the media sense feature 
being broken with multicast
            (even after disabling media sense) set the UDP protocol's 
loopback attribute to true
           
            - If your CacheMode is set to REPL_SYNC we recommend you 
comment
            out the FC (flow control) protocol
            -->
            <config>
                <UDP mcast_addr="${jboss.partition.udpGroup:224.0.0.20}"
                     mcast_port="45577"
                     ucast_recv_buf_size="20000000"
                     ucast_send_buf_size="640000"
                     mcast_recv_buf_size="25000000"
                     mcast_send_buf_size="640000"
                     loopback="false"
                     max_bundle_size="64000"
                     max_bundle_timeout="30"
                     use_incoming_packet_handler="true"
                     use_outgoing_packet_handler="true"
                     ip_ttl="${jgroups.mcast.ip_ttl:2}"
                     enable_bundling="false"/>
                <PING timeout="2000" num_initial_members="3"/>
                <MERGE2 max_interval="100000" min_interval="20000"/>
                <FD_SOCK />
                <FD shun="true" timeout="20000" max_tries="5"/>
                <VERIFY_SUSPECT timeout="1500" />
                <pbcast.NAKACK max_xmit_size="60000"
                               use_mcast_xmit="false" gc_lag="50"
                               retransmit_timeout="300,600,1200,2400,4800"
                               discard_delivered_msgs="true"/>
                <UNICAST timeout="300,600,1200,2400,3600" />
                <pbcast.STABLE stability_delay="1000" 
desired_avg_gossip="50000"
                               max_bytes="400000"/>
                <pbcast.GMS print_local_addr="true" join_timeout="3000"
                            join_retry_timeout="2000" shun="true"/>
              <!--  commented out based on recommendation above about 
using REPL_SYNC instead of REPL_ASYNC <FC max_credits="2000000" 
min_threshold="0.10"/> -->
                <FRAG2 frag_size="60000" />
                <pbcast.STATE_TRANSFER />
           </config>

           <!-- Alternate TCP stack: customize it for your environment, 
change bind_addr and initial_hosts -->
           <!--
           <config>
              <TCP bind_addr="thishost" start_port="7810" loopback="true"
                   tcp_nodelay="false" down_thread="false" 
up_thread="false"/>
              <TCPPING initial_hosts="thishost[7810],otherhost[7810]" 
port_range="3" timeout="3500"
                 num_initial_members="3" up_thread="false" 
down_thread="false"/>
              <MERGE2 min_interval="5000" max_interval="10000"
                 up_thread="false" down_thread="false"/>
              <FD_SOCK down_thread="false" up_thread="false"/>
              <FD shun="true" up_thread="false" down_thread="false"
                 timeout="10000" max_tries="5"/>
              <VERIFY_SUSPECT timeout="1500" down_thread="false" 
up_thread="false" />
              <pbcast.NAKACK down_thread="false" up_thread="false" 
gc_lag="100"
                 retransmit_timeout="3000"/>
              <pbcast.STABLE desired_avg_gossip="20000" 
down_thread="false" up_thread="false" />
              <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" 
shun="true"
                 print_local_addr="true" down_thread="false" 
up_thread="false"/>
              <FC max_credits="2000000" down_thread="false" 
up_thread="false"
                 min_threshold="0.10"/>
              <FRAG2 frag_size="60000" down_thread="false" 
up_thread="false"/>
              <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
           </config>
           -->

        </attribute>

        <!--
            Number of milliseconds to wait until all responses for a
            synchronous call have been received.
        -->
        <attribute name="SyncReplTimeout">20000</attribute>

        <!-- Max number of milliseconds to wait for a lock acquisition -->
        <attribute name="LockAcquisitionTimeout">15000</attribute>

        <!-- Buddy Replication config.
       
             See 
http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossCacheBuddyReplicationDesign
             and the JBoss Cache docs for more on buddy replication.
            
             By default, buddy replication is disabled.
            
             Following are the configuration elements likely to be changed:
            
             buddyReplicationEnabled  true if you want buddy 
replication; false if data
                                      should be replicated to all nodes 
in the cluster
                                     
             numBuddies               to how many backup nodes should 
each node replicate
                                      its state
                                     
             buddyPoolName            allows logical subgrouping of 
nodes within the cluster;
                                      if possible, buddies will be 
chosen from nodes in the
                                      same buddy pool
                                     
             Do not change the data gravitation related 
options.            
        -->
        <attribute name="BuddyReplicationConfig">
            <config>
                <buddyReplicationEnabled>false</buddyReplicationEnabled>
                
<buddyLocatorClass>org.jboss.cache.buddyreplication.NextMemberBuddyLocator</buddyLocatorClass>
                <buddyLocatorProperties>
                    numBuddies = 1
                    ignoreColocatedBuddies = true
                </buddyLocatorProperties>

                <buddyPoolName>default</buddyPoolName>
                <buddyCommunicationTimeout>2000</buddyCommunicationTimeout>

                <autoDataGravitation>false</autoDataGravitation>
                
<dataGravitationRemoveOnFind>true</dataGravitationRemoveOnFind>
                
<dataGravitationSearchBackupTrees>true</dataGravitationSearchBackupTrees>

            </config>
        </attribute>
          
    </mbean>

</server>

the server.xml files for both apps contains the following valve:

        <Valve 
className="org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn" 
cacheConfig="BMC-Session-${jboss.partition.name:Cluster}"/>


We are behind physical load balancers performing round robin load balancing.

The sequence of events is as follow:

User opens browers and goes to url for app1.

app1 displays the login page and the user hits the submit button after 
entering a valid user id and password

app1 validates the user id and password

app1 logs into app2

app2 validates the user id and password

app2 returns successful login to app1

app1 returns a redirect (302) with the url of app2 to the browser

the browser then connects to app2.

if the browser hits app2 on the same server that app1 used for the 
login, the user is successfuly logged into app2 and shown the home page 
for app2.  the user can then move around app2 without any knowledge of 
what server they are on and app2 can be stopped on either server and the 
user does not encounter any dropped connections.

if the browser hits app2 on a different server than app1 used for the 
login, the user is returned back to the app1 login page.

am I missing something in the configuration files?
yes, this is older release of jboss, but we are unable to upgrade the to 
a newer version at this time.

Thanks In Advance,

Rick Feldmann
work at rf-consulting.com



More information about the jboss-user mailing list