Thanks, Manik. I'll be trying to produce a unit test. I'm also attaching a copy of my servers' cache configuration file if it is useful for debugging. After all, it could be just a matter of misconfiguration. 

Smaller server's cache configuration. It uses a ClusteredCacheLoader to fetch cache states from far memory (the bigger server) if it doesn't find it in local cache:
<?xml version="1.0" encoding="UTF-8"?>

<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:jboss:jbosscache-core:config:3.0">
    <!--
       isolation levels supported: READ_COMMITTED and REPEATABLE_READ
       nodeLockingSchemes: mvcc, pessimistic (deprecated), optimistic (deprecated)
    -->
    <locking
            isolationLevel="READ_COMMITTED"
            lockParentForChildInsertRemove="false"
            lockAcquisitionTimeout="20000"
            nodeLockingScheme="mvcc"
            writeSkewCheck="false"
            concurrencyLevel="500"/>

    <!--
    Used to register a transaction manager and participate in ongoing transactions.
    -->
    <transaction syncRollbackPhase="false" syncCommitPhase="false"/>

    <!--
    Used to register JMX statistics in any available MBean server
    -->
    <jmxStatistics
            enabled="true"/>

    <!--
       If region based marshalling is used, defines whether new regions are inactive on startup.
    -->
    <startup
            regionsInactiveOnStartup="false"/>

    <!--
       Used to register JVM shutdown hooks.
       hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
    -->
    <shutdown
            hookBehavior="DEFAULT"/>

    <!--
       Used to define async listener notification thread pool size
    -->
    <listeners
            asyncPoolSize="1"
            asyncQueueSize="1000000"/>

    <!--
       Used to enable invocation batching and allow the use of Cache.startBatch()/endBatch() methods.
    -->
    <invocationBatching
            enabled="false"/>

    <!--
       serialization related configuration, used for replication and cache loading
    -->
    <serialization
            objectInputStreamPoolSize="12"
            objectOutputStreamPoolSize="14"
            version="3.0.0"
            useLazyDeserialization="false"
            useRegionBasedMarshalling="false"/>

    <!--
       This element specifies that the cache is clustered.
       modes supported: replication (r) or invalidation (i).
    -->
    <clustering mode="replication" clusterName="dsat-cluster-2-cache">

        <!--
           Defines whether to retrieve state on startup
        -->
        <stateRetrieval timeout="180000" fetchInMemoryState="false"/>
   
    <!--
           Network calls are synchronous. Uncomment this the comment out the async part to 
           return to synchronous replication
        
        <sync replTimeout="20000"/> 
       --> 
        <!--
           Uncomment this for async replication.
        -->
        <async useReplQueue="true" replQueueInterval="10000" replQueueMaxElements="500" serializationExecutorPoolSize="0" serializationExecutorQueueSize="5000000"/>

    
        <!--
           Configures the JGroups channel.  Looks up a JGroups config file on the classpath or filesystem.  udp.xml
           ships with jgroups.jar and will be picked up by the class loader.
        -->
        <jgroupsConfig>
            <UDP discard_incompatible_packets="true" enable_bundling="true" enable_diagnostics="true" ip_ttl="32" ip_mcast="true"
                 loopback="false" max_bundle_size="64000" max_bundle_timeout="30" mcast_addr="228.10.10.11" bind_addr="10.210.102.101" 
                 mcast_port="45588" mcast_recv_buf_size="25000000" mcast_send_buf_size="640000"
                 oob_thread_pool.enabled="true" oob_thread_pool.keep_alive_time="10000" oob_thread_pool.max_threads="4"
                 oob_thread_pool.min_threads="1" oob_thread_pool.queue_enabled="true"
                 oob_thread_pool.queue_max_size="10"
                 oob_thread_pool.rejection_policy="Run" thread_naming_pattern="pl" thread_pool.enabled="true"
                 thread_pool.keep_alive_time="30000" thread_pool.max_threads="25" thread_pool.min_threads="1"
                 thread_pool.queue_enabled="true" thread_pool.queue_max_size="10" thread_pool.rejection_policy="Run"
                 tos="8" ucast_recv_buf_size="20000000" ucast_send_buf_size="640000" use_concurrent_stack="true"
                 use_incoming_packet_handler="true"/>
            <PING num_initial_members="3" timeout="2000"/>
            <MERGE2 max_interval="30000" min_interval="10000"/>
            <FD_SOCK/>
            <FD max_tries="5" shun="true" timeout="10000"/>
            <VERIFY_SUSPECT timeout="1500"/>
            <pbcast.NAKACK discard_delivered_msgs="true" gc_lag="0" retransmit_timeout="300,600,1200,2400,4800"
                           use_mcast_xmit="false"/>
            <UNICAST timeout="300,600,1200,2400,3600"/>
            <pbcast.STABLE desired_avg_gossip="50000" max_bytes="400000" stability_delay="1000"/>
            <pbcast.GMS join_timeout="5000" print_local_addr="true" shun="false" view_ack_collection_timeout="5000"
                        view_bundling="true"/>
            <FRAG2 frag_size="60000"/>
            <pbcast.STREAMING_STATE_TRANSFER/>
            <pbcast.FLUSH timeout="0"/>

        </jgroupsConfig>
    </clustering>

    <!--
       Eviction configuration.  WakeupInterval defines how often the eviction thread runs, in milliseconds.  0 means
       the eviction thread will never run.
    -->
    <eviction wakeUpInterval="500">
        <default algorithmClass="org.jboss.cache.eviction.LRUAlgorithm" eventQueueSize="200000">
        </default>
<region name="/">
<property name="timeToLive" value="-1"/>
<property name="maxNodes" value="-1"/>
</region>
        <region name="/CONTENT_MATCH_OPTIMIZER">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
        </region>
        <region name="/CONTENT_MATCH_PROCESSOR" eventQueueSize="100000">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
        </region>
         <region name="/DEFAULT_NODE" eventQueueSize="100000">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
        </region> 
    </eviction>

    <!--
       Cache loaders.

       If passivation is enabled, state is offloaded to the cache loaders ONLY when evicted.  Similarly, when the state
       is accessed again, it is removed from the cache loader and loaded into memory.

       Otherwise, state is always maintained in the cache loader as well as in memory.

       Set 'shared' to true if all instances in the cluster use the same cache loader instance, e.g., are talking to the
       same database.
    -->
    <loaders passivation="false" shared="true">
<loader class="org.jboss.cache.loader.ClusteredCacheLoader">
<properties>
timeout=1500
</properties>
</loader>
    </loaders>

</jbosscache>

Bigger server's cache configuration. It uses a JDBC cache loader to persist cache states:

<?xml version="1.0" encoding="UTF-8"?>

<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:jboss:jbosscache-core:config:3.0">
    <!--
       isolation levels supported: READ_COMMITTED and REPEATABLE_READ
       nodeLockingSchemes: mvcc, pessimistic (deprecated), optimistic (deprecated)
    -->
    <locking
            isolationLevel="READ_COMMITTED"
            lockParentForChildInsertRemove="false"
            lockAcquisitionTimeout="20000"
            nodeLockingScheme="mvcc"
            writeSkewCheck="false"
            concurrencyLevel="500"/>

    <!--
    Used to register a transaction manager and participate in ongoing transactions.
    -->
    <transaction syncRollbackPhase="false" syncCommitPhase="false"/>

    <!--
    Used to register JMX statistics in any available MBean server
    -->
    <jmxStatistics
            enabled="true"/>

    <!--
       If region based marshalling is used, defines whether new regions are inactive on startup.
    -->
    <startup
            regionsInactiveOnStartup="false"/>

    <!--
       Used to register JVM shutdown hooks.
       hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
    -->
    <shutdown
            hookBehavior="DEFAULT"/>

    <!--
       Used to define async listener notification thread pool size
    -->
    <listeners
            asyncPoolSize="1"
            asyncQueueSize="1000000"/>

    <!--
       Used to enable invocation batching and allow the use of Cache.startBatch()/endBatch() methods.
    -->
    <invocationBatching
            enabled="false"/>

    <!--
       serialization related configuration, used for replication and cache loading
    -->
    <serialization
            objectInputStreamPoolSize="12"
            objectOutputStreamPoolSize="14"
            version="3.0.0"
            useLazyDeserialization="false"
            useRegionBasedMarshalling="false"/>

    <!--
       This element specifies that the cache is clustered.
       modes supported: replication (r) or invalidation (i).
    -->
    <clustering mode="replication" clusterName="JBossCache-cluster">

        <!--
           Defines whether to retrieve state on startup
        -->
        <stateRetrieval timeout="180000" fetchInMemoryState="false"/>

        <!--
           Network calls are synchronous. Uncomment this the comment out the async part to 
           return to synchronous replication
        -->
        <!-- <sync replTimeout="20000"/> -->
        
        <!--
           Uncomment this for async replication.
        -->
        <async useReplQueue="true" replQueueInterval="10000" replQueueMaxElements="500" serializationExecutorPoolSize="20" serializationExecutorQueueSize="5000000"/>

        <!--
           Configures the JGroups channel.  Looks up a JGroups config file on the classpath or filesystem.  udp.xml
           ships with jgroups.jar and will be picked up by the class loader.
        -->
        <jgroupsConfig>
            <UDP discard_incompatible_packets="true" enable_bundling="true" enable_diagnostics="true" ip_ttl="32" ip_mcast="true"
                 loopback="false" max_bundle_size="64000" max_bundle_timeout="30" mcast_addr="228.10.10.10" bind_addr="10.210.102.100" 
                 mcast_port="45588" mcast_recv_buf_size="25000000" mcast_send_buf_size="640000"
                 oob_thread_pool.enabled="true" oob_thread_pool.keep_alive_time="10000" oob_thread_pool.max_threads="4"
                 oob_thread_pool.min_threads="1" oob_thread_pool.queue_enabled="true"
                 oob_thread_pool.queue_max_size="10"
                 oob_thread_pool.rejection_policy="Run" thread_naming_pattern="pl" thread_pool.enabled="true"
                 thread_pool.keep_alive_time="30000" thread_pool.max_threads="25" thread_pool.min_threads="1"
                 thread_pool.queue_enabled="true" thread_pool.queue_max_size="10" thread_pool.rejection_policy="Run"
                 tos="8" ucast_recv_buf_size="20000000" ucast_send_buf_size="640000" use_concurrent_stack="true"
                 use_incoming_packet_handler="true"/>
            <PING num_initial_members="3" timeout="8000"/>
            <MERGE2 max_interval="30000" min_interval="10000"/>
            <FD_SOCK/>
            <FD max_tries="5" shun="true" timeout="10000"/>
            <VERIFY_SUSPECT timeout="1500"/>
            <pbcast.NAKACK discard_delivered_msgs="true" gc_lag="0" retransmit_timeout="300,600,1200,2400,4800"
                           use_mcast_xmit="false"/>
            <UNICAST timeout="300,600,1200,2400,3600"/>
            <pbcast.STABLE desired_avg_gossip="50000" max_bytes="400000" stability_delay="1000"/>
            <pbcast.GMS join_timeout="50000" print_local_addr="true" shun="false" view_ack_collection_timeout="50000"
                        view_bundling="true"/>
            <FRAG2 frag_size="60000"/>
            <pbcast.STREAMING_STATE_TRANSFER/>
            <pbcast.FLUSH timeout="0"/>

        </jgroupsConfig>
    </clustering>

    <!--
       Eviction configuration.  WakeupInterval defines how often the eviction thread runs, in milliseconds.  0 means
       the eviction thread will never run.
    -->
    <eviction wakeUpInterval="500">
        <default algorithmClass="org.jboss.cache.eviction.LRUAlgorithm" eventQueueSize="200000">
        </default>
        <region name="/CONTENT_MATCH_OPTIMIZER">
        </region>
        <region name="/CONTENT_MATCH_PROCESSOR" eventQueueSize="100000">
        </region>
         <region name="/DEFAULT_NODE" eventQueueSize="100000">
        </region> 
    </eviction>

    <!--
       Cache loaders.

       If passivation is enabled, state is offloaded to the cache loaders ONLY when evicted.  Similarly, when the state
       is accessed again, it is removed from the cache loader and loaded into memory.

       Otherwise, state is always maintained in the cache loader as well as in memory.

       Set 'shared' to true if all instances in the cluster use the same cache loader instance, e.g., are talking to the
       same database.
    -->
    <loaders passivation="false" shared="true">
        <preload>
            <node fqn="/org/jboss"/>
            <node fqn="/org/tempdata"/>
        </preload>

        <!--
           we can have multiple cache loaders, which get chained
        -->
        <loader class="org.jboss.cache.loader.JDBCCacheLoader" async="false" fetchPersistentState="false"
                ignoreModifications="false" purgeOnStartup="false">

            <properties>
             cache.jdbc.table.name=dsatcache
cache.jdbc.table.create=true
cache.jdbc.table.drop=false
cache.jdbc.table.primarykey=dsatcache_pk
cache.jdbc.fqn.column=fqn
cache.jdbc.fqn.type=varchar(512)
cache.jdbc.node.column=node
cache.jdbc.node.type=LONGBLOB
cache.jdbc.parent.column=parent
cache.jdbc.driver=com.mysql.jdbc.Driver
cache.jdbc.url=jdbc:mysql://1x.xxx.xxx.xxx:9213/cache
cache.jdbc.connection.factory=org.jboss.cache.loader.C3p0ConnectionFactory
cache.jdbc.user=demo
cache.jdbc.password=c1b3rd3m0
            </properties>
            <!-- alternatively use a connection from a data source, as per the code sample below-->
            <!--<properties>-->
            <!--cache.jdbc.datasource=AllSampleDS-->
            <!--cache.jdbc.table.name=jbosscache-->
            <!--cache.jdbc.table.create=true-->
            <!--cache.jdbc.table.drop=true-->
            <!--</properties>-->
            <singletonStore enabled="false" class="org.jboss.cache.loader.SingletonStoreCacheLoader">
                <properties>
                    pushStateWhenCoordinator=true
                    pushStateWhenCoordinatorTimeout=20000
                </properties>
            </singletonStore>
        </loader>
    </loaders>
</jbosscache>


On Thu, Apr 23, 2009 at 2:51 AM, Manik Surtani <manik@jboss.org> wrote:
Do you have a unit test for this?

On 23 Apr 2009, at 03:53, Yong Yuan wrote:

Hi, 

I have two servers that use JBoss Cache 3.0.3. Each server is configured to replicate its states to the other. The bigger server has more memory and thus contains most of the cache states, while the smaller server uses a ClusteredCacheLoader to get cached states if it can't find them in its own cache instance: 
   
         Smaller server -> a cache instance
              ||
              ||   using ClusteredCacheLoader to get cached state   
            \\ //
         Bigger server -> a cache instance
   

The problem is, the ClusteredCacheLoader never retrieved any cached state. Turning on trace, I found that ClusteredCacheLoader always returned null if cache mode was asynchronous replication and there was at least one serialization executor. Therefore, I set serializationExecutorPoolSize to 0. This time, ClusteredCacheLoader did attempt to send out remote request, but it ended up getting a replication exception in the response, as the following log shows: 

(CommandAwareRpcDispatcher.java:291) 2009-04-22 19:47:50,151 TRACE responses: [sender=10.210.102.104:45592, retval=[true, false], received=true, suspected=false]
[sender=10.210.102.104:37132, retval=null, received=false, suspected=false]

The bigger server didn't have any exception logged, though. 

Any suggestion how I should solve this problem? Thanks a lot!
_______________________________________________
jbosscache-users mailing list
jbosscache-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-users

--
Manik Surtani
Lead, Infinispan
Lead, JBoss Cache