4:53 a.m.
Hi,
I have two servers that use JBoss Cache 3.0.3. Each server is configured to
replicate its states to the other. The bigger server has more memory and
thus contains most of the cache states, while the smaller server uses a
ClusteredCacheLoader to get cached states if it can't find them in its own
cache instance:
Smaller server -> a cache instance
||
|| using ClusteredCacheLoader to get cached state
\\ //
Bigger server -> a cache instance
The problem is, the ClusteredCacheLoader never retrieved any cached state.
Turning on trace, I found that ClusteredCacheLoader always returned null if
cache mode was asynchronous replication and there was at least one
serialization executor. Therefore, I set serializationExecutorPoolSize to 0.
This time, ClusteredCacheLoader did attempt to send out remote request, but
it ended up getting a replication exception in the response, as the
following log shows:
(CommandAwareRpcDispatcher.java:291) 2009-04-22 19:47:50,151 TRACE
responses: [sender=10.210.102.104:45592, retval=[true, false],
received=true, suspected=false]
[sender=10.210.102.104:37132, retval=null, received=false, suspected=false]
The bigger server didn't have any exception logged, though.
Any suggestion how I should solve this problem? Thanks a lot!
11:51 a.m.
Do you have a unit test for this?
On 23 Apr 2009, at 03:53, Yong Yuan wrote:
Hi,
I have two servers that use JBoss Cache 3.0.3. Each server is
configured to replicate its states to the other. The bigger server
has more memory and thus contains most of the cache states, while
the smaller server uses a ClusteredCacheLoader to get cached states
if it can't find them in its own cache instance:
Smaller server -> a cache instance
||
|| using ClusteredCacheLoader to get cached state
\\ //
Bigger server -> a cache instance
The problem is, the ClusteredCacheLoader never retrieved any cached
state. Turning on trace, I found that ClusteredCacheLoader always
returned null if cache mode was asynchronous replication and there
was at least one serialization executor. Therefore, I set
serializationExecutorPoolSize to 0. This time, ClusteredCacheLoader
did attempt to send out remote request, but it ended up getting a
replication exception in the response, as the following log shows:
(CommandAwareRpcDispatcher.java:291) 2009-04-22 19:47:50,151 TRACE
responses: [sender=10.210.102.104:45592, retval=[true, false],
received=true, suspected=false]
[sender=10.210.102.104:37132, retval=null, received=false,
suspected=false]
The bigger server didn't have any exception logged, though.
Any suggestion how I should solve this problem? Thanks a lot!
_______________________________________________
jbosscache-users mailing list
jbosscache-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-users
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
8:37 p.m.
New subject: ClusteredCacheLoader doesn't retrieve cached state after server is restarted
Thanks, Manik. I'll be trying to produce a unit test. I'm also attaching a
copy of my servers' cache configuration file if it is useful for debugging.
After all, it could be just a matter of misconfiguration.
Smaller server's cache configuration. It uses a ClusteredCacheLoader to
fetch cache states from far memory (the bigger server) if it doesn't find it
in local cache:
<?xml version="1.0" encoding="UTF-8"?>
<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:jboss:jbosscache-core:config:3.0">
<!--
isolation levels supported: READ_COMMITTED and REPEATABLE_READ
nodeLockingSchemes: mvcc, pessimistic (deprecated), optimistic
(deprecated)
-->
<locking
isolationLevel="READ_COMMITTED"
lockParentForChildInsertRemove="false"
lockAcquisitionTimeout="20000"
nodeLockingScheme="mvcc"
writeSkewCheck="false"
concurrencyLevel="500"/>
<!--
Used to register a transaction manager and participate in ongoing
transactions.
-->
<transaction syncRollbackPhase="false"
syncCommitPhase="false"/>
<!--
Used to register JMX statistics in any available MBean server
-->
<jmxStatistics
enabled="true"/>
<!--
If region based marshalling is used, defines whether new regions are
inactive on startup.
-->
<startup
regionsInactiveOnStartup="false"/>
<!--
Used to register JVM shutdown hooks.
hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
-->
<shutdown
hookBehavior="DEFAULT"/>
<!--
Used to define async listener notification thread pool size
-->
<listeners
asyncPoolSize="1"
asyncQueueSize="1000000"/>
<!--
Used to enable invocation batching and allow the use of
Cache.startBatch()/endBatch() methods.
-->
<invocationBatching
enabled="false"/>
<!--
serialization related configuration, used for replication and cache
loading
-->
<serialization
objectInputStreamPoolSize="12"
objectOutputStreamPoolSize="14"
version="3.0.0"
useLazyDeserialization="false"
useRegionBasedMarshalling="false"/>
<!--
This element specifies that the cache is clustered.
modes supported: replication (r) or invalidation (i).
-->
<clustering mode="replication"
clusterName="dsat-cluster-2-cache">
<!--
Defines whether to retrieve state on startup
-->
<stateRetrieval timeout="180000"
fetchInMemoryState="false"/>
<!--
Network calls are synchronous. Uncomment this the comment out the
async part to
return to synchronous replication
<sync replTimeout="20000"/>
-->
<!--
Uncomment this for async replication.
-->
<async useReplQueue="true" replQueueInterval="10000"
replQueueMaxElements="500" serializationExecutorPoolSize="0"
serializationExecutorQueueSize="5000000"/>
<!--
Configures the JGroups channel. Looks up a JGroups config file
on the classpath or filesystem. udp.xml
ships with jgroups.jar and will be picked up by the class loader.
-->
<jgroupsConfig>
<UDP discard_incompatible_packets="true"
enable_bundling="true"
enable_diagnostics="true" ip_ttl="32" ip_mcast="true"
loopback="false" max_bundle_size="64000"
max_bundle_timeout="30" mcast_addr="228.10.10.11"
bind_addr="10.210.102.101"
mcast_port="45588" mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
oob_thread_pool.enabled="true"
oob_thread_pool.keep_alive_time="10000"
oob_thread_pool.max_threads="4"
oob_thread_pool.min_threads="1"
oob_thread_pool.queue_enabled="true"
oob_thread_pool.queue_max_size="10"
oob_thread_pool.rejection_policy="Run"
thread_naming_pattern="pl" thread_pool.enabled="true"
thread_pool.keep_alive_time="30000"
thread_pool.max_threads="25" thread_pool.min_threads="1"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="10" thread_pool.rejection_policy="Run"
tos="8" ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000" use_concurrent_stack="true"
use_incoming_packet_handler="true"/>
<PING num_initial_members="3" timeout="2000"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD max_tries="5" shun="true"
timeout="10000"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK discard_delivered_msgs="true"
gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
use_mcast_xmit="false"/>
<UNICAST timeout="300,600,1200,2400,3600"/>
<pbcast.STABLE desired_avg_gossip="50000"
max_bytes="400000"
stability_delay="1000"/>
<pbcast.GMS join_timeout="5000"
print_local_addr="true"
shun="false" view_ack_collection_timeout="5000"
view_bundling="true"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER/>
<pbcast.FLUSH timeout="0"/>
</jgroupsConfig>
</clustering>
<!--
Eviction configuration. WakeupInterval defines how often the
eviction thread runs, in milliseconds. 0 means
the eviction thread will never run.
-->
<eviction wakeUpInterval="500">
<default algorithmClass="org.jboss.cache.eviction.LRUAlgorithm"
eventQueueSize="200000">
</default>
<region name="/">
<property name="timeToLive" value="-1"/>
<property name="maxNodes" value="-1"/>
</region>
<region name="/CONTENT_MATCH_OPTIMIZER">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
</region>
<region name="/CONTENT_MATCH_PROCESSOR"
eventQueueSize="100000">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
</region>
<region name="/DEFAULT_NODE" eventQueueSize="100000">
<property name="maxNodes" value="2000000"/>
<property name="timeToLive" value="-1"/>
</region>
</eviction>
<!--
Cache loaders.
If passivation is enabled, state is offloaded to the cache loaders
ONLY when evicted. Similarly, when the state
is accessed again, it is removed from the cache loader and loaded
into memory.
Otherwise, state is always maintained in the cache loader as well as
in memory.
Set 'shared' to true if all instances in the cluster use the same
cache loader instance, e.g., are talking to the
same database.
-->
<loaders passivation="false" shared="true">
<loader class="org.jboss.cache.loader.ClusteredCacheLoader">
<properties>
timeout=1500
</properties>
</loader>
</loaders>
</jbosscache>
Bigger server's cache configuration. It uses a JDBC cache loader to persist
cache states:
<?xml version="1.0" encoding="UTF-8"?>
<jbosscache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:jboss:jbosscache-core:config:3.0">
<!--
isolation levels supported: READ_COMMITTED and REPEATABLE_READ
nodeLockingSchemes: mvcc, pessimistic (deprecated), optimistic
(deprecated)
-->
<locking
isolationLevel="READ_COMMITTED"
lockParentForChildInsertRemove="false"
lockAcquisitionTimeout="20000"
nodeLockingScheme="mvcc"
writeSkewCheck="false"
concurrencyLevel="500"/>
<!--
Used to register a transaction manager and participate in ongoing
transactions.
-->
<transaction syncRollbackPhase="false"
syncCommitPhase="false"/>
<!--
Used to register JMX statistics in any available MBean server
-->
<jmxStatistics
enabled="true"/>
<!--
If region based marshalling is used, defines whether new regions are
inactive on startup.
-->
<startup
regionsInactiveOnStartup="false"/>
<!--
Used to register JVM shutdown hooks.
hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
-->
<shutdown
hookBehavior="DEFAULT"/>
<!--
Used to define async listener notification thread pool size
-->
<listeners
asyncPoolSize="1"
asyncQueueSize="1000000"/>
<!--
Used to enable invocation batching and allow the use of
Cache.startBatch()/endBatch() methods.
-->
<invocationBatching
enabled="false"/>
<!--
serialization related configuration, used for replication and cache
loading
-->
<serialization
objectInputStreamPoolSize="12"
objectOutputStreamPoolSize="14"
version="3.0.0"
useLazyDeserialization="false"
useRegionBasedMarshalling="false"/>
<!--
This element specifies that the cache is clustered.
modes supported: replication (r) or invalidation (i).
-->
<clustering mode="replication"
clusterName="JBossCache-cluster">
<!--
Defines whether to retrieve state on startup
-->
<stateRetrieval timeout="180000"
fetchInMemoryState="false"/>
<!--
Network calls are synchronous. Uncomment this the comment out the
async part to
return to synchronous replication
-->
<!-- <sync replTimeout="20000"/> -->
<!--
Uncomment this for async replication.
-->
<async useReplQueue="true" replQueueInterval="10000"
replQueueMaxElements="500" serializationExecutorPoolSize="20"
serializationExecutorQueueSize="5000000"/>
<!--
Configures the JGroups channel. Looks up a JGroups config file
on the classpath or filesystem. udp.xml
ships with jgroups.jar and will be picked up by the class loader.
-->
<jgroupsConfig>
<UDP discard_incompatible_packets="true"
enable_bundling="true"
enable_diagnostics="true" ip_ttl="32" ip_mcast="true"
loopback="false" max_bundle_size="64000"
max_bundle_timeout="30" mcast_addr="228.10.10.10"
bind_addr="10.210.102.100"
mcast_port="45588" mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
oob_thread_pool.enabled="true"
oob_thread_pool.keep_alive_time="10000"
oob_thread_pool.max_threads="4"
oob_thread_pool.min_threads="1"
oob_thread_pool.queue_enabled="true"
oob_thread_pool.queue_max_size="10"
oob_thread_pool.rejection_policy="Run"
thread_naming_pattern="pl" thread_pool.enabled="true"
thread_pool.keep_alive_time="30000"
thread_pool.max_threads="25" thread_pool.min_threads="1"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="10" thread_pool.rejection_policy="Run"
tos="8" ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000" use_concurrent_stack="true"
use_incoming_packet_handler="true"/>
<PING num_initial_members="3" timeout="8000"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD max_tries="5" shun="true"
timeout="10000"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK discard_delivered_msgs="true"
gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
use_mcast_xmit="false"/>
<UNICAST timeout="300,600,1200,2400,3600"/>
<pbcast.STABLE desired_avg_gossip="50000"
max_bytes="400000"
stability_delay="1000"/>
<pbcast.GMS join_timeout="50000"
print_local_addr="true"
shun="false" view_ack_collection_timeout="50000"
view_bundling="true"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER/>
<pbcast.FLUSH timeout="0"/>
</jgroupsConfig>
</clustering>
<!--
Eviction configuration. WakeupInterval defines how often the
eviction thread runs, in milliseconds. 0 means
the eviction thread will never run.
-->
<eviction wakeUpInterval="500">
<default algorithmClass="org.jboss.cache.eviction.LRUAlgorithm"
eventQueueSize="200000">
</default>
<region name="/CONTENT_MATCH_OPTIMIZER">
</region>
<region name="/CONTENT_MATCH_PROCESSOR"
eventQueueSize="100000">
</region>
<region name="/DEFAULT_NODE" eventQueueSize="100000">
</region>
</eviction>
<!--
Cache loaders.
If passivation is enabled, state is offloaded to the cache loaders
ONLY when evicted. Similarly, when the state
is accessed again, it is removed from the cache loader and loaded
into memory.
Otherwise, state is always maintained in the cache loader as well as
in memory.
Set 'shared' to true if all instances in the cluster use the same
cache loader instance, e.g., are talking to the
same database.
-->
<loaders passivation="false" shared="true">
<preload>
<node fqn="/org/jboss"/>
<node fqn="/org/tempdata"/>
</preload>
<!--
we can have multiple cache loaders, which get chained
-->
<loader class="org.jboss.cache.loader.JDBCCacheLoader"
async="false"
fetchPersistentState="false"
ignoreModifications="false"
purgeOnStartup="false">
<properties>
cache.jdbc.table.name=dsatcache
cache.jdbc.table.create=true
cache.jdbc.table.drop=false
cache.jdbc.table.primarykey=dsatcache_pk
cache.jdbc.fqn.column=fqn
cache.jdbc.fqn.type=varchar(512)
cache.jdbc.node.column=node
cache.jdbc.node.type=LONGBLOB
cache.jdbc.parent.column=parent
cache.jdbc.driver=com.mysql.jdbc.Driver
cache.jdbc.url=jdbc:mysql://1x.xxx.xxx.xxx:9213/cache
cache.jdbc.connection.factory=org.jboss.cache.loader.C3p0ConnectionFactory
cache.jdbc.user=demo
cache.jdbc.password=c1b3rd3m0
</properties>
<!-- alternatively use a connection from a data source, as per
the code sample below-->
<!--<properties>-->
<!--cache.jdbc.datasource=AllSampleDS-->
<!--cache.jdbc.table.name=jbosscache-->
<!--cache.jdbc.table.create=true-->
<!--cache.jdbc.table.drop=true-->
<!--</properties>-->
<singletonStore enabled="false"
class="org.jboss.cache.loader.SingletonStoreCacheLoader">
<properties>
pushStateWhenCoordinator=true
pushStateWhenCoordinatorTimeout=20000
</properties>
</singletonStore>
</loader>
</loaders>
</jbosscache>
On Thu, Apr 23, 2009 at 2:51 AM, Manik Surtani <manik(a)jboss.org> wrote:
Do you have a unit test for this?
On 23 Apr 2009, at 03:53, Yong Yuan wrote:
Hi,
I have two servers that use JBoss Cache 3.0.3. Each server is configured to
replicate its states to the other. The bigger server has more memory and
thus contains most of the cache states, while the smaller server uses a
ClusteredCacheLoader to get cached states if it can't find them in its own
cache instance:
Smaller server -> a cache instance
||
|| using ClusteredCacheLoader to get cached state
\\ //
Bigger server -> a cache instance
The problem is, the ClusteredCacheLoader never retrieved any cached state.
Turning on trace, I found that ClusteredCacheLoader always returned null if
cache mode was asynchronous replication and there was at least one
serialization executor. Therefore, I set serializationExecutorPoolSize to 0.
This time, ClusteredCacheLoader did attempt to send out remote request, but
it ended up getting a replication exception in the response, as the
following log shows:
(CommandAwareRpcDispatcher.java:291) 2009-04-22 19:47:50,151 TRACE
responses: [sender=10.210.102.104:45592, retval=[true, false],
received=true, suspected=false]
[sender=10.210.102.104:37132, retval=null, received=false,
suspected=false]
The bigger server didn't have any exception logged, though.
Any suggestion how I should solve this problem? Thanks a lot!
_______________________________________________
jbosscache-users mailing list
jbosscache-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosscache-users
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
5491
days inactive
5491
days old
jbosscache-users@lists.jboss.org
2 comments
2 participants
participants (2)
-
Manik Surtani -
Yong Yuan