[JBoss JIRA] (ISPN-4585) Prioritize commands in the remote executor
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-4585?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-4585.
--------------------------------
Resolution: Won't Do
No longer necessary since lock acquisition doesn't block a remote-executor thread.
> Prioritize commands in the remote executor
> ------------------------------------------
>
> Key: ISPN-4585
> URL: https://issues.jboss.org/browse/ISPN-4585
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core
> Affects Versions: 7.0.0.Alpha5
> Reporter: Dan Berindei
> Priority: Major
>
> The remote executor currently has an unlimited queue of blocked task, but the underlying executor cannot use a queue. With a queue, we wouldn't need to overflow remote commands to the OOB threads, and the OOB threads would be free to process response messages.
> The problem is that {{ThreadPoolExecutor}} executes tasks in the order they are in the queue. If a node has a remote executor thread pool of 100 threads and receives a prepare(tx1, put(k, v1) comand, then 1000 prepare(tx_i, put(k, v_i)) commands, and finally a commit(tx1) command, the commit(tx1) command will block until all but 99 of the the prepare(tx_i, put(k, v_i)) commands have timed out.
> I think we could help this by using a {{PriorityBlockingQueue}} for the underlying executor, with commands ordered so that state transfer commands < commit/tx completion notification < prepare/lock. The commit command would still have to wait for one of the prepare commands currently running to time out, but it wouldn't have to wait for all of them.
> The current code, without a queue, would fill the remote executor and OOB thread pools, and it would discard the commit message (along with most of the prepare commands). The time it would take to process the commit successfully would depend on the timing of the retransmitted messages.
> Another possible improvement would be to keep track of the commands currently being executed, and always keep some threads free for commands with higher priority. But I'm not sure how easy it would be to do that on top of an injected {{ExecutorService}}.
> I believe there is also a problem with {{BlockingTaskAwareExecutorServiceImpl.checkForReadyTasks()}} after a topology change. Commands with the new topology id are all unblocked by submitting them to the underlying executor in FIFO order, on a single thread, so {{CallerRunsPolicy}} is not a valid rejection policy here.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-1796) Out-of-memory adding a lot of elements in cache with AsyncStore
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-1796?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-1796.
--------------------------------
Fix Version/s: 9.2.0.Final
Resolution: Done
Fixed with ISPN-2293
> Out-of-memory adding a lot of elements in cache with AsyncStore
> ---------------------------------------------------------------
>
> Key: ISPN-1796
> URL: https://issues.jboss.org/browse/ISPN-1796
> Project: Infinispan
> Issue Type: Task
> Components: Loaders and Stores, Transactions
> Affects Versions: 5.1.0.CR3
> Environment: We plan to use Infinispan as a large distributed write-behind cache of terabytes of data, with a little fraction cached in RAM, so OOM is real threat for us.
> Reporter: Andrew Pushkin
> Priority: Major
> Labels: executor, threads
> Fix For: 9.2.0.Final
>
> Original Estimate: 3 days
> Remaining Estimate: 3 days
>
> OOM occure on peaks of putting objects in cache configured to use AsyncStore.
> See Steps to Reproduce.
>
> Profiling shows that the gc path is through AsyncStore.state field.
> The AsyncStore.executor initialized to ThreadPoolExecutor with DiscardPolicy to silently discard tasks if the queue is full, which delays async processing of entries in *state* map, which continues to grow.
> Suggested solution.
> Instead of DiscardPolicy use customized behavior, which is to estimate accumulated state size and (probably comparing it with modificationQueueSize) decide to discard or to block until it is processed.
> The downside of suggested solution is the necessity to lock to estimate state size every time the task is rejected. Possibly it can be alleviated by increasing workingQueue size, so that it survive peaks without rejection.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9300) ReplicationQueue ISPN000072: Failed replicating XXX elements in replication queue org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9300?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-9300.
--------------------------------
Fix Version/s: 9.0.0.Final
Resolution: Out of Date
The replication queue was removed in 9.0 with ISPN-6417
> ReplicationQueue ISPN000072: Failed replicating XXX elements in replication queue org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9300
> URL: https://issues.jboss.org/browse/ISPN-9300
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.4.Final
> Reporter: Nilesh Deshpande
> Priority: Major
> Fix For: 9.0.0.Final
>
>
> *I am getting continuous error in the log for following.*
> ERROR 06/16 21:38:03.910 [replicationQueue-thread--p4-t1] o.i.r.ReplicationQueue ISPN000072: Failed replicating 114 elements in replication queue
> org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:657)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:250)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
> at org.infinispan.remoting.ReplicationQueueImpl.flush(ReplicationQueueImpl.java:152)
> at org.infinispan.remoting.ReplicationQueueImpl$1.run(ReplicationQueueImpl.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:417)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:432)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241)
> ... 11 common frames omitted
> Caused by: java.util.ConcurrentModificationException: null
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
> at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
> at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:24)
> at org.infinispan.marshall.exts.SetExternalizer.writeObject(SetExternalizer.java:44)
> at org.infinispan.marshall.exts.SetExternalizer.writeObject(SetExternalizer.java:26)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:144)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1062)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1018)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:884)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:43)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:31)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:144)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:124)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:108)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:65)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:144)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:124)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:108)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:65)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:144)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)
> at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:415)
> ... 13 common frames omitted
> Caused by: org.jboss.marshalling.TraceInformation: null
> *My cache setting is following*
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd"
> xmlns="urn:infinispan:config:7.0">
> <jgroups>
> <stack-file name="configurationFile" path="cache-replicator.conf"/>
> </jgroups>
> <cache-container default-cache="defaultCache">
> <!--
> Enables collection of global JMX statistics, and allows you to customize the JMX domain name under which
> MBeans are registered.
> -->
>
> <jmx>
> <property name="enabled">false</property>
> <property name="jmxDomain">org.infinispan</property>
> <property name="cacheManagerName">CacheManager</property>
> <property name="statistics">false</property>
> </jmx>
>
>
> <!--
> If the transport is omitted, there is no way to create distributed or clustered caches.
> There is no added cost to defining a transport and not creating a cache that uses one, since the transport
> is created and initialized lazily. By default, the JGroupsTransport is used.
> Note that the JGroups transport uses sensible defaults if no configuration property is defined.
> See:
> http://community.jboss.org/wiki/ClusteredConfigurationQuickStart
> http://docs.jboss.org/infinispan/5.0/apidocs/org/infinispan/remoting/tran...
> -->
>
>
>
>
> <transport cluster="TrueSight-Cache-cluster" stack="configurationFile">
> </transport>
> <!--
> Used to register JVM shutdown hooks.
> hookBehavior: DEFAULT, REGISTER, DONT_REGISTER
> -->
> <!-- <shutdown hookBehavior="DONT_REGISTER"/> -->
>
> <local-cache name="defaultCache" statistics="false">
> <!-- <storeAsBinary /> -->
> <!--
> Tunes the locking and concurrency characteristics of the cache.
> See:
> http://community.jboss.org/wiki/LockingandConcurrency
> -->
> <locking
> isolation="READ_COMMITTED"
> acquire-timeout="20000"
> write-skew="false"
> concurrency-level="5000"
> striping="false"
> />
> <!--
> Used to register JMX statistics in any available MBean server
>
> <jmxStatistics enabled="false"/>-->
> <!--
> Used to control whether objects stored in Infinispan nodes are primarily stored as rehydrated Objects or as
> raw byte arrays. See:
> http://community.jboss.org/wiki/Marshalling#Lazy_Deserialization
> -->
> <!-- <storeAsBinary enabled="true"/> -->
> <!--
> Used to enable invocation batching and allow the use of Cache.startBatch()/endBatch() methods.
> http://community.jboss.org/wiki/Batching
>
> <invocationBatching enabled="true"/> -->
>
> <transaction mode="BATCH"/>
> <!--
> This element specifies that the cache is clustered.
> modes supported: distribution (d), replication (r) or invalidation (i).
> http://community.jboss.org/wiki/Clusteringmodes
> -->
>
>
> </local-cache>
> <replicated-cache name="defaultCache" mode="ASYNC" queue-size="500" queue-flush-interval="10000">
>
> <state-transfer await-initial-transfer="false"
> timeout="240000"
> enabled="true"
> chunk-size="10000"
> />
> <!--Default write-skew is false, concurrency-level is 16 and acquire-timeout is 20 seconds in infinispan. Increasing -->
> <locking concurrency-level="1000" striping="true" acquire-timeout="240000"
> isolation="READ_COMMITTED" write-skew="false"/>
> <!-- <sync replTimeout="20000"/> -->
> <!-- async replication. -->
>
> </replicated-cache>
> </cache-container>
> </infinispan>
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years