3:56 p.m.
Hi guys,
sorry for the long post, but I do think I ran into an important problem
and we need to fix it ... :-)
I've spent the last couple of days running the IspnPerfTest [1] perftest
on Google Compute Engine (GCE), and I've run into a problem with
Infinispan. It is a design problem and can be mitigated by sizing thread
pools correctly, but cannot be eliminated entirely.
Symptom:
--------
IspnPerfTest has every node in a cluster perform 20'000 requests on keys
in range [1..20000].
80% of the requests are reads and 20% writes.
By default, we have 25 requester threads per node and 100 nodes in a
cluster, so a total of 2500 requester threads.
The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
<namedCache name="clusteredCache">
<clustering mode="distribution">
<stateTransfer awaitInitialTransfer="true"/>
<hash numOwners="2"/>
<sync replTimeout="20000"/>
</clustering>
<transaction transactionMode="NON_TRANSACTIONAL"
useEagerLocking="true"
eagerLockSingleNode="true" />
<locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
isolationLevel="READ_COMMITTED" useLockStriping="false"
/>
</namedCache>
It has 2 owners, a lock acquisition timeout of 5s and a repl timeout of
20s. Lock stripting is off, so we have 1 lock per key.
When I run the test, I always get errors like those below:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [19386] for requestor [Thread[invoker-3,5,main]]!
Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
and
org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !
I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.
This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.
I learned (from Pedro and Mircea) that GETs and PUTs work as follows in
dist-sync / 2 owners:
- GETs are sent to the primary and backup owners and the first response
received is returned to the caller. No locks are acquired, so GETs
shouldn't cause problems.
- A PUT(K) is sent to the primary owner of K
- The primary owner
(1) locks K
(2) updates the backup owner synchronously *while holding the lock*
(3) releases the lock
Hypothesis
----------
(2) above is done while holding the lock. The sync update of the backup
owner is done with the lock held to guarantee that the primary and
backup owner of K have the same values for K.
However, the sync update *inside the lock scope* slows things down (can
it also lead to deadlocks?); there's the risk that the request is
dropped due to a full incoming thread pool, or that the response is not
received because of the same, or that the locking at the backup owner
blocks for some time.
If we have many threads modifying the same key, then we have a backlog
of locking work against that key. Say we have 100 requester threads and
a 100 node cluster. This means that we have 10'000 threads accessing
keys; with 2'000 writers there's a big chance that some writers pick the
same key at the same time.
For example, if we have 100 threads accessing key K and it takes 3ms to
replicate K to the backup owner, then the last of the 100 threads waits
~300ms before it gets a chance to lock K on the primary owner and
replicate it as well.
Just a small hiccup in sending the PUT to the primary owner, sending the
modification to the backup owner, waitting for the response, or GC, and
the delay will quickly become bigger.
Verification
----------
To verify the above, I set numOwners to 1. This means that the primary
owner of K does *not* send the modification to the backup owner, it only
locks K, modifies K and unlocks K again.
I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
PROBLEM !
I then increased the requesters to 100, 150 and 200 and the test
completed flawlessly ! Performance was around *40'000 requests per node
per sec* on 4-core boxes !
Root cause
---------
*******************
The root cause is the sync RPC of K to the backup owner(s) of K while
the primary owner holds the lock for K.
*******************
This causes a backlog of threads waiting for the lock and that backlog
can grow to exhaust the thread pools. First the Infinispan internal
thread pool, then the JGroups OOB thread pool. The latter causes
retransmissions to get dropped, which compounds the problem...
Goal
----
The goal is to make sure that primary and backup owner(s) of K have the
same value for K.
Simply sending the modification to the backup owner(s) asynchronously
won't guarantee this, as modification messages might get processed out
of order as they're OOB !
Suggested solution
----------------
The modification RPC needs to be invoked *outside of the lock scope*:
- lock K
- modify K
- unlock K
- send modification to backup owner(s) // outside the lock scope
The primary owner puts the modification of K into a queue from where a
separate thread/task removes it. The thread then invokes the PUT(K) on
the backup owner(s).
The queue has the modified keys in FIFO order, so the modifications
arrive at the backup owner(s) in the right order.
This requires that the way GET is implemented changes slightly: instead
of invoking a GET on all owners of K, we only invoke it on the primary
owner, then the next-in-line etc.
The reason for this is that the backup owner(s) may not yet have
received the modification of K.
This is a better impl anyway (we discussed this before) becuse it
generates less traffic; in the normal case, all but 1 GET requests are
unnecessary.
Improvement
-----------
The above solution can be simplified and even made more efficient.
Re-using concepts from IRAC [2], we can simply store the modified *keys*
in the modification queue. The modification replication thread removes
the key, gets the current value and invokes a PUT/REMOVE on the backup
owner(s).
Even better: a key is only ever added *once*, so if we have [5,2,17,3],
adding key 2 is a no-op because the processing of key 2 (in second
position in the queue) will fetch the up-to-date value anyway !
Misc
----
- Could we possibly use total order to send the updates in TO ? TBD (Pedro?)
Thoughts ?
[1] https://github.com/belaban/IspnPerfTest
[2]
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
--
Bela Ban, JGroups lead (http://www.jgroups.org)
4:38 p.m.
Following up on my own email, I changed the config to use Pedro's
excellent total order implementation:
<transaction transactionMode="TRANSACTIONAL"
transactionProtocol="TOTAL_ORDER" lockingMode="OPTIMISTIC"
useEagerLocking="true" eagerLockSingleNode="true">
<recovery enabled="false"/>
With 100 nodes and 25 requester threads/node, I did NOT run into any
locking issues !
I could even go up to 200 requester threads/node and the perf was ~
7'000-8'000 requests/sec/node. Not too bad !
This really validates the concept of lockless total-order dissemination
of TXs; for the first time, this has been tested on a large(r) scale
(previously only on 25 nodes) and IT WORKS ! :-)
I still believe we should implement my suggested solution for non-TO
configs, but short of configuring thread pools of 1000 threads or
higher, I hope TO will allow me to finally test a 500 node Infinispan
cluster !
On 29/07/14 15:56, Bela Ban wrote:
Hi guys,
sorry for the long post, but I do think I ran into an important problem
and we need to fix it ... :-)
I've spent the last couple of days running the IspnPerfTest [1] perftest
on Google Compute Engine (GCE), and I've run into a problem with
Infinispan. It is a design problem and can be mitigated by sizing thread
pools correctly, but cannot be eliminated entirely.
Symptom:
--------
IspnPerfTest has every node in a cluster perform 20'000 requests on keys
in range [1..20000].
80% of the requests are reads and 20% writes.
By default, we have 25 requester threads per node and 100 nodes in a
cluster, so a total of 2500 requester threads.
The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
<namedCache name="clusteredCache">
<clustering mode="distribution">
<stateTransfer awaitInitialTransfer="true"/>
<hash numOwners="2"/>
<sync replTimeout="20000"/>
</clustering>
<transaction transactionMode="NON_TRANSACTIONAL"
useEagerLocking="true"
eagerLockSingleNode="true" />
<locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
isolationLevel="READ_COMMITTED"
useLockStriping="false" />
</namedCache>
It has 2 owners, a lock acquisition timeout of 5s and a repl timeout of
20s. Lock stripting is off, so we have 1 lock per key.
When I run the test, I always get errors like those below:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [19386] for requestor [Thread[invoker-3,5,main]]!
Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
and
org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !
I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.
This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.
I learned (from Pedro and Mircea) that GETs and PUTs work as follows in
dist-sync / 2 owners:
- GETs are sent to the primary and backup owners and the first response
received is returned to the caller. No locks are acquired, so GETs
shouldn't cause problems.
- A PUT(K) is sent to the primary owner of K
- The primary owner
(1) locks K
(2) updates the backup owner synchronously *while holding the lock*
(3) releases the lock
Hypothesis
----------
(2) above is done while holding the lock. The sync update of the backup
owner is done with the lock held to guarantee that the primary and
backup owner of K have the same values for K.
However, the sync update *inside the lock scope* slows things down (can
it also lead to deadlocks?); there's the risk that the request is
dropped due to a full incoming thread pool, or that the response is not
received because of the same, or that the locking at the backup owner
blocks for some time.
If we have many threads modifying the same key, then we have a backlog
of locking work against that key. Say we have 100 requester threads and
a 100 node cluster. This means that we have 10'000 threads accessing
keys; with 2'000 writers there's a big chance that some writers pick the
same key at the same time.
For example, if we have 100 threads accessing key K and it takes 3ms to
replicate K to the backup owner, then the last of the 100 threads waits
~300ms before it gets a chance to lock K on the primary owner and
replicate it as well.
Just a small hiccup in sending the PUT to the primary owner, sending the
modification to the backup owner, waitting for the response, or GC, and
the delay will quickly become bigger.
Verification
----------
To verify the above, I set numOwners to 1. This means that the primary
owner of K does *not* send the modification to the backup owner, it only
locks K, modifies K and unlocks K again.
I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
PROBLEM !
I then increased the requesters to 100, 150 and 200 and the test
completed flawlessly ! Performance was around *40'000 requests per node
per sec* on 4-core boxes !
Root cause
---------
*******************
The root cause is the sync RPC of K to the backup owner(s) of K while
the primary owner holds the lock for K.
*******************
This causes a backlog of threads waiting for the lock and that backlog
can grow to exhaust the thread pools. First the Infinispan internal
thread pool, then the JGroups OOB thread pool. The latter causes
retransmissions to get dropped, which compounds the problem...
Goal
----
The goal is to make sure that primary and backup owner(s) of K have the
same value for K.
Simply sending the modification to the backup owner(s) asynchronously
won't guarantee this, as modification messages might get processed out
of order as they're OOB !
Suggested solution
----------------
The modification RPC needs to be invoked *outside of the lock scope*:
- lock K
- modify K
- unlock K
- send modification to backup owner(s) // outside the lock scope
The primary owner puts the modification of K into a queue from where a
separate thread/task removes it. The thread then invokes the PUT(K) on
the backup owner(s).
The queue has the modified keys in FIFO order, so the modifications
arrive at the backup owner(s) in the right order.
This requires that the way GET is implemented changes slightly: instead
of invoking a GET on all owners of K, we only invoke it on the primary
owner, then the next-in-line etc.
The reason for this is that the backup owner(s) may not yet have
received the modification of K.
This is a better impl anyway (we discussed this before) becuse it
generates less traffic; in the normal case, all but 1 GET requests are
unnecessary.
Improvement
-----------
The above solution can be simplified and even made more efficient.
Re-using concepts from IRAC [2], we can simply store the modified *keys*
in the modification queue. The modification replication thread removes
the key, gets the current value and invokes a PUT/REMOVE on the backup
owner(s).
Even better: a key is only ever added *once*, so if we have [5,2,17,3],
adding key 2 is a no-op because the processing of key 2 (in second
position in the queue) will fetch the up-to-date value anyway !
Misc
----
- Could we possibly use total order to send the updates in TO ? TBD (Pedro?)
Thoughts ?
[1] https://github.com/belaban/IspnPerfTest
[2]
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
--
Bela Ban, JGroups lead (http://www.jgroups.org)
4:42 p.m.
Have you tried regular optimistic/pessimistic transactions as well? They
*should* have less issues with the OOB thread pool than non-tx mode, and
I'm quite curious how they stack against TO in such a large cluster.
On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com> wrote:
Following up on my own email, I changed the config to use
Pedro's
excellent total order implementation:
<transaction transactionMode="TRANSACTIONAL"
transactionProtocol="TOTAL_ORDER" lockingMode="OPTIMISTIC"
useEagerLocking="true" eagerLockSingleNode="true">
<recovery enabled="false"/>
With 100 nodes and 25 requester threads/node, I did NOT run into any
locking issues !
I could even go up to 200 requester threads/node and the perf was ~
7'000-8'000 requests/sec/node. Not too bad !
This really validates the concept of lockless total-order dissemination
of TXs; for the first time, this has been tested on a large(r) scale
(previously only on 25 nodes) and IT WORKS ! :-)
I still believe we should implement my suggested solution for non-TO
configs, but short of configuring thread pools of 1000 threads or
higher, I hope TO will allow me to finally test a 500 node Infinispan
cluster !
On 29/07/14 15:56, Bela Ban wrote:
> Hi guys,
>
> sorry for the long post, but I do think I ran into an important problem
> and we need to fix it ... :-)
>
> I've spent the last couple of days running the IspnPerfTest [1] perftest
> on Google Compute Engine (GCE), and I've run into a problem with
> Infinispan. It is a design problem and can be mitigated by sizing thread
> pools correctly, but cannot be eliminated entirely.
>
>
> Symptom:
> --------
> IspnPerfTest has every node in a cluster perform 20'000 requests on keys
> in range [1..20000].
>
> 80% of the requests are reads and 20% writes.
>
> By default, we have 25 requester threads per node and 100 nodes in a
> cluster, so a total of 2500 requester threads.
>
> The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
>
> <namedCache name="clusteredCache">
> <clustering mode="distribution">
> <stateTransfer awaitInitialTransfer="true"/>
> <hash numOwners="2"/>
> <sync replTimeout="20000"/>
> </clustering>
>
> <transaction transactionMode="NON_TRANSACTIONAL"
> useEagerLocking="true"
> eagerLockSingleNode="true" />
> <locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
> isolationLevel="READ_COMMITTED"
useLockStriping="false"
/>
> </namedCache>
>
> It has 2 owners, a lock acquisition timeout of 5s and a repl timeout of
> 20s. Lock stripting is off, so we have 1 lock per key.
>
> When I run the test, I always get errors like those below:
>
> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
> after [10 seconds] on key [19386] for requestor
[Thread[invoker-3,5,main]]!
> Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
>
> and
>
> org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
>
>
> Investigation:
> ------------
> When I looked at UNICAST3, I saw a lot of missing messages on the
> receive side and unacked messages on the send side. This caused me to
> look into the (mainly OOB) thread pools and - voila - maxed out !
>
> I learned from Pedro that the Infinispan internal thread pool (with a
> default of 32 threads) can be configured, so I increased it to 300 and
> increased the OOB pools as well.
>
> This mitigated the problem somewhat, but when I increased the requester
> threads to 100, I had the same problem again. Apparently, the Infinispan
> internal thread pool uses a rejection policy of "run" and thus uses the
> JGroups (OOB) thread when exhausted.
>
> I learned (from Pedro and Mircea) that GETs and PUTs work as follows in
> dist-sync / 2 owners:
> - GETs are sent to the primary and backup owners and the first response
> received is returned to the caller. No locks are acquired, so GETs
> shouldn't cause problems.
>
> - A PUT(K) is sent to the primary owner of K
> - The primary owner
> (1) locks K
> (2) updates the backup owner synchronously *while holding the
lock*
> (3) releases the lock
>
>
> Hypothesis
> ----------
> (2) above is done while holding the lock. The sync update of the backup
> owner is done with the lock held to guarantee that the primary and
> backup owner of K have the same values for K.
>
> However, the sync update *inside the lock scope* slows things down (can
> it also lead to deadlocks?); there's the risk that the request is
> dropped due to a full incoming thread pool, or that the response is not
> received because of the same, or that the locking at the backup owner
> blocks for some time.
>
> If we have many threads modifying the same key, then we have a backlog
> of locking work against that key. Say we have 100 requester threads and
> a 100 node cluster. This means that we have 10'000 threads accessing
> keys; with 2'000 writers there's a big chance that some writers pick the
> same key at the same time.
>
> For example, if we have 100 threads accessing key K and it takes 3ms to
> replicate K to the backup owner, then the last of the 100 threads waits
> ~300ms before it gets a chance to lock K on the primary owner and
> replicate it as well.
>
> Just a small hiccup in sending the PUT to the primary owner, sending the
> modification to the backup owner, waitting for the response, or GC, and
> the delay will quickly become bigger.
>
>
> Verification
> ----------
> To verify the above, I set numOwners to 1. This means that the primary
> owner of K does *not* send the modification to the backup owner, it only
> locks K, modifies K and unlocks K again.
>
> I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
> PROBLEM !
>
> I then increased the requesters to 100, 150 and 200 and the test
> completed flawlessly ! Performance was around *40'000 requests per node
> per sec* on 4-core boxes !
>
>
> Root cause
> ---------
> *******************
> The root cause is the sync RPC of K to the backup owner(s) of K while
> the primary owner holds the lock for K.
> *******************
>
> This causes a backlog of threads waiting for the lock and that backlog
> can grow to exhaust the thread pools. First the Infinispan internal
> thread pool, then the JGroups OOB thread pool. The latter causes
> retransmissions to get dropped, which compounds the problem...
>
>
> Goal
> ----
> The goal is to make sure that primary and backup owner(s) of K have the
> same value for K.
>
> Simply sending the modification to the backup owner(s) asynchronously
> won't guarantee this, as modification messages might get processed out
> of order as they're OOB !
>
>
> Suggested solution
> ----------------
> The modification RPC needs to be invoked *outside of the lock scope*:
> - lock K
> - modify K
> - unlock K
> - send modification to backup owner(s) // outside the lock scope
>
> The primary owner puts the modification of K into a queue from where a
> separate thread/task removes it. The thread then invokes the PUT(K) on
> the backup owner(s).
>
> The queue has the modified keys in FIFO order, so the modifications
> arrive at the backup owner(s) in the right order.
>
> This requires that the way GET is implemented changes slightly: instead
> of invoking a GET on all owners of K, we only invoke it on the primary
> owner, then the next-in-line etc.
>
> The reason for this is that the backup owner(s) may not yet have
> received the modification of K.
>
> This is a better impl anyway (we discussed this before) becuse it
> generates less traffic; in the normal case, all but 1 GET requests are
> unnecessary.
>
>
>
> Improvement
> -----------
> The above solution can be simplified and even made more efficient.
> Re-using concepts from IRAC [2], we can simply store the modified *keys*
> in the modification queue. The modification replication thread removes
> the key, gets the current value and invokes a PUT/REMOVE on the backup
> owner(s).
>
> Even better: a key is only ever added *once*, so if we have [5,2,17,3],
> adding key 2 is a no-op because the processing of key 2 (in second
> position in the queue) will fetch the up-to-date value anyway !
>
>
> Misc
> ----
> - Could we possibly use total order to send the updates in TO ? TBD
(Pedro?)
>
>
> Thoughts ?
>
>
> [1] https://github.com/belaban/IspnPerfTest
> [2]
>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
>
--
Bela Ban, JGroups lead (http://www.jgroups.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:50 p.m.
On 29/07/14 16:42, Dan Berindei wrote:
Have you tried regular optimistic/pessimistic transactions as well?
Yes, in my first impl. but since I'm making only 1 change per request, I
thought a TX is overkill.
They *should* have less issues with the OOB thread pool than non-tx
mode, and
I'm quite curious how they stack against TO in such a large cluster.
Why would they have fewer issues with the thread pools ? AIUI, a TX
involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not using
TXs. And we're sync anyway...
On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
<mailto:bban@redhat.com>> wrote:
Following up on my own email, I changed the config to use Pedro's
excellent total order implementation:
<transaction transactionMode="TRANSACTIONAL"
transactionProtocol="TOTAL_ORDER" lockingMode="OPTIMISTIC"
useEagerLocking="true" eagerLockSingleNode="true">
<recovery enabled="false"/>
With 100 nodes and 25 requester threads/node, I did NOT run into any
locking issues !
I could even go up to 200 requester threads/node and the perf was ~
7'000-8'000 requests/sec/node. Not too bad !
This really validates the concept of lockless total-order dissemination
of TXs; for the first time, this has been tested on a large(r) scale
(previously only on 25 nodes) and IT WORKS ! :-)
I still believe we should implement my suggested solution for non-TO
configs, but short of configuring thread pools of 1000 threads or
higher, I hope TO will allow me to finally test a 500 node Infinispan
cluster !
On 29/07/14 15:56, Bela Ban wrote:
> Hi guys,
>
> sorry for the long post, but I do think I ran into an important
problem
> and we need to fix it ... :-)
>
> I've spent the last couple of days running the IspnPerfTest [1]
perftest
> on Google Compute Engine (GCE), and I've run into a problem with
> Infinispan. It is a design problem and can be mitigated by sizing
thread
> pools correctly, but cannot be eliminated entirely.
>
>
> Symptom:
> --------
> IspnPerfTest has every node in a cluster perform 20'000 requests
on keys
> in range [1..20000].
>
> 80% of the requests are reads and 20% writes.
>
> By default, we have 25 requester threads per node and 100 nodes in a
> cluster, so a total of 2500 requester threads.
>
> The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
>
> <namedCache name="clusteredCache">
> <clustering mode="distribution">
> <stateTransfer awaitInitialTransfer="true"/>
> <hash numOwners="2"/>
> <sync replTimeout="20000"/>
> </clustering>
>
> <transaction transactionMode="NON_TRANSACTIONAL"
> useEagerLocking="true"
> eagerLockSingleNode="true" />
> <locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
> isolationLevel="READ_COMMITTED"
useLockStriping="false" />
> </namedCache>
>
> It has 2 owners, a lock acquisition timeout of 5s and a repl
timeout of
> 20s. Lock stripting is off, so we have 1 lock per key.
>
> When I run the test, I always get errors like those below:
>
> org.infinispan.util.concurrent.TimeoutException: Unable to
acquire lock
> after [10 seconds] on key [19386] for requestor
[Thread[invoker-3,5,main]]!
> Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
>
> and
>
> org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
>
>
> Investigation:
> ------------
> When I looked at UNICAST3, I saw a lot of missing messages on the
> receive side and unacked messages on the send side. This caused me to
> look into the (mainly OOB) thread pools and - voila - maxed out !
>
> I learned from Pedro that the Infinispan internal thread pool (with a
> default of 32 threads) can be configured, so I increased it to
300 and
> increased the OOB pools as well.
>
> This mitigated the problem somewhat, but when I increased the
requester
> threads to 100, I had the same problem again. Apparently, the
Infinispan
> internal thread pool uses a rejection policy of "run" and thus
uses the
> JGroups (OOB) thread when exhausted.
>
> I learned (from Pedro and Mircea) that GETs and PUTs work as
follows in
> dist-sync / 2 owners:
> - GETs are sent to the primary and backup owners and the first
response
> received is returned to the caller. No locks are acquired, so GETs
> shouldn't cause problems.
>
> - A PUT(K) is sent to the primary owner of K
> - The primary owner
> (1) locks K
> (2) updates the backup owner synchronously *while holding
the lock*
> (3) releases the lock
>
>
> Hypothesis
> ----------
> (2) above is done while holding the lock. The sync update of the
backup
> owner is done with the lock held to guarantee that the primary and
> backup owner of K have the same values for K.
>
> However, the sync update *inside the lock scope* slows things
down (can
> it also lead to deadlocks?); there's the risk that the request is
> dropped due to a full incoming thread pool, or that the response
is not
> received because of the same, or that the locking at the backup owner
> blocks for some time.
>
> If we have many threads modifying the same key, then we have a
backlog
> of locking work against that key. Say we have 100 requester
threads and
> a 100 node cluster. This means that we have 10'000 threads accessing
> keys; with 2'000 writers there's a big chance that some writers
pick the
> same key at the same time.
>
> For example, if we have 100 threads accessing key K and it takes
3ms to
> replicate K to the backup owner, then the last of the 100 threads
waits
> ~300ms before it gets a chance to lock K on the primary owner and
> replicate it as well.
>
> Just a small hiccup in sending the PUT to the primary owner,
sending the
> modification to the backup owner, waitting for the response, or
GC, and
> the delay will quickly become bigger.
>
>
> Verification
> ----------
> To verify the above, I set numOwners to 1. This means that the
primary
> owner of K does *not* send the modification to the backup owner,
it only
> locks K, modifies K and unlocks K again.
>
> I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
> PROBLEM !
>
> I then increased the requesters to 100, 150 and 200 and the test
> completed flawlessly ! Performance was around *40'000 requests
per node
> per sec* on 4-core boxes !
>
>
> Root cause
> ---------
> *******************
> The root cause is the sync RPC of K to the backup owner(s) of K while
> the primary owner holds the lock for K.
> *******************
>
> This causes a backlog of threads waiting for the lock and that
backlog
> can grow to exhaust the thread pools. First the Infinispan internal
> thread pool, then the JGroups OOB thread pool. The latter causes
> retransmissions to get dropped, which compounds the problem...
>
>
> Goal
> ----
> The goal is to make sure that primary and backup owner(s) of K
have the
> same value for K.
>
> Simply sending the modification to the backup owner(s) asynchronously
> won't guarantee this, as modification messages might get
processed out
> of order as they're OOB !
>
>
> Suggested solution
> ----------------
> The modification RPC needs to be invoked *outside of the lock scope*:
> - lock K
> - modify K
> - unlock K
> - send modification to backup owner(s) // outside the lock scope
>
> The primary owner puts the modification of K into a queue from
where a
> separate thread/task removes it. The thread then invokes the
PUT(K) on
> the backup owner(s).
>
> The queue has the modified keys in FIFO order, so the modifications
> arrive at the backup owner(s) in the right order.
>
> This requires that the way GET is implemented changes slightly:
instead
> of invoking a GET on all owners of K, we only invoke it on the
primary
> owner, then the next-in-line etc.
>
> The reason for this is that the backup owner(s) may not yet have
> received the modification of K.
>
> This is a better impl anyway (we discussed this before) becuse it
> generates less traffic; in the normal case, all but 1 GET
requests are
> unnecessary.
>
>
>
> Improvement
> -----------
> The above solution can be simplified and even made more efficient.
> Re-using concepts from IRAC [2], we can simply store the modified
*keys*
> in the modification queue. The modification replication thread
removes
> the key, gets the current value and invokes a PUT/REMOVE on the
backup
> owner(s).
>
> Even better: a key is only ever added *once*, so if we have
[5,2,17,3],
> adding key 2 is a no-op because the processing of key 2 (in second
> position in the queue) will fetch the up-to-date value anyway !
>
>
> Misc
> ----
> - Could we possibly use total order to send the updates in TO ?
TBD (Pedro?)
>
>
> Thoughts ?
>
>
> [1] https://github.com/belaban/IspnPerfTest
> [2]
>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
>
--
Bela Ban, JGroups lead (http://www.jgroups.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
8:06 p.m.
This is a nasty problem and I also feel passionately we need to get
rid of it ASAP.
I did have the same problems many times, and we discussed this also in
Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
You don't need TO, and you don't need to lock at all as long as you
guarantee the backup owners are getting the number with some
monotonicity sequence attached to it,
all that backup owners need to do is ignore incoming commands which
are outdated.
Another aspect is that the "user thread" on the primary owner needs to
wait (at least until we improve further) and only proceed after ACK
from backup nodes, but this is better modelled through a state
machine. (Also discussed in Farnborough).
It's also conceptually linked to:
- https://issues.jboss.org/browse/ISPN-1599
As you need to separate the locks of entries from the effective user
facing lock, at least to implement transactions on top of this model.
I expect this to improve performance in a very significant way, but
it's getting embarrassing that it's still not done; at the next face
to face meeting we should also reserve some time for retrospective
sessions.
Sanne
On 29 July 2014 15:50, Bela Ban <bban(a)redhat.com> wrote:
On 29/07/14 16:42, Dan Berindei wrote:
> Have you tried regular optimistic/pessimistic transactions as well?
Yes, in my first impl. but since I'm making only 1 change per request, I
thought a TX is overkill.
> They *should* have less issues with the OOB thread pool than non-tx mode, and
> I'm quite curious how they stack against TO in such a large cluster.
Why would they have fewer issues with the thread pools ? AIUI, a TX
involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not using
TXs. And we're sync anyway...
> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
> <mailto:bban@redhat.com>> wrote:
>
> Following up on my own email, I changed the config to use Pedro's
> excellent total order implementation:
>
> <transaction transactionMode="TRANSACTIONAL"
> transactionProtocol="TOTAL_ORDER" lockingMode="OPTIMISTIC"
> useEagerLocking="true" eagerLockSingleNode="true">
> <recovery enabled="false"/>
>
> With 100 nodes and 25 requester threads/node, I did NOT run into any
> locking issues !
>
> I could even go up to 200 requester threads/node and the perf was ~
> 7'000-8'000 requests/sec/node. Not too bad !
>
> This really validates the concept of lockless total-order dissemination
> of TXs; for the first time, this has been tested on a large(r) scale
> (previously only on 25 nodes) and IT WORKS ! :-)
>
> I still believe we should implement my suggested solution for non-TO
> configs, but short of configuring thread pools of 1000 threads or
> higher, I hope TO will allow me to finally test a 500 node Infinispan
> cluster !
>
>
> On 29/07/14 15:56, Bela Ban wrote:
> > Hi guys,
> >
> > sorry for the long post, but I do think I ran into an important
> problem
> > and we need to fix it ... :-)
> >
> > I've spent the last couple of days running the IspnPerfTest [1]
> perftest
> > on Google Compute Engine (GCE), and I've run into a problem with
> > Infinispan. It is a design problem and can be mitigated by sizing
> thread
> > pools correctly, but cannot be eliminated entirely.
> >
> >
> > Symptom:
> > --------
> > IspnPerfTest has every node in a cluster perform 20'000 requests
> on keys
> > in range [1..20000].
> >
> > 80% of the requests are reads and 20% writes.
> >
> > By default, we have 25 requester threads per node and 100 nodes in a
> > cluster, so a total of 2500 requester threads.
> >
> > The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
> >
> > <namedCache name="clusteredCache">
> > <clustering mode="distribution">
> > <stateTransfer awaitInitialTransfer="true"/>
> > <hash numOwners="2"/>
> > <sync replTimeout="20000"/>
> > </clustering>
> >
> > <transaction transactionMode="NON_TRANSACTIONAL"
> > useEagerLocking="true"
> > eagerLockSingleNode="true" />
> > <locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
> > isolationLevel="READ_COMMITTED"
> useLockStriping="false" />
> > </namedCache>
> >
> > It has 2 owners, a lock acquisition timeout of 5s and a repl
> timeout of
> > 20s. Lock stripting is off, so we have 1 lock per key.
> >
> > When I run the test, I always get errors like those below:
> >
> > org.infinispan.util.concurrent.TimeoutException: Unable to
> acquire lock
> > after [10 seconds] on key [19386] for requestor
> [Thread[invoker-3,5,main]]!
> > Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
> >
> > and
> >
> > org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
> >
> >
> > Investigation:
> > ------------
> > When I looked at UNICAST3, I saw a lot of missing messages on the
> > receive side and unacked messages on the send side. This caused me to
> > look into the (mainly OOB) thread pools and - voila - maxed out !
> >
> > I learned from Pedro that the Infinispan internal thread pool (with a
> > default of 32 threads) can be configured, so I increased it to
> 300 and
> > increased the OOB pools as well.
> >
> > This mitigated the problem somewhat, but when I increased the
> requester
> > threads to 100, I had the same problem again. Apparently, the
> Infinispan
> > internal thread pool uses a rejection policy of "run" and thus
> uses the
> > JGroups (OOB) thread when exhausted.
> >
> > I learned (from Pedro and Mircea) that GETs and PUTs work as
> follows in
> > dist-sync / 2 owners:
> > - GETs are sent to the primary and backup owners and the first
> response
> > received is returned to the caller. No locks are acquired, so GETs
> > shouldn't cause problems.
> >
> > - A PUT(K) is sent to the primary owner of K
> > - The primary owner
> > (1) locks K
> > (2) updates the backup owner synchronously *while holding
> the lock*
> > (3) releases the lock
> >
> >
> > Hypothesis
> > ----------
> > (2) above is done while holding the lock. The sync update of the
> backup
> > owner is done with the lock held to guarantee that the primary and
> > backup owner of K have the same values for K.
> >
> > However, the sync update *inside the lock scope* slows things
> down (can
> > it also lead to deadlocks?); there's the risk that the request is
> > dropped due to a full incoming thread pool, or that the response
> is not
> > received because of the same, or that the locking at the backup owner
> > blocks for some time.
> >
> > If we have many threads modifying the same key, then we have a
> backlog
> > of locking work against that key. Say we have 100 requester
> threads and
> > a 100 node cluster. This means that we have 10'000 threads accessing
> > keys; with 2'000 writers there's a big chance that some writers
> pick the
> > same key at the same time.
> >
> > For example, if we have 100 threads accessing key K and it takes
> 3ms to
> > replicate K to the backup owner, then the last of the 100 threads
> waits
> > ~300ms before it gets a chance to lock K on the primary owner and
> > replicate it as well.
> >
> > Just a small hiccup in sending the PUT to the primary owner,
> sending the
> > modification to the backup owner, waitting for the response, or
> GC, and
> > the delay will quickly become bigger.
> >
> >
> > Verification
> > ----------
> > To verify the above, I set numOwners to 1. This means that the
> primary
> > owner of K does *not* send the modification to the backup owner,
> it only
> > locks K, modifies K and unlocks K again.
> >
> > I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
> > PROBLEM !
> >
> > I then increased the requesters to 100, 150 and 200 and the test
> > completed flawlessly ! Performance was around *40'000 requests
> per node
> > per sec* on 4-core boxes !
> >
> >
> > Root cause
> > ---------
> > *******************
> > The root cause is the sync RPC of K to the backup owner(s) of K while
> > the primary owner holds the lock for K.
> > *******************
> >
> > This causes a backlog of threads waiting for the lock and that
> backlog
> > can grow to exhaust the thread pools. First the Infinispan internal
> > thread pool, then the JGroups OOB thread pool. The latter causes
> > retransmissions to get dropped, which compounds the problem...
> >
> >
> > Goal
> > ----
> > The goal is to make sure that primary and backup owner(s) of K
> have the
> > same value for K.
> >
> > Simply sending the modification to the backup owner(s) asynchronously
> > won't guarantee this, as modification messages might get
> processed out
> > of order as they're OOB !
> >
> >
> > Suggested solution
> > ----------------
> > The modification RPC needs to be invoked *outside of the lock scope*:
> > - lock K
> > - modify K
> > - unlock K
> > - send modification to backup owner(s) // outside the lock scope
> >
> > The primary owner puts the modification of K into a queue from
> where a
> > separate thread/task removes it. The thread then invokes the
> PUT(K) on
> > the backup owner(s).
> >
> > The queue has the modified keys in FIFO order, so the modifications
> > arrive at the backup owner(s) in the right order.
> >
> > This requires that the way GET is implemented changes slightly:
> instead
> > of invoking a GET on all owners of K, we only invoke it on the
> primary
> > owner, then the next-in-line etc.
> >
> > The reason for this is that the backup owner(s) may not yet have
> > received the modification of K.
> >
> > This is a better impl anyway (we discussed this before) becuse it
> > generates less traffic; in the normal case, all but 1 GET
> requests are
> > unnecessary.
> >
> >
> >
> > Improvement
> > -----------
> > The above solution can be simplified and even made more efficient.
> > Re-using concepts from IRAC [2], we can simply store the modified
> *keys*
> > in the modification queue. The modification replication thread
> removes
> > the key, gets the current value and invokes a PUT/REMOVE on the
> backup
> > owner(s).
> >
> > Even better: a key is only ever added *once*, so if we have
> [5,2,17,3],
> > adding key 2 is a no-op because the processing of key 2 (in second
> > position in the queue) will fetch the up-to-date value anyway !
> >
> >
> > Misc
> > ----
> > - Could we possibly use total order to send the updates in TO ?
> TBD (Pedro?)
> >
> >
> > Thoughts ?
> >
> >
> > [1] https://github.com/belaban/IspnPerfTest
> > [2]
> >
>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
--
Bela Ban, JGroups lead (http://www.jgroups.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:14 p.m.
On Tue, Jul 29, 2014 at 9:06 PM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
This is a nasty problem and I also feel passionately we need to get
rid of it ASAP.
I did have the same problems many times, and we discussed this also in
Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
You don't need TO, and you don't need to lock at all as long as you
guarantee the backup owners are getting the number with some
monotonicity sequence attached to it,
all that backup owners need to do is ignore incoming commands which
are outdated.
This is more or less what TOA does - assign a monotonic sequence number to
txs, and only apply them after all the previous txs in the sequence have
been applied. The problem is getting that monotonic sequence when there are
multiple originators and multiple primary owners also requires some extra
RPCs.
Another aspect is that the "user thread" on the primary
owner needs to
wait (at least until we improve further) and only proceed after ACK
from backup nodes, but this is better modelled through a state
machine. (Also discussed in Farnborough).
To be clear, I don't think keeping the user thread on the originator
blocked until we have the write confirmations from all the backups is a
problem - a sync operation has to block, and it also serves to rate-limit
user operations.
The problem appears when the originator is not the primary owner, and the
thread blocking for backup ACKs is from the remote-executor pool (or OOB,
when the remote-executor pool is exhausted).
It's also conceptually linked to:
- https://issues.jboss.org/browse/ISPN-1599
As you need to separate the locks of entries from the effective user
facing lock, at least to implement transactions on top of this model.
I think we fixed ISPN-1599 when we changed passivation to use
DataContainer.compute(). WDYT Pedro, is there anything else you'd like to
do in the scope of ISPN-1599?
I expect this to improve performance in a very significant way, but
it's getting embarrassing that it's still not done; at the next face
to face meeting we should also reserve some time for retrospective
sessions.
Implementing the state machine-based interceptor stack may give us a
performance boost, but I'm much more certain that it's a very complex, high
risk task... and we don't have a stable test suite yet :)
Sanne
On 29 July 2014 15:50, Bela Ban <bban(a)redhat.com> wrote:
>
>
> On 29/07/14 16:42, Dan Berindei wrote:
>> Have you tried regular optimistic/pessimistic transactions as well?
>
> Yes, in my first impl. but since I'm making only 1 change per request, I
> thought a TX is overkill.
>
>> They *should* have less issues with the OOB thread pool than non-tx
mode, and
>> I'm quite curious how they stack against TO in such a large cluster.
>
> Why would they have fewer issues with the thread pools ? AIUI, a TX
> involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not using
> TXs. And we're sync anyway...
>
>
>> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
>> <mailto:bban@redhat.com>> wrote:
>>
>> Following up on my own email, I changed the config to use Pedro's
>> excellent total order implementation:
>>
>> <transaction transactionMode="TRANSACTIONAL"
>> transactionProtocol="TOTAL_ORDER"
lockingMode="OPTIMISTIC"
>> useEagerLocking="true" eagerLockSingleNode="true">
>> <recovery enabled="false"/>
>>
>> With 100 nodes and 25 requester threads/node, I did NOT run into any
>> locking issues !
>>
>> I could even go up to 200 requester threads/node and the perf was ~
>> 7'000-8'000 requests/sec/node. Not too bad !
>>
>> This really validates the concept of lockless total-order
dissemination
>> of TXs; for the first time, this has been tested on a large(r) scale
>> (previously only on 25 nodes) and IT WORKS ! :-)
>>
>> I still believe we should implement my suggested solution for non-TO
>> configs, but short of configuring thread pools of 1000 threads or
>> higher, I hope TO will allow me to finally test a 500 node
Infinispan
>> cluster !
>>
>>
>> On 29/07/14 15:56, Bela Ban wrote:
>> > Hi guys,
>> >
>> > sorry for the long post, but I do think I ran into an important
>> problem
>> > and we need to fix it ... :-)
>> >
>> > I've spent the last couple of days running the IspnPerfTest [1]
>> perftest
>> > on Google Compute Engine (GCE), and I've run into a problem with
>> > Infinispan. It is a design problem and can be mitigated by sizing
>> thread
>> > pools correctly, but cannot be eliminated entirely.
>> >
>> >
>> > Symptom:
>> > --------
>> > IspnPerfTest has every node in a cluster perform 20'000 requests
>> on keys
>> > in range [1..20000].
>> >
>> > 80% of the requests are reads and 20% writes.
>> >
>> > By default, we have 25 requester threads per node and 100 nodes
in a
>> > cluster, so a total of 2500 requester threads.
>> >
>> > The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
>> >
>> > <namedCache name="clusteredCache">
>> > <clustering mode="distribution">
>> > <stateTransfer
awaitInitialTransfer="true"/>
>> > <hash numOwners="2"/>
>> > <sync replTimeout="20000"/>
>> > </clustering>
>> >
>> > <transaction transactionMode="NON_TRANSACTIONAL"
>> > useEagerLocking="true"
>> > eagerLockSingleNode="true" />
>> > <locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
>> > isolationLevel="READ_COMMITTED"
>> useLockStriping="false" />
>> > </namedCache>
>> >
>> > It has 2 owners, a lock acquisition timeout of 5s and a repl
>> timeout of
>> > 20s. Lock stripting is off, so we have 1 lock per key.
>> >
>> > When I run the test, I always get errors like those below:
>> >
>> > org.infinispan.util.concurrent.TimeoutException: Unable to
>> acquire lock
>> > after [10 seconds] on key [19386] for requestor
>> [Thread[invoker-3,5,main]]!
>> > Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
>> >
>> > and
>> >
>> > org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed
out
>> >
>> >
>> > Investigation:
>> > ------------
>> > When I looked at UNICAST3, I saw a lot of missing messages on the
>> > receive side and unacked messages on the send side. This caused
me to
>> > look into the (mainly OOB) thread pools and - voila - maxed out !
>> >
>> > I learned from Pedro that the Infinispan internal thread pool
(with a
>> > default of 32 threads) can be configured, so I increased it to
>> 300 and
>> > increased the OOB pools as well.
>> >
>> > This mitigated the problem somewhat, but when I increased the
>> requester
>> > threads to 100, I had the same problem again. Apparently, the
>> Infinispan
>> > internal thread pool uses a rejection policy of "run" and
thus
>> uses the
>> > JGroups (OOB) thread when exhausted.
>> >
>> > I learned (from Pedro and Mircea) that GETs and PUTs work as
>> follows in
>> > dist-sync / 2 owners:
>> > - GETs are sent to the primary and backup owners and the first
>> response
>> > received is returned to the caller. No locks are acquired, so
GETs
>> > shouldn't cause problems.
>> >
>> > - A PUT(K) is sent to the primary owner of K
>> > - The primary owner
>> > (1) locks K
>> > (2) updates the backup owner synchronously *while holding
>> the lock*
>> > (3) releases the lock
>> >
>> >
>> > Hypothesis
>> > ----------
>> > (2) above is done while holding the lock. The sync update of the
>> backup
>> > owner is done with the lock held to guarantee that the primary
and
>> > backup owner of K have the same values for K.
>> >
>> > However, the sync update *inside the lock scope* slows things
>> down (can
>> > it also lead to deadlocks?); there's the risk that the request is
>> > dropped due to a full incoming thread pool, or that the response
>> is not
>> > received because of the same, or that the locking at the backup
owner
>> > blocks for some time.
>> >
>> > If we have many threads modifying the same key, then we have a
>> backlog
>> > of locking work against that key. Say we have 100 requester
>> threads and
>> > a 100 node cluster. This means that we have 10'000 threads
accessing
>> > keys; with 2'000 writers there's a big chance that some
writers
>> pick the
>> > same key at the same time.
>> >
>> > For example, if we have 100 threads accessing key K and it takes
>> 3ms to
>> > replicate K to the backup owner, then the last of the 100 threads
>> waits
>> > ~300ms before it gets a chance to lock K on the primary owner and
>> > replicate it as well.
>> >
>> > Just a small hiccup in sending the PUT to the primary owner,
>> sending the
>> > modification to the backup owner, waitting for the response, or
>> GC, and
>> > the delay will quickly become bigger.
>> >
>> >
>> > Verification
>> > ----------
>> > To verify the above, I set numOwners to 1. This means that the
>> primary
>> > owner of K does *not* send the modification to the backup owner,
>> it only
>> > locks K, modifies K and unlocks K again.
>> >
>> > I ran the IspnPerfTest again on 100 nodes, with 25 requesters,
and NO
>> > PROBLEM !
>> >
>> > I then increased the requesters to 100, 150 and 200 and the test
>> > completed flawlessly ! Performance was around *40'000 requests
>> per node
>> > per sec* on 4-core boxes !
>> >
>> >
>> > Root cause
>> > ---------
>> > *******************
>> > The root cause is the sync RPC of K to the backup owner(s) of K
while
>> > the primary owner holds the lock for K.
>> > *******************
>> >
>> > This causes a backlog of threads waiting for the lock and that
>> backlog
>> > can grow to exhaust the thread pools. First the Infinispan
internal
>> > thread pool, then the JGroups OOB thread pool. The latter causes
>> > retransmissions to get dropped, which compounds the problem...
>> >
>> >
>> > Goal
>> > ----
>> > The goal is to make sure that primary and backup owner(s) of K
>> have the
>> > same value for K.
>> >
>> > Simply sending the modification to the backup owner(s)
asynchronously
>> > won't guarantee this, as modification messages might get
>> processed out
>> > of order as they're OOB !
>> >
>> >
>> > Suggested solution
>> > ----------------
>> > The modification RPC needs to be invoked *outside of the lock
scope*:
>> > - lock K
>> > - modify K
>> > - unlock K
>> > - send modification to backup owner(s) // outside the lock scope
>> >
>> > The primary owner puts the modification of K into a queue from
>> where a
>> > separate thread/task removes it. The thread then invokes the
>> PUT(K) on
>> > the backup owner(s).
>> >
>> > The queue has the modified keys in FIFO order, so the
modifications
>> > arrive at the backup owner(s) in the right order.
>> >
>> > This requires that the way GET is implemented changes slightly:
>> instead
>> > of invoking a GET on all owners of K, we only invoke it on the
>> primary
>> > owner, then the next-in-line etc.
>> >
>> > The reason for this is that the backup owner(s) may not yet have
>> > received the modification of K.
>> >
>> > This is a better impl anyway (we discussed this before) becuse it
>> > generates less traffic; in the normal case, all but 1 GET
>> requests are
>> > unnecessary.
>> >
>> >
>> >
>> > Improvement
>> > -----------
>> > The above solution can be simplified and even made more
efficient.
>> > Re-using concepts from IRAC [2], we can simply store the modified
>> *keys*
>> > in the modification queue. The modification replication thread
>> removes
>> > the key, gets the current value and invokes a PUT/REMOVE on the
>> backup
>> > owner(s).
>> >
>> > Even better: a key is only ever added *once*, so if we have
>> [5,2,17,3],
>> > adding key 2 is a no-op because the processing of key 2 (in
second
>> > position in the queue) will fetch the up-to-date value anyway !
>> >
>> >
>> > Misc
>> > ----
>> > - Could we possibly use total order to send the updates in TO ?
>> TBD (Pedro?)
>> >
>> >
>> > Thoughts ?
>> >
>> >
>> > [1] https://github.com/belaban/IspnPerfTest
>> > [2]
>> >
>>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
>> >
>>
>> --
>> Bela Ban, JGroups lead (http://www.jgroups.org)
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org <mailto:
infinispan-dev(a)lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:35 p.m.
On 29 July 2014 22:14, Dan Berindei <dan.berindei(a)gmail.com> wrote:
On Tue, Jul 29, 2014 at 9:06 PM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
>
> This is a nasty problem and I also feel passionately we need to get
> rid of it ASAP.
> I did have the same problems many times, and we discussed this also in
> Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
>
> You don't need TO, and you don't need to lock at all as long as you
> guarantee the backup owners are getting the number with some
> monotonicity sequence attached to it,
> all that backup owners need to do is ignore incoming commands which
> are outdated.
This is more or less what TOA does - assign a monotonic sequence number to
txs, and only apply them after all the previous txs in the sequence have
been applied. The problem is getting that monotonic sequence when there are
multiple originators and multiple primary owners also requires some extra
RPCs.
Let's not mix this up with requirements for other areas.
The strategy I've proposed is only to be applied for the communication
from the primary owner to its backups:
the value to be written is well known as it's the primary owner which
defines it unilaterally (for example if there is an atomic replacement
to be computed)
and there is no need for extra RPCs as the sequence is not related to
a group of changes but for the specific entry only.
There is no such thing as a need for consensus across owners, nor need
for a central source for sequences.
Also I don't see it as an alternative to TOA, I rather expect it to
work nicely together: when TOA is enabled you could trust the
originating sequence source rather than generate a per-entry sequence,
and in neither case you need to actually use a Lock.
I haven't thought how the sequences would need to interact (if they
need), but they seem complementary to resolve different aspects, and
also both benefit from the same cleanup and basic structure.
> Another aspect is that the "user thread" on the primary
owner needs to
> wait (at least until we improve further) and only proceed after ACK
> from backup nodes, but this is better modelled through a state
> machine. (Also discussed in Farnborough).
To be clear, I don't think keeping the user thread on the originator blocked
until we have the write confirmations from all the backups is a problem - a
sync operation has to block, and it also serves to rate-limit user
operations.
There are better ways to rate-limit than to make all operations slow;
we don't need to block a thread, we need to react on the reply from
the backup owners.
You still have an inherent rate-limit in the outgoing packet queues:
if these fill up, then and only then it's nice to introduce some back
pressure.
The problem appears when the originator is not the primary owner, and
the
thread blocking for backup ACKs is from the remote-executor pool (or OOB,
when the remote-executor pool is exhausted).
Not following. I guess this is out of scope now that I clarified the
proposed solution is only to be applied between primary and backups?
>
> It's also conceptually linked to:
> - https://issues.jboss.org/browse/ISPN-1599
> As you need to separate the locks of entries from the effective user
> facing lock, at least to implement transactions on top of this model.
I think we fixed ISPN-1599 when we changed passivation to use
DataContainer.compute(). WDYT Pedro, is there anything else you'd like to do
in the scope of ISPN-1599?
>
> I expect this to improve performance in a very significant way, but
> it's getting embarrassing that it's still not done; at the next face
> to face meeting we should also reserve some time for retrospective
> sessions.
Implementing the state machine-based interceptor stack may give us a
performance boost, but I'm much more certain that it's a very complex, high
risk task... and we don't have a stable test suite yet :)
Cleaning up and removing some complexity such as
TooManyExecutorsException might help to get it stable, and keep it
there :)
BTW it was quite stable for me until you changed the JGroups UDP
default configuration.
Sanne
>
>
> Sanne
>
> On 29 July 2014 15:50, Bela Ban <bban(a)redhat.com> wrote:
> >
> >
> > On 29/07/14 16:42, Dan Berindei wrote:
> >> Have you tried regular optimistic/pessimistic transactions as well?
> >
> > Yes, in my first impl. but since I'm making only 1 change per request, I
> > thought a TX is overkill.
> >
> >> They *should* have less issues with the OOB thread pool than non-tx
> >> mode, and
> >> I'm quite curious how they stack against TO in such a large cluster.
> >
> > Why would they have fewer issues with the thread pools ? AIUI, a TX
> > involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not using
> > TXs. And we're sync anyway...
> >
> >
> >> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
> >> <mailto:bban@redhat.com>> wrote:
> >>
> >> Following up on my own email, I changed the config to use Pedro's
> >> excellent total order implementation:
> >>
> >> <transaction transactionMode="TRANSACTIONAL"
> >> transactionProtocol="TOTAL_ORDER"
lockingMode="OPTIMISTIC"
> >> useEagerLocking="true"
eagerLockSingleNode="true">
> >> <recovery enabled="false"/>
> >>
> >> With 100 nodes and 25 requester threads/node, I did NOT run into
> >> any
> >> locking issues !
> >>
> >> I could even go up to 200 requester threads/node and the perf was ~
> >> 7'000-8'000 requests/sec/node. Not too bad !
> >>
> >> This really validates the concept of lockless total-order
> >> dissemination
> >> of TXs; for the first time, this has been tested on a large(r)
> >> scale
> >> (previously only on 25 nodes) and IT WORKS ! :-)
> >>
> >> I still believe we should implement my suggested solution for
> >> non-TO
> >> configs, but short of configuring thread pools of 1000 threads or
> >> higher, I hope TO will allow me to finally test a 500 node
> >> Infinispan
> >> cluster !
> >>
> >>
> >> On 29/07/14 15:56, Bela Ban wrote:
> >> > Hi guys,
> >> >
> >> > sorry for the long post, but I do think I ran into an important
> >> problem
> >> > and we need to fix it ... :-)
> >> >
> >> > I've spent the last couple of days running the IspnPerfTest
[1]
> >> perftest
> >> > on Google Compute Engine (GCE), and I've run into a problem
with
> >> > Infinispan. It is a design problem and can be mitigated by
> >> sizing
> >> thread
> >> > pools correctly, but cannot be eliminated entirely.
> >> >
> >> >
> >> > Symptom:
> >> > --------
> >> > IspnPerfTest has every node in a cluster perform 20'000
requests
> >> on keys
> >> > in range [1..20000].
> >> >
> >> > 80% of the requests are reads and 20% writes.
> >> >
> >> > By default, we have 25 requester threads per node and 100 nodes
> >> in a
> >> > cluster, so a total of 2500 requester threads.
> >> >
> >> > The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
> >> >
> >> > <namedCache name="clusteredCache">
> >> > <clustering mode="distribution">
> >> > <stateTransfer
awaitInitialTransfer="true"/>
> >> > <hash numOwners="2"/>
> >> > <sync replTimeout="20000"/>
> >> > </clustering>
> >> >
> >> > <transaction
transactionMode="NON_TRANSACTIONAL"
> >> > useEagerLocking="true"
> >> > eagerLockSingleNode="true" />
> >> > <locking lockAcquisitionTimeout="5000"
> >> concurrencyLevel="1000"
> >> > isolationLevel="READ_COMMITTED"
> >> useLockStriping="false" />
> >> > </namedCache>
> >> >
> >> > It has 2 owners, a lock acquisition timeout of 5s and a repl
> >> timeout of
> >> > 20s. Lock stripting is off, so we have 1 lock per key.
> >> >
> >> > When I run the test, I always get errors like those below:
> >> >
> >> > org.infinispan.util.concurrent.TimeoutException: Unable to
> >> acquire lock
> >> > after [10 seconds] on key [19386] for requestor
> >> [Thread[invoker-3,5,main]]!
> >> > Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
> >> >
> >> > and
> >> >
> >> > org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed
> >> out
> >> >
> >> >
> >> > Investigation:
> >> > ------------
> >> > When I looked at UNICAST3, I saw a lot of missing messages on
> >> the
> >> > receive side and unacked messages on the send side. This caused
> >> me to
> >> > look into the (mainly OOB) thread pools and - voila - maxed out
> >> !
> >> >
> >> > I learned from Pedro that the Infinispan internal thread pool
> >> (with a
> >> > default of 32 threads) can be configured, so I increased it to
> >> 300 and
> >> > increased the OOB pools as well.
> >> >
> >> > This mitigated the problem somewhat, but when I increased the
> >> requester
> >> > threads to 100, I had the same problem again. Apparently, the
> >> Infinispan
> >> > internal thread pool uses a rejection policy of "run"
and thus
> >> uses the
> >> > JGroups (OOB) thread when exhausted.
> >> >
> >> > I learned (from Pedro and Mircea) that GETs and PUTs work as
> >> follows in
> >> > dist-sync / 2 owners:
> >> > - GETs are sent to the primary and backup owners and the first
> >> response
> >> > received is returned to the caller. No locks are acquired, so
> >> GETs
> >> > shouldn't cause problems.
> >> >
> >> > - A PUT(K) is sent to the primary owner of K
> >> > - The primary owner
> >> > (1) locks K
> >> > (2) updates the backup owner synchronously *while holding
> >> the lock*
> >> > (3) releases the lock
> >> >
> >> >
> >> > Hypothesis
> >> > ----------
> >> > (2) above is done while holding the lock. The sync update of the
> >> backup
> >> > owner is done with the lock held to guarantee that the primary
> >> and
> >> > backup owner of K have the same values for K.
> >> >
> >> > However, the sync update *inside the lock scope* slows things
> >> down (can
> >> > it also lead to deadlocks?); there's the risk that the
request
> >> is
> >> > dropped due to a full incoming thread pool, or that the response
> >> is not
> >> > received because of the same, or that the locking at the backup
> >> owner
> >> > blocks for some time.
> >> >
> >> > If we have many threads modifying the same key, then we have a
> >> backlog
> >> > of locking work against that key. Say we have 100 requester
> >> threads and
> >> > a 100 node cluster. This means that we have 10'000 threads
> >> accessing
> >> > keys; with 2'000 writers there's a big chance that some
writers
> >> pick the
> >> > same key at the same time.
> >> >
> >> > For example, if we have 100 threads accessing key K and it takes
> >> 3ms to
> >> > replicate K to the backup owner, then the last of the 100
> >> threads
> >> waits
> >> > ~300ms before it gets a chance to lock K on the primary owner
> >> and
> >> > replicate it as well.
> >> >
> >> > Just a small hiccup in sending the PUT to the primary owner,
> >> sending the
> >> > modification to the backup owner, waitting for the response, or
> >> GC, and
> >> > the delay will quickly become bigger.
> >> >
> >> >
> >> > Verification
> >> > ----------
> >> > To verify the above, I set numOwners to 1. This means that the
> >> primary
> >> > owner of K does *not* send the modification to the backup owner,
> >> it only
> >> > locks K, modifies K and unlocks K again.
> >> >
> >> > I ran the IspnPerfTest again on 100 nodes, with 25 requesters,
> >> and NO
> >> > PROBLEM !
> >> >
> >> > I then increased the requesters to 100, 150 and 200 and the test
> >> > completed flawlessly ! Performance was around *40'000
requests
> >> per node
> >> > per sec* on 4-core boxes !
> >> >
> >> >
> >> > Root cause
> >> > ---------
> >> > *******************
> >> > The root cause is the sync RPC of K to the backup owner(s) of K
> >> while
> >> > the primary owner holds the lock for K.
> >> > *******************
> >> >
> >> > This causes a backlog of threads waiting for the lock and that
> >> backlog
> >> > can grow to exhaust the thread pools. First the Infinispan
> >> internal
> >> > thread pool, then the JGroups OOB thread pool. The latter causes
> >> > retransmissions to get dropped, which compounds the problem...
> >> >
> >> >
> >> > Goal
> >> > ----
> >> > The goal is to make sure that primary and backup owner(s) of K
> >> have the
> >> > same value for K.
> >> >
> >> > Simply sending the modification to the backup owner(s)
> >> asynchronously
> >> > won't guarantee this, as modification messages might get
> >> processed out
> >> > of order as they're OOB !
> >> >
> >> >
> >> > Suggested solution
> >> > ----------------
> >> > The modification RPC needs to be invoked *outside of the lock
> >> scope*:
> >> > - lock K
> >> > - modify K
> >> > - unlock K
> >> > - send modification to backup owner(s) // outside the lock scope
> >> >
> >> > The primary owner puts the modification of K into a queue from
> >> where a
> >> > separate thread/task removes it. The thread then invokes the
> >> PUT(K) on
> >> > the backup owner(s).
> >> >
> >> > The queue has the modified keys in FIFO order, so the
> >> modifications
> >> > arrive at the backup owner(s) in the right order.
> >> >
> >> > This requires that the way GET is implemented changes slightly:
> >> instead
> >> > of invoking a GET on all owners of K, we only invoke it on the
> >> primary
> >> > owner, then the next-in-line etc.
> >> >
> >> > The reason for this is that the backup owner(s) may not yet have
> >> > received the modification of K.
> >> >
> >> > This is a better impl anyway (we discussed this before) becuse
> >> it
> >> > generates less traffic; in the normal case, all but 1 GET
> >> requests are
> >> > unnecessary.
> >> >
> >> >
> >> >
> >> > Improvement
> >> > -----------
> >> > The above solution can be simplified and even made more
> >> efficient.
> >> > Re-using concepts from IRAC [2], we can simply store the
> >> modified
> >> *keys*
> >> > in the modification queue. The modification replication thread
> >> removes
> >> > the key, gets the current value and invokes a PUT/REMOVE on the
> >> backup
> >> > owner(s).
> >> >
> >> > Even better: a key is only ever added *once*, so if we have
> >> [5,2,17,3],
> >> > adding key 2 is a no-op because the processing of key 2 (in
> >> second
> >> > position in the queue) will fetch the up-to-date value anyway !
> >> >
> >> >
> >> > Misc
> >> > ----
> >> > - Could we possibly use total order to send the updates in TO ?
> >> TBD (Pedro?)
> >> >
> >> >
> >> > Thoughts ?
> >> >
> >> >
> >> > [1] https://github.com/belaban/IspnPerfTest
> >> > [2]
> >> >
> >>
> >>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
> >> >
> >>
> >> --
> >> Bela Ban, JGroups lead (http://www.jgroups.org)
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> <mailto:infinispan-dev@lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> > --
> > Bela Ban, JGroups lead (http://www.jgroups.org)
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
10:02 a.m.
On Wed, Jul 30, 2014 at 12:35 AM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
On 29 July 2014 22:14, Dan Berindei <dan.berindei(a)gmail.com>
wrote:
>
> On Tue, Jul 29, 2014 at 9:06 PM, Sanne Grinovero <sanne(a)infinispan.org>
> wrote:
>>
>> This is a nasty problem and I also feel passionately we need to get
>> rid of it ASAP.
>> I did have the same problems many times, and we discussed this also in
>> Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
>>
>> You don't need TO, and you don't need to lock at all as long as you
>> guarantee the backup owners are getting the number with some
>> monotonicity sequence attached to it,
>> all that backup owners need to do is ignore incoming commands which
>> are outdated.
>
>
> This is more or less what TOA does - assign a monotonic sequence number
to
> txs, and only apply them after all the previous txs in the sequence have
> been applied. The problem is getting that monotonic sequence when there
are
> multiple originators and multiple primary owners also requires some extra
> RPCs.
Let's not mix this up with requirements for other areas.
The strategy I've proposed is only to be applied for the communication
from the primary owner to its backups:
the value to be written is well known as it's the primary owner which
defines it unilaterally (for example if there is an atomic replacement
to be computed)
and there is no need for extra RPCs as the sequence is not related to
a group of changes but for the specific entry only.
There is no such thing as a need for consensus across owners, nor need
for a central source for sequences.
Just to make sure I understand correctly: your proposal is that in non-tx
caches, the primary owner should generate some sort of version number while
holding the entry lock, and replicate the write to the backup owners
synchronously, but without holding the lock? Then the backup owners would
check the version of the entry and only apply a newer write?
if your proposal is only meant to apply to non-tx caches, you are right you
don't have to worry about multiple primary owners... most of the time. But
when the primary owner changes, then you do have 2 primary owners (if the
new primary owner installs the new topology first), and you do need to
coordinate between the 2.
Slightly related: we also considered generating a version number on the
client for consistency when the HotRod client retries after a primary owner
failure [1]. But the clients can't create a monotonic sequence number, so
we couldn't use that version number for this.
[1] https://issues.jboss.org/browse/ISPN-2956
Also I don't see it as an alternative to TOA, I rather expect it
to
work nicely together: when TOA is enabled you could trust the
originating sequence source rather than generate a per-entry sequence,
and in neither case you need to actually use a Lock.
I haven't thought how the sequences would need to interact (if they
need), but they seem complementary to resolve different aspects, and
also both benefit from the same cleanup and basic structure.
We don't acquire locks at all on the backup owners - either in tx or non-tx
caches. If state transfer is in progress, we use
ConcurrentHashMap.compute() to store tracking information, which uses a
synchronized block, so I suppose we do acquire locks. I assume your
proposal would require a DataContainer.compute() or something similar on
the backups, to ensure that the version check and the replacement are
atomic.
I still think TOA does what you want for tx caches. Your proposal would
only work for non-tx caches, so you couldn't use them together.
>> Another aspect is that the "user thread" on the
primary owner needs to
>> wait (at least until we improve further) and only proceed after ACK
>> from backup nodes, but this is better modelled through a state
>> machine. (Also discussed in Farnborough).
>
>
> To be clear, I don't think keeping the user thread on the originator
blocked
> until we have the write confirmations from all the backups is a problem
- a
> sync operation has to block, and it also serves to rate-limit user
> operations.
There are better ways to rate-limit than to make all operations slow;
we don't need to block a thread, we need to react on the reply from
the backup owners.
You still have an inherent rate-limit in the outgoing packet queues:
if these fill up, then and only then it's nice to introduce some back
pressure.
Sorry, you got me confused when you called the thread on the primary owner
a "user thread". I agree that internal stuff can and should be
asynchronous, callback based, but the user still has to see a synchronous
blocking operation.
> The problem appears when the originator is not the primary
owner, and the
> thread blocking for backup ACKs is from the remote-executor pool (or OOB,
> when the remote-executor pool is exhausted).
Not following. I guess this is out of scope now that I clarified the
proposed solution is only to be applied between primary and backups?
Yeah, I was just trying to clarify that there is no danger of exhausting
the remote executor/OOB thread pools when the originator of the write
command is the primary owner (as it happens in the HotRod server).
>>
>> It's also conceptually linked to:
>> - https://issues.jboss.org/browse/ISPN-1599
>> As you need to separate the locks of entries from the effective user
>> facing lock, at least to implement transactions on top of this model.
>
>
> I think we fixed ISPN-1599 when we changed passivation to use
> DataContainer.compute(). WDYT Pedro, is there anything else you'd like
to do
> in the scope of ISPN-1599?
>
>>
>> I expect this to improve performance in a very significant way, but
>> it's getting embarrassing that it's still not done; at the next face
>> to face meeting we should also reserve some time for retrospective
>> sessions.
>
>
> Implementing the state machine-based interceptor stack may give us a
> performance boost, but I'm much more certain that it's a very complex,
high
> risk task... and we don't have a stable test suite yet :)
Cleaning up and removing some complexity such as
TooManyExecutorsException might help to get it stable, and keep it
there :)
BTW it was quite stable for me until you changed the JGroups UDP
default configuration.
Do you really use UDP to run the tests? The default is TCP, but maybe the
some tests doesn't use TestCacheManagerFactory...
I was just aligning our configs with Bela's recommandations: MERGE3 instead
of MERGE2 and the removal of UFC in TCP stacks. If they cause problems on
your machine, you should make more noise :)
Dan
Sanne
>
>
>>
>>
>> Sanne
>>
>> On 29 July 2014 15:50, Bela Ban <bban(a)redhat.com> wrote:
>> >
>> >
>> > On 29/07/14 16:42, Dan Berindei wrote:
>> >> Have you tried regular optimistic/pessimistic transactions as well?
>> >
>> > Yes, in my first impl. but since I'm making only 1 change per
request, I
>> > thought a TX is overkill.
>> >
>> >> They *should* have less issues with the OOB thread pool than non-tx
>> >> mode, and
>> >> I'm quite curious how they stack against TO in such a large
cluster.
>> >
>> > Why would they have fewer issues with the thread pools ? AIUI, a TX
>> > involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not
using
>> > TXs. And we're sync anyway...
>> >
>> >
>> >> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
>> >> <mailto:bban@redhat.com>> wrote:
>> >>
>> >> Following up on my own email, I changed the config to use
Pedro's
>> >> excellent total order implementation:
>> >>
>> >> <transaction transactionMode="TRANSACTIONAL"
>> >> transactionProtocol="TOTAL_ORDER"
lockingMode="OPTIMISTIC"
>> >> useEagerLocking="true"
eagerLockSingleNode="true">
>> >> <recovery enabled="false"/>
>> >>
>> >> With 100 nodes and 25 requester threads/node, I did NOT run into
>> >> any
>> >> locking issues !
>> >>
>> >> I could even go up to 200 requester threads/node and the perf
was ~
>> >> 7'000-8'000 requests/sec/node. Not too bad !
>> >>
>> >> This really validates the concept of lockless total-order
>> >> dissemination
>> >> of TXs; for the first time, this has been tested on a large(r)
>> >> scale
>> >> (previously only on 25 nodes) and IT WORKS ! :-)
>> >>
>> >> I still believe we should implement my suggested solution for
>> >> non-TO
>> >> configs, but short of configuring thread pools of 1000 threads or
>> >> higher, I hope TO will allow me to finally test a 500 node
>> >> Infinispan
>> >> cluster !
>> >>
>> >>
>> >> On 29/07/14 15:56, Bela Ban wrote:
>> >> > Hi guys,
>> >> >
>> >> > sorry for the long post, but I do think I ran into an
important
>> >> problem
>> >> > and we need to fix it ... :-)
>> >> >
>> >> > I've spent the last couple of days running the
IspnPerfTest
[1]
>> >> perftest
>> >> > on Google Compute Engine (GCE), and I've run into a
problem
with
>> >> > Infinispan. It is a design problem and can be mitigated by
>> >> sizing
>> >> thread
>> >> > pools correctly, but cannot be eliminated entirely.
>> >> >
>> >> >
>> >> > Symptom:
>> >> > --------
>> >> > IspnPerfTest has every node in a cluster perform 20'000
requests
>> >> on keys
>> >> > in range [1..20000].
>> >> >
>> >> > 80% of the requests are reads and 20% writes.
>> >> >
>> >> > By default, we have 25 requester threads per node and 100
nodes
>> >> in a
>> >> > cluster, so a total of 2500 requester threads.
>> >> >
>> >> > The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
>> >> >
>> >> > <namedCache name="clusteredCache">
>> >> > <clustering mode="distribution">
>> >> > <stateTransfer
awaitInitialTransfer="true"/>
>> >> > <hash numOwners="2"/>
>> >> > <sync replTimeout="20000"/>
>> >> > </clustering>
>> >> >
>> >> > <transaction
transactionMode="NON_TRANSACTIONAL"
>> >> > useEagerLocking="true"
>> >> > eagerLockSingleNode="true" />
>> >> > <locking lockAcquisitionTimeout="5000"
>> >> concurrencyLevel="1000"
>> >> > isolationLevel="READ_COMMITTED"
>> >> useLockStriping="false" />
>> >> > </namedCache>
>> >> >
>> >> > It has 2 owners, a lock acquisition timeout of 5s and a repl
>> >> timeout of
>> >> > 20s. Lock stripting is off, so we have 1 lock per key.
>> >> >
>> >> > When I run the test, I always get errors like those below:
>> >> >
>> >> > org.infinispan.util.concurrent.TimeoutException: Unable to
>> >> acquire lock
>> >> > after [10 seconds] on key [19386] for requestor
>> >> [Thread[invoker-3,5,main]]!
>> >> > Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
>> >> >
>> >> > and
>> >> >
>> >> > org.infinispan.util.concurrent.TimeoutException: Node m8.1
timed
>> >> out
>> >> >
>> >> >
>> >> > Investigation:
>> >> > ------------
>> >> > When I looked at UNICAST3, I saw a lot of missing messages
on
>> >> the
>> >> > receive side and unacked messages on the send side. This
caused
>> >> me to
>> >> > look into the (mainly OOB) thread pools and - voila - maxed
out
>> >> !
>> >> >
>> >> > I learned from Pedro that the Infinispan internal thread
pool
>> >> (with a
>> >> > default of 32 threads) can be configured, so I increased it
to
>> >> 300 and
>> >> > increased the OOB pools as well.
>> >> >
>> >> > This mitigated the problem somewhat, but when I increased
the
>> >> requester
>> >> > threads to 100, I had the same problem again. Apparently,
the
>> >> Infinispan
>> >> > internal thread pool uses a rejection policy of
"run" and thus
>> >> uses the
>> >> > JGroups (OOB) thread when exhausted.
>> >> >
>> >> > I learned (from Pedro and Mircea) that GETs and PUTs work as
>> >> follows in
>> >> > dist-sync / 2 owners:
>> >> > - GETs are sent to the primary and backup owners and the
first
>> >> response
>> >> > received is returned to the caller. No locks are acquired,
so
>> >> GETs
>> >> > shouldn't cause problems.
>> >> >
>> >> > - A PUT(K) is sent to the primary owner of K
>> >> > - The primary owner
>> >> > (1) locks K
>> >> > (2) updates the backup owner synchronously *while
holding
>> >> the lock*
>> >> > (3) releases the lock
>> >> >
>> >> >
>> >> > Hypothesis
>> >> > ----------
>> >> > (2) above is done while holding the lock. The sync update of
the
>> >> backup
>> >> > owner is done with the lock held to guarantee that the
primary
>> >> and
>> >> > backup owner of K have the same values for K.
>> >> >
>> >> > However, the sync update *inside the lock scope* slows
things
>> >> down (can
>> >> > it also lead to deadlocks?); there's the risk that the
request
>> >> is
>> >> > dropped due to a full incoming thread pool, or that the
response
>> >> is not
>> >> > received because of the same, or that the locking at the
backup
>> >> owner
>> >> > blocks for some time.
>> >> >
>> >> > If we have many threads modifying the same key, then we have
a
>> >> backlog
>> >> > of locking work against that key. Say we have 100 requester
>> >> threads and
>> >> > a 100 node cluster. This means that we have 10'000
threads
>> >> accessing
>> >> > keys; with 2'000 writers there's a big chance that
some
writers
>> >> pick the
>> >> > same key at the same time.
>> >> >
>> >> > For example, if we have 100 threads accessing key K and it
takes
>> >> 3ms to
>> >> > replicate K to the backup owner, then the last of the 100
>> >> threads
>> >> waits
>> >> > ~300ms before it gets a chance to lock K on the primary
owner
>> >> and
>> >> > replicate it as well.
>> >> >
>> >> > Just a small hiccup in sending the PUT to the primary owner,
>> >> sending the
>> >> > modification to the backup owner, waitting for the response,
or
>> >> GC, and
>> >> > the delay will quickly become bigger.
>> >> >
>> >> >
>> >> > Verification
>> >> > ----------
>> >> > To verify the above, I set numOwners to 1. This means that
the
>> >> primary
>> >> > owner of K does *not* send the modification to the backup
owner,
>> >> it only
>> >> > locks K, modifies K and unlocks K again.
>> >> >
>> >> > I ran the IspnPerfTest again on 100 nodes, with 25
requesters,
>> >> and NO
>> >> > PROBLEM !
>> >> >
>> >> > I then increased the requesters to 100, 150 and 200 and the
test
>> >> > completed flawlessly ! Performance was around *40'000
requests
>> >> per node
>> >> > per sec* on 4-core boxes !
>> >> >
>> >> >
>> >> > Root cause
>> >> > ---------
>> >> > *******************
>> >> > The root cause is the sync RPC of K to the backup owner(s)
of
K
>> >> while
>> >> > the primary owner holds the lock for K.
>> >> > *******************
>> >> >
>> >> > This causes a backlog of threads waiting for the lock and
that
>> >> backlog
>> >> > can grow to exhaust the thread pools. First the Infinispan
>> >> internal
>> >> > thread pool, then the JGroups OOB thread pool. The latter
causes
>> >> > retransmissions to get dropped, which compounds the
problem...
>> >> >
>> >> >
>> >> > Goal
>> >> > ----
>> >> > The goal is to make sure that primary and backup owner(s) of
K
>> >> have the
>> >> > same value for K.
>> >> >
>> >> > Simply sending the modification to the backup owner(s)
>> >> asynchronously
>> >> > won't guarantee this, as modification messages might get
>> >> processed out
>> >> > of order as they're OOB !
>> >> >
>> >> >
>> >> > Suggested solution
>> >> > ----------------
>> >> > The modification RPC needs to be invoked *outside of the
lock
>> >> scope*:
>> >> > - lock K
>> >> > - modify K
>> >> > - unlock K
>> >> > - send modification to backup owner(s) // outside the lock
scope
>> >> >
>> >> > The primary owner puts the modification of K into a queue
from
>> >> where a
>> >> > separate thread/task removes it. The thread then invokes the
>> >> PUT(K) on
>> >> > the backup owner(s).
>> >> >
>> >> > The queue has the modified keys in FIFO order, so the
>> >> modifications
>> >> > arrive at the backup owner(s) in the right order.
>> >> >
>> >> > This requires that the way GET is implemented changes
slightly:
>> >> instead
>> >> > of invoking a GET on all owners of K, we only invoke it on
the
>> >> primary
>> >> > owner, then the next-in-line etc.
>> >> >
>> >> > The reason for this is that the backup owner(s) may not yet
have
>> >> > received the modification of K.
>> >> >
>> >> > This is a better impl anyway (we discussed this before)
becuse
>> >> it
>> >> > generates less traffic; in the normal case, all but 1 GET
>> >> requests are
>> >> > unnecessary.
>> >> >
>> >> >
>> >> >
>> >> > Improvement
>> >> > -----------
>> >> > The above solution can be simplified and even made more
>> >> efficient.
>> >> > Re-using concepts from IRAC [2], we can simply store the
>> >> modified
>> >> *keys*
>> >> > in the modification queue. The modification replication
thread
>> >> removes
>> >> > the key, gets the current value and invokes a PUT/REMOVE on
the
>> >> backup
>> >> > owner(s).
>> >> >
>> >> > Even better: a key is only ever added *once*, so if we have
>> >> [5,2,17,3],
>> >> > adding key 2 is a no-op because the processing of key 2 (in
>> >> second
>> >> > position in the queue) will fetch the up-to-date value
anyway
!
>> >> >
>> >> >
>> >> > Misc
>> >> > ----
>> >> > - Could we possibly use total order to send the updates in
TO
?
>> >> TBD (Pedro?)
>> >> >
>> >> >
>> >> > Thoughts ?
>> >> >
>> >> >
>> >> > [1] https://github.com/belaban/IspnPerfTest
>> >> > [2]
>> >> >
>> >>
>> >>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
>> >> >
>> >>
>> >> --
>> >> Bela Ban, JGroups lead (http://www.jgroups.org)
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev(a)lists.jboss.org
>> >> <mailto:infinispan-dev@lists.jboss.org>
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev(a)lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >
>> > --
>> > Bela Ban, JGroups lead (http://www.jgroups.org)
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11 a.m.
On 07/30/2014 09:02 AM, Dan Berindei wrote:
> if your proposal is only meant to apply to non-tx caches,
you are right
> you don't have to worry about multiple primary owners... most of the
> time. But when the primary owner changes, then you do have 2 primary
> owners (if the new primary owner installs the new topology first), and
> you do need to coordinate between the 2.
I think it is the same for transactional cache. I.e. the commands wait
for the transaction data from the new topology to be installed. In the
non-tx caches, the old primary owner will send the next "sequence
number" to the new primary owner and only after that, the new primary
owner starts to give the orders.
Otherwise, I can implement a total order version for non-tx caches and
all the write serialization would be done in JGroups and Infinispan only
has to apply the updates as soon as they are delivered.
> Slightly related: we also considered generating a version number on the
> client for consistency when the HotRod client retries after a primary
> owner failure [1]. But the clients can't create a monotonic sequence
> number, so we couldn't use that version number for this.
> [1] https://issues.jboss.org/browse/ISPN-2956
> Also I don't see it as
an alternative to TOA, I rather expect it to
> work nicely together: when TOA is enabled you could trust the
> originating sequence source rather than generate a per-entry sequence,
> and in neither case you need to actually use a Lock.
> I haven't thought how the sequences would need to interact (if they
> need), but they seem complementary to resolve different aspects, and
> also both benefit from the same cleanup and basic structure.
> We don't acquire locks at
all on the backup owners - either in tx or
> non-tx caches. If state transfer is in progress, we use
> ConcurrentHashMap.compute() to store tracking information, which uses a
> synchronized block, so I suppose we do acquire locks. I assume your
> proposal would require a DataContainer.compute() or something similar on
> the backups, to ensure that the version check and the replacement are
> atomic.
> I still think TOA does what you want for tx caches. Your
proposal would
> only work for non-tx caches, so you couldn't use them together.
> >> Another aspect is
that the "user thread" on the primary owner
> needs to
> >> wait (at least until we improve further) and only proceed after ACK
> >> from backup nodes, but this is better modelled through a state
> >> machine. (Also discussed in Farnborough).
> > > > To be clear, I don't think keeping the user
thread on the
> originator blocked
> > until we have the write confirmations from all the backups is a
> problem - a
> > sync operation has to block, and it also serves to rate-limit user
> > operations.
> There are better ways to
rate-limit than to make all operations slow;
> we don't need to block a thread, we need to react on the reply from
> the backup owners.
> You still have an inherent rate-limit in the outgoing packet queues:
> if these fill up, then and only then it's nice to introduce some back
> pressure.
> Sorry, you got me confused when
you called the thread on the primary
> owner a "user thread". I agree that internal stuff can and should be
> asynchronous, callback based, but the user still has to see a
> synchronous blocking operation.
> > The problem appears
when the originator is not the primary owner,
> and the
> > thread blocking for backup ACKs is from the remote-executor pool
> (or OOB,
> > when the remote-executor pool is exhausted).
> Not following. I guess this is out of scope now that I
clarified the
> proposed solution is only to be applied between primary and backups?
> Yeah, I was just trying to
clarify that there is no danger of exhausting
> the remote executor/OOB thread pools when the originator of the write
> command is the primary owner (as it happens in the HotRod server).
> > > >> It's also conceptually linked to:
> >> - https://issues.jboss.org/browse/ISPN-1599
> >> As you need to separate the locks of entries from the effective user
> >> facing lock, at least to implement transactions on top of this
> model.
> > > > I think we fixed ISPN-1599 when we changed
passivation to use
> > DataContainer.compute(). WDYT Pedro, is there anything else you'd
> like to do
> > in the scope of ISPN-1599?
> > > > >> I expect this to improve performance in a
very significant way, but
> >> it's getting embarrassing that it's still not done; at the next
face
> >> to face meeting we should also reserve some time for retrospective
> >> sessions.
> > > > Implementing the state machine-based interceptor
stack may give us a
> > performance boost, but I'm much more certain that it's a very
> complex, high
> > risk task... and we don't have a stable test suite yet :)
> Cleaning up and removing some complexity such as
> TooManyExecutorsException might help to get it stable, and keep it
> there :)
> BTW it was quite stable for me until you changed the JGroups UDP
> default configuration.
> Do you really use UDP to run
the tests? The default is TCP, but maybe
> the some tests doesn't use TestCacheManagerFactory...
> I was just aligning our configs with Bela's
recommandations: MERGE3
> instead of MERGE2 and the removal of UFC in TCP stacks. If they cause
> problems on your machine, you should make more noise :)
> Dan
> Sanne
> >
> > >
> > >> Sanne
> > > >> On 29 July 2014
15:50, Bela Ban <bban(a)redhat.com
> <mailto:bban@redhat.com>> wrote:
> >> > >> > >> > On 29/07/14 16:42, Dan Berindei wrote:
> >> >> Have you tried regular optimistic/pessimistic transactions as
> well?
> >> > >> > Yes, in my
first impl. but since I'm making only 1 change per
> request, I
> >> > thought a TX is overkill.
> >> > >> >> They
*should* have less issues with the OOB thread pool than
> non-tx
> >> >> mode, and
> >> >> I'm quite curious how they stack against TO in such a
large
> cluster.
> >> > >> > Why would
they have fewer issues with the thread pools ? AIUI,
> a TX
> >> > involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when
> not using
> >> > TXs. And we're sync anyway...
> >> > >> > >> >> On Tue, Jul 29, 2014 at 5:38 PM,
Bela Ban <bban(a)redhat.com
> <mailto:bban@redhat.com >
>> >> <mailto:bban@redhat.com <mailto:bban@redhat.com>>>
wrote:
> >> > > >> >>
Following up on my own email, I changed the config to use
> Pedro's
> >> >> excellent total order implementation:
> >> > > >> >>
<transaction transactionMode="TRANSACTIONAL"
> >> >> transactionProtocol="TOTAL_ORDER"
lockingMode="OPTIMISTIC"
> >> >> useEagerLocking="true"
eagerLockSingleNode="true" >
>> >> <recovery enabled="false"/ > >> > >
>> >> With 100 nodes and 25 requester threads/node, I did NOT
> run into
> >> >> any
> >> >> locking issues !
> >> > > >> >> I
could even go up to 200 requester threads/node and the
> perf was ~
> >> >> 7'000-8'000 requests/sec/node. Not too bad !
> >> > > >> >> This
really validates the concept of lockless total-order
> >> >> dissemination
> >> >> of TXs; for the first time, this has been tested on a
> large(r)
> >> >> scale
> >> >> (previously only on 25 nodes) and IT WORKS ! :-)
> >> > > >> >> I
still believe we should implement my suggested solution for
> >> >> non-TO
> >> >> configs, but short of configuring thread pools of 1000
> threads or
> >> >> higher, I hope TO will allow me to finally test a 500
node
> >> >> Infinispan
> >> >> cluster !
> >> > > >> > > >> >> On 29/07/14 15:56, Bela Ban
wrote:
> >> >> > Hi guys,
> >> >> > >> >>
> sorry for the long post, but I do think I ran into an
> important
> >> >> problem
> >> >> > and we need to fix it ... :-)
> >> >> > >> >>
> I've spent the last couple of days running the
> IspnPerfTest [1]
> >> >> perftest
> >> >> > on Google Compute Engine (GCE), and I've run
into a
> problem with
> >> >> > Infinispan. It is a design problem and can be
mitigated by
> >> >> sizing
> >> >> thread
> >> >> > pools correctly, but cannot be eliminated entirely.
> >> >> > >> >>
> >> >> > Symptom:
> >> >> > --------
> >> >> > IspnPerfTest has every node in a cluster perform
> 20'000 requests
> >> >> on keys
> >> >> > in range [1..20000].
> >> >> > >> >>
> 80% of the requests are reads and 20% writes.
> >> >> > >> >>
> By default, we have 25 requester threads per node and
> 100 nodes
> >> >> in a
> >> >> > cluster, so a total of 2500 requester threads.
> >> >> > >> >>
> The cache used is NON-TRANSACTIONAL / dist-sync / 2
> owners:
> >> >> > >> >>
> <namedCache name="clusteredCache" >
>> >> > <clustering mode="distribution" > >> >> >
<stateTransfer awaitInitialTransfer="true"/ >
>> >> > <hash numOwners="2"/ > >> >> > <sync
replTimeout="20000"/ > >> >>
> </clustering > >> >>
> >> >> > <transaction
transactionMode="NON_TRANSACTIONAL"
> >> >> > useEagerLocking="true"
> >> >> > eagerLockSingleNode="true"
/ > >> >> > <locking
lockAcquisitionTimeout="5000"
> >> >> concurrencyLevel="1000"
> >> >> >
isolationLevel="READ_COMMITTED"
> >> >> useLockStriping="false" / > >> >> > </namedCache > >> >> >
>> >> > It has 2 owners, a lock acquisition timeout of 5s and
> a repl
> >> >> timeout of
> >> >> > 20s. Lock stripting is off, so we have 1 lock per
key.
> >> >> > >> >>
> When I run the test, I always get errors like those below:
> >> >> > >> >>
> org.infinispan.util.concurrent.TimeoutException: Unable to
> >> >> acquire lock
> >> >> > after [10 seconds] on key [19386] for requestor
> >> >> [Thread[invoker-3,5,main]]!
> >> >> > Lock held by
[Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
> >> >> > >> >>
> and
> >> >> > >> >>
> org.infinispan.util.concurrent.TimeoutException: Node
> m8.1 timed
> >> >> out
> >> >> > >> >>
> >> >> > Investigation:
> >> >> > ------------
> >> >> > When I looked at UNICAST3, I saw a lot of missing
> messages on
> >> >> the
> >> >> > receive side and unacked messages on the send side.
> This caused
> >> >> me to
> >> >> > look into the (mainly OOB) thread pools and - voila
-
> maxed out
> >> >> !
> >> >> > >> >>
> I learned from Pedro that the Infinispan internal
> thread pool
> >> >> (with a
> >> >> > default of 32 threads) can be configured, so I
> increased it to
> >> >> 300 and
> >> >> > increased the OOB pools as well.
> >> >> > >> >>
> This mitigated the problem somewhat, but when I
> increased the
> >> >> requester
> >> >> > threads to 100, I had the same problem again.
> Apparently, the
> >> >> Infinispan
> >> >> > internal thread pool uses a rejection policy of
"run"
> and thus
> >> >> uses the
> >> >> > JGroups (OOB) thread when exhausted.
> >> >> > >> >>
> I learned (from Pedro and Mircea) that GETs and PUTs
> work as
> >> >> follows in
> >> >> > dist-sync / 2 owners:
> >> >> > - GETs are sent to the primary and backup owners
and
> the first
> >> >> response
> >> >> > received is returned to the caller. No locks are
> acquired, so
> >> >> GETs
> >> >> > shouldn't cause problems.
> >> >> > >> >>
> - A PUT(K) is sent to the primary owner of K
> >> >> > - The primary owner
> >> >> > (1) locks K
> >> >> > (2) updates the backup owner synchronously
> *while holding
> >> >> the lock*
> >> >> > (3) releases the lock
> >> >> > >> >>
> >> >> > Hypothesis
> >> >> > ----------
> >> >> > (2) above is done while holding the lock. The sync
> update of the
> >> >> backup
> >> >> > owner is done with the lock held to guarantee that
the
> primary
> >> >> and
> >> >> > backup owner of K have the same values for K.
> >> >> > >> >>
> However, the sync update *inside the lock scope* slows
> things
> >> >> down (can
> >> >> > it also lead to deadlocks?); there's the risk
that the
> request
> >> >> is
> >> >> > dropped due to a full incoming thread pool, or that
> the response
> >> >> is not
> >> >> > received because of the same, or that the locking
at
> the backup
> >> >> owner
> >> >> > blocks for some time.
> >> >> > >> >>
> If we have many threads modifying the same key, then
> we have a
> >> >> backlog
> >> >> > of locking work against that key. Say we have 100
> requester
> >> >> threads and
> >> >> > a 100 node cluster. This means that we have
10'000 threads
> >> >> accessing
> >> >> > keys; with 2'000 writers there's a big
chance that
> some writers
> >> >> pick the
> >> >> > same key at the same time.
> >> >> > >> >>
> For example, if we have 100 threads accessing key K
> and it takes
> >> >> 3ms to
> >> >> > replicate K to the backup owner, then the last of
the 100
> >> >> threads
> >> >> waits
> >> >> > ~300ms before it gets a chance to lock K on the
> primary owner
> >> >> and
> >> >> > replicate it as well.
> >> >> > >> >>
> Just a small hiccup in sending the PUT to the primary
> owner,
> >> >> sending the
> >> >> > modification to the backup owner, waitting for the
> response, or
> >> >> GC, and
> >> >> > the delay will quickly become bigger.
> >> >> > >> >>
> >> >> > Verification
> >> >> > ----------
> >> >> > To verify the above, I set numOwners to 1. This
means
> that the
> >> >> primary
> >> >> > owner of K does *not* send the modification to the
> backup owner,
> >> >> it only
> >> >> > locks K, modifies K and unlocks K again.
> >> >> > >> >>
> I ran the IspnPerfTest again on 100 nodes, with 25
> requesters,
> >> >> and NO
> >> >> > PROBLEM !
> >> >> > >> >>
> I then increased the requesters to 100, 150 and 200
> and the test
> >> >> > completed flawlessly ! Performance was around
*40'000
> requests
> >> >> per node
> >> >> > per sec* on 4-core boxes !
> >> >> > >> >>
> >> >> > Root cause
> >> >> > ---------
> >> >> > *******************
> >> >> > The root cause is the sync RPC of K to the backup
> owner(s) of K
> >> >> while
> >> >> > the primary owner holds the lock for K.
> >> >> > *******************
> >> >> > >> >>
> This causes a backlog of threads waiting for the lock
> and that
> >> >> backlog
> >> >> > can grow to exhaust the thread pools. First the
Infinispan
> >> >> internal
> >> >> > thread pool, then the JGroups OOB thread pool. The
> latter causes
> >> >> > retransmissions to get dropped, which compounds the
> problem...
> >> >> > >> >>
> >> >> > Goal
> >> >> > ----
> >> >> > The goal is to make sure that primary and backup
> owner(s) of K
> >> >> have the
> >> >> > same value for K.
> >> >> > >> >>
> Simply sending the modification to the backup owner(s)
> >> >> asynchronously
> >> >> > won't guarantee this, as modification messages
might get
> >> >> processed out
> >> >> > of order as they're OOB !
> >> >> > >> >>
> >> >> > Suggested solution
> >> >> > ----------------
> >> >> > The modification RPC needs to be invoked *outside
of
> the lock
> >> >> scope*:
> >> >> > - lock K
> >> >> > - modify K
> >> >> > - unlock K
> >> >> > - send modification to backup owner(s) // outside
the
> lock scope
> >> >> > >> >>
> The primary owner puts the modification of K into a
> queue from
> >> >> where a
> >> >> > separate thread/task removes it. The thread then
> invokes the
> >> >> PUT(K) on
> >> >> > the backup owner(s).
> >> >> > >> >>
> The queue has the modified keys in FIFO order, so the
> >> >> modifications
> >> >> > arrive at the backup owner(s) in the right order.
> >> >> > >> >>
> This requires that the way GET is implemented changes
> slightly:
> >> >> instead
> >> >> > of invoking a GET on all owners of K, we only
invoke
> it on the
> >> >> primary
> >> >> > owner, then the next-in-line etc.
> >> >> > >> >>
> The reason for this is that the backup owner(s) may
> not yet have
> >> >> > received the modification of K.
> >> >> > >> >>
> This is a better impl anyway (we discussed this
> before) becuse
> >> >> it
> >> >> > generates less traffic; in the normal case, all but
1 GET
> >> >> requests are
> >> >> > unnecessary.
> >> >> > >> >>
> >> >> >
>> >> > Improvement
> >> >> > -----------
> >> >> > The above solution can be simplified and even made
more
> >> >> efficient.
> >> >> > Re-using concepts from IRAC [2], we can simply store
the
> >> >> modified
> >> >> *keys*
> >> >> > in the modification queue. The modification
> replication thread
> >> >> removes
> >> >> > the key, gets the current value and invokes a
> PUT/REMOVE on the
> >> >> backup
> >> >> > owner(s).
> >> >> > >> >>
> Even better: a key is only ever added *once*, so if we
> have
> >> >> [5,2,17,3],
> >> >> > adding key 2 is a no-op because the processing of
key
> 2 (in
> >> >> second
> >> >> > position in the queue) will fetch the up-to-date
value
> anyway !
> >> >> > >> >>
> >> >> > Misc
> >> >> > ----
> >> >> > - Could we possibly use total order to send the
> updates in TO ?
> >> >> TBD (Pedro?)
> >> >> > >> >>
> >> >> > Thoughts ?
> >> >> > >> >>
> >> >> > [1]
https://github.com/belaban/IspnPerfTest
> >> >> > [2]
> >> >> > >> > > >> > >
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
> >> >> > >> > > >> >> --
> >> >> Bela Ban, JGroups lead (http://www.jgroups.org)
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org >
>> >> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org> >
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> > > >> > > >> > >
>> > > >> >>
_______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org >
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> > > >> > >> > --
> >> > Bela Ban, JGroups lead (http://www.jgroups.org)
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org >
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org >
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org >
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:13 a.m.
On Wed, Jul 30, 2014 at 12:00 PM, Pedro Ruivo <pedro(a)infinispan.org> wrote:
On 07/30/2014 09:02 AM, Dan Berindei wrote:
>
>
> if your proposal is only meant to apply to non-tx caches, you are right
> you don't have to worry about multiple primary owners... most of the
> time. But when the primary owner changes, then you do have 2 primary
> owners (if the new primary owner installs the new topology first), and
> you do need to coordinate between the 2.
>
I think it is the same for transactional cache. I.e. the commands wait
for the transaction data from the new topology to be installed. In the
non-tx caches, the old primary owner will send the next "sequence
number" to the new primary owner and only after that, the new primary
owner starts to give the orders.
I'm not sure that's related, commands that wait for a newer topology do not
block a thread since the ISPN-3527 fix.
Otherwise, I can implement a total order version for non-tx caches and
all the write serialization would be done in JGroups and Infinispan only
has to apply the updates as soon as they are delivered.
Right, that sounds quite interesting. But you'd also need a less-blocking
state transfer ;)
> Slightly related: we also considered generating a version number
on the
> client for consistency when the HotRod client retries after a primary
> owner failure [1]. But the clients can't create a monotonic sequence
> number, so we couldn't use that version number for this.
>
> [1] https://issues.jboss.org/browse/ISPN-2956
>
>
> Also I don't see it as an alternative to TOA, I rather expect it to
> work nicely together: when TOA is enabled you could trust the
> originating sequence source rather than generate a per-entry
sequence,
> and in neither case you need to actually use a Lock.
> I haven't thought how the sequences would need to interact (if they
> need), but they seem complementary to resolve different aspects, and
> also both benefit from the same cleanup and basic structure.
>
>
> We don't acquire locks at all on the backup owners - either in tx or
> non-tx caches. If state transfer is in progress, we use
> ConcurrentHashMap.compute() to store tracking information, which uses a
> synchronized block, so I suppose we do acquire locks. I assume your
> proposal would require a DataContainer.compute() or something similar on
> the backups, to ensure that the version check and the replacement are
> atomic.
>
> I still think TOA does what you want for tx caches. Your proposal would
> only work for non-tx caches, so you couldn't use them together.
>
>
> >> Another aspect is that the "user thread" on the primary
owner
> needs to
> >> wait (at least until we improve further) and only proceed after
ACK
> >> from backup nodes, but this is better modelled through a state
> >> machine. (Also discussed in Farnborough).
> >
> >
> > To be clear, I don't think keeping the user thread on the
> originator blocked
> > until we have the write confirmations from all the backups is a
> problem - a
> > sync operation has to block, and it also serves to rate-limit user
> > operations.
>
>
> There are better ways to rate-limit than to make all operations slow;
> we don't need to block a thread, we need to react on the reply from
> the backup owners.
> You still have an inherent rate-limit in the outgoing packet queues:
> if these fill up, then and only then it's nice to introduce some back
> pressure.
>
>
> Sorry, you got me confused when you called the thread on the primary
> owner a "user thread". I agree that internal stuff can and should be
> asynchronous, callback based, but the user still has to see a
> synchronous blocking operation.
>
>
> > The problem appears when the originator is not the primary owner,
> and the
> > thread blocking for backup ACKs is from the remote-executor pool
> (or OOB,
> > when the remote-executor pool is exhausted).
>
> Not following. I guess this is out of scope now that I clarified the
> proposed solution is only to be applied between primary and backups?
>
>
> Yeah, I was just trying to clarify that there is no danger of exhausting
> the remote executor/OOB thread pools when the originator of the write
> command is the primary owner (as it happens in the HotRod server).
>
>
> >>
> >> It's also conceptually linked to:
> >> - https://issues.jboss.org/browse/ISPN-1599
> >> As you need to separate the locks of entries from the effective
user
> >> facing lock, at least to implement transactions on top of this
> model.
> >
> >
> > I think we fixed ISPN-1599 when we changed passivation to use
> > DataContainer.compute(). WDYT Pedro, is there anything else you'd
> like to do
> > in the scope of ISPN-1599?
> >
> >>
> >> I expect this to improve performance in a very significant way,
but
> >> it's getting embarrassing that it's still not done; at the
next
face
> >> to face meeting we should also reserve some time for
retrospective
> >> sessions.
> >
> >
> > Implementing the state machine-based interceptor stack may give
us a
> > performance boost, but I'm much more certain that it's a very
> complex, high
> > risk task... and we don't have a stable test suite yet :)
>
> Cleaning up and removing some complexity such as
> TooManyExecutorsException might help to get it stable, and keep it
> there :)
> BTW it was quite stable for me until you changed the JGroups UDP
> default configuration.
>
>
> Do you really use UDP to run the tests? The default is TCP, but maybe
> the some tests doesn't use TestCacheManagerFactory...
>
> I was just aligning our configs with Bela's recommandations: MERGE3
> instead of MERGE2 and the removal of UFC in TCP stacks. If they cause
> problems on your machine, you should make more noise :)
>
> Dan
>
> Sanne
>
> >
> >
> >>
> >>
> >> Sanne
> >>
> >> On 29 July 2014 15:50, Bela Ban <bban(a)redhat.com
> <mailto:bban@redhat.com>> wrote:
> >> >
> >> >
> >> > On 29/07/14 16:42, Dan Berindei wrote:
> >> >> Have you tried regular optimistic/pessimistic transactions
as
> well?
> >> >
> >> > Yes, in my first impl. but since I'm making only 1 change
per
> request, I
> >> > thought a TX is overkill.
> >> >
> >> >> They *should* have less issues with the OOB thread pool than
> non-tx
> >> >> mode, and
> >> >> I'm quite curious how they stack against TO in such a
large
> cluster.
> >> >
> >> > Why would they have fewer issues with the thread pools ? AIUI,
> a TX
> >> > involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when
> not using
> >> > TXs. And we're sync anyway...
> >> >
> >> >
> >> >> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban
<bban(a)redhat.com
> <mailto:bban@redhat.com>
> >> >> <mailto:bban@redhat.com
<mailto:bban@redhat.com>>> wrote:
> >> >>
> >> >> Following up on my own email, I changed the config to
use
> Pedro's
> >> >> excellent total order implementation:
> >> >>
> >> >> <transaction
transactionMode="TRANSACTIONAL"
> >> >> transactionProtocol="TOTAL_ORDER"
lockingMode="OPTIMISTIC"
> >> >> useEagerLocking="true"
eagerLockSingleNode="true">
> >> >> <recovery
enabled="false"/>
> >> >>
> >> >> With 100 nodes and 25 requester threads/node, I did NOT
> run into
> >> >> any
> >> >> locking issues !
> >> >>
> >> >> I could even go up to 200 requester threads/node and the
> perf was ~
> >> >> 7'000-8'000 requests/sec/node. Not too bad !
> >> >>
> >> >> This really validates the concept of lockless
total-order
> >> >> dissemination
> >> >> of TXs; for the first time, this has been tested on a
> large(r)
> >> >> scale
> >> >> (previously only on 25 nodes) and IT WORKS ! :-)
> >> >>
> >> >> I still believe we should implement my suggested
solution
for
> >> >> non-TO
> >> >> configs, but short of configuring thread pools of 1000
> threads or
> >> >> higher, I hope TO will allow me to finally test a 500
node
> >> >> Infinispan
> >> >> cluster !
> >> >>
> >> >>
> >> >> On 29/07/14 15:56, Bela Ban wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > sorry for the long post, but I do think I ran into
an
> important
> >> >> problem
> >> >> > and we need to fix it ... :-)
> >> >> >
> >> >> > I've spent the last couple of days running the
> IspnPerfTest [1]
> >> >> perftest
> >> >> > on Google Compute Engine (GCE), and I've run
into a
> problem with
> >> >> > Infinispan. It is a design problem and can be
mitigated by
> >> >> sizing
> >> >> thread
> >> >> > pools correctly, but cannot be eliminated
entirely.
> >> >> >
> >> >> >
> >> >> > Symptom:
> >> >> > --------
> >> >> > IspnPerfTest has every node in a cluster perform
> 20'000 requests
> >> >> on keys
> >> >> > in range [1..20000].
> >> >> >
> >> >> > 80% of the requests are reads and 20% writes.
> >> >> >
> >> >> > By default, we have 25 requester threads per node
and
> 100 nodes
> >> >> in a
> >> >> > cluster, so a total of 2500 requester threads.
> >> >> >
> >> >> > The cache used is NON-TRANSACTIONAL / dist-sync /
2
> owners:
> >> >> >
> >> >> > <namedCache name="clusteredCache">
> >> >> > <clustering
mode="distribution">
> >> >> > <stateTransfer
awaitInitialTransfer="true"/>
> >> >> > <hash numOwners="2"/>
> >> >> > <sync
replTimeout="20000"/>
> >> >> > </clustering>
> >> >> >
> >> >> > <transaction
transactionMode="NON_TRANSACTIONAL"
> >> >> > useEagerLocking="true"
> >> >> > eagerLockSingleNode="true"
/>
> >> >> > <locking
lockAcquisitionTimeout="5000"
> >> >> concurrencyLevel="1000"
> >> >> >
isolationLevel="READ_COMMITTED"
> >> >> useLockStriping="false" />
> >> >> > </namedCache>
> >> >> >
> >> >> > It has 2 owners, a lock acquisition timeout of 5s
and
> a repl
> >> >> timeout of
> >> >> > 20s. Lock stripting is off, so we have 1 lock per
key.
> >> >> >
> >> >> > When I run the test, I always get errors like
those
below:
> >> >> >
> >> >> > org.infinispan.util.concurrent.TimeoutException:
Unable to
> >> >> acquire lock
> >> >> > after [10 seconds] on key [19386] for requestor
> >> >> [Thread[invoker-3,5,main]]!
> >> >> > Lock held by
[Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
> >> >> >
> >> >> > and
> >> >> >
> >> >> > org.infinispan.util.concurrent.TimeoutException:
Node
> m8.1 timed
> >> >> out
> >> >> >
> >> >> >
> >> >> > Investigation:
> >> >> > ------------
> >> >> > When I looked at UNICAST3, I saw a lot of missing
> messages on
> >> >> the
> >> >> > receive side and unacked messages on the send
side.
> This caused
> >> >> me to
> >> >> > look into the (mainly OOB) thread pools and - voila
-
> maxed out
> >> >> !
> >> >> >
> >> >> > I learned from Pedro that the Infinispan internal
> thread pool
> >> >> (with a
> >> >> > default of 32 threads) can be configured, so I
> increased it to
> >> >> 300 and
> >> >> > increased the OOB pools as well.
> >> >> >
> >> >> > This mitigated the problem somewhat, but when I
> increased the
> >> >> requester
> >> >> > threads to 100, I had the same problem again.
> Apparently, the
> >> >> Infinispan
> >> >> > internal thread pool uses a rejection policy of
"run"
> and thus
> >> >> uses the
> >> >> > JGroups (OOB) thread when exhausted.
> >> >> >
> >> >> > I learned (from Pedro and Mircea) that GETs and
PUTs
> work as
> >> >> follows in
> >> >> > dist-sync / 2 owners:
> >> >> > - GETs are sent to the primary and backup owners
and
> the first
> >> >> response
> >> >> > received is returned to the caller. No locks are
> acquired, so
> >> >> GETs
> >> >> > shouldn't cause problems.
> >> >> >
> >> >> > - A PUT(K) is sent to the primary owner of K
> >> >> > - The primary owner
> >> >> > (1) locks K
> >> >> > (2) updates the backup owner synchronously
> *while holding
> >> >> the lock*
> >> >> > (3) releases the lock
> >> >> >
> >> >> >
> >> >> > Hypothesis
> >> >> > ----------
> >> >> > (2) above is done while holding the lock. The sync
> update of the
> >> >> backup
> >> >> > owner is done with the lock held to guarantee that
the
> primary
> >> >> and
> >> >> > backup owner of K have the same values for K.
> >> >> >
> >> >> > However, the sync update *inside the lock scope*
slows
> things
> >> >> down (can
> >> >> > it also lead to deadlocks?); there's the risk
that the
> request
> >> >> is
> >> >> > dropped due to a full incoming thread pool, or
that
> the response
> >> >> is not
> >> >> > received because of the same, or that the locking
at
> the backup
> >> >> owner
> >> >> > blocks for some time.
> >> >> >
> >> >> > If we have many threads modifying the same key,
then
> we have a
> >> >> backlog
> >> >> > of locking work against that key. Say we have 100
> requester
> >> >> threads and
> >> >> > a 100 node cluster. This means that we have
10'000
threads
> >> >> accessing
> >> >> > keys; with 2'000 writers there's a big
chance that
> some writers
> >> >> pick the
> >> >> > same key at the same time.
> >> >> >
> >> >> > For example, if we have 100 threads accessing key
K
> and it takes
> >> >> 3ms to
> >> >> > replicate K to the backup owner, then the last of
the
100
> >> >> threads
> >> >> waits
> >> >> > ~300ms before it gets a chance to lock K on the
> primary owner
> >> >> and
> >> >> > replicate it as well.
> >> >> >
> >> >> > Just a small hiccup in sending the PUT to the
primary
> owner,
> >> >> sending the
> >> >> > modification to the backup owner, waitting for the
> response, or
> >> >> GC, and
> >> >> > the delay will quickly become bigger.
> >> >> >
> >> >> >
> >> >> > Verification
> >> >> > ----------
> >> >> > To verify the above, I set numOwners to 1. This
means
> that the
> >> >> primary
> >> >> > owner of K does *not* send the modification to the
> backup owner,
> >> >> it only
> >> >> > locks K, modifies K and unlocks K again.
> >> >> >
> >> >> > I ran the IspnPerfTest again on 100 nodes, with 25
> requesters,
> >> >> and NO
> >> >> > PROBLEM !
> >> >> >
> >> >> > I then increased the requesters to 100, 150 and
200
> and the test
> >> >> > completed flawlessly ! Performance was around
*40'000
> requests
> >> >> per node
> >> >> > per sec* on 4-core boxes !
> >> >> >
> >> >> >
> >> >> > Root cause
> >> >> > ---------
> >> >> > *******************
> >> >> > The root cause is the sync RPC of K to the backup
> owner(s) of K
> >> >> while
> >> >> > the primary owner holds the lock for K.
> >> >> > *******************
> >> >> >
> >> >> > This causes a backlog of threads waiting for the
lock
> and that
> >> >> backlog
> >> >> > can grow to exhaust the thread pools. First the
Infinispan
> >> >> internal
> >> >> > thread pool, then the JGroups OOB thread pool. The
> latter causes
> >> >> > retransmissions to get dropped, which compounds
the
> problem...
> >> >> >
> >> >> >
> >> >> > Goal
> >> >> > ----
> >> >> > The goal is to make sure that primary and backup
> owner(s) of K
> >> >> have the
> >> >> > same value for K.
> >> >> >
> >> >> > Simply sending the modification to the backup
owner(s)
> >> >> asynchronously
> >> >> > won't guarantee this, as modification messages
might
get
> >> >> processed out
> >> >> > of order as they're OOB !
> >> >> >
> >> >> >
> >> >> > Suggested solution
> >> >> > ----------------
> >> >> > The modification RPC needs to be invoked *outside
of
> the lock
> >> >> scope*:
> >> >> > - lock K
> >> >> > - modify K
> >> >> > - unlock K
> >> >> > - send modification to backup owner(s) // outside
the
> lock scope
> >> >> >
> >> >> > The primary owner puts the modification of K into
a
> queue from
> >> >> where a
> >> >> > separate thread/task removes it. The thread then
> invokes the
> >> >> PUT(K) on
> >> >> > the backup owner(s).
> >> >> >
> >> >> > The queue has the modified keys in FIFO order, so
the
> >> >> modifications
> >> >> > arrive at the backup owner(s) in the right order.
> >> >> >
> >> >> > This requires that the way GET is implemented
changes
> slightly:
> >> >> instead
> >> >> > of invoking a GET on all owners of K, we only
invoke
> it on the
> >> >> primary
> >> >> > owner, then the next-in-line etc.
> >> >> >
> >> >> > The reason for this is that the backup owner(s)
may
> not yet have
> >> >> > received the modification of K.
> >> >> >
> >> >> > This is a better impl anyway (we discussed this
> before) becuse
> >> >> it
> >> >> > generates less traffic; in the normal case, all but
1
GET
> >> >> requests are
> >> >> > unnecessary.
> >> >> >
> >> >> >
> >> >> >
> >> >> > Improvement
> >> >> > -----------
> >> >> > The above solution can be simplified and even made
more
> >> >> efficient.
> >> >> > Re-using concepts from IRAC [2], we can simply
store
the
> >> >> modified
> >> >> *keys*
> >> >> > in the modification queue. The modification
> replication thread
> >> >> removes
> >> >> > the key, gets the current value and invokes a
> PUT/REMOVE on the
> >> >> backup
> >> >> > owner(s).
> >> >> >
> >> >> > Even better: a key is only ever added *once*, so if
we
> have
> >> >> [5,2,17,3],
> >> >> > adding key 2 is a no-op because the processing of
key
> 2 (in
> >> >> second
> >> >> > position in the queue) will fetch the up-to-date
value
> anyway !
> >> >> >
> >> >> >
> >> >> > Misc
> >> >> > ----
> >> >> > - Could we possibly use total order to send the
> updates in TO ?
> >> >> TBD (Pedro?)
> >> >> >
> >> >> >
> >> >> > Thoughts ?
> >> >> >
> >> >> >
> >> >> > [1] https://github.com/belaban/IspnPerfTest
> >> >> > [2]
> >> >> >
> >> >>
> >> >>
>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
> >> >> >
> >> >>
> >> >> --
> >> >> Bela Ban, JGroups lead (http://www.jgroups.org)
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> infinispan-dev mailing list
> >> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> >
> >> > --
> >> > Bela Ban, JGroups lead (http://www.jgroups.org)
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:
infinispan-dev(a)lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
12:34 p.m.
On 29/07/14 23:35, Sanne Grinovero wrote:
The strategy I've proposed is only to be applied for the
communication
from the primary owner to its backups:
the value to be written is well known as it's the primary owner which
defines it unilaterally (for example if there is an atomic replacement
to be computed)
and there is no need for extra RPCs as the sequence is not related to
a group of changes but for the specific entry only.
How would this work with TXs involving multiple keys on different
primary owners ? Each owner replicates with seqnos to the backup owners,
so changes for single keys are received in order, but do we (need to)
guarantee that TXs consistency is preserved ? In other words, do we
preserve isolation: all changes of a TX are observed at the same logical
time, across multiple backup owners ?
There is no such thing as a need for consensus across owners, nor
need
for a central source for sequences.
Also I don't see it as an alternative to TOA, I rather expect it to
work nicely together: when TOA is enabled you could trust the
originating sequence source rather than generate a per-entry sequence,
and in neither case you need to actually use a Lock.
I haven't thought how the sequences would need to interact (if they
need), but they seem complementary to resolve different aspects, and
also both benefit from the same cleanup and basic structure.
>> Another aspect is that the "user thread" on the primary owner needs to
>> wait (at least until we improve further) and only proceed after ACK
>> from backup nodes, but this is better modelled through a state
>> machine. (Also discussed in Farnborough).
>
>
> To be clear, I don't think keeping the user thread on the originator blocked
> until we have the write confirmations from all the backups is a problem - a
> sync operation has to block, and it also serves to rate-limit user
> operations.
There are better ways to rate-limit than to make all operations slow;
we don't need to block a thread, we need to react on the reply from
the backup owners.
Agreed. I think Dan mentioned it as a side effect.
You still have an inherent rate-limit in the outgoing packet queues:
if these fill up, then and only then it's nice to introduce some back
pressure.
> The problem appears when the originator is not the primary owner, and the
> thread blocking for backup ACKs is from the remote-executor pool (or OOB,
> when the remote-executor pool is exhausted).
Not following. I guess this is out of scope now that I clarified the
proposed solution is only to be applied between primary and backups?
>>
>> It's also conceptually linked to:
>> - https://issues.jboss.org/browse/ISPN-1599
>> As you need to separate the locks of entries from the effective user
>> facing lock, at least to implement transactions on top of this model.
>
>
> I think we fixed ISPN-1599 when we changed passivation to use
> DataContainer.compute(). WDYT Pedro, is there anything else you'd like to do
> in the scope of ISPN-1599?
>
>>
>> I expect this to improve performance in a very significant way, but
>> it's getting embarrassing that it's still not done; at the next face
>> to face meeting we should also reserve some time for retrospective
>> sessions.
>
>
> Implementing the state machine-based interceptor stack may give us a
> performance boost, but I'm much more certain that it's a very complex, high
> risk task... and we don't have a stable test suite yet :)
Cleaning up and removing some complexity such as
TooManyExecutorsException might help to get it stable, and keep it
there :)
BTW it was quite stable for me until you changed the JGroups UDP
default configuration.
Sanne
--
Bela Ban, JGroups lead (http://www.jgroups.org)
10:49 a.m.
On 07/29/2014 10:14 PM, Dan Berindei wrote:
It's also conceptually linked to:
- https://issues.jboss.org/browse/ISPN-1599
As you need to separate the locks of entries from the effective user
facing lock, at least to implement transactions on top of this model.
I think we fixed ISPN-1599 when we changed passivation to use
DataContainer.compute(). WDYT Pedro, is there anything else you'd like
to do in the scope of ISPN-1599?
In the scope of ISPN-1599 I have nothing more in mind so far. But
related to this discussion, I have ISPN-2849.
12:26 p.m.
On 29/07/14 23:14, Dan Berindei wrote:
On Tue, Jul 29, 2014 at 9:06 PM, Sanne Grinovero <sanne(a)infinispan.org
<mailto:sanne@infinispan.org>> wrote:
This is a nasty problem and I also feel passionately we need to get
rid of it ASAP.
I did have the same problems many times, and we discussed this also in
Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
You don't need TO, and you don't need to lock at all as long as you
guarantee the backup owners are getting the number with some
monotonicity sequence attached to it,
all that backup owners need to do is ignore incoming commands which
are outdated.
This is more or less what TOA does - assign a monotonic sequence number
to txs, and only apply them after all the previous txs in the sequence
have been applied. The problem is getting that monotonic sequence when
there are multiple originators and multiple primary owners also requires
some extra RPCs.
Yes - for a TX involving multiple keys, TOA's probably the way to go.
However, for non-TXs caches and accessing single (or only few) keys,
TOA's probably overkill.
As long as we move the sync update RPC out of the lock scope, I'm fine
with whatever solution you guys come up with.
Another aspect is that the "user thread" on the primary
owner needs to
wait (at least until we improve further) and only proceed after ACK
from backup nodes, but this is better modelled through a state
machine. (Also discussed in Farnborough).
To be clear, I don't think keeping the user thread on the originator
blocked until we have the write confirmations from all the backups is a
problem - a sync operation has to block, and it also serves to
rate-limit user operations.
I agree; sync mode implies user threads are blocking until an operation
has completed.
The problem appears when the originator is not the primary owner,
and
the thread blocking for backup ACKs is from the remote-executor pool (or
OOB, when the remote-executor pool is exhausted).
It's also conceptually linked to:
- https://issues.jboss.org/browse/ISPN-1599
As you need to separate the locks of entries from the effective user
facing lock, at least to implement transactions on top of this model.
I think we fixed ISPN-1599 when we changed passivation to use
DataContainer.compute(). WDYT Pedro, is there anything else you'd like
to do in the scope of ISPN-1599?
I expect this to improve performance in a very significant way, but
it's getting embarrassing that it's still not done; at the next face
to face meeting we should also reserve some time for retrospective
sessions.
Implementing the state machine-based interceptor stack may give us a
performance boost, but I'm much more certain that it's a very complex,
high risk task... and we don't have a stable test suite yet :)
Yes - this is something major, let's add it to the agenda
--
Bela Ban, JGroups lead (http://www.jgroups.org)
12:13 p.m.
On 29/07/14 20:06, Sanne Grinovero wrote:
This is a nasty problem and I also feel passionately we need to get
rid of it ASAP.
+1
I did have the same problems many times, and we discussed this also
in
Farnborough; AFAIR Dan and Pedro had some excellent ideas to fix this.
You don't need TO, and you don't need to lock at all as long as you
guarantee the backup owners are getting the number with some
monotonicity sequence attached to it,
all that backup owners need to do is ignore incoming commands which
are outdated.
Right. And we need to handle the scenario where we get updates from
multiple members, e.g. P40,P41,Q6,Q7 in the case where the primary owner
changed from P to Q (or from Q to P ?)
Another aspect is that the "user thread" on the primary
owner needs to
wait (at least until we improve further) and only proceed after ACK
from backup nodes, but this is better modelled through a state
machine. (Also discussed in Farnborough).
It's also conceptually linked to:
- https://issues.jboss.org/browse/ISPN-1599
As you need to separate the locks of entries from the effective user
facing lock, at least to implement transactions on top of this model.
I expect this to improve performance in a very significant way, but
it's getting embarrassing that it's still not done; at the next face
to face meeting we should also reserve some time for retrospective
sessions.
Yes - there's a link to the agenda of the 2015 team meeting, please feel
free to update the agenda. I'll send out an email re dates and location
shortly.
[1] https://mojo.redhat.com/docs/DOC-977279
--
Bela Ban, JGroups lead (http://www.jgroups.org)
10:29 p.m.
On Tue, Jul 29, 2014 at 5:50 PM, Bela Ban <bban(a)redhat.com> wrote:
On 29/07/14 16:42, Dan Berindei wrote:
> Have you tried regular optimistic/pessimistic transactions as well?
Yes, in my first impl. but since I'm making only 1 change per request, I
thought a TX is overkill.
You are using txs with TO, right?
> They *should* have less issues with the OOB thread pool than non-tx
mode, and
> I'm quite curious how they stack against TO in such a large cluster.
Why would they have fewer issues with the thread pools ? AIUI, a TX
involves 2 RPCs (PREPARE-COMMIT/ROLLBACK) compared to one when not using
TXs. And we're sync anyway...
Actually, 2 sync RPCs (prepare + commit) and 1 async RPC (tx completion
notification). But we only keep the user thread busy across RPCs (unless L1
is enabled), so we actually need less OOB/remote threads.
> On Tue, Jul 29, 2014 at 5:38 PM, Bela Ban <bban(a)redhat.com
> <mailto:bban@redhat.com>> wrote:
>
> Following up on my own email, I changed the config to use Pedro's
> excellent total order implementation:
>
> <transaction transactionMode="TRANSACTIONAL"
> transactionProtocol="TOTAL_ORDER" lockingMode="OPTIMISTIC"
> useEagerLocking="true" eagerLockSingleNode="true">
> <recovery enabled="false"/>
>
> With 100 nodes and 25 requester threads/node, I did NOT run into any
> locking issues !
>
> I could even go up to 200 requester threads/node and the perf was ~
> 7'000-8'000 requests/sec/node. Not too bad !
>
> This really validates the concept of lockless total-order
dissemination
> of TXs; for the first time, this has been tested on a large(r) scale
> (previously only on 25 nodes) and IT WORKS ! :-)
>
> I still believe we should implement my suggested solution for non-TO
> configs, but short of configuring thread pools of 1000 threads or
> higher, I hope TO will allow me to finally test a 500 node Infinispan
> cluster !
>
>
> On 29/07/14 15:56, Bela Ban wrote:
> > Hi guys,
> >
> > sorry for the long post, but I do think I ran into an important
> problem
> > and we need to fix it ... :-)
> >
> > I've spent the last couple of days running the IspnPerfTest [1]
> perftest
> > on Google Compute Engine (GCE), and I've run into a problem with
> > Infinispan. It is a design problem and can be mitigated by sizing
> thread
> > pools correctly, but cannot be eliminated entirely.
> >
> >
> > Symptom:
> > --------
> > IspnPerfTest has every node in a cluster perform 20'000 requests
> on keys
> > in range [1..20000].
> >
> > 80% of the requests are reads and 20% writes.
> >
> > By default, we have 25 requester threads per node and 100 nodes
in a
> > cluster, so a total of 2500 requester threads.
> >
> > The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
> >
> > <namedCache name="clusteredCache">
> > <clustering mode="distribution">
> > <stateTransfer awaitInitialTransfer="true"/>
> > <hash numOwners="2"/>
> > <sync replTimeout="20000"/>
> > </clustering>
> >
> > <transaction transactionMode="NON_TRANSACTIONAL"
> > useEagerLocking="true"
> > eagerLockSingleNode="true" />
> > <locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
> > isolationLevel="READ_COMMITTED"
> useLockStriping="false" />
> > </namedCache>
> >
> > It has 2 owners, a lock acquisition timeout of 5s and a repl
> timeout of
> > 20s. Lock stripting is off, so we have 1 lock per key.
> >
> > When I run the test, I always get errors like those below:
> >
> > org.infinispan.util.concurrent.TimeoutException: Unable to
> acquire lock
> > after [10 seconds] on key [19386] for requestor
> [Thread[invoker-3,5,main]]!
> > Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
> >
> > and
> >
> > org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed
out
> >
> >
> > Investigation:
> > ------------
> > When I looked at UNICAST3, I saw a lot of missing messages on the
> > receive side and unacked messages on the send side. This caused
me to
> > look into the (mainly OOB) thread pools and - voila - maxed out !
> >
> > I learned from Pedro that the Infinispan internal thread pool
(with a
> > default of 32 threads) can be configured, so I increased it to
> 300 and
> > increased the OOB pools as well.
> >
> > This mitigated the problem somewhat, but when I increased the
> requester
> > threads to 100, I had the same problem again. Apparently, the
> Infinispan
> > internal thread pool uses a rejection policy of "run" and thus
> uses the
> > JGroups (OOB) thread when exhausted.
> >
> > I learned (from Pedro and Mircea) that GETs and PUTs work as
> follows in
> > dist-sync / 2 owners:
> > - GETs are sent to the primary and backup owners and the first
> response
> > received is returned to the caller. No locks are acquired, so GETs
> > shouldn't cause problems.
> >
> > - A PUT(K) is sent to the primary owner of K
> > - The primary owner
> > (1) locks K
> > (2) updates the backup owner synchronously *while holding
> the lock*
> > (3) releases the lock
> >
> >
> > Hypothesis
> > ----------
> > (2) above is done while holding the lock. The sync update of the
> backup
> > owner is done with the lock held to guarantee that the primary and
> > backup owner of K have the same values for K.
> >
> > However, the sync update *inside the lock scope* slows things
> down (can
> > it also lead to deadlocks?); there's the risk that the request is
> > dropped due to a full incoming thread pool, or that the response
> is not
> > received because of the same, or that the locking at the backup
owner
> > blocks for some time.
> >
> > If we have many threads modifying the same key, then we have a
> backlog
> > of locking work against that key. Say we have 100 requester
> threads and
> > a 100 node cluster. This means that we have 10'000 threads
accessing
> > keys; with 2'000 writers there's a big chance that some writers
> pick the
> > same key at the same time.
> >
> > For example, if we have 100 threads accessing key K and it takes
> 3ms to
> > replicate K to the backup owner, then the last of the 100 threads
> waits
> > ~300ms before it gets a chance to lock K on the primary owner and
> > replicate it as well.
> >
> > Just a small hiccup in sending the PUT to the primary owner,
> sending the
> > modification to the backup owner, waitting for the response, or
> GC, and
> > the delay will quickly become bigger.
> >
> >
> > Verification
> > ----------
> > To verify the above, I set numOwners to 1. This means that the
> primary
> > owner of K does *not* send the modification to the backup owner,
> it only
> > locks K, modifies K and unlocks K again.
> >
> > I ran the IspnPerfTest again on 100 nodes, with 25 requesters,
and NO
> > PROBLEM !
> >
> > I then increased the requesters to 100, 150 and 200 and the test
> > completed flawlessly ! Performance was around *40'000 requests
> per node
> > per sec* on 4-core boxes !
> >
> >
> > Root cause
> > ---------
> > *******************
> > The root cause is the sync RPC of K to the backup owner(s) of K
while
> > the primary owner holds the lock for K.
> > *******************
> >
> > This causes a backlog of threads waiting for the lock and that
> backlog
> > can grow to exhaust the thread pools. First the Infinispan
internal
> > thread pool, then the JGroups OOB thread pool. The latter causes
> > retransmissions to get dropped, which compounds the problem...
> >
> >
> > Goal
> > ----
> > The goal is to make sure that primary and backup owner(s) of K
> have the
> > same value for K.
> >
> > Simply sending the modification to the backup owner(s)
asynchronously
> > won't guarantee this, as modification messages might get
> processed out
> > of order as they're OOB !
> >
> >
> > Suggested solution
> > ----------------
> > The modification RPC needs to be invoked *outside of the lock
scope*:
> > - lock K
> > - modify K
> > - unlock K
> > - send modification to backup owner(s) // outside the lock scope
> >
> > The primary owner puts the modification of K into a queue from
> where a
> > separate thread/task removes it. The thread then invokes the
> PUT(K) on
> > the backup owner(s).
> >
> > The queue has the modified keys in FIFO order, so the
modifications
> > arrive at the backup owner(s) in the right order.
> >
> > This requires that the way GET is implemented changes slightly:
> instead
> > of invoking a GET on all owners of K, we only invoke it on the
> primary
> > owner, then the next-in-line etc.
> >
> > The reason for this is that the backup owner(s) may not yet have
> > received the modification of K.
> >
> > This is a better impl anyway (we discussed this before) becuse it
> > generates less traffic; in the normal case, all but 1 GET
> requests are
> > unnecessary.
> >
> >
> >
> > Improvement
> > -----------
> > The above solution can be simplified and even made more efficient.
> > Re-using concepts from IRAC [2], we can simply store the modified
> *keys*
> > in the modification queue. The modification replication thread
> removes
> > the key, gets the current value and invokes a PUT/REMOVE on the
> backup
> > owner(s).
> >
> > Even better: a key is only ever added *once*, so if we have
> [5,2,17,3],
> > adding key 2 is a no-op because the processing of key 2 (in second
> > position in the queue) will fetch the up-to-date value anyway !
> >
> >
> > Misc
> > ----
> > - Could we possibly use total order to send the updates in TO ?
> TBD (Pedro?)
> >
> >
> > Thoughts ?
> >
> >
> > [1] https://github.com/belaban/IspnPerfTest
> > [2]
> >
>
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
> >
>
> --
> Bela Ban, JGroups lead (http://www.jgroups.org)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:
infinispan-dev(a)lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
--
Bela Ban, JGroups lead (http://www.jgroups.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:39 p.m.
On Tue, Jul 29, 2014 at 4:56 PM, Bela Ban <bban(a)redhat.com> wrote:
Hi guys,
sorry for the long post, but I do think I ran into an important problem
and we need to fix it ... :-)
I've spent the last couple of days running the IspnPerfTest [1] perftest
on Google Compute Engine (GCE), and I've run into a problem with
Infinispan. It is a design problem and can be mitigated by sizing thread
pools correctly, but cannot be eliminated entirely.
Symptom:
--------
IspnPerfTest has every node in a cluster perform 20'000 requests on keys
in range [1..20000].
80% of the requests are reads and 20% writes.
By default, we have 25 requester threads per node and 100 nodes in a
cluster, so a total of 2500 requester threads.
The cache used is NON-TRANSACTIONAL / dist-sync / 2 owners:
<namedCache name="clusteredCache">
<clustering mode="distribution">
<stateTransfer awaitInitialTransfer="true"/>
<hash numOwners="2"/>
<sync replTimeout="20000"/>
</clustering>
<transaction transactionMode="NON_TRANSACTIONAL"
useEagerLocking="true"
eagerLockSingleNode="true" />
<locking lockAcquisitionTimeout="5000"
concurrencyLevel="1000"
isolationLevel="READ_COMMITTED"
useLockStriping="false" />
</namedCache>
It has 2 owners, a lock acquisition timeout of 5s and a repl timeout of
20s. Lock stripting is off, so we have 1 lock per key.
When I run the test, I always get errors like those below:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [10 seconds] on key [19386] for requestor [Thread[invoker-3,5,main]]!
Lock held by [Thread[OOB-194,ispn-perf-test,m5.1,5,main]]
and
org.infinispan.util.concurrent.TimeoutException: Node m8.1 timed out
Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !
I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.
This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.
We can't use another rejection policy in the remote executor because the
message won't be re-delivered by JGroups, and we can't use a queue either.
Pedro is working on ISPN-2849, which should help with the remote/OOB thread
pool exhaustion. It is a bit tricky, though, because our interceptors
assume they will be able to access stack variables after replication.
I learned (from Pedro and Mircea) that GETs and PUTs work as follows in
dist-sync / 2 owners:
- GETs are sent to the primary and backup owners and the first response
received is returned to the caller. No locks are acquired, so GETs
shouldn't cause problems.
- A PUT(K) is sent to the primary owner of K
- The primary owner
(1) locks K
(2) updates the backup owner synchronously *while holding the lock*
(3) releases the lock
Hypothesis
----------
(2) above is done while holding the lock. The sync update of the backup
owner is done with the lock held to guarantee that the primary and
backup owner of K have the same values for K.
And something else: if the primary owner reports that a write was
successful and then dies, a read should find the updated value on the
backup owner(s).
However, the sync update *inside the lock scope* slows things down
(can
it also lead to deadlocks?); there's the risk that the request is
dropped due to a full incoming thread pool, or that the response is not
received because of the same, or that the locking at the backup owner
blocks for some time.
There is no locking on the backup owner, so there are no deadlocks.
There is indeed a risk of the OOB/remote thread pools being full.
If we have many threads modifying the same key, then we have a backlog
of locking work against that key. Say we have 100 requester threads and
a 100 node cluster. This means that we have 10'000 threads accessing
keys; with 2'000 writers there's a big chance that some writers pick the
same key at the same time.
For example, if we have 100 threads accessing key K and it takes 3ms to
replicate K to the backup owner, then the last of the 100 threads waits
~300ms before it gets a chance to lock K on the primary owner and
replicate it as well.
Just a small hiccup in sending the PUT to the primary owner, sending the
modification to the backup owner, waitting for the response, or GC, and
the delay will quickly become bigger.
Verification
----------
To verify the above, I set numOwners to 1. This means that the primary
owner of K does *not* send the modification to the backup owner, it only
locks K, modifies K and unlocks K again.
I ran the IspnPerfTest again on 100 nodes, with 25 requesters, and NO
PROBLEM !
I then increased the requesters to 100, 150 and 200 and the test
completed flawlessly ! Performance was around *40'000 requests per node
per sec* on 4-core boxes !
Root cause
---------
*******************
The root cause is the sync RPC of K to the backup owner(s) of K while
the primary owner holds the lock for K.
*******************
This causes a backlog of threads waiting for the lock and that backlog
can grow to exhaust the thread pools. First the Infinispan internal
thread pool, then the JGroups OOB thread pool. The latter causes
retransmissions to get dropped, which compounds the problem...
Goal
----
The goal is to make sure that primary and backup owner(s) of K have the
same value for K.
Simply sending the modification to the backup owner(s) asynchronously
won't guarantee this, as modification messages might get processed out
of order as they're OOB !
Suggested solution
----------------
The modification RPC needs to be invoked *outside of the lock scope*:
- lock K
- modify K
- unlock K
- send modification to backup owner(s) // outside the lock scope
The primary owner puts the modification of K into a queue from where a
separate thread/task removes it. The thread then invokes the PUT(K) on
the backup owner(s).
Does the replication thread execute the PUT(k) synchronously, or
asynchronously? I assume asynchronously, otherwise the replication thread
wouldn't be able to keep up with the writers.
The queue has the modified keys in FIFO order, so the modifications
arrive at the backup owner(s) in the right order.
Sending the RPC to the backup owners asynchronously, while holding the key
lock, would do the same thing.
This requires that the way GET is implemented changes slightly: instead
of invoking a GET on all owners of K, we only invoke it on the primary
owner, then the next-in-line etc.
What's the next-in-line owner? A backup won't have the last version of the
data.
The reason for this is that the backup owner(s) may not yet have
received the modification of K.
OTOH, if the primary owner dies, we have to ask a backup, and we can lose
the modifications not yet replicated by the primary.
This is a better impl anyway (we discussed this before) becuse it
generates less traffic; in the normal case, all but 1 GET requests are
unnecessary.
I have a WIP branch for this and it seemed to work fine. Test suite speed
seemed about the same, but I didn't get to do a real performance test.
Improvement
-----------
The above solution can be simplified and even made more efficient.
Re-using concepts from IRAC [2], we can simply store the modified *keys*
in the modification queue. The modification replication thread removes
the key, gets the current value and invokes a PUT/REMOVE on the backup
owner(s).
Even better: a key is only ever added *once*, so if we have [5,2,17,3],
adding key 2 is a no-op because the processing of key 2 (in second
position in the queue) will fetch the up-to-date value anyway !
Misc
----
- Could we possibly use total order to send the updates in TO ? TBD
(Pedro?)
Thoughts ?
[1] https://github.com/belaban/IspnPerfTest
[2]
https://github.com/infinispan/infinispan/wiki/RAC:-Reliable-Asynchronous-...
--
Bela Ban, JGroups lead (http://www.jgroups.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:22 a.m.
Investigation:
------------
When I looked at UNICAST3, I saw a lot of missing messages on the
receive side and unacked messages on the send side. This caused me to
look into the (mainly OOB) thread pools and - voila - maxed out !
I learned from Pedro that the Infinispan internal thread pool (with a
default of 32 threads) can be configured, so I increased it to 300 and
increased the OOB pools as well.
This mitigated the problem somewhat, but when I increased the
requester
threads to 100, I had the same problem again. Apparently, the
Infinispan
internal thread pool uses a rejection policy of "run" and thus
uses the
JGroups (OOB) thread when exhausted.
We can't use another rejection policy in the remote executor because
the message won't be re-delivered by JGroups, and we can't use a queue
either.
Can't we just send response "Node is busy" and cancel the operation? (at
least in cases where this is possible - we can't do that safely for
CommitCommand, but usually it could be doable, right?) And what's the
problem with queues, besides that these can grow out of memory?
Radim
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
12:05 p.m.
Quick update.
Even when I changed the number of segments from 60 (default) to 1000 (on
Dan's suggestion), I still got lock aquisition timeouts with regular TXs
and in the TX-less case, too.
<hash numOwners="2" numSegments="1000"/>
With total order, I had not single issue in a couple of runs, one of
them on 500 nodes with 200 requester threads each (100'000 threads) !
I didn't go back and measure performance of total order with 1000 segments.
--
Bela Ban, JGroups lead (http://www.jgroups.org)
1:59 p.m.
On Wed, Jul 30, 2014 at 12:22 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
Investigation:
> ------------
> When I looked at UNICAST3, I saw a lot of missing messages on the
> receive side and unacked messages on the send side. This caused me to
> look into the (mainly OOB) thread pools and - voila - maxed out !
>
> I learned from Pedro that the Infinispan internal thread pool (with a
> default of 32 threads) can be configured, so I increased it to 300 and
> increased the OOB pools as well.
>
> This mitigated the problem somewhat, but when I increased the requester
> threads to 100, I had the same problem again. Apparently, the Infinispan
> internal thread pool uses a rejection policy of "run" and thus uses the
> JGroups (OOB) thread when exhausted.
>
We can't use another rejection policy in the remote executor because the
message won't be re-delivered by JGroups, and we can't use a queue either.
Can't we just send response "Node is busy" and cancel the operation? (at
least in cases where this is possible - we can't do that safely for
CommitCommand, but usually it could be doable, right?) And what's the
problem with queues, besides that these can grow out of memory?
No commit commands here, the cache is not transactional :)
If the remote thread pool gets full on a backup node, there is no way to
safely cancel the operation - other backup owners may have already applied
the write. And even with numOwners=2, there are multiple backup owners
during state transfer.
We do throw an OutdatedTopologyException on the backups and retry the
operation when the topology changes, we could do something similar when the
remote executor thread pool is full. But 1) we have trouble preserving
consistency when we retry, so we'd rather do it only when we really have
to, and 2) repeated retries can be costly, as the primary needs to
re-acquire the lock.
The problem with queues is that commands are executed in the order they are
in the queue. If a node has a remote executor thread pool of 100 threads
and receives a prepare(tx1, put(k, v1) comand, then 1000 prepare(tx_i,
put(k, v_i)) commands, and finally a commit(tx1) command, the commit(tx1)
command will block until all but 99 of the the prepare(tx_i, put(k, v_i))
commands have timed out.
I have some thoughts on improving that independently of Pedro's work on
locking [1], and I've just written that up as ISPN-4585 [2]
[1] https://issues.jboss.org/browse/ISPN-2849
[2] https://issues.jboss.org/browse/ISPN-4585
Radim
--
Radim Vansa <rvansa(a)redhat.com> <rvansa(a)redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
2:47 p.m.
On 07/30/2014 01:59 PM, Dan Berindei wrote:
On Wed, Jul 30, 2014 at 12:22 PM, Radim Vansa <rvansa(a)redhat.com
<mailto:rvansa@redhat.com>> wrote:
> Investigation:
> ------------
> When I looked at UNICAST3, I saw a lot of missing messages on the
> receive side and unacked messages on the send side. This
> caused me to
> look into the (mainly OOB) thread pools and - voila - maxed out !
>
> I learned from Pedro that the Infinispan internal thread pool
> (with a
> default of 32 threads) can be configured, so I increased it
> to 300 and
> increased the OOB pools as well.
>
> This mitigated the problem somewhat, but when I increased the
> requester
> threads to 100, I had the same problem again. Apparently, the
> Infinispan
> internal thread pool uses a rejection policy of "run" and
> thus uses the
> JGroups (OOB) thread when exhausted.
>
>
> We can't use another rejection policy in the remote executor
> because the message won't be re-delivered by JGroups, and we
> can't use a queue either.
Can't we just send response "Node is busy" and cancel the
operation? (at least in cases where this is possible - we can't do
that safely for CommitCommand, but usually it could be doable,
right?) And what's the problem with queues, besides that these can
grow out of memory?
No commit commands here, the cache is not transactional :)
Sure, but any change to OOB -> remote thread pool would likely affect
both non-tx and tx.
If the remote thread pool gets full on a backup node, there is no way
to safely cancel the operation - other backup owners may have already
applied the write. And even with numOwners=2, there are multiple
backup owners during state transfer.
I was thinking about delaying the write until backup responds, but
you're right, with 2 and more backups the situation is not that easy.
We do throw an OutdatedTopologyException on the backups and retry the
operation when the topology changes, we could do something similar
when the remote executor thread pool is full. But 1) we have trouble
preserving consistency when we retry, so we'd rather do it only when
we really have to, and 2) repeated retries can be costly, as the
primary needs to re-acquire the lock.
The problem with queues is that commands are executed in the order
they are in the queue. If a node has a remote executor thread pool of
100 threads and receives a prepare(tx1, put(k, v1) comand, then 1000
prepare(tx_i, put(k, v_i)) commands, and finally a commit(tx1)
command, the commit(tx1) command will block until all but 99 of the
the prepare(tx_i, put(k, v_i)) commands have timed out.
Makes sense
I have some thoughts on improving that independently of Pedro's work
on locking [1], and I've just written that up as ISPN-4585 [2]
[1] https://issues.jboss.org/browse/ISPN-2849
[2] https://issues.jboss.org/browse/ISPN-4585
ISPN-2849 sounds a lot like the state machine-based interceptor stack, I
am looking forward to that! (although it's the music of far future -
ISPN 9, 10?)
Thanks for those answers, Dan. I should realize most of that myself, but
I don't have the capacity to hold all the wisdom about NBST algorithms
online in my brain :) I hope some day I could catch a student looking
for diploma thesis willing to model at least the basic Infinispan
algorithms and formally verify that it's (in)correct ;-).
Radim
Radim
--
Radim Vansa<rvansa(a)redhat.com> <mailto:rvansa@redhat.com>
JBoss DataGrid QA
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
12:01 p.m.
On 29/07/14 16:39, Dan Berindei wrote:
Investigation:
------------
This mitigated the problem somewhat, but when I increased the requester
threads to 100, I had the same problem again. Apparently, the Infinispan
internal thread pool uses a rejection policy of "run" and thus uses the
JGroups (OOB) thread when exhausted.
We can't use another rejection policy in the remote executor because the
message won't be re-delivered by JGroups, and we can't use a queue either.
Yes I'm aware of that and "run" is our only option for the Infinispan
internal thread pool
Suggested solution
----------------
The modification RPC needs to be invoked *outside of the lock scope*:
- lock K
- modify K
- unlock K
- send modification to backup owner(s) // outside the lock scope
The primary owner puts the modification of K into a queue from where a
separate thread/task removes it. The thread then invokes the PUT(K) on
the backup owner(s).
Does the replication thread execute the PUT(k) synchronously, or
asynchronously? I assume asynchronously, otherwise the replication
thread wouldn't be able to keep up with the writers.
Async would be preferred, but the order of the updates needs to be
guaranteed. This could be done with the sequence numbers suggested by
Sanne, or using total order/TOA.
The queue has the modified keys in FIFO order, so the
modifications
arrive at the backup owner(s) in the right order.
Sending the RPC to the backup owners asynchronously, while holding the
key lock, would do the same thing.
Yes - but if there's a chance that the send() system call blocks, e.g.
on TCP when the send window is full, the async repl should be outside
the lock scope.
If those update messages to the backup owner(s) are regular (not OOB)
messages, FIFO would ensure that they're processed in the order in which
they were sent. If they're OOB messages, we'd have to somehow guarantee
ordering, e.g. using seq numbers.
This requires that the way GET is implemented changes slightly:
instead
of invoking a GET on all owners of K, we only invoke it on the primary
owner, then the next-in-line etc.
I have a WIP branch for this and it seemed to work fine. Test suite
speed seemed about the same, but I didn't get to do a real performance test.
Hmm, it should reduce overall traffic, which indirectly should lead to
better performance. I hope you wrap your work with a JIRA and commit it !
--
Bela Ban, JGroups lead (http://www.jgroups.org)
3565
days inactive
3566
days old
infinispan-dev@lists.jboss.org
20 comments
5 participants
participants (5)
-
Bela Ban -
Dan Berindei -
Pedro Ruivo -
Radim Vansa -
Sanne Grinovero