[JBoss JIRA] (ISPN-9827) The Hot Rod client should apply topology changes after the response has been processed
by Tristan Tarrant (Jira)
Tristan Tarrant created ISPN-9827:
-------------------------------------
Summary: The Hot Rod client should apply topology changes after the response has been processed
Key: ISPN-9827
URL: https://issues.jboss.org/browse/ISPN-9827
Project: Infinispan
Issue Type: Bug
Affects Versions: 9.4.4.Final, 10.0.0.Alpha2
Reporter: Tristan Tarrant
Assignee: Tristan Tarrant
The Hot Rod client applies topology changes to the connection pool while it is processing the header. This could cause the channel on which it is reading to be closed before the response has been completely processed. Topology changes should be applied when the operation completes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9817) OOM Error on ExposedByteArrayOutputStream
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9817?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-9817:
---------------------------------------
A fix would be more than welcome:
* You can provide one for the branch you need but you must also provide one for the current master (10.0.x) and stable (9.4.x)
* You will also need to provide a reproducer test
You can come on https://infinispan.zulipchat.com/ to discuss this with us if you wish
> OOM Error on ExposedByteArrayOutputStream
> -----------------------------------------
>
> Key: ISPN-9817
> URL: https://issues.jboss.org/browse/ISPN-9817
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.4.Final
> Reporter: Rakesh Vende
> Priority: Critical
> Fix For: 7.2.4.Final
>
> Attachments: 11.jpg
>
>
> Titile - OOM Error on ExposedByteArrayOutputStream
> Data -
> 1. Replication Mode is Async
> 2. queue-size="500"
> 3. queue-flush-interval="10000"
> Details -
> 1. Application threads frequently calling put method on replicated cache results in calling flush method of ReplicationQueueImpl.java
> 2. This cause application threads to wait for every 500th put call to complete the cache replication from the queue
> 3. This becomes kind of sync replication which blocks application threads.
> 4. To avoid this situation, we can increase the queue size large enough, which, apparently, does not have any side effect as queue is linked blocking queue and application threads will only get blocked when queue becomes full.
> 5. However this puts pressure on aysnc queue, which has to replicate entire queue at once.
> _replicationQueue-thread--p4-t1 tid=119 [RUNNABLE] [DAEMON] <--- OutOfMemoryError happened in this thread
> java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
> org.infinispan.commons.io.ExposedByteArrayOutputStream.write(byte[], int, int) ExposedByteArrayOutputStream.java:71
> _
> 6. This out of memory happens when JVM fails to allocate continuations chunk of memory in the form of array of 1 or 2 GB
> Summary - If we set queue size to normal or low level, application threads result in calling flush which turns out to be sync replication which blocks other application threads. And, if I increase the queue size to maximum enough so as to avoid sync flush then replication queue throws OOM
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9817) OOM Error on ExposedByteArrayOutputStream
by Rakesh Vende (Jira)
[ https://issues.jboss.org/browse/ISPN-9817?page=com.atlassian.jira.plugin.... ]
Rakesh Vende edited comment on ISPN-9817 at 12/14/18 1:09 AM:
--------------------------------------------------------------
[~NadirX],
We cannot upgrade to 9.4 as of now as we don't know what new set of challenges it brings in even if solves old issue.
I have studied this part of Infinispan source code and have identified the code fixes. Have pulled in 7.2.* branch. Is there a way, we can review the fix and I can contribute to this issue?
About Issue and Resolution -
I have given detailed analysis of issue in defect description. Will elaborate here in brief. Please check below the comment for ReplicationQueue tnterface from Infinispan Source code.
**
* Periodically (or+ when certain size is exceeded+) takes elements and replicates them.
*
* @author <a href="mailto:bela@jboss.org">Bela Ban</a>
* @author Mircea.Markus(a)jboss.com
* @since 4.0
*/
_when certain size is exceeded_ - This when size is exceeded, we dont have control on what max size we are going to replicate. Also for asycn replication queue, this size does not matters. It flushes the queue on its interval, irrespective of how much elements to be drains from queue for replication. This results in very large continious byte array chunk resulting in OOM.
FYI - [~dneels1980]
was (Author: rvende):
[~NadirX],
We cannot upgrade to 9.4 as of now as we don't know what new set of challenges it brings in even if solves old issue.
I have studied this part of Infinispan source code and have identified the code fixes. Have pulled in 7.2.* branch. Is there a way, we can review the fix and I can contribute to this issue?
About Issue and Resolution -
I have given detailed analysis of issue in defect description. Will elaborate here in brief. Please check below the comment for ReplicationQueue tnterface from Infinispan Source code.
**
* Periodically (or+ when certain size is exceeded+) takes elements and replicates them.
*
* @author <a href="mailto:bela@jboss.org">Bela Ban</a>
* @author Mircea.Markus(a)jboss.com
* @since 4.0
*/
_when certain size is exceeded_ - This when size is exceeded, we dont have control on what max size we are going to replicate. Also for asycn replication queue, this size does not matters. It flushes the queue on its interval, irrespective of how much elements to be drains from queue for replication.
> OOM Error on ExposedByteArrayOutputStream
> -----------------------------------------
>
> Key: ISPN-9817
> URL: https://issues.jboss.org/browse/ISPN-9817
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.4.Final
> Reporter: Rakesh Vende
> Priority: Critical
> Fix For: 7.2.4.Final
>
> Attachments: 11.jpg
>
>
> Titile - OOM Error on ExposedByteArrayOutputStream
> Data -
> 1. Replication Mode is Async
> 2. queue-size="500"
> 3. queue-flush-interval="10000"
> Details -
> 1. Application threads frequently calling put method on replicated cache results in calling flush method of ReplicationQueueImpl.java
> 2. This cause application threads to wait for every 500th put call to complete the cache replication from the queue
> 3. This becomes kind of sync replication which blocks application threads.
> 4. To avoid this situation, we can increase the queue size large enough, which, apparently, does not have any side effect as queue is linked blocking queue and application threads will only get blocked when queue becomes full.
> 5. However this puts pressure on aysnc queue, which has to replicate entire queue at once.
> _replicationQueue-thread--p4-t1 tid=119 [RUNNABLE] [DAEMON] <--- OutOfMemoryError happened in this thread
> java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
> org.infinispan.commons.io.ExposedByteArrayOutputStream.write(byte[], int, int) ExposedByteArrayOutputStream.java:71
> _
> 6. This out of memory happens when JVM fails to allocate continuations chunk of memory in the form of array of 1 or 2 GB
> Summary - If we set queue size to normal or low level, application threads result in calling flush which turns out to be sync replication which blocks other application threads. And, if I increase the queue size to maximum enough so as to avoid sync flush then replication queue throws OOM
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9817) OOM Error on ExposedByteArrayOutputStream
by Rakesh Vende (Jira)
[ https://issues.jboss.org/browse/ISPN-9817?page=com.atlassian.jira.plugin.... ]
Rakesh Vende commented on ISPN-9817:
------------------------------------
[~NadirX],
We cannot upgrade to 9.4 as of now as we don't know what new set of challenges it brings in even if solves old issue.
I have studied this part of Infinispan source code and have identified the code fixes. Have pulled in 7.2.* branch. Is there a way, we can review the fix and I can contribute to this issue?
About Issue and Resolution -
I have given detailed analysis of issue in defect description. Will elaborate here in brief. Please check below the comment for ReplicationQueue tnterface from Infinispan Source code.
**
* Periodically (or+ when certain size is exceeded+) takes elements and replicates them.
*
* @author <a href="mailto:bela@jboss.org">Bela Ban</a>
* @author Mircea.Markus(a)jboss.com
* @since 4.0
*/
_when certain size is exceeded_ - This when size is exceeded, we dont have control on what max size we are going to replicate. Also for asycn replication queue, this size does not matters. It flushes the queue on its interval, irrespective of how much elements to be drains from queue for replication.
> OOM Error on ExposedByteArrayOutputStream
> -----------------------------------------
>
> Key: ISPN-9817
> URL: https://issues.jboss.org/browse/ISPN-9817
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.4.Final
> Reporter: Rakesh Vende
> Priority: Critical
> Fix For: 7.2.4.Final
>
> Attachments: 11.jpg
>
>
> Titile - OOM Error on ExposedByteArrayOutputStream
> Data -
> 1. Replication Mode is Async
> 2. queue-size="500"
> 3. queue-flush-interval="10000"
> Details -
> 1. Application threads frequently calling put method on replicated cache results in calling flush method of ReplicationQueueImpl.java
> 2. This cause application threads to wait for every 500th put call to complete the cache replication from the queue
> 3. This becomes kind of sync replication which blocks application threads.
> 4. To avoid this situation, we can increase the queue size large enough, which, apparently, does not have any side effect as queue is linked blocking queue and application threads will only get blocked when queue becomes full.
> 5. However this puts pressure on aysnc queue, which has to replicate entire queue at once.
> _replicationQueue-thread--p4-t1 tid=119 [RUNNABLE] [DAEMON] <--- OutOfMemoryError happened in this thread
> java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
> org.infinispan.commons.io.ExposedByteArrayOutputStream.write(byte[], int, int) ExposedByteArrayOutputStream.java:71
> _
> 6. This out of memory happens when JVM fails to allocate continuations chunk of memory in the form of array of 1 or 2 GB
> Summary - If we set queue size to normal or low level, application threads result in calling flush which turns out to be sync replication which blocks other application threads. And, if I increase the queue size to maximum enough so as to avoid sync flush then replication queue throws OOM
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9817) OOM Error on ExposedByteArrayOutputStream
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9817?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-9817:
---------------------------------------
Infinispan 7.2.4 is unsupported. You need to upgrade to 9.4.
> OOM Error on ExposedByteArrayOutputStream
> -----------------------------------------
>
> Key: ISPN-9817
> URL: https://issues.jboss.org/browse/ISPN-9817
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.4.Final
> Reporter: Rakesh Vende
> Priority: Critical
> Fix For: 7.2.4.Final
>
> Attachments: 11.jpg
>
>
> Titile - OOM Error on ExposedByteArrayOutputStream
> Data -
> 1. Replication Mode is Async
> 2. queue-size="500"
> 3. queue-flush-interval="10000"
> Details -
> 1. Application threads frequently calling put method on replicated cache results in calling flush method of ReplicationQueueImpl.java
> 2. This cause application threads to wait for every 500th put call to complete the cache replication from the queue
> 3. This becomes kind of sync replication which blocks application threads.
> 4. To avoid this situation, we can increase the queue size large enough, which, apparently, does not have any side effect as queue is linked blocking queue and application threads will only get blocked when queue becomes full.
> 5. However this puts pressure on aysnc queue, which has to replicate entire queue at once.
> _replicationQueue-thread--p4-t1 tid=119 [RUNNABLE] [DAEMON] <--- OutOfMemoryError happened in this thread
> java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
> org.infinispan.commons.io.ExposedByteArrayOutputStream.write(byte[], int, int) ExposedByteArrayOutputStream.java:71
> _
> 6. This out of memory happens when JVM fails to allocate continuations chunk of memory in the form of array of 1 or 2 GB
> Summary - If we set queue size to normal or low level, application threads result in calling flush which turns out to be sync replication which blocks other application threads. And, if I increase the queue size to maximum enough so as to avoid sync flush then replication queue throws OOM
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9817) OOM Error on ExposedByteArrayOutputStream
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9817?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant edited comment on ISPN-9817 at 12/13/18 3:34 PM:
-----------------------------------------------------------------
Infinispan 7.2.4 is unsupported. You need to upgrade to 9.4 and see if the issue has been resolved.
was (Author: nadirx):
Infinispan 7.2.4 is unsupported. You need to upgrade to 9.4.
> OOM Error on ExposedByteArrayOutputStream
> -----------------------------------------
>
> Key: ISPN-9817
> URL: https://issues.jboss.org/browse/ISPN-9817
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.4.Final
> Reporter: Rakesh Vende
> Priority: Critical
> Fix For: 7.2.4.Final
>
> Attachments: 11.jpg
>
>
> Titile - OOM Error on ExposedByteArrayOutputStream
> Data -
> 1. Replication Mode is Async
> 2. queue-size="500"
> 3. queue-flush-interval="10000"
> Details -
> 1. Application threads frequently calling put method on replicated cache results in calling flush method of ReplicationQueueImpl.java
> 2. This cause application threads to wait for every 500th put call to complete the cache replication from the queue
> 3. This becomes kind of sync replication which blocks application threads.
> 4. To avoid this situation, we can increase the queue size large enough, which, apparently, does not have any side effect as queue is linked blocking queue and application threads will only get blocked when queue becomes full.
> 5. However this puts pressure on aysnc queue, which has to replicate entire queue at once.
> _replicationQueue-thread--p4-t1 tid=119 [RUNNABLE] [DAEMON] <--- OutOfMemoryError happened in this thread
> java.lang.OutOfMemoryError.<init>() OutOfMemoryError.java:48
> org.infinispan.commons.io.ExposedByteArrayOutputStream.write(byte[], int, int) ExposedByteArrayOutputStream.java:71
> _
> 6. This out of memory happens when JVM fails to allocate continuations chunk of memory in the form of array of 1 or 2 GB
> Summary - If we set queue size to normal or low level, application threads result in calling flush which turns out to be sync replication which blocks other application threads. And, if I increase the queue size to maximum enough so as to avoid sync flush then replication queue throws OOM
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9634) Deprecate useRawData from remote listeners
by Gustavo Fernandes (Jira)
[ https://issues.jboss.org/browse/ISPN-9634?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-9634:
------------------------------------
Status: Open (was: New)
> Deprecate useRawData from remote listeners
> ------------------------------------------
>
> Key: ISPN-9634
> URL: https://issues.jboss.org/browse/ISPN-9634
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
> Fix For: 10.0.0.Final
>
>
> Since remote listeners with converters and filters were introduced, it was assumed the user supplied implementations would receive "unmarshalled" data to operate.
> A parameter on the remote listener annotation itself called "useRawData" was later introduced to provide a way for filters/converters to use data as it was stored (i.e., without unmarshalling)
> Since transcoders were introduced, the choice of format for listeners and converters can be specified on the listeners/converters themselves in the format() method and should not rely on useRawData anymore.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months
[JBoss JIRA] (ISPN-9634) Deprecate useRawData from remote listeners
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9634?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-9634:
------------------------------------
{{useRawData}} also controls how the converter parameters are passed to the factory. It should be possible to change the format of the parameters independently of the value format.
> Deprecate useRawData from remote listeners
> ------------------------------------------
>
> Key: ISPN-9634
> URL: https://issues.jboss.org/browse/ISPN-9634
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Major
> Fix For: 10.0.0.Final
>
>
> Since remote listeners with converters and filters were introduced, it was assumed the user supplied implementations would receive "unmarshalled" data to operate.
> A parameter on the remote listener annotation itself called "useRawData" was later introduced to provide a way for filters/converters to use data as it was stored (i.e., without unmarshalling)
> Since transcoders were introduced, the choice of format for listeners and converters can be specified on the listeners/converters themselves in the format() method and should not rely on useRawData anymore.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 5 months