[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/JGRP-1868?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on JGRP-1868:
-----------------------------------
Yes, it is the old story with all threads stuck at UFC. I am getting sick&tired of the stacktraces. At least now I have more info: I dump JGroups debug info (managed attributes etc.) and threadpool sizes every 10 seconds, so I can see how this happened. Not that I would have any clues yet.
> Size of XMIT_REQ is not limited
> -------------------------------
>
> Key: JGRP-1868
> URL: https://issues.jboss.org/browse/JGRP-1868
> Project: JGroups
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 3.4.4, 3.5
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6
>
>
> When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
> {code}
> JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
> {code}
> It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/JGRP-1868?page=com.atlassian.jira.plugin.... ]
Radim Vansa updated JGRP-1868:
------------------------------
Affects Version/s: 3.4.4
3.5
> Size of XMIT_REQ is not limited
> -------------------------------
>
> Key: JGRP-1868
> URL: https://issues.jboss.org/browse/JGRP-1868
> Project: JGroups
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 3.4.4, 3.5
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6
>
>
> When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
> {code}
> JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
> {code}
> It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1868?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1868 at 8/6/14 5:18 AM:
--------------------------------------------------------
To prevent having to use a seond {{FRAG}}, we could
* Send multiple XMIT-REQs, whose SeqnoList arg is size-limited to 60K (if transport == UDP). {{SeqnoList.size()}} computes the exact size.
* Send am XMIT-REQ only for the oldest N seqnos (as mentioned by Radim)
* (Re-)implement retransmission backoffs to reduce xmit traffic. However, this requires additional state in the xmit tables
* Compress SeqnoList to use less space, e.g. by using bitmaps/sets
was (Author: belaban):
To prevent having to use a seond {{FRAG}}, we could
* Send multiple XMIT-REQs, whose SeqnoList arg is size-limited to 60K (if transport == UDP). {{SeqnoList.size()}} computes the exact size.
* Send am XMIT-REQ only for the oldest N seqnos (as mentioned by Radim)
* (Re-)implement retransmission backoffs to reduce xmit traffic. However, this requires additional state in the xmit tables.
> Size of XMIT_REQ is not limited
> -------------------------------
>
> Key: JGRP-1868
> URL: https://issues.jboss.org/browse/JGRP-1868
> Project: JGroups
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6
>
>
> When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
> {code}
> JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
> {code}
> It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1868?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1868:
--------------------------------
To prevent having to use a seond {{FRAG}}, we could
* Send multiple XMIT-REQs, whose SeqnoList arg is size-limited to 60K (if transport == UDP). {{SeqnoList.size()}} computes the exact size.
* Send am XMIT-REQ only for the oldest N seqnos (as mentioned by Radim)
* (Re-)implement retransmission backoffs to reduce xmit traffic. However, this requires additional state in the xmit tables.
> Size of XMIT_REQ is not limited
> -------------------------------
>
> Key: JGRP-1868
> URL: https://issues.jboss.org/browse/JGRP-1868
> Project: JGroups
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6
>
>
> When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
> {code}
> JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
> {code}
> It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1868?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-1868:
---------------------------
Fix Version/s: 3.6
There are a lot of other things that are wrong if a system has that many retransmissions. SeqnoList is optimized for size and if an XMIT-REQ grows over 65K then the first prio should not be to look into how to fix this, but how to prevent this. I suspect this is the old story again where thread pools are getting clogged by Infinispan...
The workaround is to place {{FRAG}} just above the transport.
> Size of XMIT_REQ is not limited
> -------------------------------
>
> Key: JGRP-1868
> URL: https://issues.jboss.org/browse/JGRP-1868
> Project: JGroups
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Radim Vansa
> Assignee: Bela Ban
> Fix For: 3.6
>
>
> When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
> {code}
> JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
> {code}
> It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1868) Size of XMIT_REQ is not limited
by Radim Vansa (JIRA)
Radim Vansa created JGRP-1868:
---------------------------------
Summary: Size of XMIT_REQ is not limited
Key: JGRP-1868
URL: https://issues.jboss.org/browse/JGRP-1868
Project: JGroups
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Radim Vansa
Assignee: Bela Ban
When UNICAST3 sends XMIT_REQ, it serializes SeqnoList as payload without any limits upon its size. If there are too many missing messages, the XMIT_REQ grows over TP.max_bundle_size and cannot be sent at all.
{code}
JGRP000029: edg-perf03-10774: failed sending message to edg-perf01-63702 (64072 bytes): java.lang.Exception: message size (64072) is greater than max bundling size (64000). Set the fragmentation/bundle size in FRAG and TP correctly, headers: UNICAST3: XMIT_REQ, seqno=0, UDP: [channel_name=default]
{code}
It's also undesirable to resend thousands of messages, as the receiver likely cannot process them all at once and only few of them will be actually processed. Therefore only X oldest ones (in order to cleanup the tables) should be resent.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1867) Use logger instead of [stdout] for JGroups GMS logger
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1867?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1867:
--------------------------------
Printing to stdout turns out to be useful in a lot of support cases where customers don't have logging configured and only attach the output.
> Use logger instead of [stdout] for JGroups GMS logger
> -----------------------------------------------------
>
> Key: JGRP-1867
> URL: https://issues.jboss.org/browse/JGRP-1867
> Project: JGroups
> Issue Type: Feature Request
> Security Level: Public(Everyone can see)
> Reporter: Panagiotis Sotiropoulos
> Assignee: Bela Ban
> Priority: Trivial
>
> [JBossINF] 12:11:12,677 INFO [org.jboss.as.osgi] (MSC service thread 1-15) JBAS011907: Register module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,143 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,210 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,441 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1867) Use logger instead of [stdout] for JGroups GMS logger
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1867?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-1867.
----------------------------
Resolution: Rejected
You can set {{GMS.print_local_addr}} to {{false}} to prevent printing to stdout.
> Use logger instead of [stdout] for JGroups GMS logger
> -----------------------------------------------------
>
> Key: JGRP-1867
> URL: https://issues.jboss.org/browse/JGRP-1867
> Project: JGroups
> Issue Type: Feature Request
> Security Level: Public(Everyone can see)
> Reporter: Panagiotis Sotiropoulos
> Assignee: Bela Ban
> Priority: Trivial
>
> [JBossINF] 12:11:12,677 INFO [org.jboss.as.osgi] (MSC service thread 1-15) JBAS011907: Register module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,143 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,210 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,441 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (JGRP-1867) Use logger instead of [stdout] for JGroups GMS logger
by Panagiotis Sotiropoulos (JIRA)
Panagiotis Sotiropoulos created JGRP-1867:
---------------------------------------------
Summary: Use logger instead of [stdout] for JGroups GMS logger
Key: JGRP-1867
URL: https://issues.jboss.org/browse/JGRP-1867
Project: JGroups
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Reporter: Panagiotis Sotiropoulos
Assignee: Bela Ban
Priority: Trivial
[JBossINF] 12:11:12,677 INFO [org.jboss.as.osgi] (MSC service thread 1-15) JBAS011907: Register module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader
[JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1)
[JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
[JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200
[JBossINF] 12:11:13,143 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
[JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1)
[JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
[JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200
[JBossINF] 12:11:13,210 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
[JBossINF] 12:11:13,441 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months
[JBoss JIRA] (WFLY-3713) Use logger instead of [stdout] for JGroups GMS logger
by Dominik Pospisil (JIRA)
[ https://issues.jboss.org/browse/WFLY-3713?page=com.atlassian.jira.plugin.... ]
Dominik Pospisil deleted WFLY-3713:
-----------------------------------
> Use logger instead of [stdout] for JGroups GMS logger
> -----------------------------------------------------
>
> Key: WFLY-3713
> URL: https://issues.jboss.org/browse/WFLY-3713
> Project: WildFly
> Issue Type: Feature Request
> Security Level: Public(Everyone can see)
> Reporter: Panagiotis Sotiropoulos
> Assignee: Radoslav Husar
> Priority: Trivial
>
> ...and possibly remove the unnecessary dash lines on the way.
> {noformat}
> [JBossINF] 12:11:12,677 INFO [org.jboss.as.osgi] (MSC service thread 1-15) JBAS011907: Register module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,142 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,143 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1)
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,209 INFO [stdout] (ChannelService lifecycle - 1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200
> [JBossINF] 12:11:13,210 INFO [stdout] (ChannelService lifecycle - 1) -------------------------------------------------------------------
> [JBossINF] 12:11:13,441 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
11 years, 2 months