September 2015 - jboss-jira - Jboss List Archives

[JBoss JIRA] (JGRP-1958) RequestCorrelator "channel is not connected" error during shutdown

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1958?page=com.atlassian.jira.plugin.... ] Bela Ban updated JGRP-1958: --------------------------- Fixed in {{UNICAST2}} on master by removing the stack trace in the WARN log message. If a backport is needed, please cherry-pick the changes yourself. > RequestCorrelator "channel is not connected" error during shutdown > ------------------------------------------------------------------ > > Key: JGRP-1958 > URL: https://issues.jboss.org/browse/JGRP-1958 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.2.12 > Reporter: Dennis Reed > Assignee: Bela Ban > Priority: Minor > Fix For: 3.6.5 > > > Error logged during shutdown of a channel due to RequestCorrelator failing to send a reply: > ERROR [org.jgroups.protocols.UNICAST2] (OOB-17,shared=tcp) couldn't deliver OOB message [dst: server1/web, src: server2/web (4 headers), size=62 bytes, flags=OOB|DONT_BUNDLE|RSVP]: java.lang.IllegalStateException: channel is not connected > at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:617) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:544) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > [incoming JGroups message] > It appears to just be a timing issue between shutdown of the channel and RequestCorrelator processing the message, which triggers a response message. > It would be good to either avoid triggering the exception in the first place, or suppress the error log during shutdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1957) S3_PING: Nodes never removed from .list file

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1957?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1957: -------------------------------- This is similar to https://issues.jboss.org/browse/JGRP-1956: nodes cannot be removed from the {{x.list}} file because they might not have crashed, but were separated away by a network partition. If you enable the 2 attributes I listed in JGRP-1956 and reduce the size and expiration time of the logical address cache, the old members will get removed. Just tested this locally. Can you close the issue if this works for you ? Otherwise, contact me on IRC, as I want to release a 3.6.5 before before the end of the week. Otherwise, this issue gets pushed into 3.6.6. > S3_PING: Nodes never removed from .list file > -------------------------------------------- > > Key: JGRP-1957 > URL: https://issues.jboss.org/browse/JGRP-1957 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.6.4 > Environment: JGroups client running on Mac OS X - Yosemite > JDK 1.7.71 > Reporter: Nick Sawadsky > Assignee: Bela Ban > Priority: Minor > Fix For: 3.6.5 > > > I'm not 100% sure, but it seems like there might be a defect here. > I'm using TCP, S3_PING, and MERGE3. > I've set logical_addr_cache_max_size to 2 for testing purposes, although I don't think the value of this setting affects my test results. > I start a single node, node A. Then I start a second node, node B. > I then repeatedly shutdown and restart node B. > Each time node B starts, a new row is added to the .list file stored in S3. > But even if I continue this process for 15 minutes, old rows are never removed from the .list file, so it continues to grow in size. > I've read the docs and mailing list threads, so I'm aware that the list is not immediately updated as soon as a member leaves. But I was expecting that when a view change occurs, nodes no longer in the view would be marked for removal (line 2193 of TP.java) and then after the logical_addr_cache_expiration has been reached and the reaper kicks in, once a new node joins, the expired cache entries would be purged from the file. > I dug in to the code a bit, and what seems to be happening is that the MERGE3 protocol periodically generates a FIND_MBRS event. S3_PING retrieves the membership from the .list file, which includes expired nodes. And then all of these members are re-added to the logical address cache (line 157 of S3_PING.java, line 533 of Discovery.java, line 2263 of TP.java). > So expired nodes are continually re-added to the logical address cache, preventing them from ever being reaped. -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1956) S3_PING / FILE_PING: remove failed members

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1956?page=com.atlassian.jira.plugin.... ] Bela Ban resolved JGRP-1956. ---------------------------- Resolution: Won't Fix I'm closing this as the instructions in my last comment should fix this issue. If this is not the case, please re-open. > S3_PING / FILE_PING: remove failed members > ------------------------------------------ > > Key: JGRP-1956 > URL: https://issues.jboss.org/browse/JGRP-1956 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.6.4 > Reporter: Karsten Ohme > Assignee: Bela Ban > Fix For: 3.6.5 > > > When we terminate a member (EC2's "terminate" function) or kill -9 it, then the file (or bucket data in S3) won't get removed. This leads to stale data. On EC2, I expect that virtualized instances are often simply terminated, so this problem is compounded there. > SOLUTION: > - Periodically write own data to the file system (FILE_PING) or S3 (S3_PING) > - On a view change: remove all data that's not in the current view -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1956) S3_PING / FILE_PING: remove failed members

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1956?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1956 at 9/2/15 2:47 AM: -------------------------------------------------------- Can you try with the following attributes enabled: ? * {{remove_old_coords_on_view_change}} * {{remove_all_files_on_view_change}} The reason old members are not immediately removed is that these members could have been split away, in a network partition, rather than crashed. If we want a merge to succeed in such a case, it is better to leave information about them in the store. Note that {{TP.logical_addr_cache_max_size}} and {{TP.logical_addr_cache_expiration}} govern when stale entries will be removed. By default, you won't have more than 2000 stale elements in the cache. Take a look at https://issues.jboss.org/browse/JGRP-1917 for details. Doc: http://www.jgroups.org/manual/index.html#FILE_PING (removal of zombie files) was (Author: belaban): Can you try with the following attributes enabled: ? * {{remove_old_coords_on_view_change}} * {{remove_all_files_on_view_change}} The reason old members are not immediately removed is that these members could have been split away, in a network partition, rather than crashed. If we want a merge to succeed in such a case, it is better to leave information about them in the store. Note that {{TP.logical_addr_cache_max_size}} and {{TP.logical_addr_cache_expiration}} govern when stale entries will be removed. By default, you won't have more than 2000 stale elements in the cache. Take a look at https://issues.jboss.org/browse/JGRP-1917 for details > S3_PING / FILE_PING: remove failed members > ------------------------------------------ > > Key: JGRP-1956 > URL: https://issues.jboss.org/browse/JGRP-1956 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.6.4 > Reporter: Karsten Ohme > Assignee: Bela Ban > Fix For: 3.6.5 > > > When we terminate a member (EC2's "terminate" function) or kill -9 it, then the file (or bucket data in S3) won't get removed. This leads to stale data. On EC2, I expect that virtualized instances are often simply terminated, so this problem is compounded there. > SOLUTION: > - Periodically write own data to the file system (FILE_PING) or S3 (S3_PING) > - On a view change: remove all data that's not in the current view -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1956) S3_PING / FILE_PING: remove failed members

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1956?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1956 at 9/2/15 2:44 AM: -------------------------------------------------------- Can you try with the following attributes enabled: ? * {{remove_old_coords_on_view_change}} * {{remove_all_files_on_view_change}} The reason old members are not immediately removed is that these members could have been split away, in a network partition, rather than crashed. If we want a merge to succeed in such a case, it is better to leave information about them in the store. Note that {{TP.logical_addr_cache_max_size}} and {{TP.logical_addr_cache_expiration}} govern when stale entries will be removed. By default, you won't have more than 2000 stale elements in the cache. Take a look at https://issues.jboss.org/browse/JGRP-1917 for details was (Author: belaban): Can you try with the following attributes enabled: ? * {{remove_old_coords_on_view_change}} * {{remove_all_files_on_view_change}} The reason old members are not immediately removed is that these members could have been split away, in a network partition, rather than crashed. If we want a merge to succeed in such a case, it is better to leave information about them in the store. Note that {{TP.logical_addr_cache_max_size}} and {{TP.logical_addr_cache_expiration}} govern when stale entries will be removed. By default, you won't have more than 2000 stale elements in the cache. > S3_PING / FILE_PING: remove failed members > ------------------------------------------ > > Key: JGRP-1956 > URL: https://issues.jboss.org/browse/JGRP-1956 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.6.4 > Reporter: Karsten Ohme > Assignee: Bela Ban > Fix For: 3.6.5 > > > When we terminate a member (EC2's "terminate" function) or kill -9 it, then the file (or bucket data in S3) won't get removed. This leads to stale data. On EC2, I expect that virtualized instances are often simply terminated, so this problem is compounded there. > SOLUTION: > - Periodically write own data to the file system (FILE_PING) or S3 (S3_PING) > - On a view change: remove all data that's not in the current view -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1956) S3_PING / FILE_PING: remove failed members

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1956?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1956: -------------------------------- Can you try with the following attributes enabled: ? * {{remove_old_coords_on_view_change}} * {{remove_all_files_on_view_change}} The reason old members are not immediately removed is that these members could have been split away, in a network partition, rather than crashed. If we want a merge to succeed in such a case, it is better to leave information about them in the store. Note that {{TP.logical_addr_cache_max_size}} and {{TP.logical_addr_cache_expiration}} govern when stale entries will be removed. By default, you won't have more than 2000 stale elements in the cache. > S3_PING / FILE_PING: remove failed members > ------------------------------------------ > > Key: JGRP-1956 > URL: https://issues.jboss.org/browse/JGRP-1956 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.6.4 > Reporter: Karsten Ohme > Assignee: Bela Ban > Fix For: 3.6.5 > > > When we terminate a member (EC2's "terminate" function) or kill -9 it, then the file (or bucket data in S3) won't get removed. This leads to stale data. On EC2, I expect that virtualized instances are often simply terminated, so this problem is compounded there. > SOLUTION: > - Periodically write own data to the file system (FILE_PING) or S3 (S3_PING) > - On a view change: remove all data that's not in the current view -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (WFLY-5220) Tests in testsuite/integration/basic runs more time when -Dtest maven argument is used

by Josef Cacek (JIRA)

[ https://issues.jboss.org/browse/WFLY-5220?page=com.atlassian.jira.plugin.... ] Josef Cacek commented on WFLY-5220: ----------------------------------- Stuart, how it breaks the usage? The correct is only one active surefire exeuction (with {{standalone-full.xml}} profile}}) when the {{-Dtest}} is used - as it's described in the profile comment. {code:xml}  {code} It worked as described in the past, but stopped to work after this change: https://github.com/wildfly/wildfly/commit/005567fe438049426292e4ca270060e... > Tests in testsuite/integration/basic runs more time when -Dtest maven argument is used > -------------------------------------------------------------------------------------- > > Key: WFLY-5220 > URL: https://issues.jboss.org/browse/WFLY-5220 > Project: WildFly > Issue Type: Bug > Components: Test Suite > Affects Versions: 10.0.0.Beta2 > Reporter: Josef Cacek > Assignee: Josef Cacek > > Multiple maven-surefire-plugin executions are started when {{-Dtest}} maven switch is used in {{testsuite/integration/basic}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (WFLY-5220) Tests in testsuite/integration/basic runs more time when -Dtest maven argument is used

by Josef Cacek (JIRA)

[ https://issues.jboss.org/browse/WFLY-5220?page=com.atlassian.jira.plugin.... ] Josef Cacek edited comment on WFLY-5220 at 9/2/15 2:24 AM: ----------------------------------------------------------- Stuart, how it breaks the usage? The correct is only one active surefire exeuction (with {{standalone-full.xml}} profile) when the {{-Dtest}} is used - as it's described in the profile comment. {code:xml}  {code} It worked as described in the past, but stopped to work after this change: https://github.com/wildfly/wildfly/commit/005567fe438049426292e4ca270060e... was (Author: jcacek): Stuart, how it breaks the usage? The correct is only one active surefire exeuction (with {{standalone-full.xml}} profile}}) when the {{-Dtest}} is used - as it's described in the profile comment. {code:xml}  {code} It worked as described in the past, but stopped to work after this change: https://github.com/wildfly/wildfly/commit/005567fe438049426292e4ca270060e... > Tests in testsuite/integration/basic runs more time when -Dtest maven argument is used > -------------------------------------------------------------------------------------- > > Key: WFLY-5220 > URL: https://issues.jboss.org/browse/WFLY-5220 > Project: WildFly > Issue Type: Bug > Components: Test Suite > Affects Versions: 10.0.0.Beta2 > Reporter: Josef Cacek > Assignee: Josef Cacek > > Multiple maven-surefire-plugin executions are started when {{-Dtest}} maven switch is used in {{testsuite/integration/basic}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (WFLY-5259) truststore path is ignored if provider is not JKS

by Arto Huusko (JIRA)

Arto Huusko created WFLY-5259: --------------------------------- Summary: truststore path is ignored if provider is not JKS Key: WFLY-5259 URL: https://issues.jboss.org/browse/WFLY-5259 Project: WildFly Issue Type: Feature Request Components: ConfigAdmin, Security Affects Versions: 9.0.1.Final Reporter: Arto Huusko Assignee: Thomas Diesler truststore configuration ignores the path and relative-to parameters if the truststore provider is anything else than JKS. This works as documented, but it is not correct. There can be and are truststore implementations that need to load parameters or whatever data from a file, and the current implementation prevents these truststore providers from working. We have a custom truststore that is loaded from database, and database access parameters are read from a properties file. When trying to use this with Wildfly 9, the keystore engineLoad parameter is passed in as null, even though path and relative-to are configured. Even standard java supports PKCS12 truststores, where the same problem would occur. So I would suggest that - if provider is JKS, path is mandatory - if provider is not JKS, but path is specified, it is passed to the provider -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

[JBoss JIRA] (JGRP-1958) RequestCorrelator "channel is not connected" error during shutdown

by Bela Ban (JIRA)

[ https://issues.jboss.org/browse/JGRP-1958?page=com.atlassian.jira.plugin.... ] Bela Ban updated JGRP-1958: --------------------------- Fix Version/s: 3.6.5 > RequestCorrelator "channel is not connected" error during shutdown > ------------------------------------------------------------------ > > Key: JGRP-1958 > URL: https://issues.jboss.org/browse/JGRP-1958 > Project: JGroups > Issue Type: Bug > Affects Versions: 3.2.12 > Reporter: Dennis Reed > Assignee: Bela Ban > Priority: Minor > Fix For: 3.6.5 > > > Error logged during shutdown of a channel due to RequestCorrelator failing to send a reply: > ERROR [org.jgroups.protocols.UNICAST2] (OOB-17,shared=tcp) couldn't deliver OOB message [dst: server1/web, src: server2/web (4 headers), size=62 bytes, flags=OOB|DONT_BUNDLE|RSVP]: java.lang.IllegalStateException: channel is not connected > at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:617) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:544) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600) [jgroups-3.2.12.Final-redhat-1.jar:3.2.12.Final-redhat-1] > [incoming JGroups message] > It appears to just be a timing issue between shutdown of the channel and RequestCorrelator processing the message, which triggers a response message. > It would be good to either avoid triggering the exception in the first place, or suppress the error log during shutdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346)

10 years, 10 months

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

jboss-jira September 2015