[JBoss JIRA] (JGRP-1957) S3_PING: Nodes never removed from .list file
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1957?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-1957:
---------------------------
Fix Version/s: 3.6.6
(was: 3.6.5)
Pushed into 3.6.6 for now, so I can release 3.6.5. Not clear though what the solution should be...
> S3_PING: Nodes never removed from .list file
> --------------------------------------------
>
> Key: JGRP-1957
> URL: https://issues.jboss.org/browse/JGRP-1957
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.4
> Environment: JGroups client running on Mac OS X - Yosemite
> JDK 1.7.71
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Minor
> Fix For: 3.6.6
>
>
> I'm not 100% sure, but it seems like there might be a defect here.
> I'm using TCP, S3_PING, and MERGE3.
> I've set logical_addr_cache_max_size to 2 for testing purposes, although I don't think the value of this setting affects my test results.
> I start a single node, node A. Then I start a second node, node B.
> I then repeatedly shutdown and restart node B.
> Each time node B starts, a new row is added to the .list file stored in S3.
> But even if I continue this process for 15 minutes, old rows are never removed from the .list file, so it continues to grow in size.
> I've read the docs and mailing list threads, so I'm aware that the list is not immediately updated as soon as a member leaves. But I was expecting that when a view change occurs, nodes no longer in the view would be marked for removal (line 2193 of TP.java) and then after the logical_addr_cache_expiration has been reached and the reaper kicks in, once a new node joins, the expired cache entries would be purged from the file.
> I dug in to the code a bit, and what seems to be happening is that the MERGE3 protocol periodically generates a FIND_MBRS event. S3_PING retrieves the membership from the .list file, which includes expired nodes. And then all of these members are re-added to the logical address cache (line 157 of S3_PING.java, line 533 of Discovery.java, line 2263 of TP.java).
> So expired nodes are continually re-added to the logical address cache, preventing them from ever being reaped.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (JGRP-1957) S3_PING: Nodes never removed from .list file
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1957?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1957:
--------------------------------
What do you want me to change ?
> S3_PING: Nodes never removed from .list file
> --------------------------------------------
>
> Key: JGRP-1957
> URL: https://issues.jboss.org/browse/JGRP-1957
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.4
> Environment: JGroups client running on Mac OS X - Yosemite
> JDK 1.7.71
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Minor
> Fix For: 3.6.5
>
>
> I'm not 100% sure, but it seems like there might be a defect here.
> I'm using TCP, S3_PING, and MERGE3.
> I've set logical_addr_cache_max_size to 2 for testing purposes, although I don't think the value of this setting affects my test results.
> I start a single node, node A. Then I start a second node, node B.
> I then repeatedly shutdown and restart node B.
> Each time node B starts, a new row is added to the .list file stored in S3.
> But even if I continue this process for 15 minutes, old rows are never removed from the .list file, so it continues to grow in size.
> I've read the docs and mailing list threads, so I'm aware that the list is not immediately updated as soon as a member leaves. But I was expecting that when a view change occurs, nodes no longer in the view would be marked for removal (line 2193 of TP.java) and then after the logical_addr_cache_expiration has been reached and the reaper kicks in, once a new node joins, the expired cache entries would be purged from the file.
> I dug in to the code a bit, and what seems to be happening is that the MERGE3 protocol periodically generates a FIND_MBRS event. S3_PING retrieves the membership from the .list file, which includes expired nodes. And then all of these members are re-added to the logical address cache (line 157 of S3_PING.java, line 533 of Discovery.java, line 2263 of TP.java).
> So expired nodes are continually re-added to the logical address cache, preventing them from ever being reaped.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (JGRP-1956) S3_PING / FILE_PING: remove failed members
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1956?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1956:
--------------------------------
By default, neither attribute is enabled. Especially {{remove_all_files_on_view_change}} causes more network traffic. I suggest enable {{remove_old_coords_on_view_change}} only for now, as this handles members getting killed with kill -9, but doesn't cause as much traffic as enabling the other attr.
Re {{info_writer_max_writes_after_view}}: yes, you're right, 1 should be good enough in most cases.
> S3_PING / FILE_PING: remove failed members
> ------------------------------------------
>
> Key: JGRP-1956
> URL: https://issues.jboss.org/browse/JGRP-1956
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.4
> Reporter: Karsten Ohme
> Assignee: Bela Ban
> Fix For: 3.6.5
>
>
> When we terminate a member (EC2's "terminate" function) or kill -9 it, then the file (or bucket data in S3) won't get removed. This leads to stale data. On EC2, I expect that virtualized instances are often simply terminated, so this problem is compounded there.
> SOLUTION:
> - Periodically write own data to the file system (FILE_PING) or S3 (S3_PING)
> - On a view change: remove all data that's not in the current view
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (JGRP-1954) SWIFT_PING discovery protocol fatal error on OpenStack Kilo
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1954?page=com.atlassian.jira.plugin.... ]
Bela Ban resolved JGRP-1954.
----------------------------
Resolution: Done
Nick implemented the scripting engine approach suggested by Thomas, thanks !
> SWIFT_PING discovery protocol fatal error on OpenStack Kilo
> -----------------------------------------------------------
>
> Key: JGRP-1954
> URL: https://issues.jboss.org/browse/JGRP-1954
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.4
> Environment: JGroups client running on Mac OS X - Yosemite
> JDK 1.7.71
> OpenStack Kilo
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Fix For: 3.6.5
>
>
> I'm attempting to use the SWIFT_PING discovery protocol on the most recent version of OpenStack, "Kilo". An error occurs during initialization of the protocol stack, the stack trace is provided below.
> The problem appears to be that support for XML-formatted responses has been removed in the OpenStack Identity API (http://developer.openstack.org/api-ref-identity-v2.html). Even though SWIFT_PING sends an Accept header of application/xml, the response still comes back as JSON (around line 286 of SWIFT_PING.java).
> I've been able to repro the issue using Postman in Chrome. I tried providing the *request* in XML , with a Content-Type header of application/xml, but Swift returns an error: "Expecting to find application/json in Content-Type header".
> It seems like the resolution would be for SWIFT_PING to be modified so it can parse the JSON response that it is receiving. If that sounds like a reasonable approach, I can try to create a patch that fixes the issue.
> Stack Trace:
> 2015-08-21 14:30:16,123 FATAL [com.pingidentity.common.util.ErrorHandler] Problem creating factory for multiplexed cluster communications
> org.xml.sax.SAXParseException: Content is not allowed in prolog.
> at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257) ~[?:1.8.0_25]
> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:348) ~[?:1.8.0_25]
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[?:1.8.0_25]
> at org.jgroups.protocols.SWIFT_PING$Keystone_V_2_0_Auth.authenticate(SWIFT_PING.java:307) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.protocols.SWIFT_PING$SwiftClient.authenticate(SWIFT_PING.java:443) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.protocols.SWIFT_PING.init(SWIFT_PING.java:68) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:860) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:481) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.init(JChannel.java:854) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.<init>(JChannel.java:159) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.<init>(JChannel.java:120) ~[jgroups.jar:3.6.4.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (JGRP-1954) SWIFT_PING discovery protocol fatal error on OpenStack Kilo
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1954?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-1954:
---------------------------
Fix Version/s: 3.6.5
(was: 3.6.6)
> SWIFT_PING discovery protocol fatal error on OpenStack Kilo
> -----------------------------------------------------------
>
> Key: JGRP-1954
> URL: https://issues.jboss.org/browse/JGRP-1954
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.4
> Environment: JGroups client running on Mac OS X - Yosemite
> JDK 1.7.71
> OpenStack Kilo
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Fix For: 3.6.5
>
>
> I'm attempting to use the SWIFT_PING discovery protocol on the most recent version of OpenStack, "Kilo". An error occurs during initialization of the protocol stack, the stack trace is provided below.
> The problem appears to be that support for XML-formatted responses has been removed in the OpenStack Identity API (http://developer.openstack.org/api-ref-identity-v2.html). Even though SWIFT_PING sends an Accept header of application/xml, the response still comes back as JSON (around line 286 of SWIFT_PING.java).
> I've been able to repro the issue using Postman in Chrome. I tried providing the *request* in XML , with a Content-Type header of application/xml, but Swift returns an error: "Expecting to find application/json in Content-Type header".
> It seems like the resolution would be for SWIFT_PING to be modified so it can parse the JSON response that it is receiving. If that sounds like a reasonable approach, I can try to create a patch that fixes the issue.
> Stack Trace:
> 2015-08-21 14:30:16,123 FATAL [com.pingidentity.common.util.ErrorHandler] Problem creating factory for multiplexed cluster communications
> org.xml.sax.SAXParseException: Content is not allowed in prolog.
> at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257) ~[?:1.8.0_25]
> at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:348) ~[?:1.8.0_25]
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[?:1.8.0_25]
> at org.jgroups.protocols.SWIFT_PING$Keystone_V_2_0_Auth.authenticate(SWIFT_PING.java:307) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.protocols.SWIFT_PING$SwiftClient.authenticate(SWIFT_PING.java:443) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.protocols.SWIFT_PING.init(SWIFT_PING.java:68) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:860) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:481) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.init(JChannel.java:854) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.<init>(JChannel.java:159) ~[jgroups.jar:3.6.4.Final]
> at org.jgroups.JChannel.<init>(JChannel.java:120) ~[jgroups.jar:3.6.4.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (WFCORE-949) Removing http-interface autocompletes but fails
by Petr Kremensky (JIRA)
[ https://issues.jboss.org/browse/WFCORE-949?page=com.atlassian.jira.plugin... ]
Petr Kremensky moved JBEAP-981 to WFCORE-949:
---------------------------------------------
Project: WildFly Core (was: JBoss Enterprise Application Platform)
Key: WFCORE-949 (was: JBEAP-981)
Workflow: GIT Pull Request workflow (was: CDW v1)
Component/s: CLI
Domain Management
(was: CLI)
Target Release: (was: 7.0.0.GA)
Affects Version/s: 2.0.0.Beta5
(was: 7.0.0.DR9)
> Removing http-interface autocompletes but fails
> -----------------------------------------------
>
> Key: WFCORE-949
> URL: https://issues.jboss.org/browse/WFCORE-949
> Project: WildFly Core
> Issue Type: Bug
> Components: CLI, Domain Management
> Affects Versions: 2.0.0.Beta5
> Reporter: Petr Kremensky
> Assignee: Alexey Loubyansky
>
> Executing /core-service=management/management-interface=http-interface:remove() autocompletes but does not succeed.
> This is a link to a paste of the server stacktrace.
> http://pastebin.test.redhat.com/310136
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (WFCORE-628) Support for 'query' operation on legacy slaves
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-628?page=com.atlassian.jira.plugin... ]
Brian Stansberry reassigned WFCORE-628:
---------------------------------------
Assignee: Brian Stansberry
> Support for 'query' operation on legacy slaves
> ----------------------------------------------
>
> Key: WFCORE-628
> URL: https://issues.jboss.org/browse/WFCORE-628
> Project: WildFly Core
> Issue Type: Feature Request
> Components: Domain Management
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Priority: Critical
> Fix For: 2.0.0.CR1
>
>
> WFCORE-287 will introduce a new 'query' operation. But in a mixed domain, slaves running legacy versions will not understand that operation. So the console, when managing such a domain will either need to analyze the management API version of each host and not target 'query' ops to legacy hosts, or we need to figure out a way to emulate support on the DC.
> One thought I had was to use a transformer. My understanding of how query works is it's essentially a read-resource and then some post-processing is applied to filter the result. So my thought is an operation transformer can convert the query op to a read-resource op, which the slave will understand, and then a result transformer on the DC does the filtering work.
> There may of course be gotchas there.
> One issue I see is I think this query op is going to be a global op. I don't believe we currently have hooks for registering transformers for global ops. That is, there's nothing like the 'inherited' handling we use for registering the handler for the global op and then discovering it when resolving the operation name against child resource registrations.
> Another issue to remember is 'query' op will support wildcard addresses, so the result transformer will need to understand how to detect and deal with the different result format from a wildcard read-resource (result is a LIST of OBJECT) versus a simple read-resource (just an OBJECT).
> It's better if the filtering is done as soon as possible (to save the cost of transmitting data from servers to slaves to the DC just so the DC can filter) but for this mixed domain case I think it's ok to send the full read-resource result back to the DC.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months
[JBoss JIRA] (WFCORE-621) Support legacy slaves when invoking wildcard reads in a domain
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-621?page=com.atlassian.jira.plugin... ]
Brian Stansberry reassigned WFCORE-621:
---------------------------------------
Assignee: Brian Stansberry
> Support legacy slaves when invoking wildcard reads in a domain
> --------------------------------------------------------------
>
> Key: WFCORE-621
> URL: https://issues.jboss.org/browse/WFCORE-621
> Project: WildFly Core
> Issue Type: Enhancement
> Components: Domain Management
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Priority: Critical
> Fix For: 2.0.0.CR1
>
>
> This is a follow-on to WFCORE-282.
> The WFCORE-282 solution will not work for requests with address patterns {code}/host=*/server=*{code} or {code}/host=somename/server=*{code} if the host named 'somename' is running a WFCORE version prior to 1.0.0.CR1 (or whatever release first has WFCORE-282 introduced.)
> In particular, it won't work with slaves running EAP 6.x.
> The problem is with either of those address patterns the DC will send a request addressed to {code}/host=somename/server=*{code} to the slave, and the slave will not be able to handle it, as it won't have the WFCORE-282 logic that lets it identify the relevant servers and send requests on to them.
> Potentially this could be fixed by having the DC detect these patterns and not call {code}/host=somename/server=*{code}, instead adding a step to read the server child names from /host=somename and then call /host=somename/server=a, /host=somename/server=b, etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 10 months