[JBoss JIRA] (JGRP-2296) DNS_PING is dropping port values with SRV based service discovery
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2296?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2296:
---------------------------
Fix Version/s: 4.0.15
> DNS_PING is dropping port values with SRV based service discovery
> -----------------------------------------------------------------
>
> Key: JGRP-2296
> URL: https://issues.jboss.org/browse/JGRP-2296
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.11
> Environment: JGroups version 4.0.11.Final
> Used in Keycloak 4.4.0
> Deployed as Jboss based Docker container from jboss/keycloak into AWS ECS
> Reporter: Eric Thompson
> Assignee: Bela Ban
> Priority: Blocker
> Fix For: 4.0.15
>
>
> Using DNS_PING in Jgroups 4.0.11 and SRV records the port from the SRV record is being dropped (set to zero) and the default is used instead (7600).
> I am using this Jgroups config:
> {code}
> <subsystem xmlns="urn:jboss:domain:jgroups:6.0">
> <channels default="ee">
> <channel name="ee" stack="tcp" cluster="ejb"/>
> </channels>
> <stacks>
> <stack name="tcp">
> <transport type="TCP" socket-binding="jgroups-tcp">
> <property name="external_addr">${env.EXTERNAL_ADDR}</property>
> </transport>
> <protocol type="dns.DNS_PING">
> <property name="dns_query">
> jgroups.${env.DNS_NAME}.svc.cluster.local
> </property>
> <property name="dns_record_type">
> SRV
> </property>
> </protocol>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="MFC"/>
> <protocol type="FRAG3"/>
> </stack>
> </stacks>
> </subsystem>
> {code}
> I have these service discovery DNS entries
> {code}
> $ dig jgroups.dev.auth.sonatype.com.svc.cluster.local SRV
> ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.68.rc1.58.amzn1 <<>> jgroups.dev.auth.sonatype.com.svc.cluster.local SRV
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16690
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
> ;; QUESTION SECTION:
> ;jgroups.dev.auth.sonatype.com.svc.cluster.local. IN SRV
> ;; ANSWER SECTION:
> jgroups.dev.auth.sonatype.com.svc.cluster.local. 10 IN SRV 1 1 32921 9ec82e3f-3a0e-4e30-b785-17879c63cd7d.jgroups.dev.auth.sonatype.com.svc.cluster.local.
> jgroups.dev.auth.sonatype.com.svc.cluster.local. 10 IN SRV 1 1 32923 60b5a820-9678-4bd2-84c6-00061a52bde0.jgroups.dev.auth.sonatype.com.svc.cluster.local.
> jgroups.dev.auth.sonatype.com.svc.cluster.local. 10 IN SRV 1 1 32915 9d9d78d0-8919-4b91-9df8-2e4e65afedae.jgroups.dev.auth.sonatype.com.svc.cluster.local.
> jgroups.dev.auth.sonatype.com.svc.cluster.local. 10 IN SRV 1 1 32917 161f3d66-f1e3-46f4-a44f-ebda925a25c6.jgroups.dev.auth.sonatype.com.svc.cluster.local.
> ;; Query time: 2 msec
> ;; SERVER: 10.42.3.2#53(10.42.3.2)
> ;; WHEN: Fri Sep 21 01:45:44 2018
> ;; MSG SIZE rcvd: 481
> {code}
> But I get this in the logs when running Keycloak in standalone cluster:
> {code}
> 17:45:10,121 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Performing initial discovery
> 17:45:10,154 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Entries collected from DNS: [10.42.3.56:0, 10.42.3.56:0, 10.42.3.44:0, 10.42.3.44:0]
> 17:45:10,155 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Discovered IP Address with port 0 (10.42.3.56:0). Replacing with default Transport port: 7600
> 17:45:10,159 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Discovered IP Address with port 0 (10.42.3.56:0). Replacing with default Transport port: 7600
> 17:45:10,159 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Discovered IP Address with port 0 (10.42.3.44:0). Replacing with default Transport port: 7600
> 17:45:10,159 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Discovered IP Address with port 0 (10.42.3.44:0). Replacing with default Transport port: 7600
> 17:45:10,159 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) Performing discovery of the following hosts [10.42.3.56:7600, 10.42.3.44:7600, e200a617bf7a]
> 17:45:10,159 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) e200a617bf7a: sending discovery request to 10.42.3.56:7600
> 17:45:10,160 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) e200a617bf7a: sending discovery request to 10.42.3.44:7600
> 17:45:10,160 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-10,ejb,e200a617bf7a) Received discovery from: e200a617bf7a, IP: 10.42.3.44:7600
> 17:45:10,161 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-3,null,null) e200a617bf7a: sending discovery request to e200a617bf7a
> 17:45:10,162 DEBUG [org.jgroups.protocols.dns.DNS_PING] (thread-11,ejb,e200a617bf7a) Received discovery from: e200a617bf7a, IP: 10.42.3.44:7600
> {code}
> As you can see it is resolving the DNS addresses, but discarding the ports.
> To be clear, in this example 32923 ids the port (eg:
> 1 1 32923 60b5a820-9678-4bd2-84c6-00061a52bde0.jgroups.dev.auth.sonatype.com.svc.cluster.local).
> These are dynamic ports mapped to port 7600 in order to put more Keycloak containers on each instance.
> {code}
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> f67e39f8f403 datadog/agent:latest-jmx "/init" 8 hours ago Up 8 hours (healthy) 8125/udp, 8126/tcp ecs-auth-service-dev-26-datadog-agent-a2b7f783ddd0ba9cf601
> bbb12f0c43a5 233747045000.dkr.ecr.us-east-2.amazonaws.com/ops/keycloak:latest "/opt/jboss/tools/do…" 8 hours ago Up 8 hours 0.0.0.0:32923->7600/tcp, 0.0.0.0:32922->8080/tcp ecs-auth-service-dev-26-keycloak-f4bd8f8dca9fd4cd4f00
> 932cad7c4fb9 datadog/agent:latest-jmx "/init" 8 hours ago Up 8 hours (healthy) 8125/udp, 8126/tcp ecs-auth-service-dev-26-datadog-agent-baa38a98ccaddea6f501
> e200a617bf7a 233747045000.dkr.ecr.us-east-2.amazonaws.com/ops/keycloak:latest "/opt/jboss/tools/do…" 8 hours ago Up 8 hours 0.0.0.0:32921->7600/tcp, 0.0.0.0:32920->8080/tcp ecs-auth-service-dev-26-keycloak-e6f398e6cc8db5b5f101
> 73bc0b863c73 amazon/amazon-ecs-agent:latest "/agent" 2 days ago Up 2 days ecs-agent
> {code}
> This seems like it might be where ports are getting lost:
> https://github.com/belaban/JGroups/blob/07060c3ba6e52ad4aad3ac799c2bc95ff...
> I don't see the port number being extracted from the SRV entry and appended to the IP returned from resolveAEntries.
> Let me know if I am missing any details. This is a major blocker for development.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2288:
--------------------------------
Hi [~nsawadsky] I haven't looked at this yet.. will do so within the next weeks
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Fix For: 3.6.16, 4.0.15
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
[JBoss JIRA] (JGRP-2157) KEY_EXCHANGE protocol
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2157?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2157:
---------------------------
Security Sensitive Issue: This issue is security relevant
Security: (was: Security Issue)
> KEY_EXCHANGE protocol
> ---------------------
>
> Key: JGRP-2157
> URL: https://issues.jboss.org/browse/JGRP-2157
> Project: JGroups
> Issue Type: Feature Request
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0.5
>
>
> {{ASYM_ENCRYPT}} needs to exchange the shared key used for encrypting and decrypting messages between members, when a new member joins, when a member leaves, or on a merge.
> This is currently done by code in {{ASYM_ENCRYPT}} itself. However, I want to move this functionality into a separate protocol {{KEY_EXCHANGE}}, which supports an event {{GET_SHARED_KEY}}, called by {{ASYM_ENCRYPT}}. This event could be implemented by different protocols.
> An implementation could use certificates and SSL sockets to establish a connection from the joiner to the key server (coordinator) in order to fetch the shared (symmetric) key. The joiner would not only have to check the certificate of the keyserver, but also the user CN.
> Another implementation would be the current code, moved into a separate protocol, plus certificate handling added, e.g. checking of the certificate chain, plus use CN verification.
> The goals are:
> # Refactoring of the key exchange code to support multiple implementations
> # Prevent man-in-the-middle-attacks. Because certificates are used, checking of the certificate chain prevents MitM attacks
> # Make AUTH optional. When certificate are used, obtaining the shared symmetric encryption key authenticates the client (joiner), so {{AUTH}} is not needed in this case.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
[JBoss JIRA] (JGRP-2294) ForkProtocolStack ignores UnknownForkHandler for message batches
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2294?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2294:
---------------------------
Fix Version/s: 4.0.15
> ForkProtocolStack ignores UnknownForkHandler for message batches
> ----------------------------------------------------------------
>
> Key: JGRP-2294
> URL: https://issues.jboss.org/browse/JGRP-2294
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.13
> Reporter: Dan Berindei
> Assignee: Bela Ban
> Fix For: 4.0.15
>
>
> {{ForkProtocolStack.up(MessageBatch)}} discards messages if it doesn't the fork channel id isn't registered and they are part of a batch:
> {code:java}
> log.debug("fork-channel for id=%s not found; discarding message", fork_channel_id);
> {code}
> -I believe the check should also be enhanced to invoke the {{UnknownForkHandler}} if the {{ForkChannel}} is registered but is not yet connected. Otherwise messages that arrive after the {{ForkChannel}} registers itself in the constructor and before our {{JGroupsTransport}} registers itself as an {{UpHandler}} with the {{ForkChannel}} will also be discarded.-
> I misread the {{ForkChannel}} code, the registratioon in the {{ForkProtocolStack}} only happens in {{ForkChannel.connect()}}.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months
[JBoss JIRA] (WFLY-10812) JDK11 Some security tests fail on closed channel exception
by Richard Opalka (JIRA)
[ https://issues.jboss.org/browse/WFLY-10812?page=com.atlassian.jira.plugin... ]
Richard Opalka resolved WFLY-10812.
-----------------------------------
Release Notes Text:
Sorry Honza for false alarm of course this is fixed. Just verified it on all
JDKs 8,9,10 & 11
Resolution: Done
> JDK11 Some security tests fail on closed channel exception
> ----------------------------------------------------------
>
> Key: WFLY-10812
> URL: https://issues.jboss.org/browse/WFLY-10812
> Project: WildFly
> Issue Type: Bug
> Components: Test Suite
> Reporter: Richard Opalka
> Assignee: Jan Kalina
> Priority: Blocker
> Fix For: 15.0.0.Alpha1
>
>
> With latest JDK11 (build 11-ea+25) the following tests fail due to closed channel exception: modified:
> * testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/security/loginmodules/LdapExtLikeAdvancedLdapLMTestCase.java
> * testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/security/loginmodules/LdapExtLoginModuleTestCase.java
> * testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/security/loginmodules/LdapLoginModuleTestCase.java
> * testsuite/integration/elytron/src/test/java/org/wildfly/test/integration/elytron/ssl/UndertowTwoWaySslNeedClientAuthTestCase.java
> * testsuite/integration/iiop/src/test/java/org/jboss/as/test/iiopssl/basic/IIOPSslInvocationTestCase.java
> * testsuite/integration/manualmode/src/test/java/org/jboss/as/test/manualmode/security/OutboundLdapConnectionClientCertTestCase.java
> * testsuite/integration/manualmode/src/test/java/org/jboss/as/test/manualmode/security/OutboundLdapConnectionTestCase.java
> * testsuite/integration/manualmode/src/test/java/org/jboss/as/test/manualmode/web/ssl/CertificateRolesLoginModuleTestCase.java
> * testsuite/integration/manualmode/src/test/java/org/jboss/as/test/manualmode/web/ssl/DatabaseCertLoginModuleTestCase.java
> * testsuite/integration/manualmode/src/test/java/org/jboss/as/test/manualmode/web/ssl/HTTPSWebConnectorTestCase.java
> * testsuite/integration/ws/src/test/java/org/jboss/as/test/integration/ws/wsse/trust/WSTrustTestCase.java
> Is there a known SSL issue in JDK11? Could somebody from our security team investigate this issue?
> When these tests will start passing WF test suite will be passing 100% of tests on JDK11.
> Potential investigator from our security team may want to include this PR:
> https://github.com/wildfly/wildfly/pull/11499
> to fix known JDK11 issues.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 9 months