[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by Flavia Rainone (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
Flavia Rainone commented on WFLY-11292:
---------------------------------------
I was mistaken, forget what I said. The close handler needs to create an exception to be provided as an EJBException for the result of any ongoing client invocation to the dropped server... still, I created a PR for EJBCLIENT-314.
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs-10clients.zip, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (WFLY-11495) WildFly 14-15 Create Query
by Majid Mostafavi (Jira)
[ https://issues.jboss.org/browse/WFLY-11495?page=com.atlassian.jira.plugin... ]
Majid Mostafavi commented on WFLY-11495:
----------------------------------------
{color:#00875A}13:14:29,440 INFO [stdout] (default task-15) select
13:14:29,440 INFO [stdout] (default task-15) *
13:14:29,440 INFO [stdout] (default task-15) from{color}
13:14:29,440 INFO [stdout] (default task-15) ( select
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.ID as ID1_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.active as active2_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.created as created3_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.createdBy as createdBy4_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.updated as updated5_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.updatedBy as updatedBy6_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.CM_ORGANIZATION_ID as CM_ORGANIZATION_I14_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.BANK_ACCOUNT_KIND_ID as BANK_ACCOUNT_KIND15_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.BANK_ACCOUNT_TYPE_ID as BANK_ACCOUNT_TYPE16_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.BANK_ACCOUNT_TYPE_ENUM as BANK_ACCOUNT_TYPE_7_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.BANK_BRANCH_ID as BANK_BRANCH_ID17_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.ACCOUNT_NUMBER as ACCOUNT_NUMBER8_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.DESCRIPTION as DESCRIPTION9_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.END_DATE as END_DATE10_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.NAME as NAME11_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.SHEBA as SHEBA12_417_,
13:14:29,440 INFO [stdout] (default task-15) trbankacco0_.START_DATE as START_DATE13_417_
13:14:29,440 INFO [stdout] (default task-15) from
13:14:29,440 INFO [stdout] (default task-15) NEJAT.TR_BANK_ACCOUNTS trbankacco0_
13:14:29,440 INFO [stdout] (default task-15) inner join
13:14:29,440 INFO [stdout] (default task-15) NEJAT.CM_ORGANIZATION cmorganiza1_
13:14:29,440 INFO [stdout] (default task-15) on trbankacco0_.CM_ORGANIZATION_ID=cmorganiza1_.ID
13:14:29,440 INFO [stdout] (default task-15) where
13:14:29,440 INFO [stdout] (default task-15) 1=1
13:14:29,440 INFO [stdout] (default task-15) and trbankacco0_.active=?
13:14:29,440 INFO [stdout] (default task-15) and trbankacco0_.BANK_ACCOUNT_TYPE_ENUM=? )
{color:#00875A}13:14:29,440 INFO [stdout] (default task-15) where
13:14:29,440 INFO [stdout] (default task-15) rownum <= ?{color}
> WildFly 14-15 Create Query
> --------------------------
>
> Key: WFLY-11495
> URL: https://issues.jboss.org/browse/WFLY-11495
> Project: WildFly
> Issue Type: Bug
> Components: JPA / Hibernate
> Affects Versions: 14.0.1.Final, 15.0.0.Final
> Reporter: Majid Mostafavi
> Assignee: Scott Marlow
> Priority: Major
>
> Hi
> When I use this syntax for create lazy data loader
> CriteriaBuilder builder = entityManager.getCriteriaBuilder();
> CriteriaQuery<{color:#DE350B}X{color}> criteria = builder.createQuery({color:#DE350B}X{color}.class);
> Root<{color:#DE350B}XDTO{color}> xRoot = criteria.from(XDTO.class);
> .
> .
> .
> .
> criteria.where(predicate);
> Query queryResult=entityManager.createQuery(criteria);
> queryResult.setFirstResult(first);
> if (pageSize > 0){
> queryResult.setMaxResults(pageSize);
> }
> wildfly create query by this syntax
> select
> *
> from ( {color:#14892c}select dto item1, select dto item2 from Xroot {color} )
> where
> rownum <=
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (WFLY-11292) Legacy EJB Client: High fail rate
by Flavia Rainone (Jira)
[ https://issues.jboss.org/browse/WFLY-11292?page=com.atlassian.jira.plugin... ]
Flavia Rainone commented on WFLY-11292:
---------------------------------------
I think we need to fix the close handler of the channel first. IMO, that will sove the NotOpenException by itself, but only after that fix is applied we will be certain.
> Legacy EJB Client: High fail rate
> ---------------------------------
>
> Key: WFLY-11292
> URL: https://issues.jboss.org/browse/WFLY-11292
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Flavia Rainone
> Priority: Blocker
> Attachments: jboss-ejb-client.properties, logs-10clients.zip, logs.zip
>
>
> This bug is being filed as Blocker because we are observing and elevated fail rate: roughly a thousand each run for (about 0.3%).
> h2. WildFly Built from master branch on 6 Nov 2018
> With this WildFly version (client org.jboss:jboss-ejb-client-legacy:3.0.2.Final-redhat-1) in a scenario with 4 clustered nodes where nodes are failed via jboss shut-down / restart: after node 1 of 4 is shut-down, a a series of errors start on the client side the yield to 1097 errors on a total of 340218 samples;
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 05:28:51:255 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 2018/11/07 05:28:51:270 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5118, throughput: 511.7 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 1 ms, max: 54 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5118 (100%)
> 05:28:53,257 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-2) EJBCLIENT000016: Channel Channel ID c540daf7 (outbound) of Remoting connection 4ed5523c to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <664ac6d5> can no longer process messages
> 05:28:53,277 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-2) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@3cd51445, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)302066a5,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> h2. WildFly Built from master branch on 7 Nov 2018
> Same situation.
> find [here|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/...] the complete logs;
> the start of client errors is here:
> {noformat}
> 2018/11/07 09:23:22:916 EST [INFO ][TestController] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Failing node 0 (perf18)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Total: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 2018/11/07 09:23:22:930 EST [INFO ][StatsRunner] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - unknown-node: Sessions: 2000, active: 2000, samples: 5125, throughput: 512.4 samples/s, bandwidth: 0.0 MB/s, response min: 1 ms, mean: 2 ms, max: 32 ms, sampling errors: 0, unhealthy samples: 0, valid samples: 5125 (100%)
> 09:23:24,924 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000016: Channel Channel ID d940c707 (outbound) of Remoting connection 73cfdf01 to perf18/10.16.90.54:8080 of endpoint "config-based-ejb-client-endpoint" <7501616b> can no longer process messages
> 09:23:24,949 ERROR [org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver] (Remoting "config-based-ejb-client-endpoint" task-6) Failed to open channel for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@56248025, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection(a)5269f48d,channel=jboss.ejb,nodename=perf18]}
> org.jboss.remoting3.NotOpenException: Cannot open new channel because close was initiated
> at org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen(RemoteConnectionHandler.java:198)
> at org.jboss.remoting3.remote.RemoteConnectionHandler.open(RemoteConnectionHandler.java:335)
> at org.jboss.remoting3.ConnectionImpl.openChannel(ConnectionImpl.java:109)
> at org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection.openChannel(ConnectionPool.java:292)
> at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.associate(RemotingConnectionEJBReceiver.java:180)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:399)
> at org.jboss.ejb.client.EJBClientContext.registerEJBReceiver(EJBClientContext.java:349)
> at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:67)
> at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1474)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:55 PM:
---------------------------------------------------------------
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, I do see a potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for {{data}} to be non-null, but for {{data.isCoord()}} to return false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, I do see a potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:55 PM:
---------------------------------------------------------------
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, I do see a potential for a write to the cloud store on each invocation of {{findMembers()}}. With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for {{data}} to be non-null, but for {{data.isCoord()}} to return false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, I do see a potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for {{data}} to be non-null, but for {{data.isCoord()}} to return false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:54 PM:
---------------------------------------------------------------
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then data would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:54 PM:
---------------------------------------------------------------
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, I do see a potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then {{data}} would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:53 PM:
---------------------------------------------------------------
But that if condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then data would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
Ah, I think I see what you mean. In the case where two partitions exist, {{responses.findResponseFrom(local_addr)}} may return the response from the other partition, where this node is not a coordinator. So {{data.isCoord()}} will be false, and we will keep writing the file for this coordinator, even though the file has already been created. It seems like we need to see if any of the responses for the {{local_addr}} show this node as the coordinator.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (JGRP-2288) S3_PING: Under certain conditions, subclusters fail to merge after network partition
by Nick Sawadsky (Jira)
[ https://issues.jboss.org/browse/JGRP-2288?page=com.atlassian.jira.plugin.... ]
Nick Sawadsky edited comment on JGRP-2288 at 12/15/18 5:53 PM:
---------------------------------------------------------------
But that if-condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then data would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
was (Author: nsawadsky):
But that if condition checks whether {{data == null || !data.isCoord()}}. And in the typical case, if this node is a coordinator, then data would be non-null and {{data.isCoord()}} would be true. Unless I'm missing something.
In the case where two partitions exist, there is the potential for a write to the cloud store on each invocation of findMembers(). With two partitions, {{responses.findResponseFrom(local_addr)}} may return a stale response from the other partition. In that case, there's a potential for data to be non-null, but {{data.isCoord()}} returns false. Maybe it's acceptable to incur more frequent writes to the cloud store in this case.
> S3_PING: Under certain conditions, subclusters fail to merge after network partition
> ------------------------------------------------------------------------------------
>
> Key: JGRP-2288
> URL: https://issues.jboss.org/browse/JGRP-2288
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.15
> Reporter: Nick Sawadsky
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16, 3.6.17
>
>
> Repro steps:
> 1. Set up a cluster of four nodes, two on one machine (Host 1) and two on another (Host 2). Let's call the nodes A, B, C, and D.
> 2. Configure all 4 nodes with S3_PING as the discovery mechanism. Set remove_all_files_on_view_change to true.
> 3. Start up nodes in the order A, B, C, D.
> 4. In the S3 bucket, there should be a single file with all four nodes listed. Node A should be flagged as the coordinator. Ensure that the UUID for node B is larger than the UUID for node C, when compared as two's complement integers. If this is not the case, shut down all nodes and restart in order. Repeat until the desired relationship is achieved. Note that with two's complement, a UUID having a first hex digit of 8 or higher is treated as negative for comparison purposes. So, for example, a UUID starting with 'a' is less than a UUID starting with 'b' which is less than a UUID starting with '1'.
> 5. On Host 1, use iptables to block all traffic going to and coming from Host 2.
> sudo iptables -A INPUT -s <Host 2 IP addr> -j DROP
> sudo iptables -A OUTPUT -d <Host 2 IP addr> -j DROP
> 6. Allow a few minutes for the nodes to detect the network partition. Eventually you should see two files in the S3 bucket.
> 7. Using Ctrl-C, stop node A.
> 8. You should soon find only a single file in the bucket, containing a single entry for node B. This is a result of the remove_all_files_on_view_change setting on S3_PING, which we set to true to avoid accumulation of old files in the bucket.
> 9. Resolve the network partition:
> sudo iptables -F OUTPUT
> sudo iptables -F INPUT
> 10. You will find that, even after many minutes, the subclusters are not merged.
> I believe the reason why the subclusters are never merged is as follows:
> - MERGE3 on nodes B, C and D uses S3_PING to find members to send INFO messages to. Each one finds only node B in the discovery file. As a result, only node B's view consistency checker has anything to work with.
> - On node B, the consistency checker can see that there are two coordinators, B and C. However, node C has a lower UUID, so node B defers to it to perform the merge. Node C never performs the merge because, as mentioned above, it is not receiving any INFO messages.
> I this this problem would affect FILE_PING as well, and other protocols derived from FILE_PING. Looking at the latest 4.x code, it appears the problem still exists there.
> I think the crux of the issue is that the coordinator on Host 2 (node C) does not re-create its discovery file after it is deleted by node B. Would it be reasonable for FILE_PING.findMembers() to create the discovery file if the node is a coordinator and the file doesn't exist?
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months
[JBoss JIRA] (WFLY-10417) Security API - Soteria - Jaspic - Error getting ServerAuthContext
by Alessandro Moscatelli (Jira)
[ https://issues.jboss.org/browse/WFLY-10417?page=com.atlassian.jira.plugin... ]
Alessandro Moscatelli commented on WFLY-10417:
----------------------------------------------
I confirm that this is eventually fixed.
Thank you for the support.
> Security API - Soteria - Jaspic - Error getting ServerAuthContext
> -----------------------------------------------------------------
>
> Key: WFLY-10417
> URL: https://issues.jboss.org/browse/WFLY-10417
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 13.0.0.Beta1, 14.0.0.Final
> Environment: Wildfly 13.0.0.Beta1 and an EAR Application using JavaEE 8 Security API
> Reporter: Alessandro Moscatelli
> Assignee: Ilia Vassilev
> Priority: Critical
> Fix For: 15.0.0.Final
>
> Attachments: image-2018-05-24-13-41-05-322.png, soteria.zip
>
>
> I am testing the new Wildfly release and the new Java EE8 Security API.
> I noticed this serious (I truly believe) bug, and it also accours almost randomly.
>
> The deployed application is an EAR.
>
> If I deploy the EAR with a started Wildfly 13.0.0.Beta1 everything is fine.
> Then if I stop and start / restart the application server, I can see the startup and the EAR is redeployed but sometimes (like 50% of time) the bug/error accours.
> Usually after a couple of times (stop/start/restart) I can reproduce the issue.
> Before the bug accours every call to pages and APIs works correctly. After the bug accours every call triggers the AuthException.
> After the bug accours, if I redeploy the SAME EAR everything is fine again 100% of times.
> This seems like Wildfly does something different when redeploying an application on startup and when being already startup up and then processing an application deploy.
>
> I can provide my EAR if you want, but I would prefer not to if this is not necessary.
> The security domain as defined as suggested :
> {code:xml}
> <security-domain name="auth" cache-type="default">
> <authentication-jaspi>
> <login-module-stack name="dummy">
> <login-module code="Dummy" flag="optional"/>
> </login-module-stack>
> <auth-module code="Dummy"/>
> </authentication-jaspi>
> </security-domain>
> {code}
>
> META-INF/jboss-app.xml inside the EAR :
>
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <jboss-app>
> <security-domain>auth</security-domain>
> </jboss-app>
> {code}
>
> META-INF/jboss-web.xml inside the WAR inside the EAR :
>
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?>
> <jboss-web>
> <security-domain>auth</security-domain>
> </jboss-web>
> {code}
>
> Log :
> {code}
> 17:04:18,556 ERROR [org.jboss.security] (default task-1) PBOX00374: Error getting ServerAuthContext for authContextId default-host /optoplus-services-web and security domain auth: javax.security.auth.message.AuthException
> at org.jboss.security.auth.message.config.JBossServerAuthConfig.getAuthContext(JBossServerAuthConfig.java:187)
> at org.jboss.security.plugins.auth.JASPIServerAuthenticationManager.isValid(JASPIServerAuthenticationManager.java:99)
> at org.wildfly.extension.undertow.security.jaspi.JASPICAuthenticationMechanism.authenticate(JASPICAuthenticationMechanism.java:123)
> at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
> at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
> at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
> at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
> at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:55)
> at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jaspi.JASPICSecureResponseHandler.handleRequest(JASPICSecureResponseHandler.java:48)
> at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> UPDATE !!
> Now I finally got WHY it was random !!
> The problem is related to EAR containing multiple WAR registering different contexts :
> {code}
> 13:29:37,661 INFO [org.glassfish.soteria.servlet.SamRegistrationInstaller] (ServerService Thread Pool -- 80) Initializing Soteria 1.0 for context '/soteria-web2'
> 13:29:37,661 INFO [org.glassfish.soteria.servlet.SamRegistrationInstaller] (ServerService Thread Pool -- 76) Initializing Soteria 1.0 for context '/soteria-web'
> 13:29:37,738 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 80) WFLYUT0021: Registered web context: '/soteria-web2' for server 'default-server'
> 13:29:37,745 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 76) WFLYUT0021: Registered web context: '/soteria-web' for server 'default-server'
> 13:29:37,781 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "soteria-ear-1.0.0.ear" (runtime-name : "soteria-ear-1.0.0.ear")
> 13:29:37,908 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
> 13:29:37,910 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://0.0.0.0:9990/management
> 13:29:37,910 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://0.0.0.0:9990
> 13:29:37,910 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 13.0.0.Beta1 (WildFly Core 5.0.0.Beta3) started in 10500ms - Started 707 of 931 services (430 services are lazy, passive or on-demand)
> 13:30:09,195 ERROR [org.jboss.security] (default task-1) PBOX00374: Error getting ServerAuthContext for authContextId default-host /soteria-web and security domain auth: javax.security.auth.message.AuthException
> {code}
> *+In case of a reboot (or stop and start) of Wildfly, only ONE of the TWO contexts manages to get the ServerAuthContext !+*
> I attached an example project you can use to reproduce and detailed steps !
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 7 months