[JBoss JIRA] (WFCORE-4032) Add CLI request logging option
by Martin Stefanko (JIRA)
Martin Stefanko created WFCORE-4032:
---------------------------------------
Summary: Add CLI request logging option
Key: WFCORE-4032
URL: https://issues.jboss.org/browse/WFCORE-4032
Project: WildFly Core
Issue Type: Enhancement
Components: CLI
Affects Versions: 6.0.0.Alpha5
Reporter: Martin Stefanko
Assignee: Martin Stefanko
Priority: Minor
We should improve a logging capabilities of CLI. CLI logging is disabled by default, but it can be enabled by altering the jboss-cli-logging.properties.
Here is my test scenario.
{noformat}
[pkremens@localhost bin]$ ./jboss-cli.sh -c
[standalone@localhost:9990 /] /system-property=foo:add(value=bar bar)
{"outcome" => "success"}
[standalone@localhost:9990 /] shutdown
{noformat}
Set CLI logger level to ALL (logger.org.jboss.as.cli.level=ALL) to see all logs produced by CLI
actual content of jboss-cli.log - only server response is logged
{noformat}
13:45:48,041 DEBUG [org.jboss.as.cli.CommandContext] connecting to localhost:9990 as null
13:46:04,033 INFO [org.jboss.as.cli.CommandContext] {"outcome" => "success"}
{noformat}
I can imagine content like following could be helpful for troubleshooting
{noformat}
...
<TIME> <LEVEL> [org.jboss.as.cli.CommandContext] /system-property=foo:add(value=bar bar)
{
"address" => [("system-property" => "foo")],
"operation" => "add",
"value" => "barbar"
}
<TIME> <LEVEL> [org.jboss.as.cli.CommandContext] {"outcome" => "success"}
{noformat}
AFAICT the only possibility to get the actual CLI requests in raw form is to map the content of $HOME/.jboss-cli-history file with configuration history or audit log now.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (WFCORE-1583) Enhance a CLI logging
by Martin Stefanko (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1583?page=com.atlassian.jira.plugi... ]
Martin Stefanko updated WFCORE-1583:
------------------------------------
Git Pull Request: https://github.com/wildfly/wildfly-core/pull/2780, https://github.com/wildfly/wildfly-core/pull/2841 (was: https://github.com/wildfly/wildfly-core/pull/2780, https://github.com/wildfly/wildfly-core/pull/2841, https://github.com/wildfly/wildfly-core/pull/3422)
> Enhance a CLI logging
> ---------------------
>
> Key: WFCORE-1583
> URL: https://issues.jboss.org/browse/WFCORE-1583
> Project: WildFly Core
> Issue Type: Enhancement
> Components: CLI
> Reporter: Alexey Loubyansky
> Assignee: Martin Stefanko
> Priority: Minor
> Fix For: 4.0.0.Alpha1
>
>
> We should improve a logging capabilities of CLI. CLI logging is disabled by default, but it can be enabled by altering the jboss-cli-logging.properties.
> Here is my test scenario.
> {noformat}
> [pkremens@localhost bin]$ ./jboss-cli.sh -c
> [standalone@localhost:9990 /] /system-property=foo:add(value=bar bar)
> {"outcome" => "success"}
> [standalone@localhost:9990 /] shutdown
> {noformat}
> Set CLI logger level to ALL (logger.org.jboss.as.cli.level=ALL) to see all logs produced by CLI
> actual content of jboss-cli.log - only server response is logged
> {noformat}
> 13:45:48,041 DEBUG [org.jboss.as.cli.CommandContext] connecting to localhost:9990 as null
> 13:46:04,033 INFO [org.jboss.as.cli.CommandContext] {"outcome" => "success"}
> {noformat}
> I can imagine content like following could be helpful for troubleshooting
> {noformat}
> ...
> <TIME> <LEVEL> [org.jboss.as.cli.CommandContext] /system-property=foo:add(value=bar bar)
> {
> "address" => [("system-property" => "foo")],
> "operation" => "add",
> "value" => "barbar"
> }
> <TIME> <LEVEL> [org.jboss.as.cli.CommandContext] {"outcome" => "success"}
> {noformat}
> AFAICT the only possibility to get the actual CLI requests in raw form is to map the content of $HOME/.jboss-cli-history file with configuration history or audit log now.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (SWSQE-372) Github - API rate limit exceeded for 66.187.233.206.
by Filip Brychta (JIRA)
[ https://issues.jboss.org/browse/SWSQE-372?page=com.atlassian.jira.plugin.... ]
Filip Brychta reassigned SWSQE-372:
-----------------------------------
Assignee: Filip Brychta (was: Michael Foley)
> Github - API rate limit exceeded for 66.187.233.206.
> ----------------------------------------------------
>
> Key: SWSQE-372
> URL: https://issues.jboss.org/browse/SWSQE-372
> Project: Kiali QE
> Issue Type: QE Task
> Reporter: Filip Brychta
> Assignee: Filip Brychta
>
> We have only 60 requests/hour and I already hit problem with:
> sh-4.2$ curl -L -s https://api.github.com/repos/istio/istio/releases/latest
> {
> "message": "API rate limit exceeded for 66.187.233.206. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
> "documentation_url": "https://developer.github.com/v3/#rate-limiting"
> }
> Here is more info:
> sh-4.2$ curl -i https://api.github.com/users/octocat
> HTTP/1.1 403 Forbidden
> Server: GitHub.com
> Date: Mon, 06 Aug 2018 10:49:13 GMT
> Content-Type: application/json; charset=utf-8
> Content-Length: 258
> Status: 403 Forbidden
> X-RateLimit-Limit: 60
> X-RateLimit-Remaining: 0
> X-RateLimit-Reset: 1533552848
> X-GitHub-Media-Type: github.v3; format=json
> Access-Control-Expose-Headers: ETag, Link, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
> Access-Control-Allow-Origin: *
> Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
> X-Frame-Options: deny
> X-Content-Type-Options: nosniff
> X-XSS-Protection: 1; mode=block
> Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
> Content-Security-Policy: default-src 'none'
> X-Runtime-rack: 0.014340
> X-GitHub-Request-Id: CFC8:1D6B:174D41B:2D510DA:5B6827A9
> {
> "message": "API rate limit exceeded for 66.187.233.206. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
> "documentation_url": "https://developer.github.com/v3/#rate-limiting"
> }
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (WFWIP-105) [RFE EAP7-1014] Pooled connection factory with discovery group doesn't work
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-105?page=com.atlassian.jira.plugin.... ]
Martin Styk commented on WFWIP-105:
-----------------------------------
Issue was fixed here
https://github.com/ehsavoie/wildfly/commit/2f2373962b091d9119f54d54cb5120...
> [RFE EAP7-1014] Pooled connection factory with discovery group doesn't work
> ---------------------------------------------------------------------------
>
> Key: WFWIP-105
> URL: https://issues.jboss.org/browse/WFWIP-105
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis, JMS
> Reporter: Martin Styk
> Assignee: ehsavoie Hugonnet
> Priority: Blocker
> Attachments: standalone-full-ha-node1.xml, standalone-full-ha-node2.xml
>
>
> Scenario:
> Node1 w/o Artemis server has PCF with discovery group
> {noformat}
> <subsystem xmlns="urn:jboss:domain:messaging-activemq:4.0">
> <discovery-group name="dg-group1" jgroups-channel="ee" jgroups-cluster="artemis-cluster"/>
> <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" discovery-group="dg-group1" transaction="xa"/>
> <external-jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue"/>
> <external-jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue"/>
> </subsystem>
> {noformat}
> Node2 w/ Artemis server has broadcast group
> {noformat}
> <server>
> <broadcast-group name="bg-group1" jgroups-channel="ee" jgroups-cluster="artemis-cluster" connectors="connector"/>
> <http-connector name="connector" socket-binding="http" endpoint="acceptor"/>
> <http-acceptor name="acceptor" http-listener="default"/>
> <jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue" durable="true"/>
> <jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue" durable="true"/>
> ....
> </server>
> {noformat}
> For full configuration please see attached xmls.
> MDB is deployed to Node1. It should use RA(PCF) to read messages from node2. However it can not connect.
> {noformat}
> 15:51:51,738 ERROR [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ154003: Unable to reconnect org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@64121323 destination=jms/queue/InQueue destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15): ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:749) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.setup(ActiveMQActivation.java:311) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.reconnect(ActiveMQActivation.java:658) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation$SetupActivation.run(ActiveMQActivation.java:698) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.jboss.jca.core.workmanager.WorkWrapper.runWork(WorkWrapper.java:445)
> at org.jboss.as.connector.services.workmanager.WildflyWorkWrapper.runWork(WildflyWorkWrapper.java:69)
> at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:223)
> at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
> at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
> at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
> at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> {noformat}
> Same scenario works with connectors. This is *blocker* EAP7-1014.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (WFWIP-105) [RFE EAP7-1014] Pooled connection factory with discovery group doesn't work
by Martin Styk (JIRA)
[ https://issues.jboss.org/browse/WFWIP-105?page=com.atlassian.jira.plugin.... ]
Martin Styk resolved WFWIP-105.
-------------------------------
Resolution: Done
> [RFE EAP7-1014] Pooled connection factory with discovery group doesn't work
> ---------------------------------------------------------------------------
>
> Key: WFWIP-105
> URL: https://issues.jboss.org/browse/WFWIP-105
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis, JMS
> Reporter: Martin Styk
> Assignee: ehsavoie Hugonnet
> Priority: Blocker
> Attachments: standalone-full-ha-node1.xml, standalone-full-ha-node2.xml
>
>
> Scenario:
> Node1 w/o Artemis server has PCF with discovery group
> {noformat}
> <subsystem xmlns="urn:jboss:domain:messaging-activemq:4.0">
> <discovery-group name="dg-group1" jgroups-channel="ee" jgroups-cluster="artemis-cluster"/>
> <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" discovery-group="dg-group1" transaction="xa"/>
> <external-jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue"/>
> <external-jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue"/>
> </subsystem>
> {noformat}
> Node2 w/ Artemis server has broadcast group
> {noformat}
> <server>
> <broadcast-group name="bg-group1" jgroups-channel="ee" jgroups-cluster="artemis-cluster" connectors="connector"/>
> <http-connector name="connector" socket-binding="http" endpoint="acceptor"/>
> <http-acceptor name="acceptor" http-listener="default"/>
> <jms-queue name="InQueue" entries="jms/queue/InQueue java:jboss/exported/jms/queue/InQueue" durable="true"/>
> <jms-queue name="OutQueue" entries="jms/queue/OutQueue java:jboss/exported/jms/queue/OutQueue" durable="true"/>
> ....
> </server>
> {noformat}
> For full configuration please see attached xmls.
> MDB is deployed to Node1. It should use RA(PCF) to read messages from node2. However it can not connect.
> {noformat}
> 15:51:51,738 ERROR [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ154003: Unable to reconnect org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@64121323 destination=jms/queue/InQueue destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15): ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
> at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:749) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.setup(ActiveMQActivation.java:311) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.reconnect(ActiveMQActivation.java:658) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation$SetupActivation.run(ActiveMQActivation.java:698) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
> at org.jboss.jca.core.workmanager.WorkWrapper.runWork(WorkWrapper.java:445)
> at org.jboss.as.connector.services.workmanager.WildflyWorkWrapper.runWork(WildflyWorkWrapper.java:69)
> at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:223)
> at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
> at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
> at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
> at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> {noformat}
> Same scenario works with connectors. This is *blocker* EAP7-1014.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (ELY-1622) BC FIPS with CLI: SunX509 KeyManagerFactory not available
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/ELY-1622?page=com.atlassian.jira.plugin.s... ]
Darran Lofthouse commented on ELY-1622:
---------------------------------------
Don't forget we do also support setting security properties directly in the subsystem - for the correct insertion points for FIPS that may not be usable but just something to keep in mind.
> BC FIPS with CLI: SunX509 KeyManagerFactory not available
> ---------------------------------------------------------
>
> Key: ELY-1622
> URL: https://issues.jboss.org/browse/ELY-1622
> Project: WildFly Elytron
> Issue Type: Bug
> Components: SSL
> Affects Versions: 1.5.1.Final
> Reporter: Martin Choma
> Assignee: Farah Juma
> Priority: Blocker
> Fix For: 1.5.2.Final
>
> Attachments: cli-test-wildfly-config.xml, jboss-cli.log, truststore.bcfks
>
>
> I am trying to connect from jboss-cli.sh to EAP server. To reproduce the problem it is enough BC FIPS is used only on client side.
> {code:java|titlejboss-cli.log}
> 11:50:25,147 ERROR [org.jboss.as.cli.impl.CliLauncher] Error processing CLI: org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
> at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:330)
> at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:291)
> at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
> at org.jboss.modules.Module.run(Module.java:352)
> at org.jboss.modules.Module.run(Module.java:320)
> at org.jboss.modules.Main.main(Main.java:593)
> Caused by: org.jboss.as.cli.CommandLineException: Failed to resolve host 'localhost'
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:1256)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:1203)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:1198)
> at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:328)
> ... 5 more
> Caused by: java.io.IOException: Failed to obtain SSLContext
> at org.jboss.as.cli.impl.CLIModelControllerClient.<init>(CLIModelControllerClient.java:156)
> at org.jboss.as.cli.impl.ModelControllerClientFactory$2.getClient(ModelControllerClientFactory.java:85)
> at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:1222)
> ... 8 more
> Caused by: java.security.KeyManagementException: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
> at org.bouncycastle.jsse.provider.ProvSSLContextSpi.selectKeyManager(ProvSSLContextSpi.java:589)
> at org.bouncycastle.jsse.provider.ProvSSLContextSpi.engineInit(ProvSSLContextSpi.java:531)
> at javax.net.ssl.SSLContext.init(SSLContext.java:282)
> at org.wildfly.security.ssl.SSLContextBuilder.lambda$build$0(SSLContextBuilder.java:372)
> at org.wildfly.security.OneTimeSecurityFactory.create(OneTimeSecurityFactory.java:53)
> at org.wildfly.security.auth.client.AuthenticationContextConfigurationClient.getSSLContext(AuthenticationContextConfigurationClient.java:221)
> at org.wildfly.security.auth.client.AuthenticationContextConfigurationClient.getSSLContext(AuthenticationContextConfigurationClient.java:208)
> at org.jboss.as.cli.impl.CLIModelControllerClient.<init>(CLIModelControllerClient.java:153)
> ... 10 more
> Caused by: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:137)
> at org.bouncycastle.jsse.provider.ProvSSLContextSpi.selectKeyManager(ProvSSLContextSpi.java:583)
> ... 17 more
> {code}
> When I use non-FIPS java with CLI I can make it work. It does occure also when connecting to default unsecured port 9990.
> When I use BCFKS truststore on server side, e.g. in 2-way http communication it works.
> I believe problem is I cant configure algorithm for keymanager on client side in wildfly-config.xml. (At least I don't see how could I do so).
> BC provider does not know SunX509 arlgorithm, rather X509, X.509 or PKIX could be used.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (WFCORE-4031) Additional test coverage for read-config-as-features
by Richard Janík (JIRA)
Richard Janík created WFCORE-4031:
-------------------------------------
Summary: Additional test coverage for read-config-as-features
Key: WFCORE-4031
URL: https://issues.jboss.org/browse/WFCORE-4031
Project: WildFly Core
Issue Type: Task
Components: Management, Server
Affects Versions: 6.0.0.Beta1
Reporter: Richard Janík
Assignee: Richard Janík
Issue WFCORE-3991 contains test coverage for the `nested=true` option as well as smoke tests for the `nested=false` option. The `nested=false` option could use more testing on real subsystems.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months
[JBoss JIRA] (WFLY-10736) Server in cluster hangs during start after previous kill
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10736?page=com.atlassian.jira.plugin... ]
Miroslav Novak commented on WFLY-10736:
---------------------------------------
Thanks, I've updated steps to reproduce with Lodh2TestCase#testRemoteJcaInboundOnly where this issue can be reproduced.
> Server in cluster hangs during start after previous kill
> --------------------------------------------------------
>
> Key: WFLY-10736
> URL: https://issues.jboss.org/browse/WFLY-10736
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: blocker-WF14
> Attachments: clusterKilTest.zip, logs-with-workaround.zip, node-1-thread-dump-before-kill-shutdown-sequence.txt, server-with-mdb.log, standalone-full-ha-1.xml, standalone-full-ha-2.xml
>
>
> There is regression in JGroups or Infinispan in one of our tests for fault tolerance of JMS bridges. However work on JMS bridge appears to be unrelated. Issue was hit in WF weekly run.
> Test Scenario:
> * There are two servers. InQueue is deployed on Node 1,
> * OutQueue is deployed on Node 2. Both servers are started.
> * Large byte messages are sent to InQueue deployed on Node 1. Bridge between servers/queues transfers messages from node 1 to node 2.
> * Node 1 is killed and started again.
> * All messages are received from OutQueue deployed on Node 2.
> Result:
> Node 1 does not start after kill and hangs. There is following exception logged in node 2:
> {code}
> :26:17,894 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100000: Node node-1 joined the cluster
> 09:26:18,520 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,521 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,522 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN000094: Received new cluster view for channel ejb: [node-2|7] (2) [node-2, node-1]
> 09:26:18,523 INFO [org.infinispan.CLUSTER] (thread-12,ejb,node-2) ISPN100001: Node node-1 left the cluster
> 09:26:18,868 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=17, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[node-2: 122, node-1: 134]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (3)[node-2: 84, node-1: 90, node-1: 82]}, unionCH=null, actualMembers=[node-2, node-1, node-1], persistentUUIDs=[12443bfb-e88a-46f3-919e-9213bf38ce19, 2873237f-d881-463f-8a5a-940bf1d764e5, a05ea8af-a83b-42a9-b937-dc2da1cae6d1]}
> 09:26:18,869 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-2]ISPN100002: Started rebalance with topology id 17
> 09:26:18,870 INFO [org.infinispan.CLUSTER] (transport-thread--p14-t5) [Context=default][Scope=node-2]ISPN100003: Node node-2 finished rebalance phase with topology id 17
> 09:26:18,981 INFO [org.infinispan.CLUSTER] (remote-thread--p5-t2) [Context=default][Scope=node-1]ISPN100003: Node node-1 finished rebalance phase with topology id 17
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p15-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> 09:27:18,530 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t4) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
> Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) [rt.jar:1.8.0_131]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) [rt.jar:1.8.0_131]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:71)
> at org.infinispan.topology.ClusterTopologyManagerImpl.confirmMembersAvailable(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheMembers(ClusterTopologyManagerImpl.java:523)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:334)
> at org.infinispan.topology.ClusterTopologyManagerImpl.access$500(ClusterTopologyManagerImpl.java:85)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.lambda$handleViewChange$0(ClusterTopologyManagerImpl.java:745)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
> at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
> at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
> ... 1 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1
> at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
> at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [rt.jar:1.8.0_131]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> ... 1 more
> [CIRCULAR REFERENCE:java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 4 from node-1]
> {code}
> There is default JGroups udp stack configured which is used by Infinispan. Both of the servers (jgroups udp) are bound to 127.0.0.1. Node 2 has port offset 1000.
> Attaching thread dump from node 1 when it hangs during start.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 11 months