Singleton deployment of clusterbench test application fails ----------------------------------------------------------- Key: WFLY-5335 URL: https://issues.jboss.org/browse/WFLY-5335 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.0.0.Beta2 Reporter: Michal Vinkler Assignee: Paul Ferraro Priority: Blocker When trying to deploy our clustering test application "clusterbench" as a singleton deployment, server boot fails in an unrecoverable manner. To identify clusterbench-ee7.ear as a singleton deployment, I added singleton-deployment.xml deployment descriptor into deployment /META-INF/ folder. The modified clusterbench-ee7.ear is attached. The same issue occurs when trying with latest wildfly upstream/master bits. Server logs: {code:title=Server 1} [mvinkler@localhost jboss-eap-7.0_1]$ ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /usr/work/eap/tmp2/jboss-eap-7.0_1 JAVA: /usr/work/java/jdk1.8.0_45/bin/java JAVA_OPTS: -server -verbose:gc -Xloggc:"/usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 14:49:40,976 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final 14:49:41,156 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final 14:49:41,204 INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) starting 14:49:41,957 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 2) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 14:49:41,962 INFO [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0015: Re-attempting failed deployment clusterbench-ee7.ear 14:49:41,979 INFO [org.jboss.as.repository] (ServerService Thread Pool -- 30) WFLYDR0001: Content added at location /usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/data/content/b4/8714b2fc6cba3b7ba7fac5e1100c85dc0a36ce/content 14:49:42,052 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 14:49:42,073 INFO [org.xnio] (MSC service thread 1-6) XNIO version 3.4.0.Beta1 14:49:42,078 INFO [org.xnio.nio] (MSC service thread 1-6) XNIO NIO Implementation Version 3.4.0.Beta1 14:49:42,100 INFO [org.jboss.remoting] (MSC service thread 1-6) JBoss Remoting version 4.0.9.Final 14:49:42,107 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0001: Activating Infinispan subsystem. 14:49:42,111 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors 14:49:42,116 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 36) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 14:49:42,212 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 45) WFLYCLJG0001: Activating JGroups subsystem. 14:49:42,231 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 48) WFLYJSF0007: Activated the following JSF Implementations: [main] 14:49:42,233 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 60) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. 14:49:42,245 INFO [org.jboss.as.connector] (MSC service thread 1-5) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final) 14:49:42,248 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 51) WFLYNAM0001: Activating Naming Subsystem 14:49:42,249 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0018: Started Driver service with driver-name = h2 14:49:42,254 INFO [org.jboss.as.security] (ServerService Thread Pool -- 58) WFLYSEC0002: Activating Security Subsystem 14:49:42,253 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 62) WFLYWS0002: Activating WebServices Extension 14:49:42,264 INFO [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=4.9.3.Final 14:49:42,310 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 1.3.0.Beta9 starting 14:49:42,311 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0003: Undertow 1.3.0.Beta9 starting 14:49:42,339 INFO [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default] 14:49:42,339 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service 14:49:42,419 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0014: Creating file handler for path '/usr/work/eap/tmp2/jboss-eap-7.0_1/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]'] 14:49:42,429 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0012: Started server default-server. 14:49:42,431 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0018: Host default-host starting 14:49:42,485 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0006: Undertow HTTP listener default listening on /127.0.0.1:8080 14:49:42,487 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow AJP listener ajp listening on /127.0.0.1:8009 14:49:42,490 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final 14:49:42,511 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364 14:49:42,597 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 14:49:42,641 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/deployments 14:49:42,647 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear") 14:49:42,723 INFO [org.jboss.ws.common.management] (MSC service thread 1-2) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.1.0.Beta1 14:49:42,726 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war") 14:49:42,726 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar") 14:49:42,727 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war") 14:49:42,847 WARN [org.jboss.as.dependency.private] (MSC service thread 1-7) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. 14:49:42,852 WARN [org.jboss.as.dependency.private] (MSC service thread 1-2) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. 14:49:43,066 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 14:49:43,066 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 14:49:43,067 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 14:49:43,067 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 14:49:46,454 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups channel server 14:49:46,464 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view for channel server: [node1|0] (1) [node1] 14:49:46,467 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Channel server local address is node1, physical addresses are [127.0.0.1:55200] 14:49:46,471 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Infinite Darkness' 8.0.0.CR1 14:49:46,595 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started default cache from server container 14:49:46,625 INFO [org.wildfly.clustering.server] (ServerService Thread Pool -- 64) WFLYCLSV0003: node1 elected as the singleton provider of the jboss.deployment.unit."clusterbench-ee7.ear".FIRST_MODULE_USE service 14:49:48,948 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,node1) ISPN000094: Received new cluster view for channel server: [node1|1] (2) [node1, node2] 14:49:49,477 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t1) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[node1: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[node1: 30, node2: 30]}, unionCH=null, actualMembers=[node1, node2]} 14:54:42,636 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[ ("core-service" => "management"), ("management-interface" => "http-interface") ]' 14:54:42,695 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-passivating.war) in 57ms 14:54:42,695 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-default.war) in 57ms 14:54:42,695 INFO [org.infinispan.CLUSTER] (remote-thread--p2-t1) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 1 14:54:42,698 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0003: Stopped default cache from server container 14:54:42,699 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-ejb.jar) in 61ms 14:54:42,701 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel server 14:54:42,702 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000082: Stopping the RpcDispatcher for channel server 14:54:42,707 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment clusterbench-ee7.ear (runtime-name: clusterbench-ee7.ear) in 69ms 14:54:42,716 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0022: Deploy of deployment "clusterbench-ee7.ear" was rolled back with no failure message 14:54:42,718 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTP listener default suspending 14:54:42,721 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 39) MODCLUSTER000002: Initiating mod_cluster shutdown 14:54:42,719 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0008: Undertow AJP listener ajp suspending 14:54:42,722 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTP listener default stopped, was bound to /127.0.0.1:8080 14:54:42,723 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS] 14:54:42,722 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to /127.0.0.1:8009 14:54:42,724 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping 14:54:42,728 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0004: Undertow 1.3.0.Beta9 stopping 14:54:42,728 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2 14:54:42,747 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details. 14:54:42,752 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested. 14:54:42,765 INFO [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) stopped in 9ms {code} {code:title=Server 2} [mvinkler@localhost jboss-eap-7.0_2]$ ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100 ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /usr/work/eap/tmp2/jboss-eap-7.0_2 JAVA: /usr/work/java/jdk1.8.0_45/bin/java JAVA_OPTS: -server -verbose:gc -Xloggc:"/usr/work/eap/tmp2/jboss-eap-7.0_2/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 14:49:46,623 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final 14:49:46,835 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final 14:49:46,883 INFO [org.jboss.as] (MSC service thread 1-5) WFLYSRV0049: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) starting 14:49:47,735 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 30) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 14:49:47,851 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 14:49:47,866 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.4.0.Beta1 14:49:47,874 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.4.0.Beta1 14:49:47,908 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.9.Final 14:49:47,909 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0001: Activating Infinispan subsystem. 14:49:47,916 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors 14:49:47,997 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 45) WFLYCLJG0001: Activating JGroups subsystem. 14:49:47,999 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 36) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 14:49:48,016 INFO [org.jboss.as.connector] (MSC service thread 1-5) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final) 14:49:48,024 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0018: Started Driver service with driver-name = h2 14:49:48,027 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 51) WFLYNAM0001: Activating Naming Subsystem 14:49:48,028 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 60) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique. 14:49:48,034 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 48) WFLYJSF0007: Activated the following JSF Implementations: [main] 14:49:48,059 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 62) WFLYWS0002: Activating WebServices Extension 14:49:48,079 INFO [org.jboss.as.security] (ServerService Thread Pool -- 58) WFLYSEC0002: Activating Security Subsystem 14:49:48,085 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0003: Undertow 1.3.0.Beta9 starting 14:49:48,088 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0003: Undertow 1.3.0.Beta9 starting 14:49:48,111 INFO [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.3.Final 14:49:48,123 INFO [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service 14:49:48,123 INFO [org.jboss.as.mail.extension] (MSC service thread 1-8) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default] 14:49:48,188 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0014: Creating file handler for path '/usr/work/eap/tmp2/jboss-eap-7.0_2/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]'] 14:49:48,191 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0012: Started server default-server. 14:49:48,192 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0018: Host default-host starting 14:49:48,236 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow AJP listener ajp listening on /127.0.0.1:8109 14:49:48,240 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTP listener default listening on /127.0.0.1:8180 14:49:48,243 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final 14:49:48,256 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364 14:49:48,358 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS] 14:49:48,423 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /usr/work/eap/tmp2/jboss-eap-7.0_2/standalone/deployments 14:49:48,434 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0027: Starting deployment of "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear") 14:49:48,515 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war") 14:49:48,515 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar") 14:49:48,515 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war") 14:49:48,530 INFO [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.1.0.Beta1 14:49:48,661 WARN [org.jboss.as.dependency.private] (MSC service thread 1-2) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. 14:49:48,664 WARN [org.jboss.as.dependency.private] (MSC service thread 1-4) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice. 14:49:48,875 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 14:49:48,875 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 14:49:48,876 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 14:49:48,876 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 14:49:49,291 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups channel server 14:49:49,299 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view for channel server: [node1|1] (2) [node1, node2] 14:49:49,302 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Channel server local address is node2, physical addresses are [127.0.0.1:55300] 14:49:49,345 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Infinite Darkness' 8.0.0.CR1 14:50:49,523 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 64) MSC000001: Failed to start service jboss.infinispan.server.default: org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:839) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:629) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:580) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:464) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455) at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120) at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:111) at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80) at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102) ... 4 more Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node2 at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ... 19 more 14:50:49,526 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14:50:49,529 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ ("subsystem" => "infinispan"), ("cache-container" => "server"), ("replicated-cache" => "default") ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.server.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node2"}} 14:50:49,557 INFO [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "clusterbench-ee7.ear" (runtime-name : "clusterbench-ee7.ear") 14:50:49,559 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report WFLYCTL0186: Services which failed to start: service jboss.infinispan.server.default: org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl 14:50:49,642 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:10090/management 14:50:49,643 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:10090 14:50:49,643 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) started (with errors) in 63233ms - Started 321 of 583 services (7 services failed or missing dependencies, 363 services are lazy, passive or on-demand) 14:51:49,528 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14:52:49,529 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14:53:49,531 WARN [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 14:54:42,713 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,node2) ISPN000094: Received new cluster view for channel server: [node2|2] (1) [node2] {code}