[JBoss JIRA] (ISPN-9471) Class not found when injecting a RemoteCache
by Katia Aresti (JIRA)
Katia Aresti created ISPN-9471:
----------------------------------
Summary: Class not found when injecting a RemoteCache
Key: ISPN-9471
URL: https://issues.jboss.org/browse/ISPN-9471
Project: Infinispan
Issue Type: Bug
Components: Spring Integration
Environment: Tested in the infinispan-simple-example
When injecting embedded works
```java
@Autowired
private Cache<Long, String> defaultCache;
```
Injecting remote, fails
```java
@Autowired
private RemoteCache<Long, String> defaultRemoteCache;
```
Reporter: Katia Aresti
Assignee: Katia Aresti
/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/bin/java -Dvisualvm.id=40938968519874 -XX:TieredStopAtLevel=1 -noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=56505 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=56506:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/lib/tools.jar:/Users/katiaaresti/REDHAT/infinispan-simple-tutorials/spring-boot/target/classes:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-spring-boot-starter-embedded/2.0.0.Alpha1/infinispan-spring-boot-starter-embedded-2.0.0.Alpha1.jar:/Users/katiaaresti/.m2/repository/org/springframework/boot/spring-boot-starter/1.5.9.RELEASE/spring-boot-starter-1.5.9.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/boot/spring-boot/1.5.9.RELEASE/spring-boot-1.5.9.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/boot/spring-boot-autoconfigure/1.5.9.RELEASE/spring-boot-autoconfigure-1.5.9.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/boot/spring-boot-starter-logging/1.5.9.RELEASE/spring-boot-starter-logging-1.5.9.RELEASE.jar:/Users/katiaaresti/.m2/repository/ch/qos/logback/logback-classic/1.1.11/logback-classic-1.1.11.jar:/Users/katiaaresti/.m2/repository/ch/qos/logback/logback-core/1.1.11/logback-core-1.1.11.jar:/Users/katiaaresti/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar:/Users/katiaaresti/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.25/jcl-over-slf4j-1.7.25.jar:/Users/katiaaresti/.m2/repository/org/slf4j/jul-to-slf4j/1.7.25/jul-to-slf4j-1.7.25.jar:/Users/katiaaresti/.m2/repository/org/slf4j/log4j-over-slf4j/1.7.25/log4j-over-slf4j-1.7.25.jar:/Users/katiaaresti/.m2/repository/org/springframework/spring-core/4.3.13.RELEASE/spring-core-4.3.13.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar:/Users/katiaaresti/.m2/repository/org/springframework/boot/spring-boot-configuration-processor/1.5.9.RELEASE/spring-boot-configuration-processor-1.5.9.RELEASE.jar:/Users/katiaaresti/.m2/repository/com/vaadin/external/google/android-json/0.0.20131108.vaadin1/android-json-0.0.20131108.vaadin1.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-core/9.1.0.Final/infinispan-core-9.1.0.Final.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-commons/9.1.0.Final/infinispan-commons-9.1.0.Final.jar:/Users/katiaaresti/.m2/repository/org/jgroups/jgroups/4.0.4.Final/jgroups-4.0.4.Final.jar:/Users/katiaaresti/.m2/repository/com/github/ben-manes/caffeine/caffeine/2.3.5/caffeine-2.3.5.jar:/Users/katiaaresti/.m2/repository/org/jboss/spec/javax/transaction/jboss-transaction-api_1.1_spec/1.0.1.Final/jboss-transaction-api_1.1_spec-1.0.1.Final.jar:/Users/katiaaresti/.m2/repository/org/jboss/marshalling/jboss-marshalling-osgi/2.0.0.Beta3/jboss-marshalling-osgi-2.0.0.Beta3.jar:/Users/katiaaresti/.m2/repository/org/jboss/logging/jboss-logging/3.3.1.Final/jboss-logging-3.3.1.Final.jar:/Users/katiaaresti/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-spring4-embedded/8.2.8.Final/infinispan-spring4-embedded-8.2.8.Final.jar:/Users/katiaaresti/.m2/repository/org/springframework/spring-context/4.3.13.RELEASE/spring-context-4.3.13.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/spring-aop/4.3.13.RELEASE/spring-aop-4.3.13.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/spring-beans/4.3.13.RELEASE/spring-beans-4.3.13.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/springframework/spring-expression/4.3.13.RELEASE/spring-expression-4.3.13.RELEASE.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-spring4-common/8.2.8.Final/infinispan-spring4-common-8.2.8.Final.jar:/Users/katiaaresti/.m2/repository/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-spring-boot-starter-remote/2.0.0.Alpha1/infinispan-spring-boot-starter-remote-2.0.0.Alpha1.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-client-hotrod/9.1.0.Final/infinispan-client-hotrod-9.1.0.Final.jar:/Users/katiaaresti/.m2/repository/org/infinispan/infinispan-spring4-remote/9.1.0.Final/infinispan-spring4-remote-9.1.0.Final.jar:/Users/katiaaresti/.m2/repository/javax/transaction/transaction-api/1.1/transaction-api-1.1.jar org.infinispan.tutorial.simple.spring.SpringBootApp
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.9.RELEASE)
2018-08-30 21:10:42.425 INFO 12291 --- [ main] o.i.t.simple.spring.SpringBootApp : Starting SpringBootApp on katias-macbook-pro.home with PID 12291 (/Users/katiaaresti/REDHAT/infinispan-simple-tutorials/spring-boot/target/classes started by katiaaresti in /Users/katiaaresti/REDHAT/infinispan-simple-tutorials)
2018-08-30 21:10:42.427 INFO 12291 --- [ main] o.i.t.simple.spring.SpringBootApp : No active profile set, falling back to default profiles: default
2018-08-30 21:10:42.457 INFO 12291 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7ff2a664: startup date [Thu Aug 30 21:10:42 CEST 2018]; root of context hierarchy
2018-08-30 21:10:42.949 INFO 12291 --- [ main] o.i.t.simple.spring.SpringBootApp : Defining test configuration
2018-08-30 21:10:42.960 INFO 12291 --- [ main] o.i.t.simple.spring.SpringBootApp : Defining Global Configuration
2018-08-30 21:10:43.168 WARN 12291 --- [ main] o.i.manager.DefaultCacheManager : ISPN000435: Cache manager initialized with a default cache configuration but without a name for it. Set it in the GlobalConfiguration.
2018-08-30 21:10:43.413 INFO 12291 --- [ main] o.i.r.t.jgroups.JGroupsTransport : ISPN000078: Starting JGroups channel ISPN
2018-08-30 21:10:43.414 INFO 12291 --- [ main] o.i.r.t.jgroups.JGroupsTransport : ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2018-08-30 21:10:43.459 INFO 12291 --- [ main] org.jgroups.protocols.UDP : thread pool min/max/keep-alive: 0/200/60000 use_fork_join=false, internal pool: 0/8/30000 (8 cores available)
2018-08-30 21:10:43.462 INFO 12291 --- [ main] org.jgroups.protocols.pbcast.NAKACK2 : null: set max_xmit_req_size from 0 to 67600
2018-08-30 21:10:43.463 INFO 12291 --- [ main] org.jgroups.protocols.UNICAST3 : null: set max_xmit_req_size from 0 to 67600
2018-08-30 21:10:43.464 INFO 12291 --- [ main] org.jgroups.protocols.pbcast.STABLE : katias-macbook-pro-21876: stable task started
2018-08-30 21:10:43.471 WARN 12291 --- [ main] org.jgroups.protocols.UDP : JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 6.71MB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
2018-08-30 21:10:43.472 WARN 12291 --- [ main] org.jgroups.protocols.UDP : JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 6.71MB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
2018-08-30 21:10:43.472 INFO 12291 --- [ main] org.jgroups.protocols.UDP : socket information:
mcast_addr=ff0e:0:0:0:0:0:e406:708:46655, bind_addr=/fe80:0:0:0:7725:be0b:2946:4d91%utun1, ttl=2
sock: bound to fe80:0:0:0:7725:be0b:2946:4d91%12:60381, receive buffer size=6710884, send buffer size=1000000
mcast_sock: bound to fe80:0:0:0:7725:be0b:2946:4d91%utun1:46655, send buffer size=1000000, receive buffer size=6710884
2018-08-30 21:10:48.481 INFO 12291 --- [ main] org.jgroups.protocols.pbcast.GMS : katias-macbook-pro-21876: no members discovered after 5007 ms: creating cluster as first member
2018-08-30 21:10:48.483 INFO 12291 --- [ main] org.jgroups.protocols.pbcast.GMS : katias-macbook-pro-21876: installing view [katias-macbook-pro-21876|0] (1) [katias-macbook-pro-21876]
2018-08-30 21:10:48.493 INFO 12291 --- [ main] org.infinispan.CLUSTER : ISPN000094: Received new cluster view for channel ISPN: [katias-macbook-pro-21876|0] (1) [katias-macbook-pro-21876]
2018-08-30 21:10:48.499 INFO 12291 --- [ main] org.jgroups.protocols.pbcast.GMS : katias-macbook-pro-21876: created cluster (first member). My view is [katias-macbook-pro-21876|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
2018-08-30 21:10:48.500 INFO 12291 --- [ main] o.i.r.t.jgroups.JGroupsTransport : ISPN000079: Channel ISPN local address is katias-macbook-pro-21876, physical addresses are [fe80:0:0:0:7725:be0b:2946:4d91%12:60381]
2018-08-30 21:10:48.502 INFO 12291 --- [ main] o.i.factories.GlobalComponentRegistry : ISPN000128: Infinispan version: Infinispan 'Bastille' 9.1.0.Final
2018-08-30 21:10:48.588 INFO 12291 --- [ main] o.i.client.hotrod.RemoteCacheManager : ISPN004021: Infinispan version: 9.1.0.Final
2018-08-30 21:10:48.729 WARN 12291 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'springBootApp': Unsatisfied dependency expressed through field 'defaultRemoteCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getRemoteCache' defined in class path resource [org/infinispan/tutorial/simple/spring/Config.class]: Post-processing of merged bean definition failed; nested exception is java.lang.NoClassDefFoundError: org/infinispan/query/dsl/Query
2018-08-30 21:10:48.732 INFO 12291 --- [ main] o.i.r.t.jgroups.JGroupsTransport : ISPN000080: Disconnecting JGroups channel ISPN
2018-08-30 21:10:48.748 INFO 12291 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-08-30 21:10:48.753 ERROR 12291 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'springBootApp': Unsatisfied dependency expressed through field 'defaultRemoteCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getRemoteCache' defined in class path resource [org/infinispan/tutorial/simple/spring/Config.class]: Post-processing of merged bean definition failed; nested exception is java.lang.NoClassDefFoundError: org/infinispan/query/dsl/Query
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) ~[spring-context-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.9.RELEASE.jar:1.5.9.RELEASE]
at org.infinispan.tutorial.simple.spring.SpringBootApp.main(SpringBootApp.java:96) [classes/:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getRemoteCache' defined in class path resource [org/infinispan/tutorial/simple/spring/Config.class]: Post-processing of merged bean definition failed; nested exception is java.lang.NoClassDefFoundError: org/infinispan/query/dsl/Query
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:526) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:585) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
... 18 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/infinispan/query/dsl/Query
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_161]
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[na:1.8.0_161]
at java.lang.Class.getDeclaredMethods(Class.java:1975) ~[na:1.8.0_161]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613) ~[spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.util.ReflectionUtils.doWithLocalMethods(ReflectionUtils.java:489) ~[spring-core-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.buildLifecycleMetadata(InitDestroyAnnotationBeanPostProcessor.java:206) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.findLifecycleMetadata(InitDestroyAnnotationBeanPostProcessor.java:187) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(InitDestroyAnnotationBeanPostProcessor.java:125) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessMergedBeanDefinition(CommonAnnotationBeanPostProcessor.java:295) ~[spring-context-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyMergedBeanDefinitionPostProcessors(AbstractAutowireCapableBeanFactory.java:992) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:523) ~[spring-beans-4.3.13.RELEASE.jar:4.3.13.RELEASE]
... 27 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.infinispan.query.dsl.Query
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_161]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_161]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[na:1.8.0_161]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_161]
... 38 common frames omitted
Process finished with exit code 1
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9373) Update testng to the latest version
by Diego Lovison (JIRA)
[ https://issues.jboss.org/browse/ISPN-9373?page=com.atlassian.jira.plugin.... ]
Diego Lovison commented on ISPN-9373:
-------------------------------------
Initial work: https://github.com/infinispan/infinispan/pull/6167
The PR was closed because I was not able to compare the results
> Update testng to the latest version
> -----------------------------------
>
> Key: ISPN-9373
> URL: https://issues.jboss.org/browse/ISPN-9373
> Project: Infinispan
> Issue Type: Task
> Reporter: Diego Lovison
> Assignee: Diego Lovison
> Priority: Minor
>
> Update testng to the latest version
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9373) Update testng to the latest version
by Diego Lovison (JIRA)
[ https://issues.jboss.org/browse/ISPN-9373?page=com.atlassian.jira.plugin.... ]
Diego Lovison updated ISPN-9373:
--------------------------------
Status: Open (was: Pull Request Sent)
> Update testng to the latest version
> -----------------------------------
>
> Key: ISPN-9373
> URL: https://issues.jboss.org/browse/ISPN-9373
> Project: Infinispan
> Issue Type: Task
> Reporter: Diego Lovison
> Assignee: Diego Lovison
> Priority: Minor
>
> Update testng to the latest version
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9373) Update testng to the latest version
by Diego Lovison (JIRA)
[ https://issues.jboss.org/browse/ISPN-9373?page=com.atlassian.jira.plugin.... ]
Diego Lovison updated ISPN-9373:
--------------------------------
Git Pull Request: (was: https://github.com/infinispan/infinispan/pull/6167)
> Update testng to the latest version
> -----------------------------------
>
> Key: ISPN-9373
> URL: https://issues.jboss.org/browse/ISPN-9373
> Project: Infinispan
> Issue Type: Task
> Reporter: Diego Lovison
> Assignee: Diego Lovison
> Priority: Minor
>
> Update testng to the latest version
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9465) Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9465?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9465:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6225
> Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9465
> URL: https://issues.jboss.org/browse/ISPN-9465
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.1.Final
> Reporter: Paul Ferraro
> Priority: Blocker
> Attachments: Coordinator-TRACE.log, Coordinator.java, Test-TRACE.log, Test.java
>
>
> Here's the scenario:
> 1. Start cache manager on node1.
> 2. Start cache manager on node2.
> 3. Start clustered cache on node2.
> 4. Kill node2's JVM
> 5. Start cache manager on node2.
> 6. Start clustered cache on node2.
> 7. #6 fails due to state transfer timeout.
> To reproduce:
> 1. Launch Coordinator
> 2. Launch Test
> 3. kill -9 JVM process for Test
> 4. Relaunch Test
> 5. Wait for state transfer timeout
> Seems to be due to node2 waiting for a state transfer that node1 never initiates.
> N.B. The issue is specific to the abrupt leave of node2. If node2 is stopped normally, the subsequent cache start succeeds.
> I have some TRACE logging from the test run that initially identified the issue. The relevant cache name is "client-mappings". Note that in this run there are several cache managers sharing the same channel via FORK, although that is not related to the issue (as the reproducer above uses simple JChannels):
> {noformat}
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings initialized. Persisted state? false
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Added joiner node-2 to cache client-mappings with persistent uuid a1aac693-eabb-4043-ba94-ca0eb0d3b7db: members = [node-2], joiners = [node-2]
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Initializing status for cache client-mappings
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings topology updated: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, members = [node-2], joiners = []
> 08:46:22,555 TRACE [org.infinispan.topology.CacheTopology] (remote-thread--p8-t1) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings stable topology updated: members = [node-2], joiners = [], topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending command to all: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.MarshallableTypeHints] (remote-thread--p8-t1) Cache a buffer size predictor for 'org.infinispan.topology.CacheTopologyControlCommand' assuming its serializability is unknown
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (remote-thread--p8-t1) node-1: sending node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to null, src=node-1, headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.MFC] (remote-thread--p8-t1) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,556 DEBUG [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Queueing rebalance for cache client-mappings with members [node-2]
> 08:46:22,556 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Rebalancing consistent hash for cache client-mappings, members are [node-2]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) The balanced CH is the same as the current CH, not rebalancing
> 08:46:22,558 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 2 to node-2: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,558 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 512
> 08:46:22,558 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #13, conn_id=0)
> 08:46:22,558 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t3) Attempting to execute command on self: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (4 headers), size=26 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- DATA(node-2: #20, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1: delivering node-2#20
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received request 3 from node-2: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0}
> 08:46:22,564 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0} [sender=node-2]
> 08:46:22,564 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-1) Added a new task directly: 0 task(s) are waiting
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (thread-13,ejb,node-1) node-2 used 26 credits, 1998681 remaining
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 3 to node-2: SuccessfulResponse(true)
> 08:46:22,564 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 768
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #14, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,581 TRACE [org.jgroups.protocols.UNICAST3] (Timer runner-1,null,null) node-1 --> ACK(node-2: #20)
> 08:46:22,581 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to node-2, src=node-1, headers are UNICAST3: ACK, seqno=20, ts=7, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=14, ts=5, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- ACK(node-2: #14, conn-id=0, ts=5)
> 08:46:22,593 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: <null>, src: node-2 (3 headers), size=64 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ejb]
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: received node-2#4
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: delivering node-2#4
> 08:46:22,593 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received command from node-2: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings}
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute CacheRpcCommand: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings} [sender=node-2]
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Silently ignoring that client-mappings cache is not defined
> 08:46:22,593 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-1) node-2 used 64 credits, 1999587 remaining
> 08:46:22,682 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to null, src=node-1, headers are NAKACK2: [HIGHEST_SEQNO, seqno=11], TP: [cluster_name=ejb]
> {noformat}
> TRACE logging from node-2:
> {noformat}
> 08:46:22,552 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Node node-2 joining cache client-mappings
> 08:46:22,552 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 2 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=JOIN, sender=node-2, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@43db48bd, hashFunction=MurmurHash3, numSegments=256, numOwners=2, timeout=240000, totalOrder=false, cacheMode=REPL_SYNC, persistentUUID=a1aac693-eabb-4043-ba94-ca0eb0d3b7db, persistentStateChecksum=Optional.empty}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,553 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,553 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #19, conn_id=0)
> 08:46:22,553 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=19, TP: [cluster_name=ejb]
> 08:46:22,553 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 99 credits, 1998711 remaining
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: <null>, src: node-1 (3 headers), size=1156 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: received node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: delivering node-1#11
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received command from node-1: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-2) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1} [sender=node-1]
> 08:46:22,557 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-2) Added a new task directly: 0 task(s) are waiting
> 08:46:22,557 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-2) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=2210 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #13, conn_id=0)
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#13
> 08:46:22,559 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 2 from node-1: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,559 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating local topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,559 TRACE [org.infinispan.topology.CacheTopology] (ServerService Thread Pool -- 84) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,559 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Installing new cache topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]} on cache client-mappings
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) This is the first topology 1 in which the local node is a member
> 08:46:22,560 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=true, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Received new topology for cache client-mappings, isRebalance = false, isMember = true, topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.distribution.impl.DistributionManagerImpl] (ServerService Thread Pool -- 84) Topology updated to CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Ensuring segments RangeSet(256) are started
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling topology 1 is installed
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) On cache client-mappings we have: added segments: {}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Topology update processed, stateTransferTopologyId = -1, startRebalance = false, pending CH = null
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling transaction data received for topology 1
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) No end of state transfer notification, waitingForState already set to false by another thread
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Checking for transactions originated on leavers. Current cache members are [node-1, node-2], remote transactions: 0
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) No remote transactions pertain to originator(s) who have left the cluster.
> 08:46:22,561 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Removing no longer owned entries for cache client-mappings
> 08:46:22,561 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Removing segments: {} from container
> 08:46:22,561 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=false, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,561 DEBUG [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Topology changed, recalculating minTopologyId
> 08:46:22,561 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Changing minimum topology ID from -1 to 1
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Initial state transfer complete for cache client-mappings on node node-2
> 08:46:22,561 TRACE [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating stable topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) StateTransferManager of cache client-mappings on node node-2 received initial topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,563 TRACE [org.infinispan.factories.ComponentRegistry] (ServerService Thread Pool -- 84) Invoking start method waitForInitialStateTransferToComplete(priority=1000) on component org.infinispan.statetransfer.StateTransferManager
> 08:46:22,563 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 3 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=node-2, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,563 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 480
> 08:46:22,563 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #20, conn_id=0)
> 08:46:22,563 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 26 credits, 1998685 remaining
> 08:46:22,565 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #14, conn_id=0)
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#14
> 08:46:22,565 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 3 from node-1: SuccessfulResponse(true)
> 08:46:22,567 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Waiting for initial state transfer to finish for cache client-mappings on node-2
> 08:46:22,567 TRACE [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator@1aef5a9b passing event EventImpl{type=CACHE_STARTED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}
> 08:46:22,567 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 84) WFLYCLINF0002: Started client-mappings cache from ejb container
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> 08:46:22,568 TRACE [org.infinispan.manager.DefaultCacheManager] (ServerService Thread Pool -- 84) Cache client-mappings started
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> {noformat}
> After 4 minutes, node2 eventually throws:
> {noformat}
> 08:51:00,892 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 108) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:44)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:646)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:477)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:463)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:449)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> ... 7 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache client-mappings on node-2
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9465) Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9465?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-9465:
----------------------------------
Assignee: Dan Berindei
> Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9465
> URL: https://issues.jboss.org/browse/ISPN-9465
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.1.Final
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Blocker
> Attachments: Coordinator-TRACE.log, Coordinator.java, Test-TRACE.log, Test.java
>
>
> Here's the scenario:
> 1. Start cache manager on node1.
> 2. Start cache manager on node2.
> 3. Start clustered cache on node2.
> 4. Kill node2's JVM
> 5. Start cache manager on node2.
> 6. Start clustered cache on node2.
> 7. #6 fails due to state transfer timeout.
> To reproduce:
> 1. Launch Coordinator
> 2. Launch Test
> 3. kill -9 JVM process for Test
> 4. Relaunch Test
> 5. Wait for state transfer timeout
> Seems to be due to node2 waiting for a state transfer that node1 never initiates.
> N.B. The issue is specific to the abrupt leave of node2. If node2 is stopped normally, the subsequent cache start succeeds.
> I have some TRACE logging from the test run that initially identified the issue. The relevant cache name is "client-mappings". Note that in this run there are several cache managers sharing the same channel via FORK, although that is not related to the issue (as the reproducer above uses simple JChannels):
> {noformat}
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings initialized. Persisted state? false
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Added joiner node-2 to cache client-mappings with persistent uuid a1aac693-eabb-4043-ba94-ca0eb0d3b7db: members = [node-2], joiners = [node-2]
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Initializing status for cache client-mappings
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings topology updated: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, members = [node-2], joiners = []
> 08:46:22,555 TRACE [org.infinispan.topology.CacheTopology] (remote-thread--p8-t1) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings stable topology updated: members = [node-2], joiners = [], topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending command to all: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.MarshallableTypeHints] (remote-thread--p8-t1) Cache a buffer size predictor for 'org.infinispan.topology.CacheTopologyControlCommand' assuming its serializability is unknown
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (remote-thread--p8-t1) node-1: sending node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to null, src=node-1, headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.MFC] (remote-thread--p8-t1) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,556 DEBUG [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Queueing rebalance for cache client-mappings with members [node-2]
> 08:46:22,556 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Rebalancing consistent hash for cache client-mappings, members are [node-2]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) The balanced CH is the same as the current CH, not rebalancing
> 08:46:22,558 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 2 to node-2: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,558 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 512
> 08:46:22,558 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #13, conn_id=0)
> 08:46:22,558 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t3) Attempting to execute command on self: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (4 headers), size=26 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- DATA(node-2: #20, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1: delivering node-2#20
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received request 3 from node-2: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0}
> 08:46:22,564 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0} [sender=node-2]
> 08:46:22,564 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-1) Added a new task directly: 0 task(s) are waiting
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (thread-13,ejb,node-1) node-2 used 26 credits, 1998681 remaining
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 3 to node-2: SuccessfulResponse(true)
> 08:46:22,564 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 768
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #14, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,581 TRACE [org.jgroups.protocols.UNICAST3] (Timer runner-1,null,null) node-1 --> ACK(node-2: #20)
> 08:46:22,581 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to node-2, src=node-1, headers are UNICAST3: ACK, seqno=20, ts=7, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=14, ts=5, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- ACK(node-2: #14, conn-id=0, ts=5)
> 08:46:22,593 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: <null>, src: node-2 (3 headers), size=64 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ejb]
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: received node-2#4
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: delivering node-2#4
> 08:46:22,593 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received command from node-2: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings}
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute CacheRpcCommand: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings} [sender=node-2]
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Silently ignoring that client-mappings cache is not defined
> 08:46:22,593 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-1) node-2 used 64 credits, 1999587 remaining
> 08:46:22,682 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to null, src=node-1, headers are NAKACK2: [HIGHEST_SEQNO, seqno=11], TP: [cluster_name=ejb]
> {noformat}
> TRACE logging from node-2:
> {noformat}
> 08:46:22,552 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Node node-2 joining cache client-mappings
> 08:46:22,552 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 2 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=JOIN, sender=node-2, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@43db48bd, hashFunction=MurmurHash3, numSegments=256, numOwners=2, timeout=240000, totalOrder=false, cacheMode=REPL_SYNC, persistentUUID=a1aac693-eabb-4043-ba94-ca0eb0d3b7db, persistentStateChecksum=Optional.empty}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,553 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,553 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #19, conn_id=0)
> 08:46:22,553 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=19, TP: [cluster_name=ejb]
> 08:46:22,553 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 99 credits, 1998711 remaining
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: <null>, src: node-1 (3 headers), size=1156 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: received node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: delivering node-1#11
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received command from node-1: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-2) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1} [sender=node-1]
> 08:46:22,557 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-2) Added a new task directly: 0 task(s) are waiting
> 08:46:22,557 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-2) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=2210 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #13, conn_id=0)
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#13
> 08:46:22,559 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 2 from node-1: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,559 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating local topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,559 TRACE [org.infinispan.topology.CacheTopology] (ServerService Thread Pool -- 84) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,559 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Installing new cache topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]} on cache client-mappings
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) This is the first topology 1 in which the local node is a member
> 08:46:22,560 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=true, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Received new topology for cache client-mappings, isRebalance = false, isMember = true, topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.distribution.impl.DistributionManagerImpl] (ServerService Thread Pool -- 84) Topology updated to CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Ensuring segments RangeSet(256) are started
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling topology 1 is installed
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) On cache client-mappings we have: added segments: {}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Topology update processed, stateTransferTopologyId = -1, startRebalance = false, pending CH = null
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling transaction data received for topology 1
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) No end of state transfer notification, waitingForState already set to false by another thread
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Checking for transactions originated on leavers. Current cache members are [node-1, node-2], remote transactions: 0
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) No remote transactions pertain to originator(s) who have left the cluster.
> 08:46:22,561 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Removing no longer owned entries for cache client-mappings
> 08:46:22,561 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Removing segments: {} from container
> 08:46:22,561 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=false, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,561 DEBUG [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Topology changed, recalculating minTopologyId
> 08:46:22,561 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Changing minimum topology ID from -1 to 1
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Initial state transfer complete for cache client-mappings on node node-2
> 08:46:22,561 TRACE [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating stable topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) StateTransferManager of cache client-mappings on node node-2 received initial topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,563 TRACE [org.infinispan.factories.ComponentRegistry] (ServerService Thread Pool -- 84) Invoking start method waitForInitialStateTransferToComplete(priority=1000) on component org.infinispan.statetransfer.StateTransferManager
> 08:46:22,563 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 3 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=node-2, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,563 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 480
> 08:46:22,563 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #20, conn_id=0)
> 08:46:22,563 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 26 credits, 1998685 remaining
> 08:46:22,565 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #14, conn_id=0)
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#14
> 08:46:22,565 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 3 from node-1: SuccessfulResponse(true)
> 08:46:22,567 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Waiting for initial state transfer to finish for cache client-mappings on node-2
> 08:46:22,567 TRACE [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator@1aef5a9b passing event EventImpl{type=CACHE_STARTED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}
> 08:46:22,567 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 84) WFLYCLINF0002: Started client-mappings cache from ejb container
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> 08:46:22,568 TRACE [org.infinispan.manager.DefaultCacheManager] (ServerService Thread Pool -- 84) Cache client-mappings started
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> {noformat}
> After 4 minutes, node2 eventually throws:
> {noformat}
> 08:51:00,892 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 108) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:44)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:646)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:477)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:463)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:449)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> ... 7 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache client-mappings on node-2
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9470) Refactor AsyncAPITest
by Katia Aresti (JIRA)
Katia Aresti created ISPN-9470:
----------------------------------
Summary: Refactor AsyncAPITest
Key: ISPN-9470
URL: https://issues.jboss.org/browse/ISPN-9470
Project: Infinispan
Issue Type: Task
Reporter: Katia Aresti
Assignee: Katia Aresti
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9470) Refactor AsyncAPITest
by Katia Aresti (JIRA)
[ https://issues.jboss.org/browse/ISPN-9470?page=com.atlassian.jira.plugin.... ]
Katia Aresti updated ISPN-9470:
-------------------------------
Description: Make sure all asyc methods are correctly tested
> Refactor AsyncAPITest
> ---------------------
>
> Key: ISPN-9470
> URL: https://issues.jboss.org/browse/ISPN-9470
> Project: Infinispan
> Issue Type: Task
> Reporter: Katia Aresti
> Assignee: Katia Aresti
>
> Make sure all asyc methods are correctly tested
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9464) Notify listeners without holding the shared topology lock
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9464?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-9464:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.4.0.Final
Resolution: Done
> Notify listeners without holding the shared topology lock
> ---------------------------------------------------------
>
> Key: ISPN-9464
> URL: https://issues.jboss.org/browse/ISPN-9464
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.0.CR1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.4.0.Final
>
>
> Post-write listeners are invoked from {{ClusteringDependentLogic.commitSingleEntry()}}, while holding the shared topology lock. We should be able to release the topology lock before invoking listeners, so that slow listeners (e.g. clustered/client listeners) don't delay the installation of a new topology.
> Going further, we could try to use optimistic locking (e.g. with a {{StampedLock}}) instead of the {{ReentrantReadWriteLock}}, reverting the data container write if the topology changes and we are no longer an owner. This could help in scenarios where the data container update itself is slow, e.g. when using passivation with a JDBC/remote store.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (ISPN-9465) Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9465?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9465:
-------------------------------
Status: Open (was: New)
> Initial state transfer timeout for joining node following previous kill if cache does not exist on the coordinator
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9465
> URL: https://issues.jboss.org/browse/ISPN-9465
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.3.1.Final
> Reporter: Paul Ferraro
> Priority: Blocker
> Attachments: Coordinator-TRACE.log, Coordinator.java, Test-TRACE.log, Test.java
>
>
> Here's the scenario:
> 1. Start cache manager on node1.
> 2. Start cache manager on node2.
> 3. Start clustered cache on node2.
> 4. Kill node2's JVM
> 5. Start cache manager on node2.
> 6. Start clustered cache on node2.
> 7. #6 fails due to state transfer timeout.
> To reproduce:
> 1. Launch Coordinator
> 2. Launch Test
> 3. kill -9 JVM process for Test
> 4. Relaunch Test
> 5. Wait for state transfer timeout
> Seems to be due to node2 waiting for a state transfer that node1 never initiates.
> N.B. The issue is specific to the abrupt leave of node2. If node2 is stopped normally, the subsequent cache start succeeds.
> I have some TRACE logging from the test run that initially identified the issue. The relevant cache name is "client-mappings". Note that in this run there are several cache managers sharing the same channel via FORK, although that is not related to the issue (as the reproducer above uses simple JChannels):
> {noformat}
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings initialized. Persisted state? false
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Added joiner node-2 to cache client-mappings with persistent uuid a1aac693-eabb-4043-ba94-ca0eb0d3b7db: members = [node-2], joiners = [node-2]
> 08:46:22,554 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Initializing status for cache client-mappings
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings topology updated: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, members = [node-2], joiners = []
> 08:46:22,555 TRACE [org.infinispan.topology.CacheTopology] (remote-thread--p8-t1) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,555 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Cache client-mappings stable topology updated: members = [node-2], joiners = [], topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending command to all: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.MarshallableTypeHints] (remote-thread--p8-t1) Cache a buffer size predictor for 'org.infinispan.topology.CacheTopologyControlCommand' assuming its serializability is unknown
> 08:46:22,556 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (remote-thread--p8-t1) node-1: sending node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to null, src=node-1, headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.MFC] (remote-thread--p8-t1) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,556 DEBUG [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Queueing rebalance for cache client-mappings with members [node-2]
> 08:46:22,556 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) Rebalancing consistent hash for cache client-mappings, members are [node-2]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterCacheStatus] (remote-thread--p8-t1) The balanced CH is the same as the current CH, not rebalancing
> 08:46:22,558 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 2 to node-2: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,558 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 512
> 08:46:22,558 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #13, conn_id=0)
> 08:46:22,558 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,558 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t3) Attempting to execute command on self: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=NO_REBALANCE, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.infinispan.factories.GlobalComponentRegistry] (transport-thread--p16-t3) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (4 headers), size=26 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- DATA(node-2: #20, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1: delivering node-2#20
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received request 3 from node-2: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0}
> 08:46:22,564 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=0} [sender=node-2]
> 08:46:22,564 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-1) Added a new task directly: 0 task(s) are waiting
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (thread-13,ejb,node-1) node-2 used 26 credits, 1998681 remaining
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p8-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,564 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (remote-thread--p8-t1) node-1 sending response for request 3 to node-2: SuccessfulResponse(true)
> 08:46:22,564 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (remote-thread--p8-t1) Next predicted buffer size for object type 'org.infinispan.remoting.responses.SuccessfulResponse' will be 768
> 08:46:22,564 TRACE [org.jgroups.protocols.UNICAST3] (remote-thread--p8-t1) node-1 --> DATA(node-2: #14, conn_id=0)
> 08:46:22,564 TRACE [org.jgroups.protocols.UDP] (remote-thread--p8-t1) node-1: sending msg to node-2, src=node-1, headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,581 TRACE [org.jgroups.protocols.UNICAST3] (Timer runner-1,null,null) node-1 --> ACK(node-2: #20)
> 08:46:22,581 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to node-2, src=node-1, headers are UNICAST3: ACK, seqno=20, ts=7, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: node-1, src: node-2 (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=14, ts=5, TP: [cluster_name=ejb]
> 08:46:22,589 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-1) node-1 <-- ACK(node-2: #14, conn-id=0, ts=5)
> 08:46:22,593 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-1) node-1: received [dst: <null>, src: node-2 (3 headers), size=64 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ejb]
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: received node-2#4
> 08:46:22,593 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-1) node-1: delivering node-2#4
> 08:46:22,593 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-1) node-1 received command from node-2: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings}
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Attempting to execute CacheRpcCommand: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=1, gtx=GlobalTx:node-2:3, cacheName=client-mappings} [sender=node-2]
> 08:46:22,593 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-1) Silently ignoring that client-mappings cache is not defined
> 08:46:22,593 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-1) node-2 used 64 credits, 1999587 remaining
> 08:46:22,682 TRACE [org.jgroups.protocols.UDP] (Timer runner-1,null,null) node-1: sending msg to null, src=node-1, headers are NAKACK2: [HIGHEST_SEQNO, seqno=11], TP: [cluster_name=ejb]
> {noformat}
> TRACE logging from node-2:
> {noformat}
> 08:46:22,552 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Node node-2 joining cache client-mappings
> 08:46:22,552 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 2 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=JOIN, sender=node-2, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@43db48bd, hashFunction=MurmurHash3, numSegments=256, numOwners=2, timeout=240000, totalOrder=false, cacheMode=REPL_SYNC, persistentUUID=a1aac693-eabb-4043-ba94-ca0eb0d3b7db, persistentStateChecksum=Optional.empty}, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,553 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
> 08:46:22,553 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #19, conn_id=0)
> 08:46:22,553 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=19, TP: [cluster_name=ejb]
> 08:46:22,553 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 99 credits, 1998711 remaining
> 08:46:22,556 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: <null>, src: node-1 (3 headers), size=1156 bytes, flags=OOB|NO_TOTAL_ORDER|NO_RELAY], headers are FORK: ejb:ejb, NAKACK2: [MSG, seqno=11], TP: [cluster_name=ejb]
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: received node-1#11
> 08:46:22,556 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (thread-13,ejb,node-2) node-2: delivering node-1#11
> 08:46:22,556 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received command from node-1: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1}
> 08:46:22,556 TRACE [org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler] (thread-13,ejb,node-2) Attempting to execute non-CacheRpcCommand: CacheTopologyControlCommand{cache=client-mappings, type=STABLE_TOPOLOGY_UPDATE, sender=node-1, joinInfo=null, topologyId=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, availabilityMode=null, phase=null, actualMembers=[node-2], throwable=null, viewId=1} [sender=node-1]
> 08:46:22,557 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (thread-13,ejb,node-2) Added a new task directly: 0 task(s) are waiting
> 08:46:22,557 TRACE [org.jgroups.protocols.MFC] (thread-13,ejb,node-2) node-1 used 1156 credits, 1989619 remaining
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.localTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,557 TRACE [org.infinispan.factories.GlobalComponentRegistry] (remote-thread--p5-t1) Injecting dependency for field [org.infinispan.topology.CacheTopologyControlCommand.clusterTopologyManager] on an instance of [org.infinispan.topology.CacheTopologyControlCommand]
> 08:46:22,559 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=2210 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=2, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=13, TP: [cluster_name=ejb]
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #13, conn_id=0)
> 08:46:22,559 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#13
> 08:46:22,559 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 2 from node-1: SuccessfulResponse(StatusResponse{cacheJoinInfo=null, cacheTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}, stableTopology=CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}})
> 08:46:22,559 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating local topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,559 TRACE [org.infinispan.topology.CacheTopology] (ServerService Thread Pool -- 84) Current consistent hash's routing table: node-2 primary: {0-255}
> 08:46:22,559 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Installing new cache topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]} on cache client-mappings
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) This is the first topology 1 in which the local node is a member
> 08:46:22,560 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=true, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Received new topology for cache client-mappings, isRebalance = false, isMember = true, topology = CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.distribution.impl.DistributionManagerImpl] (ServerService Thread Pool -- 84) Topology updated to CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,560 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Ensuring segments RangeSet(256) are started
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling topology 1 is installed
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) On cache client-mappings we have: added segments: {}
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Topology update processed, stateTransferTopologyId = -1, startRebalance = false, pending CH = null
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (ServerService Thread Pool -- 84) Signalling transaction data received for topology 1
> 08:46:22,560 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) No end of state transfer notification, waitingForState already set to false by another thread
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Checking for transactions originated on leavers. Current cache members are [node-1, node-2], remote transactions: 0
> 08:46:22,560 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) No remote transactions pertain to originator(s) who have left the cluster.
> 08:46:22,561 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) Removing no longer owned entries for cache client-mappings
> 08:46:22,561 TRACE [org.infinispan.container.impl.DefaultSegmentedDataContainer] (ServerService Thread Pool -- 84) Removing segments: {} from container
> 08:46:22,561 TRACE [org.infinispan.notifications.cachelistener.CacheNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.infinispan.transaction.xa.XaTransactionTable@4bdae993 passing event EventImpl{type=TOPOLOGY_CHANGED, pre=false, cache=Cache 'client-mappings'@node-2, readConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtStart=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, readConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, writeConsistentHashAtEnd=PartitionerConsistentHash:ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, unionConsistentHash=null, newTopologyId=1}
> 08:46:22,561 DEBUG [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Topology changed, recalculating minTopologyId
> 08:46:22,561 TRACE [org.infinispan.transaction.impl.TransactionTable] (ServerService Thread Pool -- 84) Changing minimum topology ID from -1 to 1
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Initial state transfer complete for cache client-mappings on node node-2
> 08:46:22,561 TRACE [org.infinispan.topology.LocalTopologyManagerImpl] (ServerService Thread Pool -- 84) Updating stable topology for cache client-mappings: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,561 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) StateTransferManager of cache client-mappings on node node-2 received initial topology CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 256, owners = (1)[node-2: 256]}, pendingCH=null, unionCH=null, actualMembers=[node-2], persistentUUIDs=[a1aac693-eabb-4043-ba94-ca0eb0d3b7db]}
> 08:46:22,563 TRACE [org.infinispan.factories.ComponentRegistry] (ServerService Thread Pool -- 84) Invoking start method waitForInitialStateTransferToComplete(priority=1000) on component org.infinispan.statetransfer.StateTransferManager
> 08:46:22,563 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 84) node-2 sending request 3 to node-1: CacheTopologyControlCommand{cache=client-mappings, type=POLICY_GET_STATUS, sender=node-2, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=1}
> 08:46:22,563 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (ServerService Thread Pool -- 84) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 480
> 08:46:22,563 TRACE [org.jgroups.protocols.UNICAST3] (ServerService Thread Pool -- 84) node-2 --> DATA(node-1: #20, conn_id=0)
> 08:46:22,563 TRACE [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 84) node-2: sending msg to node-1, src=node-2, headers are RequestCorrelator: corr_id=0, type=REQ, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=20, TP: [cluster_name=ejb]
> 08:46:22,564 TRACE [org.jgroups.protocols.UFC] (ServerService Thread Pool -- 84) node-2 used 26 credits, 1998685 remaining
> 08:46:22,565 TRACE [org.jgroups.protocols.UDP] (thread-13,ejb,node-2) node-2: received [dst: node-2, src: node-1 (4 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=0, type=RSP, req_id=3, rsp_expected=true, FORK: ejb:ejb, UNICAST3: DATA, seqno=14, TP: [cluster_name=ejb]
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2 <-- DATA(node-1: #14, conn_id=0)
> 08:46:22,565 TRACE [org.jgroups.protocols.UNICAST3] (thread-13,ejb,node-2) node-2: delivering node-1#14
> 08:46:22,565 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-13,ejb,node-2) node-2 received response for request 3 from node-1: SuccessfulResponse(true)
> 08:46:22,567 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 84) Waiting for initial state transfer to finish for cache client-mappings on node-2
> 08:46:22,567 TRACE [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] (ServerService Thread Pool -- 84) Invoking listener: org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator@1aef5a9b passing event EventImpl{type=CACHE_STARTED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}
> 08:46:22,567 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 84) WFLYCLINF0002: Started client-mappings cache from ejb container
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> 08:46:22,568 TRACE [org.infinispan.manager.DefaultCacheManager] (ServerService Thread Pool -- 84) Cache client-mappings started
> 08:46:22,568 DEBUG [org.infinispan.cache.impl.CacheImpl] (ServerService Thread Pool -- 84) Started cache client-mappings on node-2
> {noformat}
> After 4 minutes, node2 eventually throws:
> {noformat}
> 08:51:00,892 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 108) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.ejb.client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_181]
> at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.security.Security.doPrivileged(Security.java:44)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:646)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:477)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:463)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:449)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
> at org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67) [wildfly-clustering-service-14.0.0.CR1-SNAPSHOT.jar:14.0.0.CR1-SNAPSHOT]
> ... 7 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache client-mappings on node-2
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_181]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_181]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_181]
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 30 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0