tommaso borgato created WFLY-10786: -------------------------------------- Summary: ISPN000136: Error executing command PutKeyValueCommand, writing keys [SessionCreationMetaDataKey(ou_d...): java.lang.NullPointerException Key: WFLY-10786 URL: https://issues.jboss.org/browse/WFLY-10786 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 14.0.0.CR1 Reporter: tommaso borgato Assignee: Paul Ferraro Attachments: NullPointerException.txt The error was observed in scenario {{*[eap-7x-stress-remote-jdg-session-invalidation-sync-2nodes|https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/JBQA-5999/job/eap-7x-stress-remote-jdg-session-invalidation-sync-2nodes_UDP2TCP_JJB/12/console-perf18/consoleText]*}}: it is composed of a 2 Wildfly nodes Cluster where the session is offloaded to a 2 JDG nodes Cluster: {noformat} <remote-cache-container name="web-sessions" default-remote-cluster="jdg-server-cluster" protocol-version="2.6"> <remote-clusters> <remote-cluster name="jdg-server-cluster" socket-bindings="remote-jdg-server1 remote-jdg-server2"/> </remote-clusters> <security ssl-context="hotrod"/> </remote-cache-container> {noformat} While the server is starting we see the following warning: {noformat} [JBossINF] [0m[33m10:15:45,844 WARN [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 78) WFLYCLINF0031: Specified HotRod protocol version 2.6 does not support creating caches automatically. Cache named 'clusterbench-ee7.ear.cb.war' must be already created on the Infinispan Server! [JBossINF] [0m[33m10:15:45,852 WARN [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 74) WFLYCLINF0031: Specified HotRod protocol version 2.6 does not support creating caches automatically. Cache named 'clusterbench-ee7.ear.clusterbench-ee7-web-granular.war' must be already created on the Infinispan Server! [JBossINF] [0m[33m10:15:45,905 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-4) ISPN004005: Error received from the server: org.infinispan.server.hotrod.CacheNotFoundException: Cache with name 'clusterbench-ee7.ear.cb.war' not found amongst the configured caches [JBossINF] [0m[33m10:15:45,912 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-4) ISPN004005: Error received from the server: org.infinispan.server.hotrod.CacheNotFoundException: Cache with name 'clusterbench-ee7.ear.clusterbench-ee7-web-granular.war' not found amongst the configured caches {noformat} When the first request is fired at the node we see the error: {noformat} [JBossINF] [0m[31m10:16:04,481 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (default task-72) ISPN000136: Error executing command PutKeyValueCommand, writing keys [SessionCreationMetaDataKey(ou_dDccwBAeTGRzyccQUWvy4N9KMM6gZ6mBhoOLZ)]: java.lang.NullPointerException [JBossINF] at org.jboss.as.clustering.infinispan.subsystem.remote.HotRodStore.load(HotRodStore.java:119) [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.loadFromAllStores(PersistenceManagerImpl.java:553) [JBossINF] at org.infinispan.persistence.PersistenceUtil.loadAndCheckExpiration(PersistenceUtil.java:219) [JBossINF] at org.infinispan.persistence.PersistenceUtil.lambda$loadAndComputeInDataContainer$1(PersistenceUtil.java:179) [JBossINF] at org.infinispan.container.impl.AbstractInternalDataContainer.lambda$compute$3(AbstractInternalDataContainer.java:223) [JBossINF] at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$remap$16(BoundedLocalCache.java:2199) [JBossINF] at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) [JBossINF] at com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(BoundedLocalCache.java:2194) [JBossINF] at com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(BoundedLocalCache.java:2146) [JBossINF] at com.github.benmanes.caffeine.cache.LocalCache.compute(LocalCache.java:100) [JBossINF] at org.infinispan.container.impl.AbstractInternalDataContainer.compute(AbstractInternalDataContainer.java:222) [JBossINF] at org.infinispan.persistence.PersistenceUtil.loadAndComputeInDataContainer(PersistenceUtil.java:201) [JBossINF] at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:135) [JBossINF] at org.infinispan.interceptors.impl.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:310) [JBossINF] at org.infinispan.interceptors.impl.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:305) [JBossINF] at org.infinispan.interceptors.impl.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:189) [JBossINF] at org.infinispan.interceptors.impl.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:118) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98) [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:672) [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:302) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) [JBossINF] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lambda$new$0(AbstractTxLockingInterceptor.java:34) [JBossINF] at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25) [JBossINF] at org.infinispan.interceptors.SyncInvocationStage.addCallback(SyncInvocationStage.java:42) [JBossINF] at org.infinispan.interceptors.InvocationStage.thenApply(InvocationStage.java:45) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.acquireLocalLockAndInvokeNext(PessimisticLockingInterceptor.java:284) [JBossINF] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:149) [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:82) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.handleWriteCommand(TxInterceptor.java:397) [JBossINF] at org.infinispan.interceptors.impl.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:234) [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:68) [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) [JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1708) [JBossINF] at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1411) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:665) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:600) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:293) [JBossINF] at org.infinispan.cache.impl.EncoderCache.putIfAbsent(EncoderCache.java:493) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:293) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.createValue(InfinispanSessionMetaDataFactory.java:58) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.createValue(InfinispanSessionMetaDataFactory.java:39) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.createValue(InfinispanSessionFactory.java:51) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.createValue(InfinispanSessionFactory.java:37) [JBossINF] at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.createSession(InfinispanSessionManager.java:224) [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:151) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412) [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:686) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) {noformat} Find attached a more complete version of the logs (the original file is pretty big - 7.9G). -- This message was sent by Atlassian JIRA (v7.5.0#75005)