[JBoss JIRA] (ISPN-5259) Disabling rebalancing causes errors when starting the second server in a cluster
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5259?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5259:
-----------------------------------------------
Dave Stahl <dstahl(a)redhat.com> changed the Status of [bug 1198211|https://bugzilla.redhat.com/show_bug.cgi?id=1198211] from VERIFIED to CLOSED
> Disabling rebalancing causes errors when starting the second server in a cluster
> --------------------------------------------------------------------------------
>
> Key: ISPN-5259
> URL: https://issues.jboss.org/browse/ISPN-5259
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 7.2.0.Alpha1, 7.1.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Fix For: 7.2.0.Beta1, 7.2.0.Final
>
>
> Perform the following steps with Infinispan 7.2 server:
> # Start server using command: bin/clustered.sh -Djboss.node.name=node0 -Djboss.socket.binding.port-offset=0
> # Set JMX attribute jboss.infinispan/CacheManager/"clustered"/LocalTopologyManager/rebalancingEnabled to false
> # Start second serer using command: bin/clustered.sh -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=10
> Result: The first server starts successfully and shows this log message when the JMX attribute is modified:
> {noformat}
> 09:38:05,620 INFO [org.infinispan.CLUSTER] (RMI TCP Connection(3)-127.0.0.1) ISPN000309: Rebalancing suspended
> {noformat}
> The second server starts, but then shows the following error messages:
> {noformat}
> =========================================================================
> JBoss Bootstrap Environment
> JBOSS_HOME: /home/afield/Development/projects/infinispan-public/server/integration/build/target/infinispan-server-7.2.0-SNAPSHOT
> JAVA: /usr/lib/jvm/java/bin/java
> JAVA_OPTS: -server -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''
> =========================================================================
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
> 09:38:24,626 INFO [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final
> 09:38:24,895 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.2.Final
> 09:38:24,967 INFO [org.jboss.as] (MSC service thread 1-8) JBAS015899: JBoss Infinispan Server 7.2.0-SNAPSHOT (WildFly 8.1.0.Final) starting
> 09:38:26,332 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
> 09:38:26,355 INFO [org.xnio] (MSC service thread 1-1) XNIO version 3.2.2.Final
> 09:38:26,371 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.2.2.Final
> 09:38:26,424 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 21) JBAS010280: Activating Infinispan subsystem.
> 09:38:26,448 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
> 09:38:26,464 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 25) JBAS010260: Activating JGroups subsystem.
> 09:38:26,476 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 28) JBAS011800: Activating Naming Subsystem
> 09:38:26,479 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.3.Final
> 09:38:26,493 INFO [org.jboss.as.security] (ServerService Thread Pool -- 30) JBAS013171: Activating Security Subsystem
> 09:38:26,497 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 32) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.
> 09:38:26,505 INFO [org.jboss.as.security] (MSC service thread 1-3) JBAS013170: Current PicketBox version=4.0.21.Beta1
> 09:38:26,533 INFO [org.jboss.as.connector.logging] (MSC service thread 1-8) JBAS010408: Starting JCA Subsystem (IronJacamar 1.1.5.Final)
> 09:38:26,577 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) JBAS017502: Undertow 1.0.15.Final starting
> 09:38:26,579 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 33) JBAS017502: Undertow 1.0.15.Final starting
> 09:38:26,629 INFO [org.jboss.as.naming] (MSC service thread 1-6) JBAS011802: Starting Naming Service
> 09:38:26,889 INFO [org.wildfly.extension.undertow] (MSC service thread 1-6) JBAS017525: Started server default-server.
> 09:38:26,901 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017531: Host default-host starting
> 09:38:27,015 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) JBAS017519: Undertow HTTP listener default listening on /127.0.0.1:8180
> 09:38:27,032 WARN [org.jgroups.stack.Configurator] (MSC service thread 1-3) JGRP000014: TP.singleton_name has been deprecated: Use fork channels instead
> 09:38:27,266 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-8) JBAS015012: Started FileSystemDeploymentService for directory /home/afield/Development/projects/infinispan-public/server/integration/build/target/infinispan-server-7.2.0-SNAPSHOT/standalone/deployments
> 09:38:27,632 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) JDGS010000: REST starting
> 09:38:28,049 INFO [org.hibernate.validator.internal.util.Version] (MSC service thread 1-7) HV000001: Hibernate Validator 5.1.0.Final
> 09:38:28,075 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered
> 09:38:28,111 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
> 09:38:28,114 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
> 09:38:28,117 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
> 09:38:28,119 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
> 09:38:28,127 INFO [org.jboss.resteasy.plugins.validation.AbstractValidatorContextResolver] (MSC service thread 1-7) Unable to find CDI supporting ValidatorFactory. Using default ValidatorFactory
> 09:38:28,231 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) JBAS017534: Registered web context: /
> 09:38:28,232 INFO [org.infinispan.server.endpoint] (MSC service thread 1-7) JDGS010002: REST mapped to /rest
> 09:38:28,285 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [node0/clustered|1] (2) [node0/clustered, node1/clustered]
> 09:38:28,395 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is node1/clustered, physical addresses are [127.0.0.1:55300]
> 09:38:28,486 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Hoptimus Prime' 7.2.0-SNAPSHOT
> 09:38:29,017 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started ___protobuf_metadata cache from clustered container
> 09:38:29,046 INFO [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-6) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
> 09:38:29,038 INFO [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-3) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
> 09:38:29,067 INFO [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-2) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
> 09:38:29,073 INFO [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-8) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
> 09:39:29,145 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.infinispan.clustered.namedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache namedCache on node1/clustered
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> 09:39:29,153 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.infinispan.clustered.transactionalCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactionalCache on node1/clustered
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> 09:39:29,151 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.infinispan.clustered.memcachedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache memcachedCache on node1/clustered
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> 09:39:29,150 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1/clustered
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> 09:39:29,166 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "clustered"),
> ("distributed-cache" => "default")
> ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1/clustered"}}
> 09:39:29,169 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "clustered"),
> ("distributed-cache" => "memcachedCache")
> ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.memcachedCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache memcachedCache on node1/clustered"}}
> 09:39:29,172 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "clustered"),
> ("distributed-cache" => "namedCache")
> ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.namedCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache namedCache on node1/clustered"}}
> 09:39:29,174 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "clustered"),
> ("distributed-cache" => "transactionalCache")
> ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.transactionalCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactionalCache on node1/clustered"}}
> 09:39:29,211 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
> JBAS014777: Services which failed to start: service jboss.infinispan.clustered.transactionalCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
> service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
> service jboss.infinispan.clustered.namedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
> service jboss.infinispan.clustered.memcachedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
> 09:39:29,342 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10090/management
> 09:39:29,343 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10090
> 09:39:29,343 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss Infinispan Server 7.2.0-SNAPSHOT (WildFly 8.1.0.Final) started (with errors) in 65043ms - Started 147 of 175 services (7 services failed or missing dependencies, 64 services are lazy, passive or on-demand)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years
[JBoss JIRA] (ISPN-5304) WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5304?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5304:
-----------------------------------------------
Dave Stahl <dstahl(a)redhat.com> changed the Status of [bug 1202370|https://bugzilla.redhat.com/show_bug.cgi?id=1202370] from VERIFIED to CLOSED
> WildFly module "/org/infinispan/client/hotrod/jdg-6.4" is different if installed with library or HR-client
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5304
> URL: https://issues.jboss.org/browse/ISPN-5304
> Project: Infinispan
> Issue Type: Feature Request
> Affects Versions: 7.2.0.Alpha1, 7.1.1.Final
> Environment: Infinispan as WildFly module
> Reporter: Wolf-Dieter Fink
> Assignee: Adrian Nistor
> Fix For: 7.2.0.Beta2, 7.2.0.Final, 7.1.2.Final
>
>
> If Infinispan is used inside WildFly applications there are two WF modules for Infinispan.
> One for Library mode
> One for HotRod remote client mode
> Both modules contain
> modules/org/infinispan/client/hotrod/jdg-6.4/infinispan-client-hotrod-6.2.0.Final-redhat-4.jar
> modules/org/infinispan/client/hotrod/jdg-6.4/module.xml
> BUT
> the module.xml is different:
> < <module name="org.infinispan" slot="jdg-6.4"/>
> ---
> > <module name="org.infinispan.commons" slot="jdg-6.4"/>
> > <module name="org.infinispan.protostream" slot="jdg-6.4"/>
> > <module name="org.infinispan.query.dsl" slot="jdg-6.4"/>
> > <module name="org.infinispan.query.remote.client" slot="jdg-6.4"/>
> This mean it is not possible to use both modules if there are applications with library mode and others with HR remote client mode.
> Also it is odd because it depends on the installation order what dependencies are installed.
> It should be possible to install both modules and use it at the same time from different applicaitons.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years
[JBoss JIRA] (ISPN-5223) NPE in InfinispanCollections.containsAny
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5223?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5223:
-----------------------------------------------
Dave Stahl <dstahl(a)redhat.com> changed the Status of [bug 1197733|https://bugzilla.redhat.com/show_bug.cgi?id=1197733] from VERIFIED to CLOSED
> NPE in InfinispanCollections.containsAny
> ----------------------------------------
>
> Key: ISPN-5223
> URL: https://issues.jboss.org/browse/ISPN-5223
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.0.Final
> Reporter: Kurt Lehrke
> Assignee: Dan Berindei
> Fix For: 7.2.0.Beta1, 7.2.0.Final
>
>
> When having partition handling enabled on a replicated cache, a null pointer exception is thrown in the post operation partition check because the ClusteringDependentLogic.getOwners(key) returns null.
> Here's the line it's thrown:
> https://github.com/infinispan/infinispan/blob/master/commons/src/main/jav...
> Here's the calling line:
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
> Relevant stack trace:
> {code}
> java.lang.NullPointerException
> at org.infinispan.commons.util.InfinispanCollections.containsAny(InfinispanCollections.java:309) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.postOperationPartitionCheck(PartitionHandlingInterceptor.java:149) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitGetKeyValueCommand(PartitionHandlingInterceptor.java:114) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:233) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:217) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:39) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:391) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:384) ~[infinispan-embedded-7.0.3.Final.jar:7.0.3.Final]
> at com.redprairie.moca.cluster.manager.AbstractClusterRoleManager$RoleUpdater.run(AbstractClusterRoleManager.java:671) ~[moca-server.jar:?]
> at com.redprairie.moca.util.ExceptionSuppressingRunnable.run(ExceptionSuppressingRunnable.java:47) ~[moca-server.jar:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.7.0_67]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:1.7.0_67]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) ~[?:1.7.0_67]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_67]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_67]
> at java.lang.Thread.run(Unknown Source) [?:1.7.0_67]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years
[JBoss JIRA] (ISPN-4810) Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-4810?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-4810:
-----------------------------------------------
Dave Stahl <dstahl(a)redhat.com> changed the Status of [bug 1196258|https://bugzilla.redhat.com/show_bug.cgi?id=1196258] from VERIFIED to CLOSED
> Local Transactional Cache loses data when eviction is enabled and there are multiple readers and one writer
> -----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-4810
> URL: https://issues.jboss.org/browse/ISPN-4810
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: Windows 7 x64 (NTFS)
> Oracle JDK1.7.0_40
> Apache Maven 3.0.5
> Reporter: Horia Chiorean
> Assignee: William Burns
> Labels: modeshape
> Fix For: 7.2.0.Beta1, 5.2.11.Final
>
> Attachments: ispn_concurrent.zip
>
>
> Using Infinispan 6.0.2 and a local, transactional cache backed by a <singleFile> store, with eviction enabled and a small {{max-entries}} setting, we have the following scenario:
> * the main thread (i.e. the "writer") starts a transaction, adds a batch of strings into the cache and also appends the same strings into a List cache entry and then commits the transaction
> * after the above has finished (i.e. after tx.commit) it fires a number of reader threads where each reader thread
> ** checks that the string entries were added into the cache and
> ** checks that the entries were correctly appended to the List entry
> * the above steps are repeated a number of times
> On any given run, based on timing, we're seeing that at some point (after some time) some of the reader threads will not see the latest version of the List entry (i.e. will not see the latest elements that were added into the list) but rather an old, stale List (sort of a "lost update" scenario).
> If we either:
> * disable eviction or
> * set the {{max-entries}} to a large enough value (which I suspect has the same effect - not evicting anything) the problem doesn't show up.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years
[JBoss JIRA] (ISPN-5174) Transaction cannot be recommitted after ownership changes
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5174?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5174:
-----------------------------------------------
Dave Stahl <dstahl(a)redhat.com> changed the Status of [bug 1185779|https://bugzilla.redhat.com/show_bug.cgi?id=1185779] from VERIFIED to CLOSED
> Transaction cannot be recommitted after ownership changes
> ---------------------------------------------------------
>
> Key: ISPN-5174
> URL: https://issues.jboss.org/browse/ISPN-5174
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.0.CR2, 7.1.1.Final
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 7.2.0.Beta2, 7.2.0.Final
>
>
> Once transaction is completed, it cannot commit again. If it should commit more keys since it has become an owner of some new keys modified in this transaction, it just ignores the further commit.
> There is a race with state transfer which can bring an old value (with StateResponseCommand sent before it is commited) but the value is not set by the ongoing transaction either.
> This results with stale value stored on one node.
> In my case, The problematic part is transaction <edg-perf01-62141>:15066 (consisting of 10 modifications) which got prepared and committed on edg-perf04 in topology 25. Before the originator finishes, topology changes and 04 requests ongoing transactions:
> {code}
> 11:06:11,369 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (transport-thread-17) Replication task sending StateRequestCommand{cache=testCache, origin=edg-perf04-35097, type=GET_TRANSACTIONS, topologyId=28, segments=[275, 1, 278, 9, 282, 286, 17, 259, 25, 267, 171, 169, 33, 306, 175, 173, 310, 172, 314, 41, 167, 165, 318, 187, 290, 49, 185, 191, 294, 189, 179, 298, 57, 177, 183, 302, 181, 343, 205, 201, 338, 203, 336, 351, 197, 349, 199, 347, 193, 345, 195, 326, 85, 87, 322, 93, 332, 95, 330, 89, 91, 103, 101, 99, 506, 97, 105, 357, 359, 353, 355, 361]} to single recipient edg-perf01-62141 with response mode GET_ALL
> 11:06:11,495 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread-17) Applying 6 transactions for cache testCache transferred from node edg-perf01-62141
> {code}
> However I don't see how these are applied, since PrepareCommand is not created again - from the code I see only that backup locks are added. Not sure if the transaction is registered at all, since it was already completed on this node (but at that time it did not own key_00000000000002EB).
> After originator stores the entry, it sends one more CommitCommand with topology 28:
> {code}
> 11:06:11,619 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-2) Replication task sending CommitCommand {gtx=GlobalTransaction:<edg-perf01-62141>:15066:local, cacheName='testCache', topologyId=28} to addresses [edg-perf03-20530, edg-perf04-35097] with response mode GET_ALL
> {code}
> 04 receives several CommitCommands (both from originator and forwards), but all of them are ignored as the transaction is completed.
> I don't see the logs where state transfer is assembled, but it's probably before the entry is stored on originator as the state transfer contains the old entry:
> {code}
> 11:06:13,449 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (remote-thread-91) Received chunk with keys [key_000000000000065B, key_00000000000006BE, key_FFFFFFFFFFFFE62F, key_0000000000001F42, key_000000000000027B, key_000000000000159D, key_00000000000002EB, key_00000000000002BB] for segment 343 of cache testCache from node edg-perf01-62141
> 11:06:13,454 TRACE [org.infinispan.container.DefaultDataContainer] (remote-thread-91) Store ImmortalCacheEntry{key=key_00000000000002EB, value=[2 #7: 366, 544, 576, 804, 1061, 1181, 1290, ]} in container
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years