[JBoss JIRA] (ISPN-3665) SingleFileStore is not thread-safe for passivation
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3665?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3665:
------------------------------------
* The start() and stop() methods are fine, ComponentRegistry also does some synchronization that ensures any thread that can access the component will see all the values set during its startup. Plus we use that trick all over the place...
* As you said, the entries monitor guarantees visibility for the FileEntry updates. However, it is possible for a _load() to use those FileEntry fields after it released both the entries monitor (allowing a write() to proceed) and the pseudo-read lock in FileEntry (allowing allocate() to allocate the entry for a new write()). I'm making FileEntry immutable to avoid this.
* Actually I was wrong about free(). The key length bytes are zeroed in the file, but the FileEntry still has the proper keyLen, and _load() can still read the entry from the file (assuming it got a hold of it before it was removed from the entries map). The actual data is only overwritten after all the readers have released their pseudo-read lock on the FileEntry.
I did notice another problem with clear(): write()/free() could write past the end of the file when a clear() happened in parallel. This would create an invalid file with a lot of zeroes. I'm adding a global read-write lock to deal with that.
> SingleFileStore is not thread-safe for passivation
> --------------------------------------------------
>
> Key: ISPN-3665
> URL: https://issues.jboss.org/browse/ISPN-3665
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: 620
> Fix For: 6.0.0.CR2
>
> Attachments: Test.java
>
>
> SingleFileStore never makes use of FileChannel.force(...) to flush changes to disk. This causes problems for the passivation use case.
> If one thread evicts a cache entry, while immediately after another thread attempts to read the same cache entry, the Cache.get(...) can return null. This is because the entry is never flushed to disk.
> I've attached a test to reproduce the problem.
> I also ran the same test with the addition of FileChannel.force(false) to the write(...) method, and the test succeeds.
> A proper fix should probably make this a configurable property (as it was with the old file store implementation). It would be nice if the flush could defer until just before tx commit, but, off hand, I don't know how feasible that is.
> I suspect this lack of flush also accounts for much of the bold claim of a 100x performance improvement over the old file store implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3665) SingleFileStore is not thread-safe for passivation
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3665?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3665:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/2180, https://github.com/infinispan/infinispan/pull/2211 (was: https://github.com/infinispan/infinispan/pull/2180)
> SingleFileStore is not thread-safe for passivation
> --------------------------------------------------
>
> Key: ISPN-3665
> URL: https://issues.jboss.org/browse/ISPN-3665
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: 620
> Fix For: 6.0.0.CR2
>
> Attachments: Test.java
>
>
> SingleFileStore never makes use of FileChannel.force(...) to flush changes to disk. This causes problems for the passivation use case.
> If one thread evicts a cache entry, while immediately after another thread attempts to read the same cache entry, the Cache.get(...) can return null. This is because the entry is never flushed to disk.
> I've attached a test to reproduce the problem.
> I also ran the same test with the addition of FileChannel.force(false) to the write(...) method, and the test succeeds.
> A proper fix should probably make this a configurable property (as it was with the old file store implementation). It would be nice if the flush could defer until just before tx commit, but, off hand, I don't know how feasible that is.
> I suspect this lack of flush also accounts for much of the bold claim of a 100x performance improvement over the old file store implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3698) Failure during concurrent restart of cache/manager
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3698?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3698:
-----------------------------------------------
Ladislav Thon <lthon(a)redhat.com> made a comment on [bug 1027738|https://bugzilla.redhat.com/show_bug.cgi?id=1027738]
I built the current EAP "upstream" branch which has the fix, and I can confirm that the issue is indeed fixed. Please take this as a preliminary feedback; once we get official build, we will provide official QA statement.
> Failure during concurrent restart of cache/manager
> --------------------------------------------------
>
> Key: ISPN-3698
> URL: https://issues.jboss.org/browse/ISPN-3698
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 5.2.7.Final, 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Blocker
> Fix For: 5.2.8.Final, 6.0.0.Final
>
>
> These failures prevent the cache from starting successfully, and consequently the redeploy to fail. There are 3 stack traces of note in the logs:
> {noformat}
> 15:05:32,044 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-20,shared=udp) failed unmarshalling buffer into return value: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) [jboss-marshalling-river-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213) [jboss-marshalling-river-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> 15:43:48,482 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> 15:43:48,482 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)15:43:48,502 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 59) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:333)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:352)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
> ... 27 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:145)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:142)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:78)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:72)
> at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:331)
> ... 29 more
> Caused by: an exception which occurred:
> in object org.infinispan.topology.CacheTopologyControlCommand@f80b63c
> [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3698) Failure during concurrent restart of cache/manager
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3698?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3698:
-----------------------------------------------
Brian Stansberry <brian.stansberry(a)redhat.com> changed the Status of [bug 1027738|https://bugzilla.redhat.com/show_bug.cgi?id=1027738] from POST to MODIFIED
> Failure during concurrent restart of cache/manager
> --------------------------------------------------
>
> Key: ISPN-3698
> URL: https://issues.jboss.org/browse/ISPN-3698
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 5.2.7.Final, 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Blocker
> Fix For: 5.2.8.Final, 6.0.0.Final
>
>
> These failures prevent the cache from starting successfully, and consequently the redeploy to fail. There are 3 stack traces of note in the logs:
> {noformat}
> 15:05:32,044 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-20,shared=udp) failed unmarshalling buffer into return value: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) [jboss-marshalling-river-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213) [jboss-marshalling-river-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> 15:43:48,482 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> 15:43:48,482 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:150)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:140)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:96)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:698)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:381)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2534)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1389)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1540)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)15:43:48,502 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 59) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:871)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
> at org.infinispan.CacheImpl.start(CacheImpl.java:666)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:557)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:505)
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:277)
> at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:91)
> at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 18 more
> Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:333)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:352)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)
> ... 27 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:145)
> at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:142)
> at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115) [jboss-marshalling-1.4.2.Final.jar:1.4.2.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:78)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:72)
> at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:331)
> ... 29 more
> Caused by: an exception which occurred:
> in object org.infinispan.topology.CacheTopologyControlCommand@f80b63c
> [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3702) Too many threads for cleaning up infinispan transactions
by Prasanth Pallamreddy (JIRA)
Prasanth Pallamreddy created ISPN-3702:
------------------------------------------
Summary: Too many threads for cleaning up infinispan transactions
Key: ISPN-3702
URL: https://issues.jboss.org/browse/ISPN-3702
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache, Transactions
Affects Versions: 5.3.0.Final
Environment: Mac and Linux
Reporter: Prasanth Pallamreddy
Assignee: Mircea Markus
When using multiple transactional caches, we are seeing that each cache has a dedicated cleanup thread. While this is not an issue for small number of caches, when the number of caches is high as in our case (~100), we see around a 100 threads dedicated for cleanup like the following.
"TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f50d3800 nid=0x10f03 waiting on condition [0x00000001a5a5d000]
"TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e800 nid=0x10e03 waiting on condition [0x00000001a595a000]
"TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e000 nid=0x10d03 waiting on condition [0x00000001a5857000]
"TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f5817800 nid=0x10c03 waiting on condition [0x00000001a5754000]
...
Looking at the source code for
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
if (!totalOrder) {
// Periodically run a task to cleanup the transaction table from completed transactions.
ThreadFactory tf = new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
th.setDaemon(true);
return th;
}
};
executorService = Executors.newSingleThreadScheduledExecutor(tf);
This code can benefit from drawing the threads from a dedicated pool which is bounded.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3697) Improved lifecycle control of JGroupsChannelLookup
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-3697?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-3697:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/2210
> Improved lifecycle control of JGroupsChannelLookup
> --------------------------------------------------
>
> Key: ISPN-3697
> URL: https://issues.jboss.org/browse/ISPN-3697
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core API
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Fix For: 6.0.0.Final
>
>
> Currently, a JGroupsChannelLookup can indicate whether the JGroupsTransport should connect the provided channel and disconnect/close the provided channel.
> In the case of Wildfly, we'd like to distinguish between disconnect and close. We have a service that provided the channel, but we allow the JGroupsTransport to manage the lifecycle. This leads to conditions where infinispan's cache manager can restart, but the jgroups channel service does not, e.g.
> https://issues.jboss.org/browse/WFLY-2458
> To fix this, I need to introduce hacks into the channel returned by the channel service, such that Channel.close() is a no-op, and introduce a separate method invisible to Infinispan to perform the actual close().
> So what I propose is the following:
> boolean shouldConnect(); // Notice that I renamed this, because there is no such distinction between "start" and "connect".
> boolean shouldDisconnect();
> boolean shouldClose();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3697) Improved lifecycle control of JGroupsChannelLookup
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-3697?page=com.atlassian.jira.plugin.... ]
Paul Ferraro reassigned ISPN-3697:
----------------------------------
Assignee: Paul Ferraro (was: Mircea Markus)
> Improved lifecycle control of JGroupsChannelLookup
> --------------------------------------------------
>
> Key: ISPN-3697
> URL: https://issues.jboss.org/browse/ISPN-3697
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core API
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Fix For: 6.0.0.Final
>
>
> Currently, a JGroupsChannelLookup can indicate whether the JGroupsTransport should connect the provided channel and disconnect/close the provided channel.
> In the case of Wildfly, we'd like to distinguish between disconnect and close. We have a service that provided the channel, but we allow the JGroupsTransport to manage the lifecycle. This leads to conditions where infinispan's cache manager can restart, but the jgroups channel service does not, e.g.
> https://issues.jboss.org/browse/WFLY-2458
> To fix this, I need to introduce hacks into the channel returned by the channel service, such that Channel.close() is a no-op, and introduce a separate method invisible to Infinispan to perform the actual close().
> So what I propose is the following:
> boolean shouldConnect(); // Notice that I renamed this, because there is no such distinction between "start" and "connect".
> boolean shouldDisconnect();
> boolean shouldClose();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3697) Improved lifecycle control of JGroupsChannelLookup
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-3697?page=com.atlassian.jira.plugin.... ]
Work on ISPN-3697 started by Paul Ferraro.
> Improved lifecycle control of JGroupsChannelLookup
> --------------------------------------------------
>
> Key: ISPN-3697
> URL: https://issues.jboss.org/browse/ISPN-3697
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core API
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Fix For: 6.0.0.Final
>
>
> Currently, a JGroupsChannelLookup can indicate whether the JGroupsTransport should connect the provided channel and disconnect/close the provided channel.
> In the case of Wildfly, we'd like to distinguish between disconnect and close. We have a service that provided the channel, but we allow the JGroupsTransport to manage the lifecycle. This leads to conditions where infinispan's cache manager can restart, but the jgroups channel service does not, e.g.
> https://issues.jboss.org/browse/WFLY-2458
> To fix this, I need to introduce hacks into the channel returned by the channel service, such that Channel.close() is a no-op, and introduce a separate method invisible to Infinispan to perform the actual close().
> So what I propose is the following:
> boolean shouldConnect(); // Notice that I renamed this, because there is no such distinction between "start" and "connect".
> boolean shouldDisconnect();
> boolean shouldClose();
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3665) SingleFileStore is not thread-safe for passivation
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-3665?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-3665:
-------------------------------
Status: Open (was: Pull Request Sent)
> SingleFileStore is not thread-safe for passivation
> --------------------------------------------------
>
> Key: ISPN-3665
> URL: https://issues.jboss.org/browse/ISPN-3665
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Paul Ferraro
> Priority: Blocker
> Labels: 620
> Fix For: 6.0.0.CR2
>
> Attachments: Test.java
>
>
> SingleFileStore never makes use of FileChannel.force(...) to flush changes to disk. This causes problems for the passivation use case.
> If one thread evicts a cache entry, while immediately after another thread attempts to read the same cache entry, the Cache.get(...) can return null. This is because the entry is never flushed to disk.
> I've attached a test to reproduce the problem.
> I also ran the same test with the addition of FileChannel.force(false) to the write(...) method, and the test succeeds.
> A proper fix should probably make this a configurable property (as it was with the old file store implementation). It would be nice if the flush could defer until just before tx commit, but, off hand, I don't know how feasible that is.
> I suspect this lack of flush also accounts for much of the bold claim of a 100x performance improvement over the old file store implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months
[JBoss JIRA] (ISPN-3665) SingleFileStore is not thread-safe for passivation
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/ISPN-3665?page=com.atlassian.jira.plugin.... ]
Paul Ferraro updated ISPN-3665:
-------------------------------
Assignee: Dan Berindei (was: Paul Ferraro)
> SingleFileStore is not thread-safe for passivation
> --------------------------------------------------
>
> Key: ISPN-3665
> URL: https://issues.jboss.org/browse/ISPN-3665
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.CR1
> Reporter: Paul Ferraro
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: 620
> Fix For: 6.0.0.CR2
>
> Attachments: Test.java
>
>
> SingleFileStore never makes use of FileChannel.force(...) to flush changes to disk. This causes problems for the passivation use case.
> If one thread evicts a cache entry, while immediately after another thread attempts to read the same cache entry, the Cache.get(...) can return null. This is because the entry is never flushed to disk.
> I've attached a test to reproduce the problem.
> I also ran the same test with the addition of FileChannel.force(false) to the write(...) method, and the test succeeds.
> A proper fix should probably make this a configurable property (as it was with the old file store implementation). It would be nice if the flush could defer until just before tx commit, but, off hand, I don't know how feasible that is.
> I suspect this lack of flush also accounts for much of the bold claim of a 100x performance improvement over the old file store implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 5 months