[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
[ https://issues.jboss.org/browse/ISPN-9762?page=com.atlassian.jira.plugin.... ]
Sergey Chernolyas commented on ISPN-9762:
-----------------------------------------
2018-11-22 15:50:23,671 WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p2-t7) ISPN000071: Caught exception when handling command StateResponseCommand{cache=DEVICES, pushTransfer=false, stateChunks=[StateChunk{
segmentId=0, cacheEntries=10, isLastChunk=false}, StateChunk{segmentId=1, cacheEntries=7, isLastChunk=false},
org.infinispan.util.concurrent.TimeoutException: Timed out applying state
at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:583)
at org.infinispan.statetransfer.StateResponseCommand.invokeAsync(StateResponseCommand.java:88)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:117)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> Cache hangs during rebalancing
> ------------------------------
>
> Key: ISPN-9762
> URL: https://issues.jboss.org/browse/ISPN-9762
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.2.Final
> Reporter: Sergey Chernolyas
> Priority: Blocker
> Attachments: hang_node.txt, normal_node.txt, stat_bad_node.png, stat_good_node.png
>
>
> I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
> Configuration of the cache:
> {code:xml}
> <distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
> <state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
> <memory>
> <object size="300000" strategy="REMOVE"/>
> </memory>
> <rocksdb-store preload="true" path="/data/rocksdb/devices/data">
> <expiration path="/data/rocksdb/devices/expired"/>
> </rocksdb-store>
> <indexing index="LOCAL">
> <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
> <property name="default.directory_provider">infinispan</property>
> <property name="default.worker.execution">async</property>
> <property name="default.index_flush_interval">500</property>
> <property name="default.indexwriter.merge_factor">30</property>
> <property name="default.indexwriter.merge_max_size">1024</property>
> <property name="default.indexwriter.ram_buffer_size">256</property>
> <property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
> <property name="default.data_cachename">LuceneIndexesData_devices</property>
> <property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
> </indexing>
> <expiration max-idle="172800000"/>
> </distributed-cache>
> {code}
> The cache contains 70 000 elements.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
[ https://issues.jboss.org/browse/ISPN-9762?page=com.atlassian.jira.plugin.... ]
Sergey Chernolyas updated ISPN-9762:
------------------------------------
Attachment: stat_bad_node.png
stat_good_node.png
> Cache hangs during rebalancing
> ------------------------------
>
> Key: ISPN-9762
> URL: https://issues.jboss.org/browse/ISPN-9762
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.2.Final
> Reporter: Sergey Chernolyas
> Priority: Blocker
> Attachments: hang_node.txt, normal_node.txt, stat_bad_node.png, stat_good_node.png
>
>
> I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
> Configuration of the cache:
> {code:xml}
> <distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
> <state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
> <memory>
> <object size="300000" strategy="REMOVE"/>
> </memory>
> <rocksdb-store preload="true" path="/data/rocksdb/devices/data">
> <expiration path="/data/rocksdb/devices/expired"/>
> </rocksdb-store>
> <indexing index="LOCAL">
> <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
> <property name="default.directory_provider">infinispan</property>
> <property name="default.worker.execution">async</property>
> <property name="default.index_flush_interval">500</property>
> <property name="default.indexwriter.merge_factor">30</property>
> <property name="default.indexwriter.merge_max_size">1024</property>
> <property name="default.indexwriter.ram_buffer_size">256</property>
> <property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
> <property name="default.data_cachename">LuceneIndexesData_devices</property>
> <property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
> </indexing>
> <expiration max-idle="172800000"/>
> </distributed-cache>
> {code}
> The cache contains 70 000 elements.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
[ https://issues.jboss.org/browse/ISPN-9762?page=com.atlassian.jira.plugin.... ]
Sergey Chernolyas updated ISPN-9762:
------------------------------------
Attachment: hang_node.txt
normal_node.txt
> Cache hangs during rebalancing
> ------------------------------
>
> Key: ISPN-9762
> URL: https://issues.jboss.org/browse/ISPN-9762
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.2.Final
> Reporter: Sergey Chernolyas
> Priority: Blocker
> Attachments: hang_node.txt, normal_node.txt
>
>
> I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
> Configuration of the cache:
> {code:xml}
> <distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
> <state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
> <memory>
> <object size="300000" strategy="REMOVE"/>
> </memory>
> <rocksdb-store preload="true" path="/data/rocksdb/devices/data">
> <expiration path="/data/rocksdb/devices/expired"/>
> </rocksdb-store>
> <indexing index="LOCAL">
> <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
> <property name="default.directory_provider">infinispan</property>
> <property name="default.worker.execution">async</property>
> <property name="default.index_flush_interval">500</property>
> <property name="default.indexwriter.merge_factor">30</property>
> <property name="default.indexwriter.merge_max_size">1024</property>
> <property name="default.indexwriter.ram_buffer_size">256</property>
> <property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
> <property name="default.data_cachename">LuceneIndexesData_devices</property>
> <property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
> </indexing>
> <expiration max-idle="172800000"/>
> </distributed-cache>
> {code}
> The cache contains 70 000 elements.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
[ https://issues.jboss.org/browse/ISPN-9762?page=com.atlassian.jira.plugin.... ]
Sergey Chernolyas updated ISPN-9762:
------------------------------------
Description:
I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
Configuration of the cache:
{code:xml}
<distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
<state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
<partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
<memory>
<object size="300000" strategy="REMOVE"/>
</memory>
<rocksdb-store preload="true" path="/data/rocksdb/devices/data">
<expiration path="/data/rocksdb/devices/expired"/>
</rocksdb-store>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">infinispan</property>
<property name="default.worker.execution">async</property>
<property name="default.index_flush_interval">500</property>
<property name="default.indexwriter.merge_factor">30</property>
<property name="default.indexwriter.merge_max_size">1024</property>
<property name="default.indexwriter.ram_buffer_size">256</property>
<property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
<property name="default.data_cachename">LuceneIndexesData_devices</property>
<property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
</indexing>
<expiration max-idle="172800000"/>
</distributed-cache>
{code}
The cache contains 70 000 elements.
was:
I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
Configuration of the cache:
{code:xml}
<distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
<state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
<partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
<memory>
<object size="300000" strategy="REMOVE"/>
</memory>
<rocksdb-store preload="true" path="/data/rocksdb/devices/data">
<expiration path="/data/rocksdb/devices/expired"/>
</rocksdb-store>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">infinispan</property>
<property name="default.worker.execution">async</property>
<property name="default.index_flush_interval">500</property>
<property name="default.indexwriter.merge_factor">30</property>
<property name="default.indexwriter.merge_max_size">1024</property>
<property name="default.indexwriter.ram_buffer_size">256</property>
<property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
<property name="default.data_cachename">LuceneIndexesData_devices</property>
<property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
</indexing>
<expiration max-idle="172800000"/>
</distributed-cache>
{code}
> Cache hangs during rebalancing
> ------------------------------
>
> Key: ISPN-9762
> URL: https://issues.jboss.org/browse/ISPN-9762
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.2.Final
> Reporter: Sergey Chernolyas
> Priority: Blocker
>
> I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
> Configuration of the cache:
> {code:xml}
> <distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
> <state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
> <memory>
> <object size="300000" strategy="REMOVE"/>
> </memory>
> <rocksdb-store preload="true" path="/data/rocksdb/devices/data">
> <expiration path="/data/rocksdb/devices/expired"/>
> </rocksdb-store>
> <indexing index="LOCAL">
> <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
> <property name="default.directory_provider">infinispan</property>
> <property name="default.worker.execution">async</property>
> <property name="default.index_flush_interval">500</property>
> <property name="default.indexwriter.merge_factor">30</property>
> <property name="default.indexwriter.merge_max_size">1024</property>
> <property name="default.indexwriter.ram_buffer_size">256</property>
> <property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
> <property name="default.data_cachename">LuceneIndexesData_devices</property>
> <property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
> </indexing>
> <expiration max-idle="172800000"/>
> </distributed-cache>
> {code}
> The cache contains 70 000 elements.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
[ https://issues.jboss.org/browse/ISPN-9762?page=com.atlassian.jira.plugin.... ]
Sergey Chernolyas updated ISPN-9762:
------------------------------------
Description:
I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
Configuration of the cache:
{code:xml}
<distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
<state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
<partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
<memory>
<object size="300000" strategy="REMOVE"/>
</memory>
<rocksdb-store preload="true" path="/data/rocksdb/devices/data">
<expiration path="/data/rocksdb/devices/expired"/>
</rocksdb-store>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">infinispan</property>
<property name="default.worker.execution">async</property>
<property name="default.index_flush_interval">500</property>
<property name="default.indexwriter.merge_factor">30</property>
<property name="default.indexwriter.merge_max_size">1024</property>
<property name="default.indexwriter.ram_buffer_size">256</property>
<property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
<property name="default.data_cachename">LuceneIndexesData_devices</property>
<property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
</indexing>
<expiration max-idle="172800000"/>
</distributed-cache>
{code}
was:
I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
Configuration of the cache:
```
<distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
<state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
<partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
<memory>
<object size="300000" strategy="REMOVE"/>
</memory>
<rocksdb-store preload="true" path="/data/rocksdb/devices/data">
<expiration path="/data/rocksdb/devices/expired"/>
</rocksdb-store>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">infinispan</property>
<property name="default.worker.execution">async</property>
<property name="default.index_flush_interval">500</property>
<property name="default.indexwriter.merge_factor">30</property>
<property name="default.indexwriter.merge_max_size">1024</property>
<property name="default.indexwriter.ram_buffer_size">256</property>
<property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
<property name="default.data_cachename">LuceneIndexesData_devices</property>
<property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
</indexing>
<expiration max-idle="172800000"/>
</distributed-cache>
```
> Cache hangs during rebalancing
> ------------------------------
>
> Key: ISPN-9762
> URL: https://issues.jboss.org/browse/ISPN-9762
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.2.Final
> Reporter: Sergey Chernolyas
> Priority: Blocker
>
> I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
> Configuration of the cache:
> {code:xml}
> <distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
> <state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
> <partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
> <memory>
> <object size="300000" strategy="REMOVE"/>
> </memory>
> <rocksdb-store preload="true" path="/data/rocksdb/devices/data">
> <expiration path="/data/rocksdb/devices/expired"/>
> </rocksdb-store>
> <indexing index="LOCAL">
> <property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
> <property name="default.directory_provider">infinispan</property>
> <property name="default.worker.execution">async</property>
> <property name="default.index_flush_interval">500</property>
> <property name="default.indexwriter.merge_factor">30</property>
> <property name="default.indexwriter.merge_max_size">1024</property>
> <property name="default.indexwriter.ram_buffer_size">256</property>
> <property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
> <property name="default.data_cachename">LuceneIndexesData_devices</property>
> <property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
> </indexing>
> <expiration max-idle="172800000"/>
> </distributed-cache>
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9762) Cache hangs during rebalancing
by Sergey Chernolyas (Jira)
Sergey Chernolyas created ISPN-9762:
---------------------------------------
Summary: Cache hangs during rebalancing
Key: ISPN-9762
URL: https://issues.jboss.org/browse/ISPN-9762
Project: Infinispan
Issue Type: Bug
Affects Versions: 9.4.2.Final
Reporter: Sergey Chernolyas
I have a cluster with two nodes. One node starts without problem. Second node hangs on rebalancing cache DEVICES.
Configuration of the cache:
```
<distributed-cache name="DEVICES" owners="2" segments="256" mode="SYNC">
<state-transfer await-initial-transfer="true" enabled="true" timeout="2400000" chunk-size="2048"/>
<partition-handling when-split="ALLOW_READ_WRITES" merge-policy="PREFERRED_ALWAYS"/>
<memory>
<object size="300000" strategy="REMOVE"/>
</memory>
<rocksdb-store preload="true" path="/data/rocksdb/devices/data">
<expiration path="/data/rocksdb/devices/expired"/>
</rocksdb-store>
<indexing index="LOCAL">
<property name="default.indexmanager">org.infinispan.query.indexmanager.InfinispanIndexManager</property>
<property name="default.directory_provider">infinispan</property>
<property name="default.worker.execution">async</property>
<property name="default.index_flush_interval">500</property>
<property name="default.indexwriter.merge_factor">30</property>
<property name="default.indexwriter.merge_max_size">1024</property>
<property name="default.indexwriter.ram_buffer_size">256</property>
<property name="default.locking_cachename">LuceneIndexesLocking_devices</property>
<property name="default.data_cachename">LuceneIndexesData_devices</property>
<property name="default.metadata_cachename">LuceneIndexesMetadata_devices</property>
</indexing>
<expiration max-idle="172800000"/>
</distributed-cache>
```
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9746) HotRod decoder should release allocated buffers
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9746?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9746:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 10.0.0.Alpha2
9.4.3.Final
Assignee: Radim Vansa
Resolution: Done
> HotRod decoder should release allocated buffers
> -----------------------------------------------
>
> Key: ISPN-9746
> URL: https://issues.jboss.org/browse/ISPN-9746
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.4.1.Final, 10.0.0.Alpha2
> Reporter: Dan Berindei
> Assignee: Radim Vansa
> Priority: Major
> Fix For: 10.0.0.Alpha2, 9.4.3.Final
>
>
> {noformat}
> 19:09:07,279 ERROR [io.netty.util.ResourceLeakDetector] (HotRod-ServerIO-6-1) LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:113)
> org.infinispan.server.hotrod.HotRodDecoder.switch3(HotRodDecoder.java:1940)
> org.infinispan.server.hotrod.HotRodDecoder.switch1_0(HotRodDecoder.java:156)
> org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.java:143)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9761) Cannot wire or start components while the registry is not running
by Radoslav Husar (Jira)
[ https://issues.jboss.org/browse/ISPN-9761?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated ISPN-9761:
---------------------------------
Description:
Looks like yet another clean shutdown issue surfaced after we changed WFLY-11324 to avoid some graceful shutdown bugs with the XA transaction table.
{noformat}
16:28:38,874 ERROR [org.infinispan.commons.tx.TransactionImpl] (default task-2) ISPN000926: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[node-1, node-2], isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[SessionAccessMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionCreationMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionAttributesKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5)], topologyId=5, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@72} org.infinispan.transaction.synchronization.SynchronizationAdapter@91: org.infinispan.IllegalLifecycleStateException: Cannot wire or start components while the registry is not running
at org.infinispan.factories.impl.BasicComponentRegistryImpl.prepareWrapperChange(BasicComponentRegistryImpl.java:610)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.wireWrapper(BasicComponentRegistryImpl.java:158)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.wire(BasicComponentRegistryImpl.java:736)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:712)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:148)
at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:861)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33)
at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:506)
at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:338)
at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:346)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
Looks like yet another clean shutdown issue.
{noformat}
16:28:38,874 ERROR [org.infinispan.commons.tx.TransactionImpl] (default task-2) ISPN000926: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[node-1, node-2], isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[SessionAccessMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionCreationMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionAttributesKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5)], topologyId=5, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@72} org.infinispan.transaction.synchronization.SynchronizationAdapter@91: org.infinispan.IllegalLifecycleStateException: Cannot wire or start components while the registry is not running
at org.infinispan.factories.impl.BasicComponentRegistryImpl.prepareWrapperChange(BasicComponentRegistryImpl.java:610)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.wireWrapper(BasicComponentRegistryImpl.java:158)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.wire(BasicComponentRegistryImpl.java:736)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:712)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:148)
at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:861)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33)
at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:506)
at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:338)
at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:346)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> Cannot wire or start components while the registry is not running
> ------------------------------------------------------------------
>
> Key: ISPN-9761
> URL: https://issues.jboss.org/browse/ISPN-9761
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.1.Final
> Reporter: Radoslav Husar
> Priority: Major
>
> Looks like yet another clean shutdown issue surfaced after we changed WFLY-11324 to avoid some graceful shutdown bugs with the XA transaction table.
> {noformat}
> 16:28:38,874 ERROR [org.infinispan.commons.tx.TransactionImpl] (default task-2) ISPN000926: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[node-1, node-2], isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[SessionAccessMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionCreationMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionAttributesKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5)], topologyId=5, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@72} org.infinispan.transaction.synchronization.SynchronizationAdapter@91: org.infinispan.IllegalLifecycleStateException: Cannot wire or start components while the registry is not running
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.prepareWrapperChange(BasicComponentRegistryImpl.java:610)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.wireWrapper(BasicComponentRegistryImpl.java:158)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.wire(BasicComponentRegistryImpl.java:736)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:712)
> at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:148)
> at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:861)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33)
> at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:506)
> at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:338)
> at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
> at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:346)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months
[JBoss JIRA] (ISPN-9761) Cannot wire or start components while the registry is not running
by Radoslav Husar (Jira)
Radoslav Husar created ISPN-9761:
------------------------------------
Summary: Cannot wire or start components while the registry is not running
Key: ISPN-9761
URL: https://issues.jboss.org/browse/ISPN-9761
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.4.1.Final
Reporter: Radoslav Husar
Looks like yet another clean shutdown issue.
{noformat}
16:28:38,874 ERROR [org.infinispan.commons.tx.TransactionImpl] (default task-2) ISPN000926: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=[node-1, node-2], isMarkedForRollback=false, lockedKeys=[], backupKeyLocks=[SessionAccessMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionCreationMetaDataKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5), SessionAttributesKey(2sYbjnFh2n-m_gkaOP2iz53e0ms4cbuARoeIdYJ5)], topologyId=5, stateTransferFlag=null} org.infinispan.transaction.synchronization.SyncLocalTransaction@72} org.infinispan.transaction.synchronization.SynchronizationAdapter@91: org.infinispan.IllegalLifecycleStateException: Cannot wire or start components while the registry is not running
at org.infinispan.factories.impl.BasicComponentRegistryImpl.prepareWrapperChange(BasicComponentRegistryImpl.java:610)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.wireWrapper(BasicComponentRegistryImpl.java:158)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.wire(BasicComponentRegistryImpl.java:736)
at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:712)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:148)
at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:861)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33)
at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:506)
at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:338)
at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:87)
at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:346)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 4 months