March 2016 - infinispan-issues - Jboss List Archives

[JBoss JIRA] (ISPN-6357) Deadlock during server start

by Dan Berindei (JIRA)

[ https://issues.jboss.org/browse/ISPN-6357?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-6357: ------------------------------------ [~gustavonalle] can you attach some server logs as well? > Deadlock during server start > ---------------------------- > > Key: ISPN-6357 > URL: https://issues.jboss.org/browse/ISPN-6357 > Project: Infinispan > Issue Type: Bug > Components: Core, Server > Affects Versions: 8.2.0.Final > Reporter: Gustavo Fernandes > Priority: Critical > Attachments: server1.txt, server2.txt > > > This happens frequently when starting servers in parallel, the more servers, the easier to reproduce. > Attached the stack trace of server1 and server2 after hanging. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6357) Deadlock during server start

by Dan Berindei (JIRA)

[ https://issues.jboss.org/browse/ISPN-6357?page=com.atlassian.jira.plugin.... ] Dan Berindei reassigned ISPN-6357: ---------------------------------- Assignee: Dan Berindei > Deadlock during server start > ---------------------------- > > Key: ISPN-6357 > URL: https://issues.jboss.org/browse/ISPN-6357 > Project: Infinispan > Issue Type: Bug > Components: Core, Server > Affects Versions: 8.2.0.Final > Reporter: Gustavo Fernandes > Assignee: Dan Berindei > Priority: Critical > Attachments: server1.txt, server2.txt > > > This happens frequently when starting servers in parallel, the more servers, the easier to reproduce. > Attached the stack trace of server1 and server2 after hanging. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6341) StateTransferManager should be the first component to stop

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/ISPN-6341?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-6341: ----------------------------------------------- wfink(a)redhat.com changed the Status of [bug 1316132|https://bugzilla.redhat.com/show_bug.cgi?id=1316132] from ASSIGNED to MODIFIED > StateTransferManager should be the first component to stop > ---------------------------------------------------------- > > Key: ISPN-6341 > URL: https://issues.jboss.org/browse/ISPN-6341 > Project: Infinispan > Issue Type: Bug > Components: Core > Affects Versions: 8.2.0.CR1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Fix For: 9.0.0.Alpha1, 8.2.1.Final > > > When a cache stops, it first removes the component registry from the {{GlobalComponentsRegistry}}'s {{namedComponents}} map, which means the node (let's call it {{A}}) will reply with a {{CacheNotFoundResponse}} to any remote command. > Another node {{B}} trying to execute a write/transactional command will receive the {{CacheNotFoundResponse}}, assume that a new cache topology with id {{current topology id + 1}} is coming soon, and wait for that new topology before retrying. > Normally this is not a problem, because {{StateTransferManagerImpl.stop()}} sends a {{CacheTopologyControlCommand(LEAVE)}} to the coordinator quickly enough, then {{B}} receives the {{current topology id + 1}} topology and retries the command. > But in some cases, the cache components that stop before {{StateTransferManagerImpl}} can take a long time to do so. In particular, because of {{ISPN-5507}}, {{TransactionTable}} can block for {{cacheStopTimeout}} if there are remote transactions in progress, even though the cache can no longer process remote commands. > We should give {{StateTransferManagerImpl.stop()}} a priority of {{0}}, so that the {{CacheTopologyControlCommand(LEAVE)}} comand is sent as soon as possible. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-5507) Transactions committed immediately before cache stop can block shutdown

by RH Bugzilla Integration (JIRA)

[ https://issues.jboss.org/browse/ISPN-5507?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-5507: ----------------------------------------------- wfink(a)redhat.com changed the Status of [bug 1316132|https://bugzilla.redhat.com/show_bug.cgi?id=1316132] from ASSIGNED to MODIFIED > Transactions committed immediately before cache stop can block shutdown > ----------------------------------------------------------------------- > > Key: ISPN-5507 > URL: https://issues.jboss.org/browse/ISPN-5507 > Project: Infinispan > Issue Type: Bug > Components: Core, Test Suite - Core > Affects Versions: 7.2.1.Final, 8.0.0.Alpha1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Blocker > Labels: testsuite_stability > Fix For: 9.0.0.Alpha1, 8.2.1.Final > > > This is causing random failures in {{DistributedEntryRetrieverTxTest.verifyNodeLeavesBeforeGettingData}}. > The test inserts some values into the cache, starts an iteration, and then kills one of the nodes. In rare instances, the killed cache only receives the TxCompletionCommand for one of the writes after it started the shutdown, and ignores it. That leaves the remote tx on-going, and {{TransactionTable.shutDownGracefully()}} blocks for 30 seconds - causing a {{TimeoutException}} elsewhere in the test. > {noformat} > 10:52:18,129 TRACE (remote-thread-NodeAM-p12133-t6:) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=null} for command CommitCommand {gtx=GlobalTransaction:<NodeAL-45757>:22325:remote, cacheName='org.infinispan.iteration.DistributedEntryRetrieverTxTest', topologyId=4} > 10:52:18,129 TRACE (testng-DistributedEntryRetrieverTxTest:) [JGroupsTransport] dests=[NodeAM-45518, NodeAL-45757], command=TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=4, gtx=GlobalTransaction:<NodeAL-45757>:22325:local, cacheName=org.infinispan.iteration.DistributedEntryRetrieverTxTest} , mode=ASYNCHRONOUS, timeout=15000 > 10:52:18,133 DEBUG (testng-DistributedEntryRetrieverTxTest:) [CacheImpl] Stopping cache org.infinispan.iteration.DistributedEntryRetrieverTxTest on NodeAM-45518 > 10:52:18,133 TRACE (OOB-2,NodeAM-45518:) [GlobalInboundInvocationHandler] Attempting to execute CacheRpcCommand: TxCompletionNotificationCommand{ xid=null, internalId=0, topologyId=4, gtx=GlobalTransaction:<NodeAL-45757>:22325:local, cacheName=org.infinispan.iteration.DistributedEntryRetrieverTxTest} [sender=NodeAL-45757] > 10:52:18,133 TRACE (OOB-2,NodeAM-45518:) [GlobalInboundInvocationHandler] Silently ignoring that org.infinispan.iteration.DistributedEntryRetrieverTxTest cache is not defined > 10:52:18,133 DEBUG (testng-DistributedEntryRetrieverTxTest:) [TransactionTable] Wait for on-going transactions to finish for 30 seconds. > 10:52:48,139 WARN (testng-DistributedEntryRetrieverTxTest:) [TransactionTable] ISPN000100: Stopping, but there are 0 local transactions and 1 remote transactions that did not finish in time. > 10:52:48,386 ERROR (testng-DistributedEntryRetrieverTxTest:) [UnitTestTestNGListener] Test verifyNodeLeavesBeforeGettingData(org.infinispan.iteration.DistributedEntryRetrieverTxTest) failed. > java.lang.IllegalStateException: Thread already timed out waiting for event pre_send_response_released > at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:131) > at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:116) > at org.infinispan.iteration.DistributedEntryRetrieverTest.verifyNodeLeavesBeforeGettingData(DistributedEntryRetrieverTest.java:105) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6357) Deadlock during server start

by Tristan Tarrant (JIRA)

[ https://issues.jboss.org/browse/ISPN-6357?page=com.atlassian.jira.plugin.... ] Tristan Tarrant updated ISPN-6357: ---------------------------------- Priority: Critical (was: Major) > Deadlock during server start > ---------------------------- > > Key: ISPN-6357 > URL: https://issues.jboss.org/browse/ISPN-6357 > Project: Infinispan > Issue Type: Bug > Components: Core, Server > Affects Versions: 8.2.0.Final > Reporter: Gustavo Fernandes > Priority: Critical > Attachments: server1.txt, server2.txt > > > This happens frequently when starting servers in parallel, the more servers, the easier to reproduce. > Attached the stack trace of server1 and server2 after hanging. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6345) Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI

by Sebastian Łaskawiec (JIRA)

[ https://issues.jboss.org/browse/ISPN-6345?page=com.atlassian.jira.plugin.... ] Sebastian Łaskawiec updated ISPN-6345: -------------------------------------- Status: Resolved (was: Pull Request Sent) Fix Version/s: 9.0.0.Final 9.0.0.Alpha1 Resolution: Done > Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI > ------------------------------------------------------------------------ > > Key: ISPN-6345 > URL: https://issues.jboss.org/browse/ISPN-6345 > Project: Infinispan > Issue Type: Task > Components: Tasks > Reporter: Anna Manukyan > Assignee: Anna Manukyan > Labels: testsuite_stability > Fix For: 9.0.0.Final, 9.0.0.Alpha1 > > > Fix the DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6345) Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI

by Sebastian Łaskawiec (JIRA)

[ https://issues.jboss.org/browse/ISPN-6345?page=com.atlassian.jira.plugin.... ] Sebastian Łaskawiec updated ISPN-6345: -------------------------------------- Labels: testsuite_stability (was: ) > Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI > ------------------------------------------------------------------------ > > Key: ISPN-6345 > URL: https://issues.jboss.org/browse/ISPN-6345 > Project: Infinispan > Issue Type: Task > Reporter: Anna Manukyan > Assignee: Anna Manukyan > Labels: testsuite_stability > > Fix the DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6345) Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI

by Sebastian Łaskawiec (JIRA)

[ https://issues.jboss.org/browse/ISPN-6345?page=com.atlassian.jira.plugin.... ] Sebastian Łaskawiec updated ISPN-6345: -------------------------------------- Component/s: Tasks > Fix DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI > ------------------------------------------------------------------------ > > Key: ISPN-6345 > URL: https://issues.jboss.org/browse/ISPN-6345 > Project: Infinispan > Issue Type: Task > Components: Tasks > Reporter: Anna Manukyan > Assignee: Anna Manukyan > Labels: testsuite_stability > > Fix the DistributedServerTaskIT#shouldPutNewValueInRemoteCache failing on CI. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6361) ClusteredCacheConfigurationIT.testQueueFlushIntervalMemcached random failures

by Dan Berindei (JIRA)

[ https://issues.jboss.org/browse/ISPN-6361?page=com.atlassian.jira.plugin.... ] Dan Berindei updated ISPN-6361: ------------------------------- Status: Open (was: New) > ClusteredCacheConfigurationIT.testQueueFlushIntervalMemcached random failures > ----------------------------------------------------------------------------- > > Key: ISPN-6361 > URL: https://issues.jboss.org/browse/ISPN-6361 > Project: Infinispan > Issue Type: Bug > Components: Test Suite - Server > Affects Versions: 8.2.0.Final > Reporter: Dan Berindei > Assignee: Dan Berindei > Labels: testsuite_stability > Fix For: 9.0.0.Alpha1, 8.2.1.Final > > -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6361) ClusteredCacheConfigurationIT.testQueueFlushIntervalMemcached random failures

by Dan Berindei (JIRA)

[ https://issues.jboss.org/browse/ISPN-6361?page=com.atlassian.jira.plugin.... ] Dan Berindei updated ISPN-6361: ------------------------------- Priority: Blocker (was: Major) > ClusteredCacheConfigurationIT.testQueueFlushIntervalMemcached random failures > ----------------------------------------------------------------------------- > > Key: ISPN-6361 > URL: https://issues.jboss.org/browse/ISPN-6361 > Project: Infinispan > Issue Type: Bug > Components: Test Suite - Server > Affects Versions: 8.2.0.Final > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Blocker > Labels: testsuite_stability > Fix For: 9.0.0.Alpha1, 8.2.1.Final > > -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

infinispan-issues March 2016