[JBoss JIRA] (ISPN-2033) JdbcBinaryCacheStore.purgeInternal() - releaseConnection() should be called in finally block.
by Jim Dunkerton (JIRA)
Jim Dunkerton created ISPN-2033:
-----------------------------------
Summary: JdbcBinaryCacheStore.purgeInternal() - releaseConnection() should be called in finally block.
Key: ISPN-2033
URL: https://issues.jboss.org/browse/ISPN-2033
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores
Affects Versions: 5.1.2.FINAL
Environment: JBoss 7.1.1 Final
Reporter: Jim Dunkerton
Assignee: Manik Surtani
A potential memory leak of pooled DB connections is present in JdbcCacheBinaryStore. Observe the catch and finally blocks of the purgeInternal() method:
catch (SQLException ex) {
//if something happens make sure buckets locks are being release
releaseLocks(expiredBuckets);
connectionFactory.releaseConnection(conn);
log.failedClearingJdbcCacheStore(ex);
throw new CacheLoaderException("Failed clearing JdbcBinaryCacheStore", ex);
} finally {
JdbcUtil.safeClose(ps);
JdbcUtil.safeClose(rs);
}
The pooled DB connection is not released unless an SQLException has been thrown.
Contrast with JdbcStringBasedCacheStore.purgeInternal():
} catch (SQLException ex) {
log.failedClearingJdbcCacheStore(ex);
throw new CacheLoaderException("Failed clearing string based JDBC store", ex);
} finally {
JdbcUtil.safeClose(ps);
connectionFactory.releaseConnection(conn);
}
When using a JdbcMixedCacheStore, I noticed a DB connection leak. It appeared that for every two connections being acquired, only one was being released. I believe it is down to the fact that the JdbcBinaryCacheStore inside the JdbcMixedCacheStore is not releasing its connections properly (i.e. in its finally block) whereas the JdbcStringBasedCacheStore is.
I guess a workaround is to use only a JdbcStringBasedCacheStore, not a JdbcMixedCacheStore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2444) the incoming local xsite commands don't check if the local cache is started
by Mircea Markus (JIRA)
Mircea Markus created ISPN-2444:
-----------------------------------
Summary: the incoming local xsite commands don't check if the local cache is started
Key: ISPN-2444
URL: https://issues.jboss.org/browse/ISPN-2444
Project: Infinispan
Issue Type: Feature Request
Components: Cross-Site Replication
Affects Versions: 5.2.0.Beta2, 5.2.0.Final
Reporter: Mircea Markus
Assignee: Mircea Markus
Currently the CommandAwareRpcDispatcher is using different handles for commands received from a remote site(InboundInvocationHandler) vs the commands received from the local site(BackupReceiverRepository).
For the commands received from the remote site, there's no check to see if the receiving cache is started, as it is in the case of commands received from the local site (that's handled in InboundInvocationHandler.handle).
This Jira is about checking if the cache that's suposed to process a command originated from a remote site is started. If it is not started, we should throw an exception to signal that the backup call was not successful.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2417) Topology check in LocalTransaction.getCommitNodes() is wrong
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-2417:
-----------------------------------
Summary: Topology check in LocalTransaction.getCommitNodes() is wrong
Key: ISPN-2417
URL: https://issues.jboss.org/browse/ISPN-2417
Project: Infinispan
Issue Type: Feature Request
Components: State transfer, Transactions
Affects Versions: 5.2.0.Beta2
Reporter: Adrian Nistor
Assignee: Mircea Markus
Fix For: 5.2.0.CR1
Method LocalTransaction.getCommitNodes() was introduced when fixing ISPN-2317 but this aspect was overlooked during review.
The problem appears because the topology id is not incremented by an update caused by leavers. So the check getTopologyId() != currentTopologyId is not reliable.
Flagging this as an issue for now but maybe the fix will actually need to increment topologyId on every update, w/o any special treatment of leavers. That would automatically solve this issue and would also simplify the code in AbstractTxLockingInterceptor that solves the same problem as here (see usages of TransactionTable.useStrictTopologyIdComparison().
Or we can fix this here to using the same approach as AbstractTxLockingInterceptor + TransactionTable.useStrictTopologyIdComparison().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2442) Ability to take site offline needs to be extended to async replication
by Erik Salter (JIRA)
Erik Salter created ISPN-2442:
---------------------------------
Summary: Ability to take site offline needs to be extended to async replication
Key: ISPN-2442
URL: https://issues.jboss.org/browse/ISPN-2442
Project: Infinispan
Issue Type: Feature Request
Components: Cross-Site Replication
Affects Versions: 5.2.0.Beta2
Reporter: Erik Salter
Assignee: Mircea Markus
Currently the site offline implementation only accounts for synchronous caches. This needs to be extended to asynchronous replication as well.
Best case, this avoids wasted cycles and thread resources sending to a dead site. Worst case this could tie up all OOB threads, which could halt communication in the local site.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2415) Initial state transfer timed out - Fail to start 2 nodes after they were killed inside 8-node cluster
by Martin Gencur (JIRA)
Martin Gencur created ISPN-2415:
-----------------------------------
Summary: Initial state transfer timed out - Fail to start 2 nodes after they were killed inside 8-node cluster
Key: ISPN-2415
URL: https://issues.jboss.org/browse/ISPN-2415
Project: Infinispan
Issue Type: Bug
Components: State transfer
Reporter: Martin Gencur
Assignee: Mircea Markus
Priority: Critical
We start 8 nodes, keep them under load, than we kill 2 nodes and later start them again. However, when we are trying to start them, the following exception is thrown and the test fails:
{code}
10:47:52,830 ERROR [org.radargun.stages.helpers.StartHelper] (pool-1-thread-1) Issues while instantiating/starting cache wrapper
org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:879)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:650)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:639)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:198)
at org.infinispan.CacheImpl.start(CacheImpl.java:517)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:689)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:652)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:548)
at org.radargun.cachewrappers.InfinispanWrapper.setUpCache(InfinispanWrapper.java:125)
at org.radargun.cachewrappers.InfinispanWrapper.setUp(InfinispanWrapper.java:74)
at org.radargun.stages.helpers.StartHelper.start(StartHelper.java:63)
at org.radargun.stages.StartClusterStage.executeOnSlave(StartClusterStage.java:47)
at org.radargun.Slave$2.run(Slave.java:103)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache testCache on edg-perf02-25863
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
... 20 more
{code}
The problem happens at nodes edg-perf02 and edg-perf03 under this Jenkins run: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/ispn-52-radargun-res...
Debug log can be found at those machines.
A few more hints:
- there are individual exceptions/errors extracted from the log - available in the "Build artifacts"
- this job passed only once, fails otherwise
- state transfer timeout is the default one (4 min?)
- version of Infinspan: 5.2.0-SNAPSHOT, HEAD=d4581e570 - ISPN-2387 ClusteredGetCommand should not be a VisitableCommand
Infinispan configuration:
{code:xml}
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.2 http://www.infinispan.org/schemas/infinispan-config-5.2.xsd"
xmlns="urn:infinispan:config:5.2">
<global>
<globalJmxStatistics
enabled="true"
jmxDomain="jboss.infinispan"
cacheManagerName="default"/>
<transport clusterName="default" distributedSyncTimeout="600000">
<properties>
<property name="configurationFile" value="jgroups-udp-custom.xml" />
</properties>
</transport>
</global>
<default>
<transaction
transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
transactionMode="TRANSACTIONAL" />
<jmxStatistics enabled="true"/>
<clustering mode="distribution">
<l1 enabled="false" />
<hash numOwners="3" numSegments="512" />
<sync replTimeout="60000"/>
</clustering>
<locking lockAcquisitionTimeout="3000" concurrencyLevel="1000" />
</default>
<namedCache name="testCache" />
<namedCache name="memcachedCache" />
</infinispan>
{code}
Test scenario (description of RadarGun's job):
{code:xml}
<bench-config>
<master bindAddress="${127.0.0.1:master.address}" port="${2103:master.port}" />
<benchmark initSize="${8:slaves}" maxSize="${8:slaves}" increment="1">
<DestroyWrapper runOnAllSlaves="true" />
<StartCluster
staggerSlaveStartup="true"
delayAfterFirstSlaveStarts="5000"
delayBetweenStartingSlaves="500" />
<ClusterValidation
partialReplication="false" />
<StartBackgroundStats
numThreads="10"
numEntries="${1000:numEntries}"
entrySize="1024"
puts="1"
gets="2"
statsIterationDuration="${1000:statsIterationDuration}"
delayBetweenRequests="100"
transactionSize="${30:transactionSize}"
startStressors="true" />
<!-- Synchronously start stat threads -->
<StartBackgroundStats
startStats="true" />
<Sleep
time="120000" />
<Kill
slaves="1,2" />
<Sleep
time="120000" />
<StartCluster
slaves="1,2"
staggerSlaveStartup="false" />
<Sleep
time="120000" />
<StopBackgroundStats />
<ReportBackgroundStats />
</benchmark>
<products>
<infinispan52>
<config name="distributed-udp-numowners-3.xml" cache="testCache"/>
</infinispan52>
</products>
<reports />
</bench-config>
{code}
If any further information is needed, let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2437) Hash "factory" element produces CastClassException
by Erik Salter (JIRA)
Erik Salter created ISPN-2437:
---------------------------------
Summary: Hash "factory" element produces CastClassException
Key: ISPN-2437
URL: https://issues.jboss.org/browse/ISPN-2437
Project: Infinispan
Issue Type: Bug
Components: Configuration
Affects Versions: 5.2.0.Beta2
Reporter: Erik Salter
Assignee: Mircea Markus
The factory attribute for the hash element is broken. It's missing a break statement. Otherwise, all XML-defined factory CH implementations will not start due to a ClassCastException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-2428) ConcurrentModificationException in StateConsumerImpl
by Radim Vansa (JIRA)
Radim Vansa created ISPN-2428:
---------------------------------
Summary: ConcurrentModificationException in StateConsumerImpl
Key: ISPN-2428
URL: https://issues.jboss.org/browse/ISPN-2428
Project: Infinispan
Issue Type: Bug
Components: State transfer
Affects Versions: 5.2.0.Beta2
Reporter: Radim Vansa
Assignee: Adrian Nistor
In StateConsumerImpl.onTopologyUpdate(...) the transfersBySource.keySet() is iterated while under some conditions transfersBySource.remove(...) is called. As the iterated keySet is backed by the HashMap itself modifications during iteration are not permitted and I got
{code}
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:894)
at java.util.HashMap$KeyIterator.next(HashMap.java:928)
at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:215)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:186)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:117)
at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194)
at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
at org.infinispan.topology.ClusterTopologyManagerImpl$3.call(ClusterTopologyManagerImpl.java:554)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months