[JBoss JIRA] (ISPN-5174) Transaction cannot be recommitted after ownership changes
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5174?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-5174:
----------------------------------
Assignee: Dan Berindei
> Transaction cannot be recommitted after ownership changes
> ---------------------------------------------------------
>
> Key: ISPN-5174
> URL: https://issues.jboss.org/browse/ISPN-5174
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.1.0.CR2
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Critical
>
> Once transaction is completed, it cannot commit again. If it should commit more keys since it has become an owner of some new keys modified in this transaction, it just ignores the further commit.
> There is a race with state transfer which can bring an old value (with StateResponseCommand sent before it is commited) but the value is not set by the ongoing transaction either.
> This results with stale value stored on one node.
> In my case, The problematic part is transaction <edg-perf01-62141>:15066 (consisting of 10 modifications) which got prepared and committed on edg-perf04 in topology 25. Before the originator finishes, topology changes and 04 requests ongoing transactions:
> {code}
> 11:06:11,369 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (transport-thread-17) Replication task sending StateRequestCommand{cache=testCache, origin=edg-perf04-35097, type=GET_TRANSACTIONS, topologyId=28, segments=[275, 1, 278, 9, 282, 286, 17, 259, 25, 267, 171, 169, 33, 306, 175, 173, 310, 172, 314, 41, 167, 165, 318, 187, 290, 49, 185, 191, 294, 189, 179, 298, 57, 177, 183, 302, 181, 343, 205, 201, 338, 203, 336, 351, 197, 349, 199, 347, 193, 345, 195, 326, 85, 87, 322, 93, 332, 95, 330, 89, 91, 103, 101, 99, 506, 97, 105, 357, 359, 353, 355, 361]} to single recipient edg-perf01-62141 with response mode GET_ALL
> 11:06:11,495 DEBUG [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread-17) Applying 6 transactions for cache testCache transferred from node edg-perf01-62141
> {code}
> However I don't see how these are applied, since PrepareCommand is not created again - from the code I see only that backup locks are added. Not sure if the transaction is registered at all, since it was already completed on this node (but at that time it did not own key_00000000000002EB).
> After originator stores the entry, it sends one more CommitCommand with topology 28:
> {code}
> 11:06:11,619 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (DefaultStressor-2) Replication task sending CommitCommand {gtx=GlobalTransaction:<edg-perf01-62141>:15066:local, cacheName='testCache', topologyId=28} to addresses [edg-perf03-20530, edg-perf04-35097] with response mode GET_ALL
> {code}
> 04 receives several CommitCommands (both from originator and forwards), but all of them are ignored as the transaction is completed.
> I don't see the logs where state transfer is assembled, but it's probably before the entry is stored on originator as the state transfer contains the old entry:
> {code}
> 11:06:13,449 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (remote-thread-91) Received chunk with keys [key_000000000000065B, key_00000000000006BE, key_FFFFFFFFFFFFE62F, key_0000000000001F42, key_000000000000027B, key_000000000000159D, key_00000000000002EB, key_00000000000002BB] for segment 343 of cache testCache from node edg-perf01-62141
> 11:06:13,454 TRACE [org.infinispan.container.DefaultDataContainer] (remote-thread-91) Store ImmortalCacheEntry{key=key_00000000000002EB, value=[2 #7: 366, 544, 576, 804, 1061, 1181, 1290, ]} in container
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-3702) Too many threads for cleaning up infinispan transactions
by Denis Kirpichenkov (JIRA)
[ https://issues.jboss.org/browse/ISPN-3702?page=com.atlassian.jira.plugin.... ]
Denis Kirpichenkov commented on ISPN-3702:
------------------------------------------
I found a way to "customize" TransactionTable to let it use fixed thread pool.
here is my solution:
First of all, I have to inject custom TransactionTableFactory into DefaultCacheManager.
{code}
DefaultCacheManager.getGlobalComponentRegistry().getComponentMetadataRepo().injectFactoryForComponent(TransactionTable.class, CustomTransactionTableFactoryKlass.class);
{code}
Then CustomTransactionTableFactoryKlass should replace TransactionTable to my own implementation
{code}
@DefaultFactoryFor(classes = { TransactionTable.class })
public class CustomTransactionTableFactoryKlass extends TransactionTableFactory
{
@SuppressWarnings("unchecked")
@Override
public <T> T construct(Class<T> componentType)
{
T result = super.construct(componentType);
if (result instanceof TransactionTable && !TransactionTable.class.isInterface())
{
result = (T)new CustomTransactionTable();
}
return result;
}
}
{code}
And the last step to get things work is to implement CustomTransactionTable with Java Reflection used to replace executorService because it private field in superclass. Here is example implementation.
{code}
public class CustomTransactionTable extends TransactionTable
{
private static final ScheduledExecutorService executors = Executors.newScheduledThreadPool(5, new ThreadFactory()
{
private final AtomicInteger counter = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r)
{
Thread th = new Thread(r, "InfinispanTxCleanupService-" + counter.incrementAndGet());
th.setDaemon(true);
return th;
}
});
private void changeExecutorService()
{
try
{
Field field = TransactionTable.class.getDeclaredField("executorService");
field.setAccessible(true);
ScheduledExecutorService old = (ScheduledExecutorService)field.get(this);
old.shutdownNow();
field.set(this, executors);
long interval = configuration.transaction().reaperWakeUpInterval();
executors.scheduleAtFixedRate(new Runnable()
{
@Override
public void run()
{
cleanupCompletedTransactions();
}
}, interval, interval, TimeUnit.MILLISECONDS);
}
catch (NoSuchFieldException | SecurityException | IllegalArgumentException | IllegalAccessException e)
{
// something nasty happend
}
}
@Start(priority = 9)
private void start()
{
Class<?>[] classArray = new Class<?>[0];
try
{
Method m = this.getClass().getSuperclass().getDeclaredMethod("start", classArray);
m.setAccessible(true);
m.invoke(this, new Object[0]);
changeExecutorService();
}
catch (NoSuchMethodException | SecurityException | IllegalAccessException | IllegalArgumentException
| InvocationTargetException e)
{
LOG.error("Can't start transaction table", e);
}
}
}
{code}
I hope this will help someone.
In current master branch on github TransactionTable transformed to interface, so this "hack" wouldn't work.
> Too many threads for cleaning up infinispan transactions
> --------------------------------------------------------
>
> Key: ISPN-3702
> URL: https://issues.jboss.org/browse/ISPN-3702
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Transactions
> Affects Versions: 5.3.0.Final
> Environment: Mac and Linux
> Reporter: Prasanth Pallamreddy
>
> When using multiple transactional caches, we are seeing that each cache has a dedicated cleanup thread. While this is not an issue for small number of caches, when the number of caches is high as in our case (~100), we see around a 100 threads dedicated for cleanup like the following.
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f50d3800 nid=0x10f03 waiting on condition [0x00000001a5a5d000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e800 nid=0x10e03 waiting on condition [0x00000001a595a000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e000 nid=0x10d03 waiting on condition [0x00000001a5857000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f5817800 nid=0x10c03 waiting on condition [0x00000001a5754000]
> ...
> Looking at the source code for
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
> if (!totalOrder) {
> // Periodically run a task to cleanup the transaction table from completed transactions.
> ThreadFactory tf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
> Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
> th.setDaemon(true);
> return th;
> }
> };
> executorService = Executors.newSingleThreadScheduledExecutor(tf);
> This code can benefit from drawing the threads from a dedicated pool which is bounded.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5198) FuturesTest.testCombineWithCompletionErrors random failures
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5198?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5198:
------------------------------------
Status: Open (was: New)
> FuturesTest.testCombineWithCompletionErrors random failures
> -----------------------------------------------------------
>
> Key: ISPN-5198
> URL: https://issues.jboss.org/browse/ISPN-5198
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.1.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code}
> [pri:0, instance:org.infinispan.commons.util.concurrent.FuturesTest@60e31054] should have thrown an exception of class java.util.concurrent.ExecutionException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1512)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:754)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5198) FuturesTest.testCombineWithCompletionErrors random failures
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5198?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5198:
------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3244
> FuturesTest.testCombineWithCompletionErrors random failures
> -----------------------------------------------------------
>
> Key: ISPN-5198
> URL: https://issues.jboss.org/browse/ISPN-5198
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.1.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code}
> [pri:0, instance:org.infinispan.commons.util.concurrent.FuturesTest@60e31054] should have thrown an exception of class java.util.concurrent.ExecutionException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1512)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:754)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5199) TransportStackConfigurationIT failing
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-5199:
---------------------------------------
Summary: TransportStackConfigurationIT failing
Key: ISPN-5199
URL: https://issues.jboss.org/browse/ISPN-5199
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Server
Affects Versions: 7.1.0.Final
Reporter: Gustavo Fernandes
TransportStackConfigurationIT.testTCPStackAttributes and TransportStackConfigurationIT.testUDPStackAttributes fails with:
{code}
org.junit.ComparisonFailure: expected:<[d]iscard> but was:<[D]iscard>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.infinispan.server.test.transport.TransportStackConfigurationIT.assertMBeanAttributes(TransportStackConfigurationIT.java:76)
at org.infinispan.server.test.transport.TransportStackConfigurationIT.testUDPStackAttributes(TransportStackConfigurationIT.java:55)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5198) FuturesTest.testCombineWithCompletionErrors random failures
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5198?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-5198:
-----------------------------------------
To reproduce, set {{invocationCount}} count to a high number in the test
> FuturesTest.testCombineWithCompletionErrors random failures
> -----------------------------------------------------------
>
> Key: ISPN-5198
> URL: https://issues.jboss.org/browse/ISPN-5198
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.1.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code}
> [pri:0, instance:org.infinispan.commons.util.concurrent.FuturesTest@60e31054] should have thrown an exception of class java.util.concurrent.ExecutionException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1512)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:754)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5198) FuturesTest.testCombineWithCompletionErrors random failures
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5198?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5198:
------------------------------------
Issue Type: Bug (was: Feature Request)
> FuturesTest.testCombineWithCompletionErrors random failures
> -----------------------------------------------------------
>
> Key: ISPN-5198
> URL: https://issues.jboss.org/browse/ISPN-5198
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 7.1.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code}
> [pri:0, instance:org.infinispan.commons.util.concurrent.FuturesTest@60e31054] should have thrown an exception of class java.util.concurrent.ExecutionException
> at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1512)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:754)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ------- Stdout: -------
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months
[JBoss JIRA] (ISPN-5198) FuturesTest.testCombineWithCompletionErrors random failures
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-5198:
---------------------------------------
Summary: FuturesTest.testCombineWithCompletionErrors random failures
Key: ISPN-5198
URL: https://issues.jboss.org/browse/ISPN-5198
Project: Infinispan
Issue Type: Feature Request
Components: Test Suite - Core
Affects Versions: 7.1.0.Final
Reporter: Gustavo Fernandes
Assignee: Gustavo Fernandes
{code}
[pri:0, instance:org.infinispan.commons.util.concurrent.FuturesTest@60e31054] should have thrown an exception of class java.util.concurrent.ExecutionException
at org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1512)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:754)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
------- Stdout: -------
{code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 2 months