[infinispan-issues] [JBoss JIRA] (ISPN-3702) Too many threads for cleaning up infinispan transactions

[Denis Kirpichenkov (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037101#comment-13037101 ] 

Denis Kirpichenkov commented on ISPN-3702:
------------------------------------------

I found a way to "customize" TransactionTable to let it use fixed thread pool.
here is my solution:

First of all, I have to inject custom TransactionTableFactory into DefaultCacheManager. 
{code}
DefaultCacheManager.getGlobalComponentRegistry().getComponentMetadataRepo().injectFactoryForComponent(TransactionTable.class, CustomTransactionTableFactoryKlass.class);
{code}

Then CustomTransactionTableFactoryKlass should replace TransactionTable to my own implementation

{code}
@DefaultFactoryFor(classes = { TransactionTable.class })
public class CustomTransactionTableFactoryKlass extends TransactionTableFactory
{
    @SuppressWarnings("unchecked")
    @Override
    public <T> T construct(Class<T> componentType)
    {
        T result = super.construct(componentType);
        if (result instanceof TransactionTable && !TransactionTable.class.isInterface())
        {
            result = (T)new CustomTransactionTable();
        }
        return result;
    }
}
{code}

And the last step to get things work is to implement CustomTransactionTable with Java Reflection used to replace executorService because it private field in superclass. Here is example implementation.

{code}
public class CustomTransactionTable extends TransactionTable
{
    private static final ScheduledExecutorService executors = Executors.newScheduledThreadPool(5, new ThreadFactory()
    {
        private final AtomicInteger counter = new AtomicInteger(0);

        @Override
        public Thread newThread(Runnable r)
        {
            Thread th = new Thread(r, "InfinispanTxCleanupService-" + counter.incrementAndGet());
            th.setDaemon(true);
            return th;
        }
    });

    private void changeExecutorService()
    {
        try
        {
            Field field = TransactionTable.class.getDeclaredField("executorService");
            field.setAccessible(true);

            ScheduledExecutorService old = (ScheduledExecutorService)field.get(this);
            old.shutdownNow();
            field.set(this, executors);

            long interval = configuration.transaction().reaperWakeUpInterval();
            executors.scheduleAtFixedRate(new Runnable()
            {
                @Override
                public void run()
                {
                    cleanupCompletedTransactions();
                }
            }, interval, interval, TimeUnit.MILLISECONDS);
        }
        catch (NoSuchFieldException | SecurityException | IllegalArgumentException | IllegalAccessException e)
        {
            // something nasty happend
        }
    }

    @Start(priority = 9)
    private void start()
    {
        Class<?>[] classArray = new Class<?>[0];
        try
        {
            Method m = this.getClass().getSuperclass().getDeclaredMethod("start", classArray);
            m.setAccessible(true);
            m.invoke(this, new Object[0]);
            changeExecutorService();
        }
        catch (NoSuchMethodException | SecurityException | IllegalAccessException | IllegalArgumentException
                | InvocationTargetException e)
        {
            LOG.error("Can't start transaction table", e);
        }
    }
}
{code}

I hope this will help someone.

In current master branch on github TransactionTable transformed to interface, so this "hack" wouldn't work.

> Too many threads for cleaning up infinispan transactions
> --------------------------------------------------------
>
>                 Key: ISPN-3702
>                 URL: https://issues.jboss.org/browse/ISPN-3702
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core, Transactions
>    Affects Versions: 5.3.0.Final
>         Environment: Mac and Linux
>            Reporter: Prasanth Pallamreddy
>
> When using multiple transactional caches, we are seeing that each cache has a dedicated cleanup thread. While this is not an issue for small number of caches, when the number of caches is high as in our case (~100), we see around a 100 threads dedicated for cleanup like the following.
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f50d3800 nid=0x10f03 waiting on condition [0x00000001a5a5d000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e800 nid=0x10e03 waiting on condition [0x00000001a595a000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f507e000 nid=0x10d03 waiting on condition [0x00000001a5857000]
> "TxCleanupService,{default}_{XXX},user-mac-54275" daemon prio=5 tid=0x00007fa0f5817800 nid=0x10c03 waiting on condition [0x00000001a5754000]
> ...
> Looking at the source code for 
> https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/transaction/TransactionTable.java#L126
>       if (!totalOrder) {
>          // Periodically run a task to cleanup the transaction table from completed transactions.
>          ThreadFactory tf = new ThreadFactory() {
>             @Override
>             public Thread newThread(Runnable r) {
>                String address = rpcManager != null ? rpcManager.getTransport().getAddress().toString() : "local";
>                Thread th = new Thread(r, "TxCleanupService," + cacheName + "," + address);
>                th.setDaemon(true);
>                return th;
>             }
>          };
>          executorService = Executors.newSingleThreadScheduledExecutor(tf);
> This code can benefit from drawing the threads from a dedicated pool which is bounded. 



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)