[JBoss JIRA] (ISPN-6478) Netty worker thread starvation with events
by Karsten Klein (JIRA)
[ https://issues.jboss.org/browse/ISPN-6478?page=com.atlassian.jira.plugin.... ]
Karsten Klein commented on ISPN-6478:
-------------------------------------
We seem to face the issues with higher load on our Infinispan Server based system.
Here the top most part of a stack dump from an .hprof. The thread never comes back and causes severe effects on our system (as entity locks are still in place):
java.lang.Thread @ 0x6c5ee2d70
- at sun.misc.Unsafe.park(ZJ)V (Native Method)
- at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175)
- at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()V (AbstractQueuedSynchronizer.java:2039)
- at java.util.concurrent.LinkedBlockingQueue.put(Ljava/lang/Object;)V (LinkedBlockingQueue.java:350)
- at org.infinispan.server.hotrod.ClientListenerRegistry$BaseClientEventSender.sendEvent(...)V (ClientListenerRegistry.scala:296)
- at org.infinispan.server.hotrod.ClientListenerRegistry$BaseClientEventSender.onCacheEvent(...)V (ClientListenerRegistry.scala:262)
- at sun.reflect.GeneratedMethodAccessor247.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
We are using 8.2.6 at the moment. Using the latest 8.2.x version is doable. Switching to 9.x at this point in time is not regarded an option.
How and when do you plan to approach this issue?
> Netty worker thread starvation with events
> ------------------------------------------
>
> Key: ISPN-6478
> URL: https://issues.jboss.org/browse/ISPN-6478
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 8.2.1.Final
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Priority: Critical
>
> As a result of ISPN-6005, we decoupled the incoming Hot Rod server invocations from sending events by adding a intermediate queue that maintains the events to send to clients. However, this separation can lead to a Netty worker thread starvation issue since we add the events to the queue in Netty's IO thread, so if the queue is full, incoming requests are stuck.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-7643) Coordinator status is unavailable in Infinispan cluster
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-7643?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec commented on ISPN-7643:
-------------------------------------------
I think I found it.... it depends whether the node is a coordinator or not... Sometimes it works correctly..
> Coordinator status is unavailable in Infinispan cluster
> -------------------------------------------------------
>
> Key: ISPN-7643
> URL: https://issues.jboss.org/browse/ISPN-7643
> Project: Infinispan
> Issue Type: Bug
> Components: JMX, reporting and management
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Fix For: 9.1.0.Final
>
>
> It seems that Coordinator field uses JGroups Address class which can be displayed properly. Since we already have a Coordinator Address field, maybe the best way would be to remove this from JMX.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-8038) java.lang.ClassCastException with compatibility mode and text content
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8038?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8038:
------------------------------------
Status: Open (was: New)
> java.lang.ClassCastException with compatibility mode and text content
> ---------------------------------------------------------------------
>
> Key: ISPN-8038
> URL: https://issues.jboss.org/browse/ISPN-8038
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.1.0.Beta1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.1.0.Final
>
>
> Setting compat mode with a string marshaller in the server, the same marshaller in the hot rod client, writing text content via rest and reading via Hot Rod yields:
> {noformat}
> org.infinispan.client.hotrod.exceptions.HotRodClientException:
> Request for messageId=3 returned server error (status=0x85): java.lang.ClassCastException: [B cannot be cast to java.lang.String
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-7642) Administration console - remote sites are not displayed correctly on cache container page
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7642?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-7642:
-------------------------------------------
Mine too [~ryanemerson] I forgot how we got the 'sites-mixed', 'sites-online' etc structures in ICacheContainer and not in the Cache itself - so let's take this offline and come back with a solution for this issue.
> Administration console - remote sites are not displayed correctly on cache container page
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-7642
> URL: https://issues.jboss.org/browse/ISPN-7642
> Project: Infinispan
> Issue Type: Bug
> Components: Console
> Affects Versions: 9.0.0.CR2
> Reporter: Roman Macor
> Assignee: Vladimir Blagojevic
> Attachments: Screenshot-cacheContainer.png, Screenshot-detail-cache.png, clustered.xml
>
>
> Have 2 caches each configured with a different remote site.
> When you click on cache container, both remote sites are displayed on both cache cards. Clicking on a cache to see cache detail page shows correct remote site. Please see the attached screenshots.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-8036) TestResourceTracker keeps cache managers alive after they are stopped
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8036?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8036:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> TestResourceTracker keeps cache managers alive after they are stopped
> ---------------------------------------------------------------------
>
> Key: ISPN-8036
> URL: https://issues.jboss.org/browse/ISPN-8036
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.1.0.CR1, 9.0.3.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Fix For: 9.1.0.Final
>
>
> {{TestCacheManagerFactory.newDefaultCacheManager()}} adds the manager to {{TestResourceTracker}} so if the test doesn't stop it explicitly, {{TestResourceTracker}} will.
> When a test uses {{@CleanupAfterMethod}}, however, the cache managers are stopped and recreated after each method, but the list of resources is only cleared at the end of the test class. If a test has lots of methods, like {{LocalDistributedExecutorTest}} and its subclasses, the stopped managers can use a lot of memory (250KB for each {{ComponentMetadataRepo}}).
> I believe the extra memory usage is causing longer GC pauses, and in turn leads to intermittent test failures in test methods that use short timeouts:
> {noformat}
> 16:58:34,801 ERROR (testng-ReplSyncDistributedExecutorTest:[]) [TestSuiteProgress] Test failed: org.infinispan.distexec.ReplSyncDistributedExecutorTest.testInvokeAnyTimedSleepingTasks
> java.lang.AssertionError: Should have thrown an class java.util.concurrent.TimeoutException
> at org.infinispan.test.Exceptions.assertException(Exceptions.java:21) ~[test-classes/:?]
> at org.infinispan.test.Exceptions.expectException(Exceptions.java:92) ~[test-classes/:?]
> at org.infinispan.distexec.LocalDistributedExecutorTest.testInvokeAnyTimedSleepingTasks(LocalDistributedExecutorTest.java:269) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-7642) Administration console - remote sites are not displayed correctly on cache container page
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-7642?page=com.atlassian.jira.plugin.... ]
Ryan Emerson commented on ISPN-7642:
------------------------------------
[~vblagojevic] My understanding is that a cache card should only show the backups associated with that cache.
> Administration console - remote sites are not displayed correctly on cache container page
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-7642
> URL: https://issues.jboss.org/browse/ISPN-7642
> Project: Infinispan
> Issue Type: Bug
> Components: Console
> Affects Versions: 9.0.0.CR2
> Reporter: Roman Macor
> Assignee: Vladimir Blagojevic
> Attachments: Screenshot-cacheContainer.png, Screenshot-detail-cache.png, clustered.xml
>
>
> Have 2 caches each configured with a different remote site.
> When you click on cache container, both remote sites are displayed on both cache cards. Clicking on a cache to see cache detail page shows correct remote site. Please see the attached screenshots.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months
[JBoss JIRA] (ISPN-7642) Administration console - remote sites are not displayed correctly on cache container page
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-7642?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-7642:
-------------------------------------------
[~ryanemerson] I need your help here for a moment. It seems (from caches.html page) that we indeed list all the cache containers backup sites for each cache card instead of per cache backup. Is that intended?
> Administration console - remote sites are not displayed correctly on cache container page
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-7642
> URL: https://issues.jboss.org/browse/ISPN-7642
> Project: Infinispan
> Issue Type: Bug
> Components: Console
> Affects Versions: 9.0.0.CR2
> Reporter: Roman Macor
> Assignee: Vladimir Blagojevic
> Attachments: Screenshot-cacheContainer.png, Screenshot-detail-cache.png, clustered.xml
>
>
> Have 2 caches each configured with a different remote site.
> When you click on cache container, both remote sites are displayed on both cache cards. Clicking on a cache to see cache detail page shows correct remote site. Please see the attached screenshots.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
8 years, 9 months