[Red Hat JIRA] (ISPN-12607) Metrics degrade cluster performance
by Ryan Emerson (Jira)
[ https://issues.redhat.com/browse/ISPN-12607?page=com.atlassian.jira.plugi... ]
Ryan Emerson commented on ISPN-12607:
-------------------------------------
The route cause of this issue is that the [MetricsCollector|https://github.com/infinispan/infinispan/blob/master/cor... each mbean attribute individually. This results in multiple calls to [CacheContainerStatsImpl#getEnabledStats|https://github.com/infinispan/inf...], each of which creates a {{StatsImpl}} instance that then calls {{CacheMgmtInterceptor#getNumberOfEntries}}. Hence there is a call to size for each cache with statistics enabled, for every mbean attribute exposed in {{CacheContainerStatsImpl}}.
A simple workaround to this problem is to cache the collection of container stats so that a single StatsImpl instance is reused for all calls to the mbean attributes.
> Metrics degrade cluster performance
> -----------------------------------
>
> Key: ISPN-12607
> URL: https://issues.redhat.com/browse/ISPN-12607
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 12.0.0.CR1, 11.0.8.Final
> Reporter: Ryan Emerson
> Priority: Major
>
> The `/metrics` endpoint exposes Infinispan stats so that they can be periodically scraped by monitoring tools such as Prometheus. However, these stats include calls to `size` which does not perform well as the number of entries in a cache increase. Consequently, if deploying DG in a k8 environment with Prometheus monitoring the DG cluster performance rapidly declines as the number of entries in a cache increases due to time /resources spent iterating the cache container.
> This problem is only exasperated when muiltiple caches exist with many entries exist.
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12261) Protocol server transport management
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-12261?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-12261:
-----------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/8938
Status: Pull Request Sent (was: Open)
> Protocol server transport management
> -------------------------------------
>
> Key: ISPN-12261
> URL: https://issues.redhat.com/browse/ISPN-12261
> Project: Infinispan
> Issue Type: Feature Request
> Components: CLI, JMX, reporting and management, Server
> Affects Versions: 12.0.0.Dev02
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Major
> Fix For: 12.0.0.Final
>
>
> the WF-server had the ability to stop/start a transport via the CLI (ISPN-11240).
> The new server should have a similar capability.
> h4. Protocol management via CLI
> {noformat}
> $ cli.sh server connector ls
> $ cli.sh server connector describe endpoint-default
> $ cli.sh server connector stop endpoint-default
> $ cli.sh server connector start endpoint-default
> {noformat}
> Aside from start/stop, we should also leverage netty's ipfilter handler which allows filtering based on subnet so that traffic can be blocked selectively.
> h4. IP Filtering
> {code:xml}
> <endpoints socket-binding="default" security-realm="default">
> <ip-filter>
> <reject from="172.16.0.0/16"/>
> <accept from="127.0.0.0/8"/>
> </ip-filter>
> <hotrod-connector/>
> <rest-connector/>
> </endpoints>
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12261) Protocol server transport management
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-12261?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-12261:
-----------------------------------
Status: Open (was: New)
> Protocol server transport management
> -------------------------------------
>
> Key: ISPN-12261
> URL: https://issues.redhat.com/browse/ISPN-12261
> Project: Infinispan
> Issue Type: Feature Request
> Components: CLI, JMX, reporting and management, Server
> Affects Versions: 12.0.0.Dev02
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Major
> Fix For: 12.0.0.Final
>
>
> the WF-server had the ability to stop/start a transport via the CLI (ISPN-11240).
> The new server should have a similar capability.
> h4. Protocol management via CLI
> {noformat}
> $ cli.sh server connector ls
> $ cli.sh server connector describe endpoint-default
> $ cli.sh server connector stop endpoint-default
> $ cli.sh server connector start endpoint-default
> {noformat}
> Aside from start/stop, we should also leverage netty's ipfilter handler which allows filtering based on subnet so that traffic can be blocked selectively.
> h4. IP Filtering
> {code:xml}
> <endpoints socket-binding="default" security-realm="default">
> <ip-filter>
> <reject from="172.16.0.0/16"/>
> <accept from="127.0.0.0/8"/>
> </ip-filter>
> <hotrod-connector/>
> <rest-connector/>
> </endpoints>
> {code}
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12609) ClassNotFoundException: org.jboss.marshalling.TraceInformation
by Paul Ferraro (Jira)
[ https://issues.redhat.com/browse/ISPN-12609?page=com.atlassian.jira.plugi... ]
Paul Ferraro commented on ISPN-12609:
-------------------------------------
[~alessandromoscatelli] Which version of WildFly triggered this exception?
A quick workaround, add the following to {{$WILDFLY_HOME/modules/system/layers/base/org/infinispan/main/module.xml}}:
{code:xml}
<module name="org.jboss.marshalling"/>{code}
> ClassNotFoundException: org.jboss.marshalling.TraceInformation
> --------------------------------------------------------------
>
> Key: ISPN-12609
> URL: https://issues.redhat.com/browse/ISPN-12609
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 11.0.4.Final
> Reporter: Alessandro Moscatelli
> Priority: Major
>
> Sometimes, when new members are added to an HA Wildfly Cluster, I see several errors like these :
> [0m[31m11:24:02,046 ERROR [org.infinispan.CLUSTER] (thread-20,ejb,679443c9ac75) ISPN000475: Error processing response for request 53 from eb7eea3c43fe: java.lang.ClassNotFoundException: org.jboss.marshalling.TraceInformation from [Module "org.infinispan" version 11.0.4.Final from local module loader @4f966719 (finder: local module finder @18ac53e8 (roots: /opt/jboss/wildfly/modules,/opt/jboss/wildfly/modules/system/layers/base))][0m[31m11:24:02,046 ERROR [org.infinispan.CLUSTER] (thread-20,ejb,679443c9ac75) ISPN000475: Error processing response for request 53 from eb7eea3c43fe: java.lang.ClassNotFoundException: org.jboss.marshalling.TraceInformation from [Module "org.infinispan" version 11.0.4.Final from local module loader @4f966719 (finder: local module finder @18ac53e8 (roots: /opt/jboss/wildfly/modules,/opt/jboss/wildfly/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255) at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410) at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398) at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) at org.infinispan@11.0.4.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readGenericThrowable(ThrowableExternalizer.java:278) at org.infinispan@11.0.4.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:259) at org.infinispan@11.0.4.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan@11.0.4.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:226) at org.infinispan@11.0.4.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan@11.0.4.Final//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan@11.0.4.Final//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192) at org.infinispan@11.0.4.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221) at org.infinispan@11.0.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1394) at org.infinispan@11.0.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1305) at org.infinispan@11.0.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:131) at org.infinispan@11.0.4.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445) at org.jgroups@4.2.5.Final//org.jgroups.JChannel.up(JChannel.java:784) at org.jgroups@4.2.5.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:135) at org.jgroups@4.2.5.Final//org.jgroups.stack.Protocol.up(Protocol.java:306) at org.jgroups@4.2.5.Final//org.jgroups.protocols.FORK.up(FORK.java:142) at org.jgroups@4.2.5.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:174) at org.jgroups@4.2.5.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:343) at org.jgroups@4.2.5.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:868) at org.jgroups@4.2.5.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:243) at org.jgroups@4.2.5.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1049) at org.jgroups@4.2.5.Final//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:772) at org.jgroups@4.2.5.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:753) at org.jgroups@4.2.5.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups@4.2.5.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups@4.2.5.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups@4.2.5.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups@4.2.5.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups@4.2.5.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups@4.2.5.Final//org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups@4.2.5.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1385) at org.jgroups@4.2.5.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.jboss.as.clustering.common@21.0.0.Final//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49) at org.jboss.as.clustering.common@21.0.0.Final//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70) at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12614) Initial state transfer timed out for cache
by Om Giri (Jira)
[ https://issues.redhat.com/browse/ISPN-12614?page=com.atlassian.jira.plugi... ]
Om Giri updated ISPN-12614:
---------------------------
Labels: state_transfer timeout (was: )
> Initial state transfer timed out for cache
> ------------------------------------------
>
> Key: ISPN-12614
> URL: https://issues.redhat.com/browse/ISPN-12614
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.2.5.Final
> Reporter: Om Giri
> Priority: Critical
> Labels: state_transfer, timeout
> Attachments: error.log, fw-jgroups.xml, infinispan-fw-cache.xml
>
>
> In 2 node cluster environment, 1st time both node up properly with replicated cache. After down 1 node and again up the same node, getting "org.infinispan.commons.CacheException: Initial state transfer timed out for cache". Log/cache-configuration/jgroup file attached.
>
> I have tried this with Weblogic/IBM-WAS/Jetty server and issue replicated on every server and every time.
>
> org.infinispan.commons.CacheException: Initial state transfer timed out for cache GENERIC_PARAMETER_TYPE_ENTITIES$EMPTY on NII0910LEDF0031-12663
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12614) Initial state transfer timed out for cache
by Om Giri (Jira)
[ https://issues.redhat.com/browse/ISPN-12614?page=com.atlassian.jira.plugi... ]
Om Giri updated ISPN-12614:
---------------------------
Security Sensitive Issue: (was: This issue is security relevant)
> Initial state transfer timed out for cache
> ------------------------------------------
>
> Key: ISPN-12614
> URL: https://issues.redhat.com/browse/ISPN-12614
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.2.5.Final
> Reporter: Om Giri
> Priority: Critical
> Attachments: error.log, fw-jgroups.xml, infinispan-fw-cache.xml
>
>
> In 2 node cluster environment, 1st time both node up properly with replicated cache. After down 1 node and again up the same node, getting "org.infinispan.commons.CacheException: Initial state transfer timed out for cache". Log/cache-configuration/jgroup file attached.
>
> I have tried this with Weblogic/IBM-WAS/Jetty server and issue replicated on every server and every time.
>
> org.infinispan.commons.CacheException: Initial state transfer timed out for cache GENERIC_PARAMETER_TYPE_ENTITIES$EMPTY on NII0910LEDF0031-12663
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12614) Initial state transfer timed out for cache
by Om Giri (Jira)
[ https://issues.redhat.com/browse/ISPN-12614?page=com.atlassian.jira.plugi... ]
Om Giri commented on ISPN-12614:
--------------------------------
I have tried this with Infinispan-9.4.18-final and Jgroups-4.0.20-final also ut getting same issue.
> Initial state transfer timed out for cache
> ------------------------------------------
>
> Key: ISPN-12614
> URL: https://issues.redhat.com/browse/ISPN-12614
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.2.5.Final
> Reporter: Om Giri
> Priority: Critical
> Attachments: error.log, fw-jgroups.xml, infinispan-fw-cache.xml
>
>
> In 2 node cluster environment, 1st time both node up properly with replicated cache. After down 1 node and again up the same node, getting "org.infinispan.commons.CacheException: Initial state transfer timed out for cache". Log/cache-configuration/jgroup file attached.
>
> I have tried this with Weblogic/IBM-WAS/Jetty server and issue replicated on every server and every time.
>
> org.infinispan.commons.CacheException: Initial state transfer timed out for cache GENERIC_PARAMETER_TYPE_ENTITIES$EMPTY on NII0910LEDF0031-12663
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months
[Red Hat JIRA] (ISPN-12614) Initial state transfer timed out for cache
by Om Giri (Jira)
[ https://issues.redhat.com/browse/ISPN-12614?page=com.atlassian.jira.plugi... ]
Om Giri updated ISPN-12614:
---------------------------
Security: (was: Security Issue)
> Initial state transfer timed out for cache
> ------------------------------------------
>
> Key: ISPN-12614
> URL: https://issues.redhat.com/browse/ISPN-12614
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 9.2.5.Final
> Reporter: Om Giri
> Priority: Critical
> Attachments: error.log, fw-jgroups.xml, infinispan-fw-cache.xml
>
>
> In 2 node cluster environment, 1st time both node up properly with replicated cache. After down 1 node and again up the same node, getting "org.infinispan.commons.CacheException: Initial state transfer timed out for cache". Log/cache-configuration/jgroup file attached.
>
> I have tried this with Weblogic/IBM-WAS/Jetty server and issue replicated on every server and every time.
>
> org.infinispan.commons.CacheException: Initial state transfer timed out for cache GENERIC_PARAMETER_TYPE_ENTITIES$EMPTY on NII0910LEDF0031-12663
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
--
This message was sent by Atlassian Jira
(v8.13.1#813001)
3 years, 4 months